Dataflow Simulator for Long-Context, Causal Transformers

Link to Repo

Simulator Settings

Cycle Rate (micros/cycle): Min Chunk Size (Tokens):

Training Spec

Sequence Length (K): Max Attended Tokens (K): Train Token Ratio: Train Chunk Distribution: Chunk Type:

Training Environment

Num Devices: Max Device Memory (GB):

Model Spec

Bitwidth: Total Blocks: Vocab Size: Model Dim: KV Dim: Num Experts: Active Experts: Expert Dim: Attention Algo:

Hardware Specs

In practice, these will be discovered by the system

TFLOPS (at bitwidth): Dev Mem BW (GB/s): Matmul Efficiency: Attention Efficiency: Home BW (Gb/s): Peer BW (Gb/s):

Memory
&
Storage

Speed: 80

Run to Cycle:

Status: Idle | Cycle: 0

Example Sequence

Compute
&
Communication