Dataflow Simulator for Causal Transformers

Link to Repo

Simulator Settings

Cycle Rate (micros/cycle): Min Chunk Size (Tokens):

Training Spec

Num Seqs Per Ring (Per Round): Sequence Length (K): Max Attended Tokens (K): Chunk Type:

Training Environment

Num Devices Per Ring: Num Rings: Max Device Memory (GB):

Model Spec

Matmul Bitwidth: Attention Bitwidth: Head Bitwidth: Grad Bitwidth: Total Blocks: Vocab Size: Model Dim: KV Dim: Num Experts: Active Experts: Expert Dim: Attention Algo:

Hardware Specs

In practice, these will be discovered by the system. All efficiencies are relative to specified TFLOPs (e.g. if attention is in 16-bit and running at 500 TFLOPs vs 2000 specified, it has .25 efficiency). If you change the bitwidths, make sure to update these efficiency constants to be realistic...

TFLOPS (at matmul bitwidth): Dev Mem BW (GB/s): Matmul Efficiency: Attention Fwd Efficiency: Attention Bwd Efficiency: Head Efficiency: Home BW (Gb/s): Peer BW (Gb/s):

Memory
&
Storage

Speed: 80

Run to Cycle:

Status: Idle | Cycle: 0

Example Ring

Memory&Storage

Completion Details

Compute&Communication

Memory
&
Storage

Compute
&
Communication