Dataflow Simulator for Long-Context, Causal Transformers

Link to Repo

Training Spec

Sequence Length (K): Train Token Ratio: Train Chunk Distribution: Chunk Type: Min Chunk Size:

Training Environment

Num Devices: Max Device Memory (GB):

Model Spec

Bitwidth: Total Blocks: Vocab Size: Model Dim: KV Dim: Num Experts: Active Experts: Expert Dim: Attention Type:

Hardware Specs

In practice, these will be discovered by the system

TFLOPS (at bitwidth): Dev Mem BW (GB/s): Matmul Efficiency: Attention Efficiency: Home BW (Gb/s): Peer BW (Gb/s):

Memory

Speed: 50 Run to Cycle:

Status: Idle | Cycle: 0

Compute
&
Communication