Compile the agent program and the model together.

Today's compilers stop at the kernel or at one model iteration. The KV cache, batching, prefill/decode placement and prefix reuse belong to a serving scheduler, in Python, over hand-written per-vendor kernels. Agentic workloads make that split expensive.

1. 01.1Workloads

   Agentic workloads dominate inference growth

   A median coding-agent step carries about 119k prefix tokens, appends 875 and generates 214. Prefix-cache hit rates are 95 to 96 percent, and each miss, caused by a long tool wait, costs seconds of recompute.

   Source: TraceLab, June 2026

2. 01.2Silicon

   Hardware split along the prefill/decode axis

   Vera Rubin with Groq 3 LPX racks, Trainium with Cerebras: heterogeneous prefill and decode devices shipped in 2026. Placement across device classes is still configured by hand.

   Source: research notes 01 to 03

3. 01.3Software

   Tile IRs converged, and nobody sees the agent

   Tile-level IRs are vendor-owned, the neutral layers changed owners (Modular to Qualcomm, Groq to NVIDIA), and NVFP4 versus OCP MX numerics diverge. No compiler looks past one model iteration.

   Source: research note 05

KV block liveness is derived from the agent program the way register liveness is derived from a control-flow graph. The same machine files drive the cost model, the simulator and the compiler, so every policy is testable before silicon.

1. L1

   Agent Program IR

   agentforge-ir::agent

   Steps are LLM calls with structured prompt blocks marked stable or not; transitions are tool calls with latency models, probabilistic branches, or end. KV block liveness is derived from it.

   Baseline
2. L2

   Model graph IR

   agentforge-ir::model

   Attention and MoE variants as structured descriptions (dense, grouped-query, latent, sliding-window, compressed); numerics as first-class types, including block-scaled formats.

   Baseline
3. L3

   Tile IR

   agentforge-ir::tile

   Tiles with explicit memory spaces (global, shared, register, tensor memory), async copies and MMA operations. Async protocols are to be encoded in Rust types so misuse fails to compile.

   Skeleton
4. L4

   Machine model

   agentforge-machine

   TOML descriptions: devices with class, compute per precision, memory capacity and bandwidth, on-chip SRAM; memory tiers with capacity, bandwidth and latency; links.

   Baseline
5. L5

   Cost model and simulator

   agentforge-cost · agentforge-sim

   Roofline-style phase costs for prefill, decode and transfers, and a discrete-event replay of agent sessions over the machine description.

   Baseline
6. L6

   Compiler passes

   agentforge-compile

   KV liveness, residency planning (Belady-style cost comparison over tiers), prefill/decode placement. E-graph rewrites of the model graph and tile programs come next.

   Baseline
7. L7

   Backends

   PTX · CUDA Tile IR · LLVM AMDGPU · SPIR-V · MSL · tt-metal · Cranelift

   Emit public, stable low-level formats. No MLIR dependency in the baseline.

   Planned
8. L8

   Runtime

   agentforge-runtime

   Executor, KV block store with lifetimes from the plan, transports between tiers and devices; memory-safe by construction.

   Traits

• 01 Where it holds

  With a fast, large host tier the plan cuts pressure evictions by 30 to 40 percent in every seed and lowers foreground KV waits in three of four. The information doing the work is which tool the agent called: a policy that only saw the blended wait distribution lost to LRU.

• 02 What it does not fix

  Mean TTFT barely moves in any pressured regime, because the dominant cost is queueing for device memory. No residency policy changes how much KV the active sessions need; admission control, smaller KV and placement onto bigger devices do.

• 03 Where it loses

  With scarce, slow tiers (a 128 GB host tier, NVMe writing at 24 GB/s) every extra transfer competes for one channel. The plan saturates it and loses to LRU, which is why channel-aware planning is the first M1 item.

A machine is a TOML file: devices with class, compute per precision, memory capacity and bandwidth, on-chip SRAM; memory tiers with capacity, read and write bandwidth and latency; links. Five ship with the baseline. Closed silicon is a modelling target through the same files.

• gpu01 / 05

  h200

  NVIDIA H200 SXM ×8

  Memory

  141 GB HBM3e

  Bandwidth

  4.8 TB/s

  SRAM

  50 MB

  FP8

  1979 TFLOPS

  Link

  NVLink 4 · 900 GB/s

  Tiers

  host 2 TB · NVMe 30 TB

  No FP4 tensor cores; FP4 weights run through the FP8 path.

  h200.toml
• gpu02 / 05

  b200

  NVIDIA HGX B200 ×8

  Memory

  180 GB HBM3e

  Bandwidth

  8.0 TB/s

  SRAM

  164 MB

  FP8

  4500 TFLOPS

  Link

  NVLink 5 · 1800 GB/s

  Tiers

  host 2 TB · NVMe 30 TB

  NVFP4 at 9000 TFLOPS; E4M3 scales per 16 elements.

  b200.toml
• gpu03 / 05

  mi355x

  AMD Instinct MI355X ×8

  Memory

  288 GB HBM3e

  Bandwidth

  8.0 TB/s

  SRAM

  296 MB

  FP8

  5000 TFLOPS

  Link

  Infinity Fabric · 1000 GB/s

  Tiers

  host 2 TB · NVMe 30 TB

  OCP MX block scaling only; NVFP4 falls back to the MXFP4 rate.

  mi355x.toml
• gpu04 / 05

  blackhole-p150

  Tenstorrent Blackhole p150 ×1

  Memory

  32 GB GDDR6

  Bandwidth

  0.512 TB/s

  SRAM

  210 MB

  FP8

  774 TFLOPS

  Link

  800G ×4 · 400 GB/s

  Tiers

  host 512 GB

  140 Tensix cores with 1.5 MB SRAM each. The only fully open accelerator stack.

  blackhole-p150.toml
• sram_lpu05 / 05

  lpx-tier

  SRAM decode tier · Groq 3 LPX shape ×256

  Memory

  0.5 GB SRAM per LPU

  Bandwidth

  150 TB/s

  SRAM

  500 MB

  FP8

  1230 TFLOPS

  Link

  Rack fabric · 2500 GB/s

  Tiers

  DDR5 pool 12 TB

  Modelling target only; the hardware is closed.

  lpx-tier.toml

name = "h200"
description = "NVIDIA H200 SXM node: eight GPUs on NVLink 4, PCIe Gen5 host links, host DRAM and NVMe tiers"

[[devices]]
name = "h200-sxm"
class = "gpu"
count = 8
memory_gb = 141
memory_bw_tbps = 4.8
on_chip_sram_mb = 50

[devices.compute_tflops]
fp16 = 989
fp8 = 1979

[[tiers]]
name = "host_dram"
capacity_gb = 2000
read_bw_gbps = 50
write_bw_gbps = 50
latency_us = 20

[[links]]
name = "nvlink4"
bw_gbps = 900
latency_us = 2

1. MilestoneScopeDone whenStatus
2. M0

   Baseline

   2026-10

   Research notes and report, design doc, Cargo workspace with IR, machine model, cost model, compiler passes, trace simulator, runtime traits, CLI.

   cargo test, clippy and fmt pass; sim run compares LRU and program-aware residency.

   current
3. M1

   Simulator validation

   next

   FreeInference and TraceLab trace loaders, calibration against GenZ and LLMCompass, TP interference, channel-aware residency, write-through baseline, admission control.

   Simulated TTFT and throughput within 15 percent of published measurements for Llama 3.1 70B on H200 and B200.

   planned
4. M2

   CPU backend

   planned

   Tile IR to Cranelift for a tiny transformer; reference numerics for FP8, MXFP4, NVFP4.

   A 2-layer model runs end to end, bit-exact against the reference.

   planned
5. M3

   NVIDIA backend

   planned

   PTX emission and cutile-rs kernels; KV block store with planned lifetimes; agent-aware runtime on one GPU.

   An agentic trace replays on real hardware with the simulator’s predicted ordering of policies confirmed.

   planned
6. M4

   Tenstorrent backend

   planned

   tt-metal programs through the host API; Blackhole machine file validated on hardware.

   The same tiny model and trace replay on a Blackhole card.

   planned
7. M5

   Heterogeneous placement

   planned

   Prefill-class and decode-class devices in one plan; KV transfer scheduling over links.

   Simulation, and measurement where hardware allows, of a two-class deployment.

   planned
8. M6

   Numerics verification

   planned

   Block formats as verified IR types; e-graph rewrites with checked equivalence; batch-invariant kernel mode.

   Rewrite rules proven or property-tested; deterministic mode passes a 1,000-run identity test.

   planned