Executive Summary
- Agent infrastructure engineers its way out of cold-start hell — SpecBox delivers 2.9× latency cuts for MCP sandbox scheduling with speculative prewarming
- LLM-as-judge gets replaced by 100× cheaper programs — PAJAMA distills evaluation logic into auditable code, matching 13B judges
- Ion Stoica discovers a new attack class — deadline-driven distributed inference pipelines collapse accuracy under shaped workloads with no model access needed
📄 Deep Dives
METHOD Recasts the fundamental tension between persistent sandbox reservations (memory-hungry) and on-demand instantiation (cold-start latency) as a scheduling problem. SpecBox uses intent-driven prewarming — detecting pending tool calls mid-LLM token generation via keyword matching + streaming semantic embeddings — to fully overlap sandbox bootstrapping with model inference. Adds context-aware stochastic prefetching from a sandbox dependency graph, semantic result caching, and a zero-copy shared-memory transport plane.
KEY RESULT 2.9× reduction in P99 end-to-end latency vs on-demand baseline; 45.9% lower peak memory vs persistent reservations. Tested on high-concurrency multi-turn agent traces with MCP workloads.
Every bank deploying agentic AI at scale will hit this cold-start wall. SpecBox shows the solution is speculative prewarming at the infrastructure layer, not faster sandbox images. The shared-memory transport plane (bypassing network serialization for artifact transfers) is a pattern every MCP deployment should adopt. This is the kind of infrastructure paper that defines how production agent systems are built — like what Kubernetes did for containers.
METHOD Instead of calling an LLM judge at evaluation time, PAJAMA distills the judge's decision logic into a committee of interpretable programs that score candidates directly. These programmatic judges offer transparency, are inspectable/editable, and eliminate per-sample API costs. A fallback mechanism escalates low-confidence cases to a full LLM judge, advancing the accuracy-throughput Pareto frontier.
KEY RESULT Programmatic judges match a 13B LLM judge across 5 datasets and 4 model families. When used as reward signals for training, a reward model distilled from program verdicts outperforms one trained on proprietary LLM labels at 100× lower API cost. Programs are auditable, version-controllable, and debuggable.
The banking AI governance stack needs continuous evaluation — model changes, prompt updates, new tools. LLM-as-judge is too expensive and too opaque for audit. PAJAMA shows the path: distill judging logic into executable code, keep the LLM only for edge cases. This is evaluation infrastructure that satisfies both the engineering team (fast, cheap) and the risk committee (transparent, auditable). Extends arc #27 (Evaluation Infrastructure Reform) to production readiness.
METHOD Identifies a new attack surface in the ubiquitous fast-path/slow-path distributed inference architecture. The attacker shapes workload (Yo-Yo bursts) to saturate shared network/compute on the slow path, pushing benign users' high-accuracy predictions past latency deadlines. The merger component then discards them, collapsing to low-accuracy fast-path outputs — all without access to model weights or victim data.
KEY RESULT ~4,000 burst requests inflate benign P99 latency from 92ms → 2s, dropping object tracking quality by 7.0 HOTA points (up to 18.7 for targeted intervals). Rare classes (stop signs) lose ~50% of pre-attack accuracy.
Every production AI deployment that uses a fast/slow inference split (edge+cloud is the standard pattern) is vulnerable to this attack class. For banking — real-time fraud detection, KYC verification, market surveillance — the accuracy collapse means the system silently degrades under attack while appearing operational. Extends arc #17 (Agentic Control Plane Security) to the inference transport layer.
🔬 Emerging Themes
SpecBox (sandbox scheduling), Denial of Deadline (inference pipeline security), and StateAct (program-state vs. pixel paradigm) form a cluster at three layers of agent infrastructure — sandbox runtime, network transport, and interaction interface. This isn't "prompt engineering" anymore. The agent stack is becoming a real systems engineering discipline.
Codifying the Judge / PAJAMA shows evaluation moving from LLM-as-oracle to programmatic distillation. Combined with last week's RIPO and Regression Tax papers, the message is clear: evaluation infrastructure needs the same engineering rigor as the systems it evaluates.
The Physics of Multi-Turn Long-Horizon Planning provides the first controlled experimental framework for understanding how planning ability is acquired (pretraining), shaped (post-training), and integrated (multi-teacher distillation). Agent planning is shifting from alchemy to chemistry.
⚡ Quick Hits
🔮 AI-Adjacent Breakthroughs
📡 Weekly Pattern Watch
The agent infrastructure engineering cluster (SpecBox + Denial of Deadline + StateAct) suggests arc #57 (Agent Failure as Infrastructure) continues deepening. Evaluation reform (PAJAMA) extends arc #27 into an 11th session. New arc candidate: Agent Runtime Optimization as Formal Discipline — speculative scheduling, program-state interaction, and protocol distillation converge on the insight that agent infrastructure IS the problem, not the model.