METHOD: Multi-agent system triggered by a natural language research question. Agents autonomously formulate search strategies, retrieve and screen literature, extract quantitative statistics, compute standardized effect sizes (Hedges' g), and perform random-effects meta-analysis — producing a PRISMA-aligned transparent report.
KEY RESULT: Screened 28+ studies across multiple domains. Pooled effect estimates closely matched expert-conducted meta-analyses. The system also performs heterogeneity analysis and risk-of-bias assessment — the full meta-analysis pipeline, automated.
🎓 SO WHAT FOR ANDY: This is the kind of capability that transforms a Big 4 bank's regulatory and risk function. Every major regulatory submission requires systematic evidence synthesis — AML model validation, climate risk assessment, fair lending analysis. An agentic meta-analysis pipeline that produces PRISMA-compliant reports closes the gap between "we have the data" and "we have actionable evidence."
METHOD: Evaluate LLM security agents through a cost-success lens on Cybench (offensive CTF) and Splunk BOTS v1 (defensive SOC). Decompose performance into inference spend + tool spend, compare models at fixed cost levels rather than best-case success.
KEY RESULT: Offensive CTF performance scales with test-time compute — open-weight models approach frontier systems cost-competitively. Defensive SOC investigation does NOT scale the same way: success depends on disciplined tool use, telemetry navigation, and selective enrichment.
🎓 SO WHAT FOR ANDY: Banks spend disproportionately on defensive security. This paper provides rigorous evidence that throwing more LLM compute at defensive security agents is the wrong strategy — invest in tool discipline and telemetry integration instead. The cost-aware evaluation framework is a template for any bank AI deployment cost optimization.
METHOD: Binary-tree recursive partitioning: split a population into subpopulations, prompt LLMs with each, aggregate estimates back to population-level. Test whether LLM outputs satisfy the law of total probability.
KEY RESULT: Widespread violations across frontier models. The "macro fallacy" — fine-grained subpopulation estimates are MORE accurate than direct population-level prompts, but the model fails to propagate this knowledge upward. Statistical self-consistency becomes a new reference-free evaluation criterion.
🎓 SO WHAT FOR ANDY: Directly impacts how the bank uses LLMs for population-level analysis — customer segmentation, credit risk across demographics. The counterintuitive prescription: DON'T ask the LLM for aggregate estimates. Elicit fine-grained conditional estimates and aggregate them externally.
Evaluation Infrastructure as the Bottleneck (Week 4): AutoSynthesis, Beyond Success Rate, and Partition-Prompt-Aggregate converge on the same message — the way we evaluate AI systems is systematically incomplete. From benchmark hacking to cost-blindness to probability violations, the evaluation layer needs fundamental reform. This is now a 4-week sustained narrative arc spanning 18 papers.
Agentic Evidence Synthesis Arrives: AutoSynthesis is the first paper to tackle automated meta-analysis end-to-end with a multi-agent architecture. Part of a broader wave — agentic systems moving from "do the task" to "synthesize the evidence about whether the task was done correctly."
Saturday Infrastructure Theme: Valinor (memory allocation hardware), Don't Predict Prioritize (GPU reliability), and RoboTTT (robot context scaling) all attack infrastructure with architecture-level solutions rather than scale — classic weekend pattern.
Driven-dissipative superconductivity in moiré heterostructures (Huang, Imamoglu, Hafezi, Diehl; arXiv:2607.15169): A theoretical proposal for preparing superconductivity as a stationary state through dissipation rather than attraction — relevant to the long-term quantum computing hardware roadmap.
RoboTTT (Jiang, Chebotar, Fei-Fei, Fan et al., NVIDIA GEAR; 2607.15275) — Test-time training scales robot visuomotor context to 8K timesteps, 3 orders of magnitude beyond SOTA. +87% on manipulation, 10-stage assembly. 16/20.
Don't Predict, Prioritize (Ma, Pei, Lu et al.; KDD 2026; 2607.15115) — GPU failure prediction is inherently unreliable. HeaRank reframes as learning-to-rank: 64% of future failures in top 5% of nodes vs 21% for production baseline. 16/20.
Valinor (Kanellopoulos, Mutlu et al., ETH Zurich; 2607.14789) — Programmable hardware memory allocation. 17× speedup, 16% end-to-end improvement, up to 8% energy reduction for serverless/microservices. 16/20.
In-Place Tokenizer Expansion (Smith, Dakhran et al., Liquid AI; 2607.15232) — Upgrade pre-trained LLM tokenizer without full retraining. 2.4-4× token reduction for Hindi/Vietnamese/Thai, 2.2-3.7× decode speedup. 15/20.
The week of July 14-18: evaluation infrastructure is the binding constraint on agent deployment. Every layer of our evaluation stack — benchmarks, retrieval metrics, safety probes, cost models, probability assumptions — needs rebuilding for the agent era. This is no longer a research finding; it's an engineering mandate.
New Arc #51: Evidence Synthesis as AI Infrastructure — AutoSynthesis represents the first end-to-end agentic approach to perhaps the most stubbornly manual bottleneck in science and policy. For banking, regulatory evidence synthesis is a direct analog.