Casefile
Agents & tools: multi-agent orchestration · 45–60 hours · $15–40 in API spend
A bounded multi-agent system that triages 900 auto-insurance claims a day, stops when it should, and never pays out without a human.
What it proves
- You decompose a task into specialists with typed contracts and keep the graph bounded, with step, token, dollar and time ceilings enforced in code, not requested in a prompt.
- You make agent runs durable and reproducible: a checkpoint after every node, resume after a killed process, and replay from recorded model and tool outputs.
- You put humans where the risk is: four approval levels mapped to explicit triggers, and a payout tool that is unreachable except through the gate.
- You measure decision quality, not just control flow: accuracy on hand-adjudicated claims, wrong approvals, and a comparison with a single-prompt baseline.
Slotfill
Fine-tuning & model economics · 42–60 hours · $40–120: GPU rental plus API
Fine-tune vs prompt vs distill: decide, with field-level evidence, how to extract 40,000 freight invoices a month into a strict schema.
What it proves
- You check the cost premise with arithmetic before you rent a GPU.
- You separate "valid JSON" from "correct fields": constrained decoding makes schema validity free, so field-level accuracy decides.
- You run a clean experiment: layout-grouped splits, one-factor ablations, an untouched test set, eval numbers over loss curves.
- You can compute a break-even volume and accept the honest outcome where the prompt wins and you ship the prompt.
Fieldnote
Retrieval & RAG: multimodal · 40–55 hours · $20–60 API plus a few GPU hours
Multimodal RAG for 40 field-service technicians: answer questions whose evidence lives in a chart, a wiring diagram or a photo, and show the tech the cropped figure.
What it proves
- You retrieve pages as images when the evidence is visual, instead of hoping a caption kept the numbers.
- You choose a representation per modality for a reason: page-image retrieval (ColPali-style) for manuals, SigLIP-style image embeddings for photos, and transcripts for speech, because speech is language and text is the right form for it.
- You solve the hard part of late fusion: scores from different encoders aren't comparable, so you fuse by rank and rerank with a model that can see the images.
- You ground answers to a region, not just a page, and measure crop accuracy against annotated boxes.
Atlas
Retrieval & RAG: GraphRAG · 40–55 hours · $20–60 in API spend
Knowledge-graph plus vector RAG over 40 companies' public filings, answering two- and three-hop questions with citations and benchmarked by hop count against plain vector RAG.
What it proves
- You know where embeddings fail (multi-hop chains, comparisons across entities, aggregation over relationships) and can add a graph without throwing away vector search.
- You constrain extraction: a fixed ontology, schema-validated output, entity resolution and idempotent writes, with every edge pointing back to the sentence that justified it.
- You keep the model away from raw database queries. Parameterized Cypher templates are a security control and make results reproducible.
- You report an honest benchmark by hop count, including the extra ingestion cost and latency of the graph.
Redline
Safety & security: red teaming · 35–45 hours · $10–40 in API spend
An automated red-team harness that attacks your own P06 Doorman and P04 Sourcebook on every deploy, files findings with repros, and never lets a closed hole reopen.
What it proves
- You think in attack families, delivery channels and severities, mapped to a published taxonomy, rather than collecting jailbreak screenshots.
- You score with deterministic checks first (canary strings, blocked tool calls, outbound URLs) and send only the ambiguous rest to a calibrated judge and a review queue.
- You handle nondeterminism: each attack runs several times, and "closed" means it failed every time.
- You close the loop: findings become issues with repros, fixed findings become permanent regression cases, and the deploy gate enforces zero regressions.
Flywheel
Evaluation & observability · 35–47 hours · $15–40 in API spend
Turn production logs into failure forensics and new eval cases: find which pipeline step broke, then grow a human-verified golden dataset that the CI gate runs.
What it proves
- You treat logs as an eval data source and handle privacy first: redaction before storage, and no raw user text in the repo.
- You localize a failure to a pipeline step, and you measure how often you localize correctly against faults injected at known steps.
- You keep the golden dataset honest: a model may draft a reference answer, but a human verifies every one before it can gate a release.
- You close the loop into CI. New cases flow into Guide 05 · P02 Tripwire, and you track freshness and coverage, not just size.
Canary
Production & LLMOps: safe release · 35–47 hours · $10–30 in API spend
A prompt registry with deterministic assignment, A/B tests that don't peek, and staged rollouts that advance only on non-inferiority and roll back on their own.
What it proves
- You version prompts as immutable artifacts with an audit trail, and roll back by moving a label instead of deploying code.
- You assign traffic with deterministic hash bucketing at the right unit, and you check for sample ratio mismatch.
- You run experiments that survive scrutiny: a precomputed sample size with a single analysis, or a proper sequential method. No peeking.
- You release with a shadow phase, non-inferiority gates and guardrail-based auto-rollback, and you know what "not significantly worse" does and doesn't mean.