PixioLab

Portfolio projects

17 AI engineering portfolio projects

A portfolio project counts when it proves one skill a hiring manager is screening for, with numbers. These are the 17 projects in Guide 05, The Portfolio Project Catalog, in 3 tiers: each with the skill it proves, the hours it takes and what it costs to run. The guide has the full spec, architecture, ship gates and interview story for each.

By PixioLab

Tier 1: Foundations

Groundtruth

Retrieval evaluation · 18–26 hours · Under $10 in API spend ($0 with local models)

A retrieval evaluation harness that proves whether a change to a legal research assistant's search actually helped, measured on 150 human-reviewed queries and enforced in CI.

What it proves
  • You separate retrieval failure from generation failure and measure each with the right metric.
  • You can build a golden dataset with a written, human-reviewed labeling protocol that covers every relevant passage and the queries with no answer.
  • You compare configurations fairly: one factor at a time, the same queries, a fixed context budget and a paired confidence interval.
  • You can make an evaluation deterministic enough to gate pull requests without false alarms.

Tripwire

Evaluation in CI · 16–24 hours · Under $25 in API spend (about $1–3 per full run)

A CI eval gate that fails any pull request whose prompt or model change makes a support-routing feature measurably worse, and stays quiet when nothing real changed.

What it proves
  • You treat prompts, model IDs and settings as versioned code with tests that run on every change.
  • You check behaviors, not strings: code checks for structure and labels, and a judge from a different model family only where code cannot decide, checked against your own labels.
  • You can tell a regression from noise with repeated runs, a paired test on the cases that flipped and a tolerance stated in cases rather than percent.
  • You make the gate operational: a required check that cannot get stuck, no secrets for fork PRs, a known cost per run and a baseline that changes only on purpose.

Warmstart

Cost and latency optimization · 14–22 hours · Under $15 in API spend

A three-layer cache (exact match, semantic match and the provider's prompt cache) for a support assistant answering 400,000 questions a month, with a measured false-hit rate and no cross-customer leaks.

What it proves
  • A semantic cache is a retrieval system with a precision problem. You choose its threshold from a measured precision curve, not from a blog post.
  • Your cache keys carry everything that changes the answer (tenant, prompt version, model, parameters, tool set), and personalized intents never touch the shared cache.
  • You layer cheap and safe first: exact match, then semantic match, plus the provider's prompt cache for the static prefix, and you can say which layer saves what.
  • You report the result honestly: hit rate, cost and latency on replayed traffic, and a false-hit rate with its sample size and upper bound.

Tier 2: Core build

Sourcebook

Retrieval & RAG · 35–50 hours · $10–30 API spend (est.)

Cited, hybrid-search answers over 4,000 internal docs for a 600-person company.

What it proves
  • You handle identifiers and paraphrases, and show what BM25, dense, fusion and reranking each add.
  • You treat citations as claims. Every cited sentence is checked against the chunk it points to.
  • You measure retrieval and answers separately, using the harness from Guide 05 · P01 Groundtruth.
  • With the stretch goal, you enforce access control inside the query, not after it.

Switchyard

Production & LLMOps · 40–55 hours · Under $10 (mocks)

An LLM gateway for six AI features: survives outages, bills every call to a team.

What it proves
  • You design for failure: deadlines, retry budgets, breakers and fallback chains that differ by request class.
  • You make cost act, not just report: per-request attribution and budgets that hold under concurrency.
  • You prove failover with a scripted chaos run, not a diagram.
  • You know when to buy instead, and you can say what building one taught you.

Doorman

Safety & security · 25–35 hours · $10–30 API spend (est.)

Layered injection defenses for a recruiting agent that reads 2,000 uploads a week.

What it proves
  • You reason about hostile input and a gullible model: the attack surface is the tools, not the prompt.
  • You layer defenses and measure each one: attack success rate, false positives, per-layer recall.
  • You put the real control in the action layer: allowlists, taint rules, human confirmation.
  • You treat the fairness and legal risk of hiring tools as engineering requirements.

Querybound

Text-to-SQL & guardrails · 30–40 hours · $10–25 API spend (est.)

A text-to-SQL analyst for 200 questions a day that cannot change a single row.

What it proves
  • You put safety in the database: a read-only role is the boundary, AST checks a second layer.
  • You validate SQL with a real parser and allowlists, not regex or a non-validating tokenizer.
  • You measure execution accuracy on public benchmarks and your own schema, not string match.
  • You surface ambiguity, and you back confidence with checks rather than the model's self-report.

Intake

Structured extraction · 35–45 hours · $15–40 API spend (est.)

Extraction with per-field provenance and review for 15,000 invoices a month.

What it proves
  • You build extraction as a pipeline with contracts: schema, provenance, validation, routing.
  • Every value carries a page and a bounding box, so a reviewer can verify it in seconds.
  • You set auto-approval from a measured error rate and understand the automation trade-off.
  • You produce the labeled data that Guide 05 · P12 Slotfill fine-tunes on.

Plumbline

Evaluation · 25–35 hours · $10–25 API spend (est.)

An LLM judge for a docs assistant, calibrated on 200 human labels and gated in CI.

What it proves
  • You never trust an unmeasured judge: you report TPR, TNR and kappa against held-out human labels.
  • You write rubrics that can be scored reliably: binary criteria, defined in words, with anchors.
  • You quantify the known judge biases (position, length, self-preference) and design around them.
  • You turn a calibrated judge into a ship gate with a statistically honest decision rule.

Docdrift

AI in CI/CD · 25–35 hours · Under $15 API (est.)

A GitHub Action that flags docs a pull request made stale and suggests fixes.

What it proves
  • You put an LLM into a developer workflow with the right blast radius: it comments, a human merges.
  • You build the eval set from real git history and report precision and recall, not a demo.
  • You use cheap deterministic signals first and the LLM only where needed, so each PR stays cheap.
  • You handle CI realities: tokens, permissions, forks and secrets.

Tier 3: Advanced

Casefile

Agents & tools: multi-agent orchestration · 45–60 hours · $15–40 in API spend

A bounded multi-agent system that triages 900 auto-insurance claims a day, stops when it should, and never pays out without a human.

What it proves
  • You decompose a task into specialists with typed contracts and keep the graph bounded, with step, token, dollar and time ceilings enforced in code, not requested in a prompt.
  • You make agent runs durable and reproducible: a checkpoint after every node, resume after a killed process, and replay from recorded model and tool outputs.
  • You put humans where the risk is: four approval levels mapped to explicit triggers, and a payout tool that is unreachable except through the gate.
  • You measure decision quality, not just control flow: accuracy on hand-adjudicated claims, wrong approvals, and a comparison with a single-prompt baseline.

Slotfill

Fine-tuning & model economics · 42–60 hours · $40–120: GPU rental plus API

Fine-tune vs prompt vs distill: decide, with field-level evidence, how to extract 40,000 freight invoices a month into a strict schema.

What it proves
  • You check the cost premise with arithmetic before you rent a GPU.
  • You separate "valid JSON" from "correct fields": constrained decoding makes schema validity free, so field-level accuracy decides.
  • You run a clean experiment: layout-grouped splits, one-factor ablations, an untouched test set, eval numbers over loss curves.
  • You can compute a break-even volume and accept the honest outcome where the prompt wins and you ship the prompt.

Fieldnote

Retrieval & RAG: multimodal · 40–55 hours · $20–60 API plus a few GPU hours

Multimodal RAG for 40 field-service technicians: answer questions whose evidence lives in a chart, a wiring diagram or a photo, and show the tech the cropped figure.

What it proves
  • You retrieve pages as images when the evidence is visual, instead of hoping a caption kept the numbers.
  • You choose a representation per modality for a reason: page-image retrieval (ColPali-style) for manuals, SigLIP-style image embeddings for photos, and transcripts for speech, because speech is language and text is the right form for it.
  • You solve the hard part of late fusion: scores from different encoders aren't comparable, so you fuse by rank and rerank with a model that can see the images.
  • You ground answers to a region, not just a page, and measure crop accuracy against annotated boxes.

Atlas

Retrieval & RAG: GraphRAG · 40–55 hours · $20–60 in API spend

Knowledge-graph plus vector RAG over 40 companies' public filings, answering two- and three-hop questions with citations and benchmarked by hop count against plain vector RAG.

What it proves
  • You know where embeddings fail (multi-hop chains, comparisons across entities, aggregation over relationships) and can add a graph without throwing away vector search.
  • You constrain extraction: a fixed ontology, schema-validated output, entity resolution and idempotent writes, with every edge pointing back to the sentence that justified it.
  • You keep the model away from raw database queries. Parameterized Cypher templates are a security control and make results reproducible.
  • You report an honest benchmark by hop count, including the extra ingestion cost and latency of the graph.

Redline

Safety & security: red teaming · 35–45 hours · $10–40 in API spend

An automated red-team harness that attacks your own P06 Doorman and P04 Sourcebook on every deploy, files findings with repros, and never lets a closed hole reopen.

What it proves
  • You think in attack families, delivery channels and severities, mapped to a published taxonomy, rather than collecting jailbreak screenshots.
  • You score with deterministic checks first (canary strings, blocked tool calls, outbound URLs) and send only the ambiguous rest to a calibrated judge and a review queue.
  • You handle nondeterminism: each attack runs several times, and "closed" means it failed every time.
  • You close the loop: findings become issues with repros, fixed findings become permanent regression cases, and the deploy gate enforces zero regressions.

Flywheel

Evaluation & observability · 35–47 hours · $15–40 in API spend

Turn production logs into failure forensics and new eval cases: find which pipeline step broke, then grow a human-verified golden dataset that the CI gate runs.

What it proves
  • You treat logs as an eval data source and handle privacy first: redaction before storage, and no raw user text in the repo.
  • You localize a failure to a pipeline step, and you measure how often you localize correctly against faults injected at known steps.
  • You keep the golden dataset honest: a model may draft a reference answer, but a human verifies every one before it can gate a release.
  • You close the loop into CI. New cases flow into Guide 05 · P02 Tripwire, and you track freshness and coverage, not just size.

Canary

Production & LLMOps: safe release · 35–47 hours · $10–30 in API spend

A prompt registry with deterministic assignment, A/B tests that don't peek, and staged rollouts that advance only on non-inferiority and roll back on their own.

What it proves
  • You version prompts as immutable artifacts with an audit trail, and roll back by moving a label instead of deploying code.
  • You assign traffic with deterministic hash bucketing at the right unit, and you check for sample ratio mismatch.
  • You run experiments that survive scrutiny: a precomputed sample size with a single analysis, or a proper sequential method. No peeking.
  • You release with a shadow phase, non-inferiority gates and guardrail-based auto-rollback, and you know what "not significantly worse" does and doesn't mean.

The 45-day path builds one of these, Sourcebook, from Day 1. See the roadmap.

See the guide