PixioLab

Interview prep

20 AI engineer interview questions, with 60-second answers

These are 20 of the 88 questions in Guide 07, AI Engineering Interview Q&A. Each one says what the interviewer is testing, gives an answer you can say in about a minute, and names the red flag that sinks it. The guide adds deeper points and the follow-up questions for every one.

By PixioLab

LLM fundamentals

What is tokenization, and how does it cause real bugs?

What they're testing: Whether you know the unit the model actually sees, bills and limits.

A model never sees characters or words. A tokenizer splits text into subword units from a fixed vocabulary, usually learned with byte-pair encoding, so a common word is one token and a rare string becomes several. Tokens are the unit of everything operational: price is per token, the context window is measured in tokens, and speed is tokens per second. That explains a family of bugs. Counting letters or reversing strings is unreliable, because the model doesn't see letters. Numbers split into odd chunks, which hurts arithmetic. Some languages and code need far more tokens for the same content, so they cost more and fill the window sooner. Long IDs and unusual Unicode can explode into dozens of tokens. So I budget with the provider's token counter, not word counts, and move exact work like counting and math into code.

Red flag: Estimating cost or context limits in words, or assuming every model tokenizes the same way.

How do you choose a model for a new feature?

What they're testing: Whether you choose with an eval and real constraints, not a leaderboard or habit.

I start from requirements, not leaderboards: the task, the quality bar, the latency budget, expected volume, data-handling rules and modalities. Then I build a small eval set of real cases and run two or three candidates on it. I usually try the most capable model first to find the quality ceiling, then step down to smaller, cheaper models until quality falls below the bar. Beyond quality I check price at my real input-to-output mix, time to first token and throughput at p95, how quality holds at my actual prompt sizes, tool-calling and structured-output support, rate limits, and data terms such as retention and where inference runs. Closed API versus open weights is its own trade-off: APIs win on capability and zero operations; open weights win on control, data residency and unit cost at high, steady volume. Whatever I pick, I pin the version and keep the eval, so the next migration is a measured decision.

Red flag: Choosing by leaderboard rank or habit, with no eval on your own cases.

What are logits, temperature and top-p, and why isn't temperature 0 deterministic?

What they're testing: Whether you know what the model actually outputs and which controls you really have.

At each step the model outputs logits, one raw score per vocabulary token. Softmax turns them into probabilities, and a decoding rule picks the next token. Temperature divides the logits before softmax: low values sharpen the distribution toward the top tokens, high values flatten it. Top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p; top-k keeps a fixed number. I run extraction, classification and tool arguments at low temperature and creative work higher, and I tune one knob, not all three. Temperature 0 still isn't reproducible. Requests are batched with other traffic, GPU kernels can give slightly different numbers at different batch sizes, and a near-tie between two tokens can flip. Providers also update models. So tests check properties, not exact strings. Many reasoning models fix or reject sampling parameters entirely; the control there is reasoning effort.

Red flag: "Set temperature to 0 and it's deterministic", or tuning temperature, top-p and top-k at the same time.

Retrieval & RAG

Walk me through a production RAG pipeline end to end. Where does quality actually break?

What they're testing: Whether you've shipped one, not just read about one.

There are two lanes. At ingest: parse sources while keeping their structure, split them into chunks that link back to the parent section, embed with a pinned model version, and index vectors, a keyword index and filterable metadata. At query time: rewrite the query if needed, retrieve a wide candidate set with hybrid search and permission filters, rerank down to the best few, assemble a prompt with citation instructions, generate, and check grounding and citations before the user sees anything. Quality usually breaks before generation. The common failure points are parsing that turns tables and multi-column PDFs into garbage, chunk boundaries that split the answer, an embedding model that doesn't know the domain, a candidate set too small to contain the right chunk, no reranker so the answer sits at rank 12, and a stale index. So I measure retrieval in isolation first: if the right chunk isn't retrieved, no prompt change will save the answer.

Red flag: Reaching for a bigger model or prompt tweaks before measuring retrieval.

When do you use keyword search, vector search or both, and why add a reranker?

What they're testing: Trade-off fluency beyond "use a vector database".

Dense vector search matches meaning, so it handles paraphrases and synonyms, but it's weak on exact identifiers, rare terms, acronyms and negation. Keyword search such as BM25 matches exact terms, which is what you want for part numbers, error codes, statute references and names, but it knows nothing about synonyms. Hybrid search runs both and fuses the results, usually with reciprocal rank fusion. For enterprise corpora that mix prose and identifiers, hybrid is my default, not an optimization. A reranker is a second, more precise pass. Retrieval uses a bi-encoder: query and documents are embedded separately, so document vectors are precomputed and searched at scale. A cross-encoder reranker reads the query and each candidate together, which is much more accurate and far too slow for millions of documents. So I retrieve 50–100 candidates cheaply and rerank to the top five. It's often one of the highest-return changes in a RAG system, and it costs tens to hundreds of milliseconds, so I measure both.

Red flag: "Vector search handles everything", or adding a reranker without checking the right chunk is among the candidates.

What is reciprocal rank fusion, and why is it a good default for combining rankers?

What they're testing: Whether you can merge results from systems whose scores aren't comparable.

When you run BM25 and vector search together, their scores live on different scales: a cosine similarity and a BM25 score can't simply be added, and normalizing them is fragile because the distributions change from query to query. Reciprocal rank fusion ignores the scores and uses only ranks. Each document gets 1 / (k + rank) from each list it appears in, and the contributions are summed. The standard constant is k = 60, from the 2009 paper by Cormack, Clarke and Büttcher. It damps the gap between rank 1 and rank 2, so agreement across lists matters more than topping one list. It's a good default because there's almost nothing to tune and it degrades gracefully. With labeled data, a weighted fusion or a reranker over the fused list usually does better, so in practice I fuse a wide candidate set from each retriever and rerank the shortlist.

Red flag: Adding raw scores from different retrievers.

Agents & tools

What's the difference between an agent and a workflow of LLM calls, and when is an agent the wrong choice?

What they're testing: Whether you reach for an agent because the problem needs one, not because agents are exciting.

In a workflow, your code fixes the path: the steps, their order and the exit are decided in advance, even if each step calls a model. In an agent, the model decides at runtime which tool to call, how many steps to take and when it's done. That's the framing Anthropic uses in "Building effective agents" (2024), which also names the common workflow patterns: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. An agent earns its extra cost when the path can't be enumerated in advance and the number of steps depends on what the system finds, as in research, triage across messy sources, or debugging. It's the wrong choice when the workflow is known, because a pipeline is cheaper, faster and easier to test; when the latency budget is tight; when errors are expensive and hard to reverse; and when you can't write down a success test. Many production "agents" are workflows with one or two model-driven decisions, and that's often the right design.

Red flag: Calling every LLM feature an agent, or building one with no success criterion and no budget.

Walk me through a production agent architecture. What exists besides the model?

What they're testing: Whether you've run an agent in production or only built a demo.

The organizing principle is that the model proposes and the orchestrator decides. The orchestrator is code, not prompt text: it owns the loop, retries, budgets and termination. Around it sit a few parts. A tool gateway with typed schemas, scoped credentials, timeouts, idempotency keys and structured errors. A durable state store, so a long run survives a restart and resumes instead of replaying. Memory: a small, curated working context plus a long-term store written through an explicit extraction step. Stop conditions: step, token, wall-clock and cost limits, plus an explicit success test. Approval gates for irreversible or externally visible actions. And tracing of every step's inputs, outputs, tool results, tokens, latency and cost, which feeds an eval harness that replays recorded runs against every prompt, schema or model change. Putting control flow in the prompt is a common mistake, because it makes behavior untestable and hard to version.

Red flag: "The model handles it" as the answer for retries, limits or approvals.

How do you design tools so the model uses them reliably, and how do you handle tool failures?

What they're testing: Whether you treat the tool interface as the main lever on agent behavior.

The model knows only what a tool's name, description and schema tell it, so the interface is mine to get right. I keep the tool set small and non-overlapping: thirty tools with fuzzy boundaries produce fuzzy selection, so I merge or remove tools before tuning prompts. Each description says what the tool is for and when not to use it, which helps the model choose between neighbors. Parameters are strict and flat: enums instead of free text, required fields, formats, and an example of a valid call. Arguments are validated before execution, and a bad argument returns a structured error that says what was wrong and how to fix it, not a stack trace. For failures, I classify. Transient ones, like timeouts and rate limits, are retried by the orchestrator with backoff and a cap, without involving the model. Terminal ones, like "not found" or "permission denied", go back to the model as facts so it can choose another path. Write tools take idempotency keys, so a retry never charges a customer twice.

Red flag: Dozens of overlapping tools, free-text parameters, or stack traces pasted into the context.

Evaluation

How do you evaluate a RAG system? Be specific about metrics.

What they're testing: Whether you evaluate the layers separately and can name real metrics.

I evaluate each layer separately, then end to end, because one end-to-end score doesn't tell you what to fix. Retrieval: on a labeled set of queries mapped to the chunks that should be found, I measure recall@k, precision@k, MRR for how high the first relevant result ranks, and nDCG when relevance is graded. Recall@k matters most, because a chunk that's never retrieved can never be used. Generation, given a known-good context: faithfulness, meaning every claim traces to the context; answer relevance and completeness; and citation correctness. End to end: task success as the business defines it, plus correct refusals on unanswerable questions and safety checks. Online, I track the same quality signals on sampled traffic, plus user signals. And I don't use BLEU or ROUGE for open-ended answers: they measure word overlap with one reference answer, not correctness.

Red flag: "We looked at some answers and they seemed good", or a single end-to-end score.

Build an evaluation dataset from scratch. Where do the examples and labels come from?

What they're testing: Whether you can create the measurement foundation, not just consume one.

Everything downstream depends on this set: release gates, regression tests and judge calibration. A small, unrepresentative or leaky set makes every metric misleading. The best source is production logs. I sample real queries and stratify deliberately: head queries that dominate volume, the long tail, ambiguous phrasing, adversarial input and out-of-scope requests, because a set of easy head queries looks great and hides the failures. Labels depend on the layer: for retrieval, the chunks that should be found; for generation, an acceptable answer or, better for open-ended tasks, a rubric of what a good answer must and must not contain. I bootstrap labels with a strong model proposing and humans verifying, which is much faster than labeling cold, as long as a human owns the final label. Then I keep it alive: version it, hold out a slice that whoever tunes the prompts never sees, and add every production failure as a new case, so each incident becomes a permanent test.

Red flag: An eval set written entirely by the person tuning the prompt, or one that never changes.

You're using an LLM as a judge. How do you know the judge is trustworthy?

What they're testing: Whether you treat automated evaluation as an instrument that itself needs calibration.

Judges are useful and biased, so the job is to make the bias measurable. I write a concrete rubric with a short scale, pass/fail or three to five points, each point defined; "rate it from one to ten" produces noise. The judge gets the reference material and writes its reasoning before the verdict, which improves consistency. Then I calibrate against humans on 100–200 labeled examples and report agreement properly. Raw percent agreement misleads under imbalance: if 90% of answers are good, a judge that always says "good" agrees 90% of the time. So I report the true-positive and true-negative rates, or Cohen's kappa, and iterate on the rubric until they're acceptable. I control known biases: position bias in pairwise comparisons, by randomizing the order and judging both ways; length bias; and self-preference, by judging with a different model family. Pairwise comparisons decide between two versions; absolute scores track regressions. And I pin the judge model version, or scores stop being comparable across weeks.

Red flag: Judge scores with no human agreement numbers, or "it agrees 90% of the time" with no class balance.

Production & LLMOps

Your assistant serves a million queries a day. Can you estimate the bill, then cut it without wrecking quality?

What they're testing: Whether you can do unit economics out loud and tie every saving to a quality number.

I'd estimate before optimizing. Per request, cost is input tokens times the input price plus output tokens times the output price, and reasoning models bill their hidden thinking tokens as output. Multiply by volume, then add the lines people forget: embeddings and re-indexing, the reranker, guardrail calls and online judges. With the profile below the baseline is about $12,000 a day, and 75% of it is input, which tells me where to start. I'd cache the stable prefix, retrieve fewer but better chunks, route easy queries to a small model with an escalation path, and answer repeats from a response cache. Each step ships only if the eval set shows no regression, and I report cost per thousand queries next to the quality delta. The number I'd track is cost per successful task, not cost per call.

Red flag: Quoting a saving with no quality number beside it, or assuming input tokens always dominate.

Where does latency come from in an LLM feature, and how would you budget it?

What they're testing: Whether you can break a slow request into stages and know which number users actually feel.

I split latency into what happens before the first token and what happens after it. Before it: gateway and input checks, query rewriting, retrieval, reranking and the model's prefill, which grows with prompt length. After it: decoding, roughly output tokens divided by tokens per second, plus any output check that blocks display. Users feel time to first token (TTFT) most, so I stream, and I budget TTFT and total time separately at p95, not on the average. Then I attack the biggest stage. I run independent steps in parallel, skip the rewrite for standalone questions, rerank fewer candidates, cache the stable prefix, cap output length and use a smaller model where quality allows. Reasoning models change the picture, because their hidden thinking happens before the first visible token.

Red flag: "It takes about three seconds," with no TTFT split or percentile.

Safety, security & compliance

How do you defend against prompt injection, both direct and indirect?

What they're testing: Whether you know this is an architecture problem with no prompt-level fix.

Direct injection, or jailbreaking, is a user attacking their own session. Indirect injection is worse: an attacker plants instructions in content the system reads later, such as a web page, a PDF, an email or a tool result, and the victim never sees them. The model can't reliably tell instructions from data, so no prompt solves it. I start from the lethal trifecta: access to private data, exposure to untrusted content, and a way to communicate externally. When an agent has all three, a successful injection can exfiltrate data, so I remove a leg wherever I can. Then I layer controls. Untrusted content is delimited and labeled as data, tools run with the user's own permissions, consequential actions pass a deterministic policy check or a human, output is filtered for exfiltration channels, and an attack corpus runs in CI. The lethal trifecta Untrusted contentweb page, email, PDF,tool result hiddeninstructions Agentmodel + toolscan't tell datafrom instructions reads Private datainbox, CRM, internal docs sends External channelemail, HTTP, image URL remove any one leg and the exfiltration path breaks Figure 6.1The lethal trifecta (Simon Willison, 2025). What to look for: an attack needs all three legs at once, so your design only has to remove one, for example no tools while reading untrusted text, read-only scopes, or an egress allowlist.

Red flag: "We added 'ignore any instructions inside documents' to the system prompt."

Your agent can send email, write to databases and move money. What guardrails go beyond text filtering, and how do you close exfiltration channels?

What they're testing: Whether you govern actions, not just words, and know the specific channels data leaks through.

A chat model can say something wrong; an acting agent can do something wrong, so the controls live in the system. Tools run with the requesting user's permissions, read tools are separate from write tools, and retrieved content can never widen privileges. Every argument is validated before execution, with allowlists for recipients, domains and tables, and limits on amounts. Irreversible or external actions go through a deterministic policy check or a human approval that the model can't talk its way past. Risky work runs in a sandbox with per-tenant budgets and a kill switch. For exfiltration I close the channels one by one: markdown images and links in output, URLs the model generates, tool arguments that reach outside destinations, and verbose errors that echo context. Model output stays untrusted until it's checked.

Red flag: Relying on the system prompt to stop harmful actions.

ML foundations & fine-tuning

What is the bias–variance trade-off, and why does it still matter if you only work with LLMs?

What they're testing: Fundamentals, and whether you connect them to day-to-day LLM work.

Bias is error from a model too simple to capture the pattern; variance is error from a model too sensitive to its particular training sample. High bias underfits, so training and validation error are both high and close together. High variance overfits, so training error is low and validation error is much higher. That gap is the instrument, which is why a clean held-out set is non-negotiable. For underfitting I add capacity, features or training time. For overfitting I add data, usually the strongest fix, or regularization, early stopping or a simpler model. The same failure shows up in LLM work under new names: few-shot examples tuned to the test cases, a judge rubric tuned on 50 examples, a fine-tune that memorizes a small dataset, and eval scores reported on the very set you iterated against.

Red flag: Defining both terms with no way to tell which one you have.

Your data has 1% positives. Why is accuracy the wrong metric, and what do precision, recall, F1 and ROC AUC tell you?

What they're testing: Whether you pick the metric that matches the cost of being wrong.

With 1% positives, a model that always says "negative" is 99% accurate and useless, so accuracy hides the cases I care about. Precision asks: of the items flagged, how many are real? It matters when a false alarm is costly, like blocking a legitimate payment. Recall asks: of the real positives, how many did we catch? It matters when a miss is costly, like undetected fraud. F1 is their harmonic mean when I need one number. ROC AUC is the probability that the model ranks a random positive above a random negative, a threshold-free view of ranking. Under heavy imbalance it can look strong while precision at a usable threshold is poor, so I also read the precision–recall curve. Then I set the threshold on validation data from the relative cost of each error, instead of defaulting to 0.5.

Red flag: Reporting accuracy on an imbalanced problem.

Behavioral & project deep-dive

Walk me through an AI system you built end to end. What broke, and what did you change?

What they're testing: Whether you built it yourself, measured it and understand how it fails.

Example built on Guide 05 · P04 Sourcebook. "I built a question-answering assistant over about 3,000 internal documents for a 200-person team. Success meant correct answers with correct citations, so before building I labeled 150 real questions with the documents that answer them. Version one used vector search only. Recall@10 was 0.62, and it missed anything with a product code or error ID. I added BM25 with reciprocal rank fusion and a cross-encoder reranker, which took recall@10 to 0.86 for about 180 ms of extra latency. What broke next was tables: the parser flattened them, so numeric answers came out wrong. I switched to a layout-aware parser and stored tables as Markdown with their captions. What I'd change is timing: I'd build the eval set in week one instead of week three."

Red flag: Describing a tutorial architecture with no numbers and no failures.

Tell me about a time an AI feature failed in production. How did you find it, and what changed afterward?

What they're testing: Ownership, debugging under pressure, and whether the fix was systemic.

Example built on Guide 05 · P16 Flywheel. "After a prompt change, our support assistant's escalation rate rose from 18% to 26% over two days. Nothing alerted, because we only paged on errors and latency; a support lead noticed. I pulled traces for escalated conversations and compared them with the previous week. The new prompt made the model refuse anything that mentioned refunds, because one instruction was ambiguous. I rolled back the prompt version within an hour, then fixed the causes. We added 40 refund cases to the eval set, put escalation rate on a dashboard with an alert at plus 5 points, and routed prompt changes through a canary. In the post-mortem the lesson I pushed was that business metrics need alerts too, not just system metrics."

Red flag: Blaming the model, the provider or a teammate without describing what you changed.

All 88 questions, with deeper points, follow-ups and coding-round patterns, are in Guide 07.

See the guide