Guides

How Companies Are Building AI Agents in Production

Updated 2026-11-04

AI agents are moving from demos to production, and the engineering blogs coming out of that transition tell a consistent story: the hard part is not the model. This guide summarizes what practitioners and companies have shared publicly about building agents that actually work at scale: the patterns, the failure modes and the lessons that keep showing up.

If you're preparing for an AI engineering interview, agent architecture and production operations are now standard topics. Everything here is cited from published engineering blogs and reports.

The agent is ~20% of the system

One of the most common observations from practitioners who have shipped agents to production: the LLM call itself is a small share of the work. As one engineer wrote after shipping to production, "most effort went to orchestration, retries, caching, monitoring, permissions, rate limiting, tool integration, state management, evaluation and cost."

A production agent pipeline spans authentication, memory, a planner, tool selection, knowledge retrieval, a vector database, multiple APIs, the LLM, guardrails, validation and observability. Most failures occur between components, not inside the model.

This matters for interviews: when you're asked to design an agent system, the orchestration and reliability layers are what interviewers want to hear about.

Workflows vs. agents: know when to use which

Anthropic's "Building Effective Agents" guide, one of the most widely referenced resources in the field, draws a clear line between workflows (deterministic orchestration of LLM calls) and agents (models that dynamically decide which tools to use and when to stop).

The guide describes several workflow patterns:

The practical takeaway: start with workflows. Reach for agents only when the task genuinely requires dynamic tool selection and the decision space is too wide for a predetermined sequence. As the Anthropic guide puts it, the simplest solution that works is the right one.

What works in production

Engineering blogs from teams that have shipped agents converge on several patterns.

Tool calling is where the value is

Agents that execute business workflows (creating tickets, updating databases, calling APIs, sending messages) deliver more value than agents that only generate text. The tools an agent can call define what it can actually do for users.

A report analyzing millions of agent interactions on Anthropic's API found that software engineering accounts for nearly 50% of agentic tool calls. Agents perform best where outputs are easy to verify: running code, testing an API call, checking a database query.

Context outweighs model size

Multiple practitioners report the same finding: a smaller model with strong context beats a larger model with weak context. Useful context includes system state, business rules, permissions, documentation, user intent and execution history.

This is directly relevant to interview system design questions. When you're asked "how would you improve this agent's accuracy?", the answer is usually better context management before a model upgrade.

Evaluation has to be continuous

Every release should be checked for answer quality, cost, latency, tool success rate, hallucinations and user success. Without this, releases become experiments on users.

Teams that ship agents reliably describe evaluation as a first-class system: golden test sets, automated regression checks and CI-gated deployment. This matches what interviewers expect. Evaluation is a baseline, not a bonus.

Observability is non-negotiable

Teams need end-to-end tracing of prompts, tool calls, retrieved documents, model decisions, latency and user outcomes. When an agent makes a bad decision, you need to reconstruct exactly what it saw, what it decided and why.

Cost is an architectural decision

Production agent systems use several techniques to control token costs: semantic caching (avoid re-running identical queries), model routing (use a smaller model for simple tasks), context compression (send only what's relevant), selective retrieval, batching and asynchronous workflows. As one practitioner put it: "The cheapest token is the one you never sent."

What goes wrong

Agent failure modes are different from traditional software bugs, and interviewers test whether you know them.

Multi-agent coordination failures

Systems with multiple specialized agents face coordination problems that mirror microservices challenges: duplicated reasoning, conflicting decisions, token cost explosion and latency growth. Each agent needs a clear, non-overlapping responsibility.

Human oversight gaps

A study of millions of agent interactions found that about 73% of agent tool calls have a human somewhere in the loop, but about 0.8% of actions appear irreversible (sending an email to a customer, making a financial transaction, deploying to production). The report warns that "average usage masks the extremes": most actions are safe, but the ones that aren't can cause real damage.

Experienced users don't approve everything blindly. Data from Claude Code sessions shows that interrupt rates actually rise with experience, from about 5% of turns for new users to about 9% for experienced users. More experienced operators monitor and intervene more, not less.

Trust calibration

The same study found that among new users, about 20% of sessions use full auto-approve. By 750 sessions, that rises to over 40%. But the 99.9th percentile turn duration nearly doubled from October 2025 to January 2026, suggesting agents are taking on longer, more complex tasks. The report describes a "deployment overhang," where models can handle more autonomy than users currently grant them, but the gap is closing.

The reliability gap

Users notice slow responses (over 20 seconds), tool failures, lost memory, broken workflows, incorrect permissions and inconsistent answers more than small model quality gains. Reliability beats raw intelligence in production.

What this means for your interview

If you're preparing for AI engineering interviews, agent architecture is now a standard topic. Here's what the engineering blogs suggest you should be ready to discuss:

  1. When to use agents vs. workflows, and the specific patterns (chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer)
  2. Tool design: how to make tool calls safe, idempotent and observable
  3. Context management: what context the agent needs and how to keep it within token limits
  4. Failure modes: coordination failures, cost explosions, trust and safety gaps
  5. Evaluation and observability: how to know the agent is working, continuously
  6. Cost control: caching, routing, compression and batching strategies

These map directly to the production challenges companies face. Practice thinking about them on GenAIDojo's problem set. Several problems cover agent loops, tool calling bugs and evaluation failures.


Sources