Module 1 · Lesson 9

AI Agent Architecture: Model, Tools, Memory and Orchestration

A production agent is not just a prompt. It is a system with clear boundaries around the model, tools, state, control flow, permissions and observability.

Updated August 2026 · Reference architecture

A practical reference architecture

User / API request
       ↓
Authentication & input validation
       ↓
Agent runtime / orchestrator
       ├── Model + instructions
       ├── Tool registry
       ├── Session / working state
       ├── Guardrails / approvals
       └── Tracing / evaluation
       ↓
Deterministic services, APIs, databases, MCP servers
       ↓
Normalized observations returned to the agent

This architecture keeps the probabilistic model inside deterministic application boundaries. That separation makes the system easier to secure and test.

1. Model

The model interprets goals, chooses tools and synthesizes results. Model selection should follow task requirements rather than architecture fashion. Some systems use one model everywhere; others use stronger models for planning and cheaper models for classification, extraction or validation.

2. Instructions

Instructions define role, goals, constraints and behavior. Keep them focused. If a system prompt contains unrelated policies for billing, coding, legal review, deployment and support, that may be a sign the application needs clearer routing or specialization.

3. Tools

Tools connect the model to deterministic capabilities. OpenAI's Agents SDK currently supports function tools, hosted tools, local/runtime tools, agents-as-tools and MCP-backed tools. Tool boundaries are part of your architecture because they define what the model can attempt.

4. Memory and state

Memory should be divided by purpose instead of treated as one giant conversation log.

Working context

The information needed for the current run: user request, recent tool results and intermediate state.

Short-term conversational memory

Information that should persist across turns in a conversation. OpenAI's Agents SDK provides sessions as a persistent memory layer, while LangGraph uses checkpointers for thread-scoped persistence.

Long-term memory

Application-defined facts or preferences that may persist across conversations. LangGraph explicitly distinguishes long-term stores from thread-level checkpoints. Long-term memory needs its own retention, privacy and update policy.

5. Orchestration

Orchestration determines what owns the control flow. A single-agent runner may manage the loop automatically. A graph can make state transitions explicit. A manager agent may delegate to specialists. Handoffs can transfer control to another agent.

Choose the simplest orchestration model that gives you the control you need. More nodes and more agents do not automatically improve quality.

6. Guardrails and approvals

Guardrails validate model input, output or tool behavior. OpenAI's current Agents SDK supports input/output guardrails and tool guardrails. Sensitive side effects should also have deterministic permission checks and, where appropriate, human approval.

7. Tracing and observability

Agent debugging requires more than application logs. You need to see model turns, tool calls, handoffs, guardrails, latency and failures as one run. OpenAI's Agents SDK includes built-in tracing across generations, tools, handoffs and guardrails. Comparable observability is valuable regardless of framework.

8. Evaluation

Do not evaluate only the final prose. A useful agent evaluation can measure task completion, correct tool selection, number of steps, unnecessary calls, policy violations, latency and cost. Tool design should be tested using representative tasks rather than a handful of happy-path prompts.

9. Deterministic services remain the source of truth

Your agent should call existing application services instead of duplicating business logic in prompts. A billing tool should call the same validated billing service used elsewhere in your product.

10. Failure boundaries

Decide what happens when each dependency fails. Timeouts, malformed tool results, partial workflows and interrupted runs should have explicit handling. If a write can be retried, make it idempotent.

Example architecture: support agent

Authenticated Support API
    ↓
Triage / Support Agent
    ├── get_customer_profile (read)
    ├── get_order_status (read)
    ├── search_policy (read)
    └── request_refund (approval + deterministic validation)
    ↓
Audit log + trace + final answer

Notice that the agent can investigate freely within a safe read-only tool set, but a refund crosses a stronger application boundary.

Architecture checklist

• Who authenticates the user?
• Which tools are read-only vs side-effecting?
• Where are permissions enforced?
• What state persists, for how long, and why?
• What is the maximum number of steps?
• Which actions require approval?
• How are runs traced and evaluated?
• Can writes be retried safely?
• How are secrets kept out of model context?
• What happens if the run stops halfway through?

Official references