Advanced AI Agent Systems Architecture and Evaluation Training Course

Course overview

AI agents move beyond single prompts when they must plan multi-step work, call business systems, hand work between specialised agents, retain state, and operate within security and cost constraints. The engineering challenge is not merely selecting a model: teams must define bounded responsibilities, reliable tool contracts, escalation paths, evaluation datasets, and production telemetry. Without these controls, agent pilots can produce inconsistent decisions, unsafe actions, hidden latency, and costs that cannot be explained to stakeholders.

This five-day course develops the architecture and evaluation practices required to build dependable multi-agent systems. Participants design agent graphs, orchestrate tool use, implement memory and state management, apply Model Context Protocol (MCP) patterns, and establish human-in-the-loop approval controls. They create testable agent contracts, construct representative evaluation sets, score task success and trajectory quality, diagnose failure modes, and use tracing data to improve reliability, latency, and token expenditure.

Instruction combines architecture briefings with practical labs using LangGraph, OpenAI Agents SDK, LangSmith, and OpenTelemetry. Working from a realistic enterprise case—an agent system that investigates service incidents and proposes controlled remediation—participants produce an architecture decision record, agent interaction diagram, tool-risk register, evaluation plan, and production-readiness scorecard. These deliverables can be adapted directly for an internal pilot, vendor assessment, or engineering design review.

The course is designed for experienced AI, data, platform, and software professionals who are responsible for moving agentic AI from proof of concept into governed operational use. Managers gain a clearer basis for approving architectures, defining acceptance criteria, and directing investment toward systems that can be tested and observed.

Course objectives

By the end of this course, participants will be able to:

• Design a bounded multi-agent architecture using agent graphs, role contracts, and explicit handoff rules
• Implement tool-calling workflows with typed input schemas, permission boundaries, and retry policies
• Configure short-term state and long-term memory strategies for traceable agent interactions
• Apply Model Context Protocol patterns to expose enterprise data and actions through controlled tool interfaces
• Build an evaluation dataset with golden tasks, adversarial cases, expected outputs, and pass criteria
• Measure agent quality using task-success, trajectory, safety, latency, and cost metrics
• Instrument agent runs with traces, spans, structured events, and failure classifications
• Produce a production-readiness scorecard covering security, observability, escalation, and operational ownership

Benefits of attending

Target competencies

Who should attend

• AI Solution Architects — who must design governed agent platforms and defend architecture choices
• Senior Machine Learning Engineers — who build agent workflows that require repeatable evaluation and monitoring
• Software Architects — who integrate agents with enterprise APIs, identity controls, and operational systems
• Data Science Leads — who need to establish quality metrics and test datasets for agent-enabled services
• Platform Engineers — who operate the tracing, deployment, access-control, and reliability layers around agents
• Technical Product Managers — who define measurable acceptance criteria for enterprise agent use cases

Requirements and prerequisites

Participants should be comfortable reading Python, working with JSON and REST APIs, and explaining core large language model concepts such as prompts, tokens, context windows, embeddings, and function calling. Practical experience building at least one LLM application or retrieval-augmented generation prototype is strongly recommended. Familiarity with Git, virtual environments, API keys, and basic cloud deployment concepts will help during labs. This is not a beginner prompting course and does not teach Python fundamentals. Deep learning model training, advanced mathematics, Kubernetes expertise, or prior experience with LangGraph or OpenAI Agents SDK is not required.

Training methodology

Each day alternates concise instructor-led architecture sessions with guided build labs and design-review discussions. Participants work in small technical teams on an enterprise incident-response agent case, progressing from agent boundaries and tool contracts to evaluation and observability. Labs use prepared repositories, test APIs, trace data, and failure scenarios rather than isolated prompt exercises. The instructor critiques design choices against reliability, security, cost, and operability criteria. On the final day, each participant converts their work into an application plan for a live organisational use case.

Course outline

Day 1: Agent system architecture and boundaries

• Single-agent, supervisor, swarm, and hierarchical orchestration patterns
• Agent responsibility boundaries and least-privilege design
• State machines, directed acyclic graphs, and cyclic agent workflows
• LangGraph nodes, edges, conditional routing, and checkpoints
• Typed agent input and output contracts using JSON Schema
• Human-in-the-loop approval points and exception escalation
• Architecture decision records for agent system trade-offs

Workshop: Design an incident-response agent graph and produce an architecture decision record defining agents, handoffs, state, and approval gates.

Day 2: Tools, memory, and controlled enterprise access

• OpenAI Agents SDK tool definitions and function-calling lifecycle
• Tool schema validation, idempotency, retries, and timeout handling
• Model Context Protocol servers, resources, prompts, and tool exposure
• Authentication propagation and scoped credentials for agent actions
• Short-term conversation state and checkpoint recovery
• Long-term memory retrieval, write policies, and retention controls
• Tool-risk registers for read, write, and irreversible operations

Workshop: Implement controlled diagnostic and remediation tools for the case agent, then produce a tool-risk register with permissions and failure responses.

Day 3: Evaluation design for agent behaviour

• Agent evaluation units: final answer, tool use, trajectory, and system outcome
• Golden task selection from production-like workflows
• Adversarial, ambiguous, and policy-violating test cases
• Rubrics for correctness, groundedness, safety, and action appropriateness
• Deterministic assertions and LLM-as-judge evaluation patterns
• Pairwise comparison of prompts, models, and orchestration variants
• Dataset versioning, test splits, and evaluation governance

Workshop: Build a versioned evaluation pack containing golden tasks, adversarial cases, scoring rubrics, and explicit release thresholds.

Day 4: Observability, failure analysis, and optimisation

• OpenTelemetry traces, spans, attributes, and correlation identifiers
• LangSmith run tracing and dataset-linked experiment analysis
• Failure taxonomy for planning, retrieval, tool, policy, and handoff errors
• Latency decomposition across model calls, tools, and orchestration
• Token, model, and tool-cost attribution by workflow path
• Prompt, routing, and tool-contract experiments using controlled comparisons
• SLOs, error budgets, and alert thresholds for agent services

Workshop: Analyse supplied traces from failing agent runs and produce a prioritised remediation backlog supported by quality, latency, and cost evidence.

Day 5: Production governance and implementation planning

• Threat modelling for prompt injection, data exfiltration, and unsafe tool use
• Policy enforcement layers and content or action guardrails
• Fallback models, circuit breakers, and graceful degradation patterns
• Release gates, canary tests, and rollback criteria for agent workflows
• Ownership models across AI, platform, security, and business teams
• Production-readiness scorecards and operational runbooks
• Use-case prioritisation based on value, risk, integration effort, and evaluability

Workshop: Present a production-readiness scorecard and 90-day implementation plan for an agent use case, receiving instructor and peer design-review feedback.

Tools & standards covered

LangGraph, OpenAI Agents SDK, LangSmith, OpenTelemetry

A typical training day

Live online deliveries follow the same structure in the East Africa Time zone, with shorter screen blocks and longer breaks.

What the fee includes

How you can take this course

Certification

Participants who complete the full five days receive the Skillset Development Certificate of Completion, stating the course title, course code, dates and delivery format — suitable for professional-development records and employer reimbursement.

Frequently asked questions