Operate MCP Systems in Production
Design for traffic spikes, retries, partial failures, cost and observability instead of assuming every tool call succeeds.
Make a payment tool retry-safe
A network timeout after charging a card is dangerous: the client does not know whether the payment succeeded. Add an idempotency key so retrying cannot create a second charge.
@mcp.tool()
async def charge_invoice(invoice_id: str, request_id: str):
existing = await payments.by_idempotency_key(request_id)
if existing:
return existing
return await payments.charge(
invoice_id=invoice_id,
idempotency_key=request_id,
timeout_seconds=8,
)Retry only where safe
Reads are usually retryable. Writes need idempotency or a compensating action before automatic retries are safe.
Budget latency
Give each downstream call a timeout so one slow dependency cannot consume the entire agent run.
Trade-off: reliability features add storage and complexity, but duplicate payments, emails or deployments cost far more.