Monitoring an AI Agent: Run Records, Health Signals, and Alerts Worth an Interruption
Back to Blog
AI Agents· 8 min read

Monitoring an AI Agent: Run Records, Health Signals, and Alerts Worth an Interruption

A practical monitoring design for scheduled AI agents: one structured record per run, a few health signals, a review-queue check, and alert rules that avoid noise, without logging raw customer data.

ML
By Mneme Labs
NEPA AI · Building autonomous systems for creators and businesses
#ai-agents#monitoring#logging#operations

A scheduled agent can fail without anyone noticing. It runs on time and reads nothing. Its login expires and the job exits quietly. Drafts accumulate in a review queue that nobody opens. Monitoring exists to answer three questions: did the agent run, did it do something sensible, and does anything need a person right now?

This guide is about visibility. It is separate from the retry guide, which covers how an agent recovers from a failure. It reports no measured results from any deployment. The record format, signals, and alert rules below are a design you can adapt and test on your own system.

Grid of six groups of fields for a run record: identity, timing, outcome, volume, provenance, and what is left out by default
One record per run, grouped by purpose. The last group lists what the record deliberately omits. Original Mneme Labs diagram.

Write one structured record per run

Start with a single line of structured data at the end of every run, whether it succeeded or failed. The OWASP Logging Cheat Sheet says each log entry should capture when, where, who, and what, and recommends an interaction identifier that links every event belonging to one interaction. A run_id plays that role for an agent. Give every step in a run the same run_id and you can reassemble what happened later.

The figure groups fields by purpose. Here is one illustrative record, formatted across lines for readability. In a JSON Lines file, serialize each whole record on one line:

{
  "run_id": "2026-11-15T13:00Z-inbox-triage-a1b2",
  "job": "inbox_triage",
  "tenant": "shop_main",
  "trigger": "schedule",
  "started_at": "2026-11-15T13:00:02Z",
  "ended_at": "2026-11-15T13:00:41Z",
  "status": "completed_with_skips",
  "items_seen": 12,
  "items_skipped": 2,
  "drafts_created": 10,
  "error_classes": {"transient": 1},
  "provider_request_ids": ["req_example"],
  "model": "model-id-from-config",
  "prompt_version": "triage-v3",
  "input_ids": ["msg_01", "msg_02"],
  "output_ref": "queue/2026-11-15/13",
  "review_state": "pending"
}

Keep timestamps in UTC and format them the same way everywhere. Record the provider's request ID when the API returns one. Anthropic's error reference, for example, includes a request_id in error responses so that a support team can trace a specific call. If you later adopt OpenTelemetry, you will find the idea familiar: its traces documentation describes a span as a unit of work with a name, timestamps, attributes, and status, and notes that spans resemble structured logs with correlation and hierarchy added. You do not need that tooling to begin. A run record that shares an ID across steps is a useful starting point for a small agent; it does not replace cross-service traces.

Log identifiers and counts, not content

The OWASP cheat sheet lists data that should usually not appear in logs directly and should instead be removed, masked, hashed, or encrypted. The list includes access tokens, session identifiers, passwords, connection strings, encryption keys, payment data, and sensitive personal data. It treats names, phone numbers, and email addresses as data that may need special handling before they are recorded. It also advises keeping logs out of web-accessible locations and restricting file permissions.

For an agent that reads email or customer messages, that translates to a plain default: store message IDs and counts, and leave the message text in the system where it already lives. If a bug can only be diagnosed with content, make capture an explicit switch for one run, with a short retention period and limited access. The last panel in the figure above lists the fields this default leaves out. The security guide covers the broader exposure risks.

Watch five signals

Google's SRE book chapter on monitoring distributed systems says that if you can only measure four things, measure latency, traffic, errors, and saturation. An agent maps onto that list with one addition.

  1. Freshness. How long since the last successful run, compared with how often it should run? This catches an agent that stopped without saying anything.
  2. Error mix. Count errors by class: transient, permanent, needs a person, and unknown. The SRE chapter notes that errors include implicit failures, such as a successful response carrying the wrong content. Your own validators, such as a schema check or a banned-phrase check, are how an agent notices those.
  3. Volume. How many items did the run see? Zero is sometimes correct, but zero for many runs in a row, or a sudden spike, deserves a look. Decide what counts as unusual from a baseline you have actually observed.
  4. Queue age. The SRE chapter defines saturation as how full a system is. For an agent that feeds human review, the review queue is what fills up. Track the age of the oldest unreviewed item and the number waiting.
  5. Output validity. How many drafts fail validation, and why do reviewers reject the ones they do? Rejection reasons tell you more than a pass rate.

Run duration is the latency signal, and cost per run, computed from the usage figures your provider reports, belongs on a weekly view. None of these needs a dashboard to start. A script that reads the run log is enough.

Check from outside the agent

A job that has crashed cannot report that it crashed. Put the freshness check in a separate small script, on a separate schedule, that reads the run log and notifies you if no successful record has appeared within about two intervals. This is the same idea as a dead-man's switch.

import json
import pathlib
from datetime import datetime, timedelta, timezone


def last_success(log_path, job):
    best = None
    for line in pathlib.Path(log_path).read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        rec = json.loads(line)
        if rec["job"] != job or rec["status"] not in {"completed", "completed_with_skips"}:
            continue
        ended = datetime.fromisoformat(rec["ended_at"].replace("Z", "+00:00"))
        if ended.tzinfo is None:
            raise ValueError("run timestamps must include a time zone")
        best = ended if best is None else max(best, ended)
    return best


def is_stale(log_path, job, interval_minutes, now=None):
    """True means stale or monitoring data needs investigation."""
    if interval_minutes <= 0:
        raise ValueError("interval_minutes must be positive")
    now = now or datetime.now(timezone.utc)
    try:
        last = last_success(log_path, job)
    except (OSError, ValueError, KeyError, TypeError):
        return True  # a missing or corrupt log must not look healthy
    return last is None or last > now or now - last > timedelta(minutes=2 * interval_minutes)

This is a sketch, not a tested package. It reads the whole file each time, which is fine for a small log and wasteful for a large one. If the monitor and the agent run on the same machine, a power loss takes both down, so for anything important run the check somewhere else.

Decide which signals interrupt you

The SRE chapter recommends actionable alerts tied to user-visible symptoms, with separate channels for urgent and less urgent findings. Apply that distinction to the work your agent is supposed to deliver. For an agent, the symptom is "nothing reached the queue." The cause, such as an auth error or a changed API response, goes in the run record's error fields.

Routing grid placing seven agent signals into three channels: notify now, daily digest, and weekly review
A starting split of signals across three channels. Move a signal only after it has proven noisy or too quiet. Original Mneme Labs diagram.

A workable starting split, shown above:

  • Notify now for clear failures: no successful run within about two intervals, a login that needs a person to renew it, and a write left in an unknown state.
  • Daily digest for conditions that matter but can wait: the age of the oldest unreviewed item, skipped items and validation failures, and a streak of zero-item runs.
  • Weekly review for slow movement: cost per run drifting up, reviewer rejection reasons, and retry counts trending upward.

The SRE chapter also recommends removing collected signals that no dashboard or alert uses, and alert rules that rarely fire. Review your rules every few weeks and drop what you never act on.

A short routine you will keep doing

Each day, read the digest and open the oldest item in the review queue. Each week, pull five approved items at random and compare them with their sources. Scan the rejected items for a pattern, such as one input type that keeps producing bad drafts. Check whether each alert that fired was worth the interruption. If the routine takes long enough that you start skipping it, remove a signal.

What monitoring cannot tell you

Logs show that the agent ran and what it counted. They do not show whether a plausible draft was wrong. Only sampling against sources catches that. Monitoring also adds work of its own: schemas drift, scripts break, and a noisy alert gets muted. Match the effort to the stakes. A single worker that drafts notes needs a run record, a freshness check, and a queue-age line. A system that touches money or customers needs stricter controls and a person who owns them.

First steps

Add the run record to one existing job, then write the freshness check. Add queue age to a daily digest. Only after a few weeks of using those should you consider more. To see a concrete job that benefits, read the read-only Gmail polling design. Then exercise the retry policy against fake failures and verify that exhausted attempts appear in the digest. Monitoring is useful only when it leads to a clear next action.

Sources