Guide

AI Agent Audit Dashboard from the CLI

Your CISO asks which AI agents touched the mailbox last week, what they ran, and how often they failed. The raw audit log has the answers, but nobody reads 1,000 JSON lines in a meeting. This guide turns nylas audit logs into a dashboard: a per-agent activity table, error rates, a deterministic session replay for incident review, and a cron job that snapshots it all daily.

Written by Qasim Muhammad Staff SRE

Verified — CLI 3.1.16 · last tested June 6, 2026

Command references used in this guide: nylas audit init, nylas audit logs show, nylas audit logs summary, and nylas audit export.

What is an AI agent audit dashboard?

An AI agent audit dashboard is an aggregated view of every command your agents executed: which agent ran what, when, against which account, and with what result. It answers accountability questions in seconds instead of grep sessions. The underlying data is the audit log, where each entry records the command, the agent source, the status, the duration, and the Nylas request ID (8 fields in total, documented in the hub guide below).

The dashboard layer matters because raw logs don't survive contact with a meeting. A week of moderate agent activity produces hundreds of entries; a compliance review wants 4 numbers: actions per agent, error rate, blocked operations, and the trend. The OWASP AI Agent Security Cheat Sheet recommends treating the model as untrusted and enforcing controls at the tool layer — which is exactly where this log records. SOC 2 Type II auditors expect 90 days of retention for autonomous system logs, and they ask for summaries, not dumps. The agent audit guide covers enabling and filtering the log; this page covers turning it into the report.

How do you collect audit data from every agent?

Every agent that acts through the CLI writes to the same audit log once logging is initialized, so collection is one command: nylas audit init --enable. From that point each command (whether a human typed it or an agent invoked it through MCP) is recorded with a source field that identifies the caller platform.

The 3 known platforms (claude-code, github-actions, and terminal) are detected automatically. Custom agents that aren't auto-detected (an OpenClaw workflow, a Manus-driven automation, an in-house orchestrator) set the NYLAS_CLI_SOURCE environment variable so their entries stay distinguishable in every panel below. Without it, all custom callers collapse into one bucket and the per-agent table loses its meaning.

# One-time setup
nylas audit init --enable

# Label a custom agent's activity (set in the agent's environment)
export NYLAS_CLI_SOURCE="openclaw-assistant"

# Verify entries are flowing
nylas audit logs show --limit 5

How do you get a dashboard without writing any code?

The nylas audit logs summary command is the zero-setup dashboard: total commands, success rate, top commands by frequency, and a breakdown by agent source for any window you pass with --days. The data is local, so even a month of heavy agent activity summarizes in well under a second, with no API round-trip involved.

Run it as a table for humans, or add --json when a script consumes the aggregates. The weekly health check below is the form most teams pin to a Monday reminder.

# Weekly view, human-readable
nylas audit logs summary --days 7

# Monthly view as JSON for downstream tooling
nylas audit logs summary --days 30 --json

Which jq aggregations make the per-agent panels?

Custom panels come from piping nylas audit logs show --json through jq's group_by and select filters. The 3 aggregations below cover what reviews actually ask for: a list of active agents with command counts, an error-rate column, and the slowest operations. Each one runs locally in well under a second on a 1,000-entry log.

The first expression is also the literal answer to "list every agent writing audit logs": it groups entries by source and counts them, so any agent that executed even one command appears as a row.

# Panel 1: active agents, command counts, error rates
nylas audit logs show --limit 1000 --json | jq '
  group_by(.source) | map({
    agent: .[0].source,
    commands: length,
    errors: [.[] | select(.status == "error")] | length
  }) | sort_by(-.commands)'

# Panel 2: top 10 commands across all agents
nylas audit logs show --limit 1000 --json | jq '
  group_by(.command) | map({command: .[0].command, runs: length})
  | sort_by(-.runs) | .[:10]'

# Panel 3: operations slower than 5 seconds
nylas audit logs show --limit 1000 --json | jq '
  [.[] | select(.duration_ms > 5000)
       | {command: .command, duration_ms: .duration_ms, source: .source}]'

An empty Panel 1 result means no agent has written an entry — usually because audit logging was enabled after the agents started, or a custom agent's environment is missing the NYLAS_CLI_SOURCE label. Both are 1-line fixes.

How do you replay an agent session deterministically?

Session replay means reconstructing the exact ordered sequence of commands an agent ran in a time window, so an incident review works from evidence instead of recollection. Audit entries carry timestamps and per-command Nylas request IDs, which makes the reconstruction deterministic: the same query always yields the same ordered trail, and each row links to the specific API request behind it.

Bound the window with --since and --until, then sort by timestamp. For any suspicious row, the request ID pivots to the full entry detail.

# Reconstruct one agent's session, in execution order
nylas audit logs show --source openclaw-assistant \
  --since 2026-06-05 --until 2026-06-06 --json | jq '
  sort_by(.timestamp)
  | .[] | {time: .timestamp, command: .command, status: .status, request_id: .request_id}'

# Pivot into a single suspicious entry
nylas audit logs show --request-id req_abc123

This trail is what makes the dashboard defensible to a security team: the read-then-send sequences that signal prompt injection (covered in the injection defense guide) show up here as adjacent rows with timestamps seconds apart.

How do you snapshot the dashboard on a schedule?

A scheduled export turns the live log into dated evidence files that survive log rotation and machine changes. The nylas audit export command writes JSON or CSV, filters by date, and defaults to 10,000 entries per run — enough headroom that a daily export never truncates. One cron line produces a file per day.

# Daily 23:55 snapshot of that day's agent activity (one line — cron
# entries cannot span lines)
55 23 * * * nylas audit export --since "$(date +\%Y-\%m-\%d)" --output "$HOME/audit-snapshots/agents-$(date +\%Y\%m\%d).json"

# Monthly compliance bundle as CSV
nylas audit export --since 2026-05-01 --until 2026-05-31 \
  --output may-2026-agent-activity.csv --format csv

Snapshots also feed whatever charting layer your team already has — a spreadsheet, Grafana over a JSON datasource, or a 20-line script that renders the jq panels into a Markdown report. Keep 90 days of files to match the SOC 2 retention expectation, and the next audit request is a directory listing, not a project.

Next steps