Real-Time Anomaly Scoring for AI Agents via OpenTelemetry

TL;DR

• Kakunin exports four standardized OTLP metrics per agent: risk score gauge, event count counter, revocation status indicator, and verification latency histogram.
• These feed directly into existing SIEM/observability stacks (Datadog, Grafana, Honeycomb, Splunk) via vendor-neutral OTLP/HTTP.
• Resource attributes (agent.id, cert.serial, tenant_id) let SOC teams correlate telemetry with the exact cryptographic identity the gateway verified.
• Trace context propagation lets analysts follow a delegation chain across multiple certificate-bearing sub-agents in one trace.
• OTLP telemetry supplements — never replaces — the WORM-backed audit_log, which remains the compliance system of record.

What metrics does Kakunin export via OpenTelemetry?

Kakunin exports four standardized OTLP metrics for every registered agent: a risk score gauge (0.00-1.00, real time), an event count counter, a revocation status indicator, and a verification latency histogram. These ship over OTLP/HTTP directly into existing SIEM and observability platforms like Datadog, Grafana, Honeycomb, and Splunk, tagged with resource attributes tying each data point back to the exact certificate identity the gateway verified.

Observability Standards for Autonomous Systems

As enterprises deploy autonomous AI agents into core production workflows, security operations centers (SOCs) and SRE teams require real-time visibility into agent behavioral metrics. Proprietary dashboards create data silos that prevent unified threat detection across enterprise infrastructure.

OpenTelemetry (OTLP) has emerged as the global CNCF standard for vendor-neutral observability telemetry. Kakunin natively exports agent behavioral risk scores, anomaly metrics, and trace spans via standard OTLP/HTTP protocols directly into existing SIEM and monitoring platforms, including Datadog, Grafana, Honeycomb, and Splunk.

---

Telemetry Schema & Endpoint Docs

Kakunin exports four standardized OTLP metrics for every registered agent: risk score gauge, event count counter, revocation status indicator, and verification latency histogram. Review complete endpoint specifications in our REST & OpenTelemetry API Documentation.

Resource and Span Attribute Schema for Agent Identity

A risk score gauge is only as useful as the identity it's attached to. In a multi-agent deployment — where a supervisor agent delegates sub-tasks to specialist agents, each holding its own X.509 certificate — the OTLP resource attributes are what let a SOC analyst answer the question "which certificate produced this behavior?" without cross-referencing a separate lookup table. Kakunin populates the OpenTelemetry Resource object for every exported metric and span with a fixed set of attributes derived directly from the agent's certificate, not from a mutable application config, so the identity in the telemetry backend always matches the identity the mTLS gateway enforced at request time.

The core resource attributes are `agent.id` (the UUID Kakunin assigns at registration), `agent.cert.serial` (the X.509 serial number, matching the value written to `audit_log` on issuance), `agent.tenant_id`, and `agent.cert.kms_key_arn` truncated to the key alias rather than the full ARN, since the full ARN is sensitive infrastructure metadata that has no analytic value in a dashboard. These four fields let a query in Grafana or a saved search in Splunk group every metric and span by the exact cryptographic identity that was verified at the gateway, rather than by an application-layer username or session token that could be spoofed independently of the certificate.

Trace Correlation Across Multi-Agent Workflows

For span-level data, Kakunin follows the OTel semantic convention of propagating a single `trace_id` across every agent that participates in a workflow, then differentiating participants with span attributes rather than separate traces. When a supervisor agent calls a specialist agent through Kakunin's mTLS-gated API gateway, the gateway injects the standard `traceparent` header so the child agent's span becomes a child of the parent's span in the same trace — this is unmodified W3C Trace Context propagation, not a Kakunin-specific mechanism, which is precisely why it interoperates with existing SIEM trace ingestion.

Each span additionally carries `agent.delegation.parent_id` and `agent.delegation.depth`, so a SOC team can reconstruct the full delegation chain for a single user-facing request even when it fans out across five or six certificate-bearing agents. Combined with the risk score gauge tagged to the same `agent.id`, this means an analyst investigating an anomalous outcome can pivot from the trace waterfall directly to the risk score history of every agent that touched the request, in the same tool, without leaving the observability platform to query Kakunin's dashboard separately.

Alerting on the Risk Score Gauge

The risk score gauge is a 0.00–1.00 float sampled on every scored behavioral event, and it's the metric most teams wire alerting to first, because it maps directly onto Kakunin's own auto-revocation threshold. Setting the alert condition to fire below the 0.85 revocation line is intentional: the alert should notify a human before or at the moment auto-revocation happens, not after, so the SOC has context loaded by the time downstream systems start rejecting that agent's mTLS handshakes.

• Warning tier: risk score sustained above 0.60 for 5 minutes — early drift, not yet actionable revocation-wise, worth a Slack notification.
• Critical tier: risk score at or above 0.85 for any single sample — this is the auto-revocation trigger; the alert exists to explain the incident, not to catch something Kakunin missed.
• Rate-of-change tier: risk score increasing by more than 0.30 within a 60-second window — catches a sudden behavioral shift even if the absolute value hasn't crossed 0.85 yet, which is often the more useful early signal for a coordinated attack.

Example: Grafana Alert Rule

A minimal Grafana alerting rule against the OTLP-sourced gauge groups by `agent.id` and `agent.tenant_id`, queries the last 60 seconds to align with Kakunin's revocation SLA, and evaluates every 15 seconds so the alert can never lag more than one evaluation cycle behind the revocation event itself. The query is a simple `max()` aggregation over the gauge rather than an average — averaging a risk score across a window can mask a brief spike that still crossed 0.85 and triggered revocation, which is exactly the event the alert exists to surface.

Datadog monitors follow the same shape using a metric monitor on `agent.risk_score` with the threshold set to 0.85, `evaluation_delay` kept low (the metric is already near-real-time), and a `notify_no_data` rule disabled — a gap in the gauge for a given agent typically means the agent was already revoked and stopped emitting, not that monitoring failed, and treating that as a data-quality alert would just create noise.

OTLP Spans and EU AI Act Article 12 Record-Keeping

Article 12 of the EU AI Act requires high-risk AI systems to maintain automatically generated logs sufficient to identify situations that may result in the system presenting a risk, and to facilitate post-market monitoring. Kakunin's WORM-backed `audit_log` in Supabase/Postgres — append-only, with database triggers that block UPDATE and DELETE — is the system of record that satisfies this requirement on its own; OTLP export doesn't replace it and isn't positioned as a compliance artifact in the same legal sense.

What OTLP spans add is investigative reach that the audit log, by design, doesn't provide. `audit_log` rows are structured and tamper-evident, but each row is a discrete event — a certificate issuance, a revocation, a score update. A trace spanning multiple agents shows the causal sequence between those events: which upstream request triggered the risk-scoring event that produced a given `audit_log` row, and which downstream agent calls happened before revocation took effect. Regulators and internal auditors reconstructing an incident timeline for Article 12 purposes routinely need both — the tamper-proof record of what happened, and the trace of why it happened in that order — and exporting spans to whatever SIEM already ingests the compliance team's other retention data means that second layer doesn't require a bespoke query against Kakunin's own database.

Because the same `trace_id` and `agent.cert.serial` attributes appear on both the OTLP spans and the corresponding `audit_log` entries (Kakunin writes the active `trace_id` into the audit row's metadata column at write time), an auditor can start from either system and land on the exact matching record in the other. Full field-level detail on how audit rows link to trace context is in the /docs/audit-log-schema reference.

SOC/SIEM Correlation Rules for Coordinated Multi-Agent Anomalies

A single agent's risk score crossing 0.85 is a contained event — auto-revocation handles it within the 60-second SLA without human intervention. The harder detection problem is a coordinated anomaly: several agents belonging to the same tenant, or the same delegation chain, each drifting upward in risk score without any individual one crossing the revocation threshold on its own. That pattern looks benign to a per-agent alert but can indicate a shared upstream compromise — a poisoned prompt template, a compromised shared credential, or an adversarial input reused across agents.

Because every metric and span carries `agent.tenant_id` and `agent.delegation.parent_id`, this is a correlation rule any SIEM's existing rule engine can express without custom integration work: group the risk score gauge by tenant, count distinct `agent.id` values with a score above the 0.60 warning tier within the same rolling window, and fire when that count exceeds a small threshold — three or more agents trending upward within five minutes is a reasonable starting point, tuned down for smaller fleets. Splunk correlation searches and Datadog security monitors both support this as a straightforward aggregation over the OTLP metric stream, no custom parser needed since the schema is already flat and typed.

The event count counter is useful here as a denominator — a tenant whose overall event volume spiked at the same time as its risk scores drifted is a different investigative case than one whose volume held steady while scores climbed on the existing traffic. Wiring both metrics into the same correlation rule, rather than alerting on the risk score gauge in isolation, is what separates a genuine coordinated-anomaly signal from a false positive caused by a traffic surge that happens to include a few naturally noisier agent interactions.

Migrating from Proprietary Logging Pipelines

Teams coming from a vendor-locked agent logging setup — a proprietary SDK that ships JSON blobs to a single vendor's ingestion endpoint — typically have three things to unwind before Kakunin's OTLP export can fully replace it: custom field names that don't map to OTel semantic conventions, dashboards built against those custom fields, and alerting rules tied to vendor-specific query syntax. None of this requires a hard cutover; Kakunin's OTLP exporter and a proprietary pipeline can run in parallel during migration since they're independent write paths against the same underlying risk-scoring events.

The practical migration sequence that avoids a monitoring gap: first, point Kakunin's OTLP/HTTP exporter at the new backend (Datadog, Grafana, Honeycomb, or Splunk) alongside the existing proprietary pipeline, and confirm all four metrics — risk score gauge, event count counter, revocation status indicator, and verification latency histogram — are landing with the expected `agent.id` and `agent.tenant_id` attributes. Second, rebuild alerting rules against the OTLP metrics using the thresholds above rather than trying to translate the proprietary vendor's alert syntax field-for-field; the underlying risk semantics are the same but the query language rarely maps cleanly. Third, once the new dashboards and alerts have run in parallel for at least one full revocation-threshold incident (so the on-call team has seen a real 0.85 crossing surface correctly in the new tooling), decommission the proprietary exporter.

• Keep the WORM `audit_log` as the compliance system of record throughout migration — it's independent of both the old and new observability pipeline and is never the thing being migrated.
• Verify attribute parity before cutover: `agent.id`, `agent.cert.serial`, `agent.tenant_id`, and `trace_id` should appear on every exported data point, not just a sample of them.
• Full exporter configuration and endpoint URLs are documented alongside the metric schema in the REST & OpenTelemetry API Documentation; teams evaluating whether their current stack is ready for this migration can run it through Kakunin's /assessment first.

Dashboarding Patterns for SOC Teams

Beyond individual alerts, most SOC teams building on Kakunin's OTLP export settle on a small set of standing dashboard panels rather than trying to surface every available attribute. A fleet-overview panel groups the risk score gauge by tenant and shows the distribution of current scores as a histogram, making it easy to spot a tenant whose agents are collectively trending upward even before any single one crosses an alert threshold. A per-agent drill-down panel plots the risk score gauge alongside the event count counter and verification latency histogram on a shared time axis, since a genuine behavioral anomaly often shows a distinctive combination — rising risk score, rising event count, stable or falling latency — that's easy to spot visually but harder to express as a single alert rule.

A revocation timeline panel, filtered to the revocation status indicator, gives compliance and engineering a shared view of every auto-revocation event across the fleet, which doubles as a lightweight audit trail for standups and postmortems even though the authoritative record remains the WORM-backed audit_log. Teams standing up these dashboards for the first time generally get more value from starting with these three views and expanding based on actual incidents encountered, rather than attempting to build a comprehensive dashboard covering every attribute Kakunin exports before any real production traffic has exercised the system.

Sampling Strategy for High-Volume Agent Fleets

For fleets running thousands of agents generating continuous telemetry, exporting every span at full fidelity can strain both network bandwidth and backend ingestion costs. Kakunin's exporter supports tail-based sampling keyed on the risk score gauge rather than uniform random sampling, so spans associated with elevated or rapidly changing risk scores are always retained at full fidelity while routine, low-risk activity can be sampled down more aggressively. This keeps the traces that actually matter for an investigation fully intact without forcing teams to choose between telemetry cost and investigative completeness across the entire fleet.