AI Agent — Introduction

The AI Agent is an opt-in subsystem that turns raw application logs into a small, curated catalog of recurring patterns. Once a service has been running long enough for the catalog to stabilize, anything that doesn't fit a known pattern is — by definition — something new, and worth a closer look.

It is off by default (agent.enable: false). Nothing extra runs, no goroutines start, and no files are created until you explicitly opt in.

Why pattern learning?

Most production logs are repetitive: a few hundred templates account for 99% of the volume. The agent solves the problem by building its own picture of "normal" from your traffic, then flagging departures from that baseline.

The pipeline

Every time the agent checks for new logs, each line travels through a short assembly line. Each station does one job; if a station rejects the line, the rest is skipped.

flowchart TD
    A[Read a log line<br/>from your app or Elasticsearch] --> B[Hide sensitive bits<br/><i>passwords, tokens, emails…</i>]
    B --> C{Is this line<br/>worth learning from?}
    C -- no --> X[(ignore)]
    C -- yes --> D[Group with similar lines<br/><i>e.g. all 'connection refused' messages</i>]
    D --> E[Remember the group<br/><i>save to the catalog</i>]
    E --> F{What do we<br/>do with it?}
    F -- training --> T[Just learn it]
    F -- shadow --> S[Pretend to alert<br/><i>log only, useful to review</i>]
    F -- detect --> R[Send a real incident<br/><i>only for unfamiliar lines</i>]

In plain English:

  1. Read a fresh log line from one of your sources.
  2. Hide sensitive parts so secrets never leave your machine.
  3. Decide if the line is interesting. Boring lines (200 OK, health checks, debug noise) are dropped here so they don't clutter the catalog.
  4. Group the line with others that look the same. The agent doesn't store every line it has ever seen — it stores one entry per "shape" of message.
  5. Remember that group, including how often it shows up.
  6. React based on the agent's mode: just learn (training), pretend to alert (shadow), or send a real incident if the line is something the agent has never seen before (detect).

Components

Each component has its own page with the full configuration reference, trade-offs, and examples.

1. Data Sources

What the agent reads from. Two source types ship today: a file tailer for local logs and an Elasticsearch reader for production clusters. Sources are cursor-aware, so the agent always picks up where it left off after a restart.

2. Redaction

Pattern-based scrubbing of secrets and PII (JWTs, AWS keys, bearer tokens, emails, UUIDs, user agents, …). Runs first so no other component — and no external AI model — ever sees the raw values.

3. Regex pre-filter

A small set of named rules plus an optional default_pattern that decide which signals are worth learning from. Lines that match nothing are dropped before the miner sees them, keeping the catalog focused. Set default_pattern: ".*" to learn from every line.

4. Miner

A Drain-style log clusterer that turns a stream of similar lines into a single template with <*> placeholders for variable parts (timestamps, IDs, IPs, etc.). Configurable similarity threshold and tree depth.

5. Catalog

Long-term memory. Every template the miner produces is stored with a first-seen timestamp, sighting count, EWMA frequency, and operator- assignable label/severity/tags. Persisted as data/patterns.json (atomic writes, rotated backups).

6. Worker & modes

The worker glues the components together and runs them on a polling ticker. Three modes:

  • training — observe only. Learn templates and persist them. No alerts of any kind.
  • shadow — same as training, but log a verdict every time a signal would have alerted in detect mode. Useful for reviewing the agent's judgement before going live.
  • detect — emit incidents for genuinely novel patterns. (AI summarization is a follow-up milestone; today this mode logs the verdict only.)

7. Admin endpoints

A small REST surface (/api/agent/*, gated by X-Gateway-Secret) for inspecting the catalog, labeling patterns, and flushing state during training reviews.

  1. Start in training mode for a few days. Confirm the catalog stabilizes and the templates make sense.
  2. Switch to shadow mode. Watch the agent[shadow]: would alert ... log lines for a release cycle.
  3. Promote to detect. Triage incidents the agent emits and keep curating the catalog through the admin endpoints.

Where to next