2026 · AI in production

Agents in production whose work can be measured

MCP, a run journal and retros instead of a demo

The problem

A typical LLM rollout looks like this: a prototype is built, shown on a call, wired into the process — and then nobody knows whether anything improved. There is no record of what the agent did, what it cost, or where it failed. A month later the tool is quietly abandoned and nobody knows why.

The model is rarely the problem. The missing loop is: without a journal and metrics an agent stays a demonstration even when it formally works.

What I built

  • An MCP server for the platform: the agent reaches tasks, discussions, members and the repository through one protocol instead of homemade scripts.
  • A run journal: every run records status, outcome, commits, changed files, tests, deployment and cost.
  • Closed value lists instead of free text: outcome is validated on write, so a typo fails loudly instead of landing silently in the database.
  • A reaper: a run nobody closed for twelve hours is closed automatically with a clear reason.
  • Retro review: a separate agent reads the journal and proposes changes to its own instructions — from data, not impressions.
  • Applied scenarios: support triage, classification, notifications, in-app support, a daily digest for mentors.

How it is measured

The point of the journal is that the result is visible without reading transcripts. A separate screen shows what was delivered next to what it cost: runs, commits, deploys, what did not land and why. Because the fields are closed lists, the numbers stay comparable week to week.

What backs it up

  • Journal schema version 3: provider, outcome, commits, tests, deployment, error class, source.
  • Daily reconciliation with the local database: tokens, price and model are pulled in automatically.
  • Agents work in the same loop as people: same tasks, same rules, same reporting.

What you can reuse

If you already have an LLM in a process and cannot answer whether it helped, you need the journal first and improvements second. In that order: without measurement, prompt and model changes are guesswork.