2026 · AI in production
Agents in production whose work can be measured
MCP, a run journal and retros instead of a demo
The problem
A typical LLM rollout looks like this: a prototype is built, shown on a call, wired into the process — and then nobody knows whether anything improved. There is no record of what the agent did, what it cost, or where it failed. A month later the tool is quietly abandoned and nobody knows why.
The model is rarely the problem. The missing loop is: without a journal and metrics an agent stays a demonstration even when it formally works.
What I built
- An MCP server for the platform: the agent reaches tasks, discussions, members and the repository through one protocol instead of homemade scripts.
- A run journal: every run records status, outcome, commits, changed files, tests, deployment and cost.
- Closed value lists instead of free text: outcome is validated on write, so a typo fails loudly instead of landing silently in the database.
- A reaper: a run nobody closed for twelve hours is closed automatically with a clear reason.
- Retro review: a separate agent reads the journal and proposes changes to its own instructions — from data, not impressions.
- Applied scenarios: support triage, classification, notifications, in-app support, a daily digest for mentors.
How it is measured
The point of the journal is that the result is visible without reading transcripts. A separate screen shows what was delivered next to what it cost: runs, commits, deploys, what did not land and why. Because the fields are closed lists, the numbers stay comparable week to week.
What backs it up
- Journal schema version 3: provider, outcome, commits, tests, deployment, error class, source.
- Daily reconciliation with the local database: tokens, price and model are pulled in automatically.
- Agents work in the same loop as people: same tasks, same rules, same reporting.
What you can reuse
If you already have an LLM in a process and cannot answer whether it helped, you need the journal first and improvements second. In that order: without measurement, prompt and model changes are guesswork.