Measuring before trusting

For a while I trusted my agents on feel. They seemed to be doing sensible work, the changes were landing, and I had no particular reason to worry. Then I looked at how much of my plan allowance had gone in a single week and could not explain it. No tool had failed, but I had no record of where it went, and a system I cannot account for is one I should not be leaving alone.

That is where the usage side of Murongi came from. I decided to measure first and extend trust afterwards.

Counting each run

I count each run, including the ones that fail and the ones I stop. I used to count only the successes, because those were the ones I could point at. It gave me a flattering picture. The failures and the abandoned attempts were where a lot of the usage went, and leaving them out hid exactly the part I needed to see.

For each run I want to know a few plain things:

• Which task it belonged to, and which agent and machine ran it.
• How much of the allowance it used, and how that compares with similar tasks.

With those two answers I can tell a task that was expensive because it was hard from a task that was expensive because something looped. Those need different responses, and before I counted, I could not tell them apart.

A meter and an alarm

The agents I use run on plans that give a certain amount of use in each window of time. When a plan runs low, an agent starts failing or slowing down, usually at an inconvenient moment. I wanted to see that coming. Monizard, a usage meter I built, reads how much of each plan has been used and writes it where other tools can read it. It also marks a reading as stale when it has not refreshed, so I do not act on an old number.

The Limited guardrail reads from that meter. If a plan is running hot, Murongi can shift new work to another plan or another vendor that has room, or let the queue grow a little until the window resets. Where a single run is using more than a task of its kind should, the run is stopped and the task goes back to the queue with a note, as I described in the post on guardrails.

Next to the meter I set a few alarms, and I choose them so that each one means something I would act on. An alarm that fires on ordinary busy days teaches me to ignore it. I would rather have fewer alarms that I believe. The ones I keep are for a queue growing faster than the machines can drain it, and for a plan close to its limit.

What the numbers changed

Once I could see the counts, a few habits changed. I stopped giving large, vague tasks to agents, because those were the ones that wandered. I broke work into smaller pieces with a clear end. I also started matching the task to the plan, sending heavy jobs to wherever there was room and keeping light ones for the cheaper corners of the set-up.

The counting also made reviewing easier. A run that looks odd in the numbers gets read first. It is a cheap way of deciding where my own attention goes, and my attention is the scarcest thing in the whole arrangement.

I should say what the measurements do not tell me. A low count does not mean the work was good, and a high one does not mean it was wasteful. They are a prompt to look, and I still read what lands. What they give me is a way to notice early, so that trust in the agents is something I have evidence for instead of a mood. The home page shows the idea in a small example, and each figure there is made up for illustration.