Skip to main content

Measuring AI Work per Prompt, Not per Seat

· 6 min read
Scott Havird
Engineer at Georgia-Pacific · ex-WarnerMedia Innovation Lab (ContentAI) · decade shipping AI-powered platforms

I ran promptconduit cost on a workflow-heavy session and trusted the number. It said $49.02. After I fixed how the CLI walks subagent transcripts, the same session was $762.05. Workflow agents write one directory deeper than the CLI used to look.

Teams usually see AI spend per license or per seat. I wanted cost and behavior per prompt, for a given kind of task.

TL;DR

Per-prompt ledgers split re-read dollars from new context; the Loop panel in the editor extension shows cost drivers, cheaper re-prices, and local history by task kind. CLI v0.36.1 fixed session cost that missed workflow agents; v0.37.0 batches hook uploads.

Motivation​

In April I wrote about measuring AI coding productivity and called raw prompt count a measurement problem. This is the spend side.

Dashboards roll license fees into one line. They do not explain why one ship-the-blog-post prompt cost seventy cents and another cost three dollars, or whether the gap was round-trips, model choice, or cache timing. The unit I care about is a prompt of a given task kind.

What​

The Loop is the main UI. In the public editor extension, PR #122 merged November 13: it is the only panel now. Per prompt you get a loop chart, notes, what would have made this cheaper, how you have done this before (local history by task kind), and how this session got here. I am not claiming a marketplace release; it is merged in the repo.

promptconduit loop, task kinds on CLI events, the platform Loop API, team comparison by task kind, a web view, and the spec are in review. This post covers what shipped and what is still open.

Each prompt becomes a ledger of model calls: model, tokens, dollars, context tokens read, tool names only. Calls after the prompt ends are background wake-ups. Totals split re-read (cache read), new context (cache write plus input), and output. Prompt text never leaves the machine; task kind does, classified locally into ship, debug, test, refactor, docs, review, explore, chore, build, or other. The earliest regex match in the prompt wins, so publish the blog post is ship, not docs. Replies like yes please inherit the session kind when they are three words or fewer and do not match on their own.

How​

Almost all agent cost is re-reading the conversation each round-trip: context size times trips. The old breakdown listed model and token rows without saying why.

My own publish-the-blog-post run on claude-opus-5-5 used eight calls for $0.699; $0.626 was re-reading. The header says $0.70 for this prompt, 89% of it was re-reading the conversation, about 391K tokens re-read eight times. Advice re-prices the same tokens: half the round-trips lands near $0.35; a fresh session uses opening context; switching to claude-sonnet-5 would cost $2.14 because the first call re-caches about 390K tokens (cheaper per token is not cheaper here). A 5m04s gap would have cost more without the one-hour cache; that gap saved about $1.95.

In the extension, how you have done this before compares this prompt to your past runs of the same kind. Team medians and habit chips by task kind are in review on the platform, paid Team plan only, not live.

Numbers were wrong​

A CLI health check found three bugs in subagent transcripts. PR #224 merged November 14 (v0.36.1), closing issue #225. Cost only read flat subagents/*.jsonl; workflow agents live under subagents/workflows/wf_*/agent-*.jsonl. updated_at followed the last applied event, not the latest. Auto-sync on the parent Stop hook never ran for subagents; on one machine 355 of 438 waiting transcripts were subagent or workflow files. I re-walked 242 files with 6,164 priced requests and found no double counting.

The fix recurses subagents/, fixes session time bounds, and syncs subagent files by mtime plus size. My first patch compared mtime to synced_at, which is written after upload, so lines added during upload could be skipped forever. CI caught it. More accuracy fixes are in review.

dir := filepath.Join(strings.TrimSuffix(sessionPath, ".jsonl"), "subagents")
return jsonlFiles(dir)
// ...and workflow agents one level deeper
root := filepath.Join(strings.TrimSuffix(sessionPath, ".jsonl"), "subagents")
var out []string
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil
}
if !d.IsDir() && strings.HasSuffix(d.Name(), ".jsonl") {
out = append(out, path)
}
return nil
})
sort.Strings(out)
return out

Task kinds on the CLI (open PR, not released):

func ClassifyTaskKind(prompt string) string {
text := strings.ToLower(prompt)
best, bestAt := TaskKindOther, -1
for _, r := range taskKindRules {
loc := r.re.FindStringIndex(text)
if loc == nil {
continue
}
if bestAt == -1 || loc[0] < bestAt {
best, bestAt = r.kind, loc[0]
}
}
return best
}

PR #226 (v0.37.0) spools hook events and uploads batches instead of one HTTP request per event; opt out with promptconduit config set --batch-upload=false. Entries leave the spool only after the server accepts or permanently rejects them, so a crash resends and the server dedupes on event id. Queue-based ingest on the private platform merged the same day.

What's Next​

I am finishing CLI Loop commands, the platform API, team comparison, and the web view. v0.36.1 and v0.37.0 are the releases for cost and batching.

Summary​

Measure per prompt and per task kind, not per seat. Ledgers show re-read spend; advice shows cheaper round-trips and when a smaller model would cost more. I trusted $49.02 until recursive transcript discovery fixed the picture. The Loop panel is in the extension source; team views are still in review. See measure AI coding productivity and PromptConduit auth hardening. I cannot link the private platform repo; public code is the CLI and editor extension.