Skip to main content

How to Measure AI Coding Assistant Productivity: A Framework for Engineering Teams

· 9 min read
Scott Havird
Scott Havird
Engineer

How to Measure AI Coding Assistant Productivity: A Framework for Engineering Teams​

Here's a question I get asked constantly: "How do you know if AI coding tools are actually making your team more productive?"

It's a fair question. Engineering leaders are investing real budget in Claude Code, Cursor, and GitHub Copilot seats. Developers are restructuring their workflows around these tools. But when someone asks for data — actual numbers on impact — most teams have nothing to show.

I've been working on this problem for over a year, first as an engineering leader trying to justify AI tooling investments at Georgia-Pacific, and then by building PromptConduit to close the analytics gap. Here's the framework I've developed for measuring what actually matters.

The Analytics Gap in AI Coding Tools​

Let me state the obvious: the AI coding tools themselves don't help you measure their impact. Here's what each major tool gives you today:

ToolBuilt-in AnalyticsWhat's Missing
Claude CodeSession transcripts (JSONL)No dashboards, no aggregation, no team view
CursorNoneNo usage data exported at all
GitHub CopilotAcceptance rate, lines suggestedNo context on what was accepted or why
Gemini CLISession logsNo structured analytics

GitHub Copilot comes closest with acceptance rate metrics, but "acceptance rate" is a vanity metric. A developer who accepts 90% of suggestions might be building the wrong thing faster. A developer who accepts 30% might be doing more complex work that requires more iteration.

The gap isn't just about data — it's about the right data.

What to Actually Measure​

After tracking my own AI coding usage for months and analyzing the patterns, I've found that useful metrics fall into three categories.

Category 1: Usage Patterns (What's Happening)​

These are descriptive metrics that establish a baseline of how your team uses AI tools.

Prompts per day per developer — Are people actually using the tools? You'd be surprised how many teams buy seats that go underutilized. In my experience, heavy users send 50-100+ prompts per day. Light users send fewer than 10. The gap between them is often a training problem, not a motivation problem.

Tool distribution — Which AI tools are people reaching for? If your team has access to Claude Code and Cursor, are they using both? Do different tasks gravitate toward different tools? I found that I use Claude Code for complex multi-file changes and Cursor for quick inline edits — but I only discovered that pattern by tracking it.

Tool invocations per session — What actions is the AI actually taking? File edits, bash commands, web searches, reads? The ratio tells you a lot. Sessions dominated by file reads suggest the AI is struggling to understand the codebase. Sessions dominated by edits suggest it's in flow.

Session duration and depth — How long are AI interactions? Are they quick Q&A sessions or deep multi-turn collaborations? Longer sessions aren't necessarily better — they can indicate friction.

Category 2: Quality Signals (Is It Working)​

These metrics tell you whether AI-assisted work is meeting your quality bar.

Iteration count — How many back-and-forth exchanges does it take to get a working result? If a developer consistently needs 8+ iterations for a simple feature, they might benefit from better prompting techniques or a different tool for that task type.

Commit attribution — What percentage of commits involved AI assistance? This isn't about replacing humans — it's about understanding the collaboration ratio. In my workflow, 70-80% of my commits now involve some AI assistance, but the nature of that assistance varies enormously.

Test pass rate on AI-generated code — Does code written with AI assistance pass tests at the same rate as code written without it? If AI-assisted commits have a higher failure rate, that's a signal that review processes need adjustment.

PR review feedback — Are PRs with AI-assisted code getting more review comments? Fewer? Different types of comments? This is harder to track automatically but reveals a lot about quality.

Category 3: Impact Metrics (Does It Matter)​

These connect AI tool usage to business outcomes.

Time to first commit on new features — Does AI assistance reduce the ramp-up time when developers start work on unfamiliar parts of the codebase? This is one of the clearest productivity signals I've found.

Context switching cost — AI tools can serve as a knowledge bridge. If a developer can ask Claude Code about a codebase instead of interrupting a colleague, that's a measurable reduction in context switching for the whole team.

Scope of individual contributions — Are developers working across more files, more languages, or more systems than before? AI assistance often expands the range of work a single developer can handle confidently.

A Practical Tracking Approach​

You don't need a custom platform to start measuring. Here's a tiered approach:

Tier 1: Git-Based Attribution (Start Here)​

The simplest measurement uses what you already have — git history.

Add AI tool attribution to your commit messages:

feat: add user authentication flow

Implemented OAuth2 PKCE flow with refresh token rotation.

AI-Tool: claude-code
AI-Session: abc123def456

Then query it:

# Count AI-assisted commits this month
git log --since="1 month ago" --grep="AI-Tool:" --oneline | wc -l

# Breakdown by tool
git log --since="1 month ago" --grep="AI-Tool:" --format="%b" | grep "AI-Tool:" | sort | uniq -c

# Compare PR merge time: AI-assisted vs not
# (requires scripting against your GitHub/GitLab API)

This costs nothing, requires no new tools, and gives you a baseline within a week.

Tier 2: Session Analytics (Go Deeper)​

Claude Code stores session transcripts as JSONL files in ~/.claude/projects/. These contain every prompt, tool invocation, and response — a goldmine for analytics.

# Find recent sessions
ls -t ~/.claude/projects/*/*.jsonl | head -10

# Count prompts per session
cat session.jsonl | jq -c 'select(.type == "human")' | wc -l

# Extract tool usage
cat session.jsonl | jq -c 'select(.type == "tool_use") | .name' | sort | uniq -c | sort -rn

This is exactly the problem I built PromptConduit to solve. The CLI captures events in real-time across Claude Code, Cursor, and other tools, normalizes them into a common schema, and pipes them to a dashboard where you can see patterns across sessions, projects, and team members.

Tier 3: Team-Wide Measurement (Scale It)​

For engineering leaders managing multiple teams, you need aggregated views:

  • Team adoption dashboards — Who's using AI tools, how often, and for what?
  • Productivity trend lines — Is AI-assisted velocity increasing over time as the team gets better at prompting?
  • Cost-per-outcome metrics — What's the API cost for an average feature? An average bug fix?
  • Cross-tool comparison — Which AI tool performs best for which task categories?

This tier requires tooling purpose-built for AI coding analytics — either custom-built internal dashboards or platforms like PromptConduit that aggregate across tools.

Metrics to Avoid​

Not everything worth measuring is worth optimizing. Avoid these traps:

Lines of code generated — More lines isn't better. AI tools excel at generating boilerplate, but the most valuable AI-assisted work often reduces code through better abstractions.

Suggestion acceptance rate — As mentioned above, this is a vanity metric. High acceptance doesn't mean high quality.

Time saved (self-reported) — Developers are terrible at estimating how long things "would have taken" without AI. Don't ask them. Measure outcomes instead.

Prompt count as a productivity score — More prompts doesn't mean more productivity. It might mean the opposite — more iteration because the AI isn't understanding the task.

The ROI Question​

When leadership asks "what's the ROI of our AI coding tools?", here's how I frame it:

Direct cost: Seat licenses + API usage. For most teams, this is $20-100/developer/month for seats plus variable API costs.

Measurable impact: Track time-to-first-commit on new features, number of files touched per PR (scope expansion), and developer satisfaction surveys. These are imperfect but directional.

The honest answer: We're in the early innings. The teams investing in measurement infrastructure now — even basic git attribution — will be the ones who can make data-driven decisions about AI tooling in 12 months. The teams flying blind will keep guessing.

What I've Learned From My Own Data​

After months of tracking my AI coding patterns through PromptConduit, here are the patterns that surprised me:

  1. My most productive sessions are short. 10-15 minute deep sessions with a clear goal outperform hour-long exploratory sessions. The AI works best when I know what I want and can articulate it precisely.

  2. File reads dominate early; edits dominate later. At the start of a feature, 70% of tool invocations are reads and searches. By the end, 80% are edits and writes. The AI's "understanding phase" is a real and measurable phenomenon.

  3. Cross-project context is the biggest friction. Switching between projects resets the AI's understanding. This is where good CLAUDE.md files and hooks for context re-injection make the biggest difference.

  4. Tool choice matters less than prompting skill. My productivity metrics are roughly similar across Claude Code and Cursor. The variable isn't the tool — it's how clearly I describe what I want.

Getting Started​

If you take one thing from this post: start with git attribution today. Add AI-Tool: trailers to your commits. After a month, you'll have enough data to see patterns. After three months, you'll have enough to make decisions.

If you want deeper analytics across tools, check out PromptConduit — it's the platform I built specifically for this problem.

And if you're an engineering leader trying to justify AI tooling spend, remember: the question isn't "are AI coding tools worth it?" (they are). The question is "which tools, for which tasks, with which teams?" That's a measurement problem, and it's solvable.