We Ran /improve on 50 Sessions, Here's What It Found

Monday, the problem: AI agents repeat the same mistakes because every session starts from zero.

Wednesday, the architecture: a 4-stage pipeline (Discovery, Extraction, Raw Traces, Synthesis) that extracts 7 types of structured insights from raw session data.

Today, the data.

We ran /improve on 50 real development sessions from pydantic-deep, approximately 3,200 messages spread across 7 days of active development. No cherry-picking, no synthetic data. Just the raw output of what a self-improving agent pipeline finds when you point it at real work.

TL;DR

• 180+ insights from 50 sessions, extracted automatically from raw session transcripts, zero manual analysis.
• AgentLearning dominates at 45%, build commands, file paths, tool chains. The mundane knowledge that prevents the most repeated failures.
• MEMORY.md: from empty to useful, the pipeline built a structured knowledge base covering user preferences, build setup, project architecture, and effective workflows.
• Confidence scores are calibrated, SOUL.md proposals averaged 0.88, MEMORY.md 0.82, AGENTS.md 0.78. Higher confidence = more consistent signal across sessions.
• 50 sessions is enough, you don’t need thousands of sessions for actionable results. The marginal value decreases quickly for environment facts, keeps growing for patterns and preferences.

The Setup

The sessions covered three types of work:

• Bug fix sessions (~15 sessions, avg 45 messages each)
• Feature implementation sessions (~20 sessions, avg 80 messages each)
• Refactoring sessions (~15 sessions, avg 30 messages each)

We ran ImprovementAnalyzer with default settings, last 7 days, all session types, confidence threshold at 0.7:

from pydantic_deep.improve.analyzer import ImprovementAnalyzer

analyzer = ImprovementAnalyzer(
    model="anthropic:claude-opus-4-6",
    sessions_dir=get_sessions_dir(),
    working_dir=Path("."),
)
report = await analyzer.analyze(days=7)

The pipeline took about 4 minutes to process all 50 sessions. Here’s what came back.

Aggregated Numbers

Insight Type Distribution

The breakdown across all 180+ extracted insights:

• AgentLearning: 45%, discovered behaviors, file paths, build commands, tool chains
• Pattern: 20%, recurring tool call sequences the agent repeated across sessions
• Failure: 15%, mistakes with root causes and resolutions
• UserFact: 10%, facts about the user (language, role, preferences)
• Preference + Context + Decision: 10%, user style preferences, project architecture facts, key decisions

AgentLearning dominating at 45% makes sense. Most of what an agent needs to “learn” is practical knowledge about the environment it operates in, which commands work, where files live, what the build system expects. This is the low-hanging fruit that causes the most repeated failures.

Session-by-Session Examples

Session 1: Bug Fix (45 messages)

The agent was fixing a test failure. Here’s what /improve extracted:

FailureInsight:

description: "Agent tried `pytest tests/` but actual command is `uv run pytest`"
root_cause: "Build system not documented in AGENTS.md"
resolution: "Discovered uv-based workflow after 3 failed attempts"
severity: medium

AgentLearningInsight:

description: "Tests located in tests/ directory, run with `uv run pytest`"
category: tool_chain
confidence: 0.95

PatternInsight:

description: "read → edit → run tests → fix → run tests"
frequency: 4

That FailureInsight is the money. The agent wasted 3 attempts running pytest directly before discovering the project uses uv. Without /improve, it would do this again in the next session. With /improve, the build command gets written to AGENTS.md, permanently.

Session 2: Feature Implementation (80 messages)

Longer session, more diverse insights:

UserFactInsight:

description: "User speaks Polish"
category: language
confidence: 1.0

PreferenceInsight:

description: "User prefers type-safe solutions with Pydantic models"
evidence: "User rejected dict-based approach twice in favor of Pydantic BaseModel"
confidence: 0.85

ContextInsight:

description: "Project uses Textual for TUI, Pydantic AI for agents"
category: architecture
confidence: 0.9

The PreferenceInsight is interesting. The agent observed that the user rejected dict-based approaches twice, not once, which could be a coincidence, but twice across different parts of the codebase. That’s enough signal to encode it as a preference.

Session 3: Refactoring (30 messages)

Shorter session, but high-value insights:

DecisionInsight:

description: "Moved from dataclasses to Pydantic models for serialization"
rationale: "Pydantic provides model_dump_json() and validation"
confirmed: true

AgentLearningInsight:

description: "model_dump_json() preferred over json.dumps(asdict())"
category: tool_chain
confidence: 0.85

DecisionInsights are the rarest type but arguably the most valuable. They capture architectural decisions that shouldn’t be reversed by accident in future sessions.

The Before/After That Matters Most

The real payoff is what happens to MEMORY.md.

Before:

(empty)

After:

## User

- User's name is Kacper
- Primary language: Polish, code in English
- Prefers type-safe Pydantic models over raw dicts

## Build & Test

- Run tests: `uv run pytest`
- Type check: `uv run ty check`
- Lint: `uv run ruff check --fix`

## Project

- TUI built with Textual
- Agent framework: Pydantic AI
- Structured output via output_type parameter

## Effective Patterns

- Read file → understand → edit → verify with tests
- When debugging: check error message first, then trace to source

This is structured, actionable, and immediately useful. Every future session starts with this context instead of a blank slate. The agent knows the build commands, the user’s language, the project architecture, and the effective debugging patterns, all extracted automatically from raw session transcripts.

What the Confidence Scores Tell Us

Not all insights are created equal:

• SOUL.md proposals (avg 0.88): Highest confidence. Personality and style insights are either consistent across many sessions or they don’t appear at all. When the pipeline proposes a SOUL.md change, it’s usually solid.
• MEMORY.md proposals (avg 0.82): Strong confidence. Facts about the user and project tend to be verifiable, the agent either found uv run pytest works or it didn’t.
• AGENTS.md proposals (avg 0.78): Slightly lower. Tool configurations and capability descriptions require more nuance, and the pipeline is appropriately less certain.

The confidence threshold of 0.7 filters out noise. Below that, you get speculative insights like “user might prefer tabs over spaces” based on a single observation. Above 0.7, you get things the pipeline is reasonably sure about.

Why AgentLearning Dominates

The 45% AgentLearning share is not a fluke. Here’s why:

Most agent failures aren’t dramatic. They’re mundane:

• Wrong build command
• Wrong file path
• Wrong import style
• Wrong testing framework invocation

These are exactly what AgentLearningInsight captures, practical, environment-specific knowledge that’s trivial for a human but invisible to an agent starting fresh.

PatternInsight at 20% is the second biggest category because agents develop habits. The “read → edit → test → fix → test” cycle appeared in 4 out of 15 bug fix sessions. That’s not noise, it’s a workflow the agent discovered works for this codebase.

FailureInsight at 15% captures the mistakes that actually happened. The pytest vs uv run pytest example is typical: the failure is specific, the root cause is identifiable, and the resolution is actionable.

Key Takeaways

1. Agents accumulate practical knowledge fast. 50 sessions was enough to build a comprehensive MEMORY.md covering build commands, user preferences, project architecture, and effective workflows.

2. Most “learning” is boring and useful. The highest-value insights aren’t clever architectural observations, they’re build commands and file paths. The mundane stuff that causes real friction.

3. Confidence scores are calibrated well. The pipeline doesn’t oversell its findings. High-confidence proposals are genuinely reliable, and the threshold filters out speculation.

4. The system finds things humans wouldn’t bother documenting. “User rejected dict-based approach twice” is not something a developer would write down. But it’s exactly what helps an agent make better suggestions next time.

5. You don’t need 1,000 sessions. 50 sessions over 7 days produced actionable results. The marginal value of additional sessions decreases quickly for MEMORY.md (there are only so many build commands to discover) but keeps growing for PatternInsight and PreferenceInsight.

Try It Yourself

pip install pydantic-deep

After a few sessions of real work:

# Inside pydantic-deep TUI
/improve

Or programmatically:

from pydantic_deep.improve.analyzer import ImprovementAnalyzer

analyzer = ImprovementAnalyzer(
    model="anthropic:claude-opus-4-6",
    sessions_dir=get_sessions_dir(),
    working_dir=Path("."),
)
report = await analyzer.analyze(days=7)

# Review proposed changes
for change in report.proposed_changes:
    print(f"{change.target_file}: {change.description} (confidence: {change.confidence})")

The review modal in the TUI lets you accept, reject, or edit each proposed change before it’s applied. No automatic writes without your approval.

This wraps up the /improve series. The problem (Monday), the architecture (Wednesday), and now the data (today). Next week, more from the pydantic-deep ecosystem: checkpointing, agent teams, and the production infrastructure that makes all of this work at scale.

GitHub: github.com/vstorm-co/pydantic-deep Vstorm OSS: oss.vstorm.co