Write the Mismatch Log Before You Rewrite the Prompt

Most prompt rewrites start too late.

Someone gets a bad output. They edit the wording. They add a sentence about being concise, careful, or accurate. The next run looks better, so the team calls it fixed.

That is not prompt debugging. That is guessing with a better user interface.

The useful artifact is the mismatch log.

A mismatch log records the gap between what the human meant, what the prompt said, what the model appeared to infer, what it returned, and what happened downstream.

That sounds small. It changes the work.

Intent is not the same thing as text

The research on intent comprehension makes this problem explicit. Measuring Intent Comprehension in LLMs defines a model as understanding intent when it responds consistently to different prompts that express the same underlying purpose, and differently when the purpose changes.

That is the bar users care about. They do not judge the model by whether it answered the literal string. They judge whether it understood what they wanted.

Another paper, Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation, argues that some multi-turn failures come from a gap between user expression and model interpretation. The model can lock onto a plausible but wrong assumption and keep building on it.

That maps directly to real work. A product manager asks for "a sharper launch brief." The model thinks sharper means shorter. The PM meant stronger objections, clearer risk language, and a better buyer narrative.

The output is not random. It is a mismatch.

If you only rewrite the prompt, you lose the evidence.

Prompt sensitivity is an operations problem

ProSA studied prompt sensitivity at the instance level and found that semantically similar prompt variants can still produce meaningfully different behavior.

That matters because prompt debugging often happens one example at a time.

One rewrite fixes the example in front of you. It might break three others. Or it might only work because the next input was easier.

This is why a prompt log should not stop at "old prompt" and "new prompt." It should include:

• what the user intended
• the exact request or trigger
• the prompt or Skill version
• rendered variables and retrieved context
• model and runtime settings
• tool calls and tool outputs
• the model output
• the failure mode
• the human note
• the downstream result
• the decision made after review

That schema turns a bad output into a test case.

Clarification is one fix, not the fix

Clarifying questions help when the problem is ambiguity. They are not a universal patch.

Clarify When Necessary breaks ambiguity repair into three subtasks: deciding when to ask, deciding what to ask, and using the answer correctly.

Asking the Right Question at the Right Time shows why this is harder than it sounds. Human clarification behavior and model uncertainty do not mirror each other cleanly.

So the practical rule is not "always ask more questions."

The rule is: log why the model was unsure or wrong, then decide whether the fix is a question, a better input form, a stronger output schema, a new example, a tool boundary, or a human checkpoint.

For reusable Skills, this is especially important.

A Skill that says "ask clarifying questions when ambiguous" is weaker than a Skill that defines what counts as ambiguity, what question to ask, when to stop, and what evidence to record.

Prompt defects should be treated like engineering defects

A Taxonomy of Prompt Defects in LLM Systems names the categories teams keep rediscovering: specification and intent defects, input and content defects, structure and formatting defects, context and memory defects, performance defects, and maintainability defects.

That taxonomy is useful because it stops every failure from becoming "the prompt was bad."

Maybe the prompt is underspecified.

Maybe the input included stale context.

Maybe the output schema was undefined.

Maybe the parser expected one field name and the model returned another.

Maybe the prompt was copied in three places and only one version changed.

Prompting in the Wild studied prompt changes across open source repositories and found that prompt evolution is a real maintenance problem, not a one-time writing task.

Prompts are software artifacts now. Skills are software artifacts too. They need version history, evals, comments, and change notes.

The field-guide workflow

For teams using AI tools inside real workflows, I would start with a simple five-step loop.

1. Capture the mismatch. Record the user intent, prompt version, model output, failure mode, and downstream result.

2. Label the failure. Use a small taxonomy. Intent mismatch, missing constraint, stale context, output format error, unsafe tool action, weak evaluator, or downstream integration mismatch.

3. Cluster before rewriting. Do not rewrite the prompt for one weird case unless the risk is high. Look for repeated patterns.

4. Choose the smallest fix. The fix might be a clarification question, an example, a schema validator, a retrieval change, an approval gate, or a Skill edit.

5. Promote the mismatch into an eval. Once a case matters, add it to the regression set.

This lines up with the direction of current observability and evaluation tooling. OpenTelemetry has GenAI spans for model calls, parameters, token usage, errors, conversations, and tool calls, while treating prompt and output capture as sensitive and opt-in.

Datadog links prompt templates and versions to traces.

MLflow Prompt Registry treats prompts as versioned artifacts with aliases, rollback, and lineage to evaluations.

LangSmith and Promptfoo show the same operational shape from the evaluation side: production traces become datasets, assertions catch regressions, and human review stays in the loop for subjective cases.

The security note

Do not blindly log everything.

Prompts and outputs can contain customer data, secrets, internal policies, source code, or malicious instructions from untrusted inputs. The log itself becomes part of the system.

That means mismatch logs need data boundaries:

• redact sensitive content when possible
• store raw traces in a controlled place
• keep references when full content capture is too risky
• mark untrusted input clearly
• separate exploratory examples from stable regression cases
• review any production log before reusing it in evals or training

The point is not to collect more data. The point is to preserve enough evidence to make the next change defensible.

Prompt rewrites should come after the mismatch is understood.

Otherwise, the team is just negotiating with a black box and calling the latest answer progress.