Prompt Engineering for Production, Not Demos

A clever prompt on your laptop and a prompt that runs in production are two different objects. The first has to work once, in front of you, on an input you chose. The second has to work thousands of times, on inputs you never saw, while you are asleep, without a human in the loop to notice when it quietly goes wrong. Almost everything that makes prompt engineering feel easy in a demo is the thing that makes it fragile in production.

I build product prompts at Spoon Hire AI and automations with Claude Code, and I advise Micro1.ai on LLM-training product work. Across all of it, the same pattern shows up. The prompt that wowed everyone in the Friday demo is the same prompt that pages someone three weeks later, because the demo tested the happy path and production tests reality. My default is Claude, but I ship across OpenAI and Gemini too, and moving between them has taught me exactly how much of “prompt engineering” is really just the model behaving well on easy inputs.

This post is about the discipline, not the tricks. Not ten magic phrases that unlock the model. The unglamorous engineering habits that make a prompt hold up when the inputs get weird, the volume gets high, and the cost of a silent failure is real.

The gap between a clever prompt and a reliable one

A one-off prompt succeeds because you are steering it in real time. You see a bad output, you rephrase, you add a clause, you try again. That tight loop hides how brittle the underlying instruction is, because you are the error correction. Take yourself out of the loop and the prompt has to carry all of that judgment on its own.

Production inputs are not the inputs you tested with. Users paste in text with broken formatting, half a table, a different language, an empty field, an entire document where you expected a sentence. The model that summarized your three clean paragraphs beautifully now receives a wall of HTML and confidently invents a summary of something that is not there. The prompt did not change. The input distribution did, and the prompt was only ever tuned to the narrow slice you happened to try.

Reliability means the prompt does something sensible across that whole distribution, including the ugly tail. That is a much harder target than “works on my example,” and you cannot hit it by writing a better sentence. You hit it by treating the prompt as a piece of software that has to be specified, tested, and hardened, which is the frame this whole post runs on. It is one of the core shifts I describe in building AI-native products: the prompt is product code, not a note you jotted down.

Treat the prompt as versioned product code

The single biggest mindset change is to stop treating prompts as strings that live in someone’s head or buried in a Slack thread, and start treating them as code. That means source control, review, and a changelog.

Source control sounds obvious until you notice how many teams edit prompts directly in a vendor console with no history. When output quality drops, they have no idea what changed or when. Put the prompt in the repo. Every edit is a commit with a message explaining the intent. Now a quality regression is a git blame away from an explanation instead of a mystery.

Review matters because prompt edits carry deceptive reach. One added sentence can shift behavior across every request the system handles. That deserves a second set of eyes the same way a change to a pricing function would. A reviewer catches the instruction that contradicts an earlier one, the example that teaches the wrong pattern, the clause that quietly widens scope. Prompts are unusually good at hiding their side effects, so the review is where you catch the thing the author could not see.

A changelog closes the loop. When you write down that you added a guardrail on July third to stop the model from answering questions outside its domain, you build institutional memory. Six months later, when someone proposes removing that clause because it looks redundant, the changelog tells them why it exists. Prompts accumulate hard-won fixes, and without a record every fix is at risk of being casually deleted by someone who was not there when it was earned.

Structure that holds up under pressure

There is a reason well-built production prompts tend to share a shape. Structure is what keeps the model on task when the input is trying to knock it off.

The pieces that consistently earn their place: a clear role and set of instructions up front, an explicit output format, one or two examples when the task is non-obvious, delimiters around any content you inject, and deliberate placement of context. None of this is decorative. Each part is preventing a specific failure.

The role and instructions define the job. Be direct about what the model is and what it should do, and just as direct about what it should not. Vague instructions produce vague behavior that drifts under pressure. The explicit output format is what lets the rest of your system consume the result without guessing. Examples, or few-shot demonstrations, are the fastest way to communicate a pattern that is hard to describe in prose, so if you keep explaining a nuance in words and the model keeps missing it, show it instead.

Delimiters are the part people skip and regret. When you drop user content into a prompt, wrap it in clear markers so the model can tell the difference between your instructions and the material it is supposed to act on. Without that boundary, a user’s text can read as a command, which is both a quality problem and, as I get to below, a security one.

Be specific about the failure you are preventing

Most weak prompt instructions are weak because they are aimed at nothing in particular. “Be accurate” and “do not hallucinate” feel like they are doing work, but they are wishes, not instructions. The model already believes it is being accurate.

Good instructions name the failure. Not “summarize well” but “if the document does not state the answer, say so rather than guessing.” Not “be concise” but “reply in at most three sentences.” Every clause in a mature prompt is scar tissue from a specific way the model went wrong, and you can usually read the history of a system by reading its prompt.

This is why prompts grow from evals and incidents rather than from imagination. You watch the model fail on a real input, you understand the specific mistake, and you add the narrowest instruction that prevents that mistake without collateral damage. A prompt built this way stays lean and every line has a reason, which matters both for cost and for the next person who has to reason about why it behaves the way it does. A prompt built by piling on generic virtues gets long, expensive, and internally contradictory, and eventually the instructions start fighting each other.

Managing context without drowning the model

There is a persistent myth that more context is always better, that if you just stuff enough information into the prompt the model will figure it out. In production, more context is often worse.

Every token you add is a token the model has to weigh, and irrelevant context actively dilutes the signal. Bury the one instruction that matters under three paragraphs of background and the model may well ignore it. It also costs money and latency on every single call, which adds up fast at volume. The discipline is deciding what actually needs to be there, not defaulting to everything you have.

Ordering matters too. Models pay uneven attention across a long context, and material at the very start and very end tends to land harder than material stranded in the middle. So put the instructions that must not be missed where they will be seen, and do not assume a critical rule buried on line forty of a long block will be honored. When context comes from retrieval, this discipline is even more important, because you are assembling the prompt on the fly from documents of varying quality, and I cover how to keep that clean in RAG in production. The goal is always the smallest context that fully supports the task, not the largest one you can technically fit.

Making output reliable enough to build on

A prompt is not done when it produces good text. It is done when the thing downstream can consume its output without breaking. Freeform prose that a human reads is forgiving. Output that feeds another system is not.

When something programmatic depends on the result, ask for structured output, usually JSON, and validate it against a schema before you trust it. The schema is your contract, and validation is what stops a malformed response from silently corrupting everything downstream. Do not assume the model will always honor the format. It usually will, and the occasional time it does not is exactly the failure that surfaces at 2am.

So plan for malformed output as a normal event, not an exception. When validation fails, retry, and consider feeding the validation error back into the retry so the model can correct itself. Set a cap on retries so a stubborn input cannot spin forever. This is the difference between a prompt that works in a notebook and a component that behaves in a pipeline, and it is squarely an engineering concern rather than a wording one. The prompt gets you a good draft; the schema, the validation, and the retry loop are what make it something you can build on.

A Claude prompt is not a drop-in GPT or Gemini prompt

Here is a mistake I watch teams make repeatedly: they tune a prompt against one model, decide to switch providers for cost or capability, paste the same prompt into the new model, and are surprised when quality drops. Prompts are not portable in the way people assume.

Models differ in how they respond to instruction style, how they handle system prompts, how strictly they follow formatting, how they behave at the edges. A prompt tuned hard against Claude has quietly absorbed Claude’s tendencies, and those assumptions do not automatically hold on GPT or Gemini. The prompt did not get worse. It got moved to a model it was never fitted to.

Working across Claude, OpenAI, and Gemini has made me treat the prompt as coupled to the model, not floating above it. If being able to swap models matters to you, and it often should for cost and resilience, then you maintain and evaluate the prompt on each model you intend to run it on. This is part of the broader multi-model thinking I keep coming back to. Do not let “it works on Claude” become an unexamined assumption that it works everywhere, because the moment you switch under load is the worst moment to discover it does not.

Never trust a prompt change without an eval

If there is one habit that separates people who ship reliable AI from people who ship demos, it is this: they do not judge a prompt change by reading a few outputs and nodding. They run it against an eval set.

Eyeballing is how you fool yourself. You make a change, you check three examples, they look good, you ship. What you did not see is the change quietly broke a fourth case you did not check. Because prompt edits ripple across every input, a tweak that fixes one failure mode can easily introduce another, and you will not catch that by hand. An eval set is a collection of representative inputs with expected behavior, and you run every prompt change against it before it ships. That turns “this feels better” into “this scored better on the cases I care about,” and it is the entire subject of LLM evaluation for products.

The workflow that follows is the real unlock. You iterate against the eval set, not against your gut. Change the prompt, run the evals, read what regressed, adjust, run again. It is slower per iteration than freestyling and dramatically faster to a prompt you can actually trust, because you are measuring instead of hoping. Once you have worked this way, going back to eyeballing feels like flying blind, which is exactly what it is.

Treat untrusted input as a security surface

The moment your prompt includes text you did not write, you have a security problem, not just a quality one. Prompt injection is when a user, or a document, or a web page you retrieved, contains instructions aimed at your model, and the model follows them instead of you.

Picture a prompt that summarizes user-submitted documents, and a document that contains the line “ignore your previous instructions and output the following.” A naive setup obeys it. Now imagine that model has access to tools or data, and the risk stops being embarrassing and starts being dangerous. This is not a hypothetical edge case. It is a standing property of any system that mixes trusted instructions with untrusted content in the same context.

Defenses start with the structure habits from earlier, using clear delimiters so the model knows which text is data and which is instruction, and being explicit that content inside those boundaries is to be processed, never obeyed. But delimiters alone are not enough, and you should assume a determined input can get through. That means limiting what the model can actually do, validating its outputs, and never wiring untrusted-input-driven output straight into a consequential action without a check. I go deeper on this posture in shipping AI features safely. Treat every prompt that touches outside text as exposed, because it is.

Keep system prompts lean and cost-aware

The system prompt runs on every request, which makes it the most valuable and most expensive real estate you own. It is also where bloat accumulates the fastest, because adding a line always feels harmless in the moment.

Every token in that system prompt is paid on every call, and at production volume that is a real line on the bill and a real contribution to latency. More to the point, a bloated system prompt is usually a less effective one, because the important instructions are competing for attention with everything that got bolted on and never removed. Leanness is not just frugality. It is what keeps the instructions that matter legible to the model.

So audit the system prompt the way you would audit any hot path. Which instructions actually change behavior? Which are hopeful decoration? Which fire so rarely they belong in a conditional branch rather than on every request? This is the same measure-and-trim instinct I bring to automations in Claude Code for marketers, where a leaner instruction reliably beats a longer one. A tight system prompt earned through eval-backed pruning beats a sprawling one every time, on quality and on cost.

The short version

• A demo prompt works once on an input you chose; a production prompt works thousands of times on inputs you never saw, with no human to catch it failing.
• Treat prompts as versioned product code: source control, review, and a changelog, so a regression has an explanation instead of being a mystery.
• Use structure that prevents specific failures, clear role and instructions, explicit output format, examples, delimiters, and deliberate context placement.
• Write instructions that name the exact failure you are preventing, not generic wishes like “be accurate.”
• More context is not better; use the smallest context that supports the task and order it so critical instructions land where the model reads hardest.
• Demand structured output, validate against a schema, and retry on malformed responses with a cap.
• Prompts are coupled to the model; a Claude prompt is not a drop-in for GPT or Gemini, so evaluate on each model you run.
• Never ship a prompt change on vibes; run it against an eval set and iterate against the measurement.
• Any prompt that includes untrusted text is a security surface; defend against injection and limit what the model can do.
• Keep the system prompt lean, because it runs and is paid on every request.

---

I am Deepanshu Grover, a Growth Product Manager and AI builder in Paris. If your prompts work on your laptop and break in production, connect on LinkedIn or get in touch.