What is a System Prompt?
A system prompt is a special message sent to an LLM at the start of a conversation that defines the model's persona, capabilities, constraints, and instructions: separate from user messages and treated by the model as higher-priority guidance throughout the conversation.
Overview
The system prompt is the most important piece of text in any production LLM application. It establishes who the model is, what it should and shouldn't do, what tools it has access to, what tone to use, and what to refuse. Anthropic publishes its consumer Claude system prompt publicly; OpenAI does not but has acknowledged ChatGPT's prompt extends to thousands of words. In our production engagements, BearPlex spends more time engineering the system prompt than any other component: small wording changes can swing accuracy by 10-20% on hard tasks.
How system prompts work
Modern LLM APIs distinguish messages by role: system, user, assistant, and tool. The system message (sent once at the start of a conversation) is treated by the model as higher-priority context that should govern all subsequent turns. The model's RLHF training reinforces this priority: if a user message contradicts the system prompt, the model is supposed to follow the system prompt. In practice, this is mostly true but not absolute: sufficiently sophisticated user prompts can sometimes override system instructions (this is the basis of jailbreaking and prompt injection attacks). Production systems should not rely solely on system prompts for security boundaries; structural defenses (input validation, output validation, capability restrictions) are required.
What goes in a good system prompt
Production system prompts typically include: (1) Identity, who the model is, what product it represents; (2) Scope: what tasks are in scope and out of scope; (3) Tone and style: formal vs casual, verbose vs concise, audience expectations; (4) Tool documentation: when to call which tools, how to format calls; (5) Format requirements: how to structure responses (markdown, JSON, plain text); (6) Refusal patterns: how to decline out-of-scope requests politely; (7) Edge cases: known tricky scenarios with explicit instructions; (8) Examples: few-shot examples of correct behavior. A typical BearPlex production system prompt is 500-2,000 words.
System prompt vs user prompt vs few-shot examples
All three influence model behavior but at different levels. System prompt = persistent, high-priority guidance for the whole conversation. User prompt = the specific request being asked right now. Few-shot examples = demonstrations of input-output pairs that teach the model the desired pattern (often embedded in the system prompt). For tasks where you can describe the behavior in words, system prompts work well. For tasks where the behavior is hard to describe but easy to demonstrate (specific output formats, subtle tone), few-shot examples in the system prompt or via prefix examples often work better than verbal instruction.
Use cases
- Defining a chatbot's persona, scope, and refusal patterns
- Establishing tool-use protocols for agent systems
- Locking output to a specific format (JSON, markdown table, structured report)
- Setting safety boundaries for user-facing AI products
- Embedding domain expertise (legal, medical, financial) the model should apply consistently
Examples in production
Anthropic
Publishes the Claude consumer system prompt publicly, typically 2,000+ words covering identity, capabilities, refusal patterns, and tool documentation.
SourceOpenAI
ChatGPT and ChatGPT Enterprise use multi-thousand-word system prompts that define behavior across hundreds of edge cases.
BearPlex
Production system prompts for client agent systems typically span 800-2,500 words with explicit scope, tool documentation, refusal patterns, and 5-10 few-shot examples.
System Prompt compared to alternatives
| Alternative | Choose System Prompt when | Choose alternative when |
|---|---|---|
Few-shot examples Demonstrations of input-output pairs embedded in the prompt | Use system prompt for persistent guidance describable in words | Use few-shot examples for behaviors easier to demonstrate than describe |
Fine-tuning Training the model on examples to bake behavior into weights | Use system prompt for behavior that might change or vary by user/tenant | Use fine-tuning for stable global behavior or to reduce per-call prompt cost |
Common pitfalls
- Treating the system prompt as a security boundary: it isn't; sophisticated prompts can override it
- Writing vague identity statements ('You are a helpful assistant') instead of specific role definitions
- Forgetting to update the system prompt when adding new tools: the model won't know how to use them well
- Stuffing too many edge cases without examples: the model can follow rules better when shown how
- Ignoring the system prompt's token cost on cost-sensitive applications: long prompts get expensive at scale
Related BearPlex services
Questions about System Prompt.
Not by default: the API doesn't expose it to user-facing chat. But sufficiently sophisticated prompt injection can sometimes leak the system prompt verbatim. Don't put secrets, API keys, or competitive information in the system prompt under the assumption it's hidden.
Often no. Production systems frequently template the system prompt with per-user or per-tenant context: user role, permission level, locale, organization-specific terminology. This makes the model behave appropriately for each request without requiring a separate fine-tuned model per tenant.
Yes: significantly. Anthropic, OpenAI, and other providers offer prompt caching that lets the same system prompt prefix be cached across requests at typically 90% cost discount. For applications where the same 2,000-word system prompt is sent thousands of times per day, prompt caching often pays for itself within hours of enabling it.
Need help implementing System Prompt?
BearPlex builds production AI systems that use System Prompt for Fortune 500s and high-growth scale-ups. Outcome-based pricing. 90-day embedded sprints.