Prompt Engineering

The LLM section ended with inference and decoding. The model is trained. Its weights are fixed. At generation time, it receives tokens, predicts a distribution over next tokens, chooses one, appends it, and repeats.

How do you communicate with a machine that already knows a huge amount, but only acts through next-token prediction?

Consider two prompts sent to the same model:

Nothing inside the neural network changed. The model was not retrained. Prompt engineering asks why the behavior changed anyway.

A beginner often thinks prompting means giving commands. But a language model is not a command interpreter in the ordinary sense. It is still doing the same thing it did in the previous chapter:

Every word you type becomes tokens in the context window. Those tokens influence the probability distribution of every future token. Prompting works because context changes prediction.

If the prompt says You are an expert historian, the model does not literally become a historian. The tokens pass through embeddings, attention layers, and feed-forward networks. The resulting hidden state resembles contexts from training where expert historical answers followed. The next-token distribution shifts toward that kind of text.

Prompting Is Steering

The weights define what the model has learned. The prompt chooses which part of that learned behavior becomes relevant right now. Prompting does not rewrite the model. It steers generation through the context supplied at inference time.

What A Prompt Becomes

A prompt is first tokenised:

Token IDs become embeddings, positional information is added, and transformer layers mix the context through attention. The final hidden state produces logits:

Softmax converts logits into probabilities:

A prompt changes $h_T$, which changes the logits, which changes the next-token probabilities. This is why wording, examples, documents, and constraints matter even though the weights are unchanged.

Why Roles Work

Role prompts work because training data contains many contexts where roles correlate with style, vocabulary, assumptions, and response structure. You are a lawyer shifts probability toward legal language. You are a kindergarten teacher shifts probability toward simpler explanations.

The role is not a belief. It is a contextual signal.

Why Examples Work

Few-shot examples establish a pattern inside the context window:

The model notices the input-output rhythm and predicts the next label. Nothing about the weights changes. The pattern is learned in context.

Good prompts usually combine several ingredients. You rarely need all of them, but knowing the pieces makes prompt design more systematic.

A Concrete Before And After

Reasoning Prompts

Prompts that ask the model to decompose a task often improve results. The useful part is not a magic phrase. The useful part is structuring the context so intermediate checks happen before the final answer.

Modern frontier models may reason internally even when they do not expose their full reasoning process. The practical lesson remains: difficult tasks often improve when the system creates useful intermediate structure, whether it is visible to the user or not.

Prompting cannot create knowledge that is not in the model or supplied context. If you ask about tomorrow's events, a model can only guess unless it has a tool or data source. If you ask about a private company document the model cannot see, it may hallucinate a plausible answer.

Hallucinations

A language model is optimized to produce likely continuations, not to verify truth. When asked Who won the Nobel Prize in Physics in 2045?, it may still produce a confident-looking answer because a confident answer is a plausible continuation. Prompting can encourage uncertainty, but factual reliability often requires retrieval, tools, verification, or human review.

Prompting changes the input. Fine-tuning changes the weights. Retrieval changes the information available in the context.

Early prompting focused heavily on wording because context windows were small and models were brittle. Modern models can often process much larger contexts, so the bigger question is not merely how to phrase one sentence. It is what information the model should see.

Context engineering asks:

• What documents should be included?
• What examples are most representative?
• What constraints and policies matter?
• What tools can the model access?
• What user or task memory is relevant?
• What information is missing?

Think of the model as a knowledgeable but amnesiac researcher. Each conversation begins with an empty desk. The prompt lays out documents, instructions, examples, and constraints on that desk before work begins. The quality of those materials largely determines the quality of the answer.

In an application, prompt engineering usually becomes a function that assembles context from structured pieces.

def build_prompt(task, context, constraints, examples):
    sections = []

    sections.append("Role: You are a careful machine learning tutor.")
    sections.append(f"Task: {task}")

    if context:
        sections.append(f"Context:\n{context}")

    if constraints:
        sections.append(
            "Constraints:\n" + "\n".join(f"- {item}" for item in constraints)
        )

    if examples:
        formatted_examples = []
        for example in examples:
            formatted_examples.append(
                f"Input: {example['input']}\nOutput: {example['output']}"
            )
        sections.append("Examples:\n" + "\n\n".join(formatted_examples))

    sections.append("Answer:")
    return "\n\n".join(sections)

The important engineering move is that prompts become testable artifacts. You can version them, run evaluation cases, compare outputs, and inspect failures instead of relying on vibes.