Do I need a separate eval set for prompt changes?

Usually, yes—if you want to know whether a prompt change actually helped, you should evaluate it on a separate held-out set that the prompt tuning process has not seen.

Why

If you use the same data to:

• design the prompt,
• choose wording/examples,
• and judge performance,

you can easily overfit to that set, even with prompts.

Good setup

A practical split is:

• Train/dev set: to iterate on prompt wording, instructions, examples
• Eval/test set: kept untouched until you want an unbiased comparison

When a separate eval set is especially important

You should definitely separate it if:

• you’re comparing multiple prompt versions,
• you expect to publish/share results,
• the task is noisy or subjective,
• the prompt includes few-shot examples chosen from data.

If you have very little data

You can still do this with:

• a small dev set for prompt iteration
• a small final eval set reserved for one-time assessment

Or use:

• cross-validation,
• repeated random splits,
• or human review on a fixed blind benchmark.

Rule of thumb

If the prompt change is being used to make a decision, don’t evaluate on the same examples you used to craft it.

If you want, I can suggest a simple evaluation workflow for prompt iteration with small datasets.