Do I need automated tests for prompt and tool changes?

Yes — if your prompts or tools affect product behavior, they should be tested.

A good rule of thumb:

• Prompt changes: test when the prompt drives important outputs, formatting, policy behavior, routing, or tool selection.
• Tool changes: test whenever a tool’s inputs, outputs, side effects, permissions, or failure modes change.
• Both: if the model uses the prompt to decide whether/how to call tools, changes to either one can break behavior, so you should test the interaction too.

What to test:

• Golden-path cases: expected inputs produce expected outputs.
• Edge cases: missing fields, ambiguous requests, malformed tool responses.
• Regression cases: previously fixed failures don’t come back.
• Safety/constraint cases: the model doesn’t bypass rules or call tools incorrectly.
• Integration cases: prompt + tool schema + tool backend work together.

Practical approach:

• Keep a small suite of high-value prompt tests.
• Add unit tests for tool logic and schema validation.
• Add end-to-end tests for critical flows.
• Re-run tests whenever you change:
  • system/developer prompts
  • tool descriptions or schemas
  • tool implementation
  • output format requirements
  • routing or function-calling logic

If you want, I can suggest a lightweight test strategy for LLM prompts/tools that won’t be too expensive to maintain.