How do I set up a test failure explanation tool for debugging failing tests in our CI pipeline?

A good way to do this is to add a failure-explanation step to your CI that runs only when tests fail, collects the relevant artifacts, and then passes them to an “explainer” tool or workflow.

1) Decide what the tool should explain

Typically you want it to summarize:

• Which test failed
• The assertion/error message
• Relevant logs/stdout/stderr
• Stack trace
• Recent code changes or diff
• Any environment/setup info that could matter

2) Capture the right artifacts in CI

Make sure your test job saves:

• JUnit/XML test results
• Raw test logs
• Stack traces
• Screenshots/video for UI tests
• Coverage and timing data if useful
• The git diff or commit metadata

In many CI systems, store these as artifacts so the explainer can read them after the run.

3) Add a failure-only post-processing step

Configure the pipeline so that:

• Tests run normally
• On failure, a separate job/step runs
• That step loads the artifacts and generates a readable explanation

Example flow:

1. Run tests
2. If tests fail, upload artifacts
3. Run explainer script
4. Post output to CI summary, Slack, PR comment, or issue tracker

4) Use either rules-based or LLM-based explanation

Option A: Rules-based

A script parses:

• assertion failures
• exception types
• known error patterns
• flaky test indicators

Good for predictable failures, fast, and cheap.

Option B: LLM-assisted

Feed the explainer:

• failing test output
• relevant logs
• git diff
• stack trace
• repo context

Then ask it to produce:

• likely root cause
• suspected file/function
• suggested next debugging steps
• confidence level

This is more flexible for complex failures.

5) Keep the prompt/output structured

If using an LLM, use a consistent schema like:

• Test name
• Failure summary
• Root cause hypothesis
• Evidence
• Suggested fix
• Confidence
• Follow-up checks

That makes it easier to display in CI or parse later.

6) Integrate into your CI system

Examples:

• GitHub Actions: use if: failure() for the explainer step, or a separate job that depends on the test job and checks needs.test.result == 'failure'.
• GitLab CI: use when: on_failure.
• Jenkins: use post { failure { ... } }.

7) Make it safe and useful

• Redact secrets before sending logs to any external service
• Limit artifact size
• Cache repo context selectively
• Avoid overwhelming the explainer with huge logs; trim to relevant sections
• Add rate limiting if many tests fail at once

8) Example CI pattern

Pseudo-flow:

test:
  script:
    - run-tests.sh
  artifacts:
    when: always
    paths:
      - test-results/
      - logs/

explain_failure:
  needs: [test]
  if: failure()
  script:
    - explain-failure.sh test-results/ logs/ > explanation.md
  artifacts:
    paths:
      - explanation.md

9) Best practices

• Start with a narrow scope: only failed unit tests
• Compare current failure to historical failures
• Include the last successful run if possible
• Track whether explanations actually help fix failures
• Add a feedback loop so developers can mark explanations as helpful/unhelpful

If you want, I can give you:

1. a GitHub Actions example,
2. a Jenkins pipeline example, or
3. a Python script that summarizes failing test logs with an LLM.