Are there any A/B testing platforms that work with offline evaluation sets for search ranking changes?

Yes. What you’re looking for is often called offline A/B testing, counterfactual evaluation, or replay evaluation for ranking systems. A few platforms and tools can help, though most “A/B testing platforms” are built for live traffic rather than purely offline datasets.

Platforms / tools to look at

1. Open source evaluation frameworks

These are often the best fit for offline search-ranking experiments:

• Microsoft Recommenders / recommender evaluation libraries
  • Support offline ranking metrics and experimentation workflows.
• RecBole
  • Research-oriented framework for ranking/recommendation evaluation.
• Evidently AI
  • More general ML monitoring/evaluation, not search-specific, but can be adapted to compare offline ranking outputs.
• PyTerrier
  • Very useful for search/ranking experiments with offline test collections.

2. Counterfactual / bandit evaluation libraries

If you have logged interaction data and want to estimate how a new ranking policy would perform without live traffic:

• Open Bandit Pipeline (OBP)
  • Designed for offline policy evaluation from logged data.
• Vowpal Wabbit
  • Supports contextual bandit / off-policy evaluation workflows.
• CausalML / EconML
  • Not search-specific, but useful for treatment-effect style offline analysis.

3. Search relevance evaluation platforms

These are more about relevance benchmarking than A/B testing, but they support offline comparison of ranking models:

• Lucene / Elasticsearch evaluation tooling
  • Can run offline relevance tests with judged datasets.
• PyTerrier + TREC-style evaluation
  • Common for search ranking changes.
• Tonic / RAGAS-like eval tools
  • More for retrieval/RAG than classic search, but offline ranking comparison is possible.

4. Enterprise experiment platforms

Some commercial experimentation platforms can ingest offline results, but they usually don’t do offline ranking evaluation natively:

• Statsig
• Optimizely
• LaunchDarkly
• Amplitude Experiment
• GrowthBook

These are primarily for live A/B tests, but you can sometimes:

• upload offline metrics,
• compare experiment variants manually,
• or use them as decision/reporting layers on top of offline evaluation.

---

If your use case is search ranking changes

The most relevant approach is usually:

1. Build or obtain a judged test set
   • queries, candidate results, relevance labels.
2. Run both ranking versions offline.
3. Compare with ranking metrics like:
   • NDCG
   • MRR
   • MAP
   • Precision@k
   • Recall@k
4. If you have logged click data, use counterfactual evaluation methods to estimate online impact.

---

Practical recommendation

If you want a platform-like experience for offline ranking evaluation, I’d start with:

• PyTerrier for classic search ranking experimentation
• Open Bandit Pipeline if you have logged interaction data
• Vowpal Wabbit if you want off-policy / bandit evaluation
• GrowthBook/Statsig only if you want to tie offline results into an experimentation workflow

If you want, I can also give you:

• a shortlist by stack (open-source vs commercial),
• a setup for Elasticsearch/OpenSearch search ranking evaluation,
• or a recommendation based on whether you have judged data or click logs.