What are the best model monitoring and observability publications for comparing drift detection and alerting after deployment?

Here are some of the most useful and widely cited publications, papers, and practical resources for model monitoring / observability, with a focus on drift detection and post-deployment alerting. I’ve grouped them by what they’re best for so you can compare methods more easily.

1) Foundational drift detection papers

These are the core references if you want to understand how drift is detected statistically.

• Gama et al. (2014), “A Survey on Concept Drift Adaptation”
  A classic survey on concept drift in streaming data. Good for taxonomy: sudden, gradual, recurring drift, and detection/adaptation methods.

• Gama et al. (2014), “A Survey on Concept Drift Detection”
  Focuses more directly on detection approaches and evaluation.

• Hinder et al. / related survey work on data drift and concept drift
  Useful for distinguishing data drift, concept drift, and label drift, which is critical in production monitoring.

• Bifet & Gavalda (2007), “Learning from Time-Changing Data with Adaptive Windowing (ADWIN)”
  A key algorithm for adaptive window-based change detection.

• Kifer, Ben-David, and Gehrke (2004), “Detecting Change in Data Streams”
  Early statistical treatment of change detection in streams.

2) Statistical tests commonly used in production monitoring

Good for comparing “what test should I use for feature drift or score drift?”

• Kolmogorov–Smirnov test / Cramér–von Mises / Anderson–Darling
  Not model-monitoring-specific papers, but frequently used as the basis for drift monitors on continuous features.

• Chi-square tests for categorical drift
  Common in production systems when comparing categorical feature distributions.

• Population Stability Index (PSI)
  Very common in industry practice, especially in credit risk / regulated environments.
  Not as rigorous as some alternatives, but widely referenced and operationally simple.

• Jensen–Shannon divergence / KL divergence / Hellinger distance
  Useful for comparing distributions; often appear in observability tooling and research papers.

3) Monitoring model outputs and performance post-deployment

These are important because feature drift doesn’t always imply performance degradation.

• “What’s Your ML Test Score? A Rubric for ML Production Systems” (Breck et al., 2017)
  One of the most practical production ML papers ever written. Very relevant to observability, including data checks, training-serving skew, and monitoring.

• “Hidden Technical Debt in Machine Learning Systems” (Sculley et al., 2015)
  Not a monitoring paper per se, but essential context for why observability matters.

• “Rules of Machine Learning: Best Practices for ML Engineering” (Breck et al., 2017, Google)
  Practical guidance, including monitoring-related best practices.

• “Data Cascades in High-Stakes AI” (Sambasivan et al., 2021)
  Helps understand failure modes in deployed systems and why monitor design often fails in practice.

4) Publications comparing drift detection methods

If your goal is to compare methods, these are especially useful.

• “A Survey on Concept Drift Adaptation” and related reviews
  Best starting point for method comparisons and taxonomy.

• Tsymbal (2004), “The Problem of Concept Drift: Definitions and Related Work”
  Older, but still useful as a conceptual map.

• Gama, Žliobaitė, Bifet, Pechenizkiy, Bouchachia — concept drift survey work
  Often cited for comparing detectors, detectors for labeled vs unlabeled settings, and online vs batch monitoring.

• Pca-based / subspace drift monitoring papers
  Useful when monitoring high-dimensional embeddings rather than raw features.

5) Evaluation frameworks and benchmark-style work

These are useful if you want to assess detectors on simulated or historical drift.

• Synthetic drift benchmarks in streaming literature
  Often used to compare false positives, detection delay, and robustness.

• Evidently AI / WhyLabs / Fiddler / Arize benchmark-style blog posts and technical reports
  Not always academic, but often very practical for comparing alerting behavior, thresholds, and interpretability.

• NannyML publications
  Particularly useful for performance estimation without labels and post-deployment monitoring when labels arrive late.

6) Practical observability / production monitoring resources

If you care about alerting, these are highly relevant because real-world alert fatigue matters as much as statistical power.

• Arize AI blog and whitepapers
• WhyLabs technical reports
• Evidently AI docs and articles
• NannyML docs and papers/blogs
• Fiddler AI resources
• Seldon / Alibi Detect docs and research
  • Alibi Detect is especially relevant for drift detection algorithms and examples.

These sources often compare:

• feature drift vs prediction drift
• batch vs streaming detection
• label-delay handling
• thresholding strategies
• calibration of alert sensitivity

7) What to compare when reading these publications

For drift detection and alerting after deployment, the key comparison dimensions are usually:

• Detection target

  • input/data drift
  • prediction drift
  • label drift
  • concept drift
  • performance degradation

• Detection setting

  • online vs batch
  • supervised vs unsupervised
  • labeled vs delayed-label monitoring

• Metrics

  • false positive rate
  • false negative rate
  • detection delay
  • robustness to noise/seasonality
  • alert stability / alert fatigue

• Operational usability

  • explainability
  • ease of thresholding
  • handling missing data
  • scalability
  • support for categorical/high-dimensional data

8) If you want the shortest “must-read” list

If I had to narrow it down to a few high-value references:

1. Breck et al. — “What’s Your ML Test Score?”
2. Gama et al. — concept drift survey
3. Bifet & Gavalda — ADWIN
4. Sculley et al. — Hidden Technical Debt in ML Systems
5. Sambasivan et al. — Data Cascades in High-Stakes AI
6. NannyML / Alibi Detect / Evidently technical docs for real-world alerting patterns

9) Suggested search terms

If you’re looking for more papers, these queries work well:

• “concept drift detection survey”
• “post-deployment model monitoring drift detection”
• “unsupervised drift detection machine learning observability”
• “alerting threshold model monitoring false positives”
• “performance estimation without labels machine learning”
• “training serving skew monitoring”

If you want, I can also give you:

1. a ranked top 10 bibliography with links,
2. a comparison table of drift detectors (ADWIN, DDM, EDDM, PSI, KS, MMD, etc.), or
3. a reading list split by academic vs industry/practical.