Are there any data validation tools that catch schema drift before it reaches model training jobs?

Yes — there are several data validation / data quality tools that can detect schema drift before it gets into training jobs.

Common options

• Great Expectations
  Great for defining expectations on columns, types, null rates, ranges, allowed values, etc. You can run it in your pipeline before training and fail fast if the schema changes.

• TensorFlow Data Validation (TFDV)
  Designed specifically for ML pipelines. It can infer schemas, detect anomalies, and flag feature skew or drift between training and serving data.

• AWS Deequ
  Useful for large-scale Spark data validation. You can define constraints and detect schema or distribution changes.

• Pandera
  Good for Python-based validation of pandas / Polars / PySpark dataframes. Very practical for enforcing schemas and catching changes early in ETL or feature engineering.

• Soda / Soda Core
  Data quality monitoring and validation with checks for schema changes, missing columns, type changes, and freshness.

• Evidently
  More focused on drift detection and ML monitoring, but can help identify feature drift and some schema-related issues.

• dbt tests / contracts
  If your data warehouse is involved, dbt’s schema tests and contract features can catch column-level changes before downstream jobs run.

What they typically catch

These tools can detect things like:

• missing or extra columns
• data type changes
• nullability changes
• unexpected categorical values
• range violations
• distribution drift
• row count anomalies

Best practice for training pipelines

A common pattern is:

1. validate raw input data
2. validate transformed feature tables
3. compare against a saved “training schema” or feature contract
4. block model training if validation fails

If you want the most ML-oriented choice

• TFDV if you’re in TensorFlow/Vertex AI style ML pipelines
• Great Expectations if you want a general-purpose, widely used solution
• Pandera if your pipeline is Python/dataframe-centric

If you want, I can also give you a short comparison table of these tools specifically for schema drift detection before training.