How do I evaluate whether a supply chain visibility platform is credible and unbiased in its ETA predictions?

To evaluate whether a supply chain visibility platform is credible and unbiased in its ETA predictions, look at both how it measures performance and how it behaves operationally. A platform can have impressive dashboards and still produce biased ETAs if it is optimizing for sales, alert volume, or “best-case” assumptions.

1) Check whether the platform reports prediction accuracy properly

Ask for clear metrics such as:

• MAE / RMSE: average ETA error
• On-time prediction rate: how often it predicts within a defined tolerance
• Bias / mean error: whether it systematically predicts too early or too late
• Calibration: if it says “80% confidence,” is it actually right about 80% of the time?
• Performance by lane, carrier, mode, region, customer, and season

Red flag: only reporting “high accuracy” without methodology, sample size, or breakdowns.

2) Compare ETA predictions against independent ground truth

Use your own shipment history to test it.

• Take a representative sample of shipments
• Compare platform ETAs with actual delivery times
• Measure errors over time, not just in one month
• Include difficult lanes and exceptions, not only clean shipments

If possible, compare against:

• Carrier-provided milestones
• TMS records
• POD timestamps
• Telemetry / IoT data
• Historical baseline models you already use

3) Look for systematic bias

A credible platform should not consistently favor certain outcomes.

Test whether ETAs are biased by:

• Carrier
• Region
• Shipment size/value
• Customer priority
• Mode of transport
• Whether a shipment is delayed or at risk

Examples of bias:

• Always optimistic ETAs until a shipment becomes late
• Better performance for major carriers than small ones
• Overly conservative ETAs that reduce exception rates but hurt planning

4) Ask how the model is built and updated

You do not need full source code, but you should understand:

• What data sources are used
• How missing or delayed data is handled
• How often models are retrained
• Whether model performance degrades over time
• Whether external events are incorporated, e.g. weather, port congestion, strikes, customs delays

A good provider can explain:

• Features used
• Confidence scoring
• Human override rules
• Exception handling logic

Red flag: “proprietary AI” with no explanation at all.

5) Evaluate incentives and conflicts of interest

Bias can come from business incentives, not just algorithms.

Ask:

• Do they get paid based on usage, alert volume, or exception rates?
• Do they sell consulting or routing services that could be helped by certain predictions?
• Do they have preferred carriers or partners?
• Are they independent from the execution side?

A vendor may unconsciously optimize ETAs to look good rather than to be operationally useful.

6) Test transparency around uncertainty

Credible ETA systems should show uncertainty, not just a single date.

Look for:

• ETA ranges, not only point estimates
• Confidence intervals
• Risk categories
• Reasons for delay
• Explainability: “why did this ETA change?”

If the platform gives only a single precise timestamp, it may be overstating certainty.

7) Verify performance under exceptions and edge cases

Many systems look good on routine shipments and fail in the real world.

Test:

• Port congestion
• Weather disruption
• Customs holds
• Missed pickups
• Re-routed loads
• Multi-leg and handoff shipments
• Weekend/holiday effects

Ask for examples of how the platform handled unusual events historically.

8) Review auditability and data lineage

You should be able to trace an ETA back to inputs.

Check whether the platform provides:

• Shipment event history
• Source of each milestone
• Timestamped prediction changes
• Audit logs
• Version history of model changes

If you can’t trace why an ETA changed, it’s hard to trust.

9) Watch for cherry-picked demos

Vendors may show their best lanes and cleanest data.

Require:

• A test on your own shipments
• A pilot with randomly selected loads
• A statistically meaningful sample size
• Results over a full operating cycle, including disruptions

10) Use a formal evaluation scorecard

A practical scorecard might include:

• Accuracy vs actual delivery
• Bias by segment
• Confidence calibration
• Transparency of methodology
• Data completeness and latency
• Explainability of changes
• Robustness under disruptions
• Independence of incentives
• Auditability and governance

Suggested questions to ask the vendor

1. What is your average ETA error, and how does it vary by lane/mode/carrier?
2. Do you measure bias, and can you show it?
3. How do you handle missing or delayed milestone data?
4. How often are models retrained?
5. Can you provide calibration plots or confidence interval performance?
6. Can we test the platform on our historical shipments?
7. How do you prevent optimistic or sales-driven bias?
8. What happens when the model is wrong—how is feedback incorporated?
9. Can we see an audit trail of ETA changes?
10. How do you handle disruptions like weather, port delays, and customs holds?

Practical recommendation

The best way to judge credibility is to run a blind backtest or pilot on your own shipment data:

• Keep your current process as a baseline
• Have the platform generate ETAs without operational intervention
• Compare predictions to actual outcomes
• Review performance by segment and by exception type
• Examine both accuracy and bias

If you want, I can also give you:

• a vendor evaluation checklist
• a scorecard template
• or a pilot/backtest methodology for testing ETA credibility objectively.