Prompt
How do I troubleshoot a sudden spike in uncaught exceptions after a deploy?
Latest observation
A sudden spike in uncaught exceptions after a deploy usually means one of four things:
- a bad code path was introduced,
- an environment/config change caused existing code to fail,
- traffic shifted to a previously rare path,
- the deployment itself exposed a dependency or compatibility issue.
Here’s a practical troubleshooting workflow.
1) Confirm the spike and scope
Start by answering:
- When exactly did it start?
- Does it line up with the deploy timestamp?
- Is it all environments or only one service/region/version?
- Is the spike in one exception type or many?
- Is it every request, a subset of users, or a specific endpoint/job?
Look for:
- error rate by version
- error rate by route/handler
- error rate by region/instance
- error rate by browser/client/version if relevant
- correlation with deploy rollout phases
If you have canary or blue-green deploys, compare the new version against the old one.
2) Read the top stack traces first
Don’t start with the entire error firehose. Sort by frequency and inspect:
- exception class/message
- top stack trace
- affected code path
- request metadata, user agent, feature flag state, tenant ID, etc.
Questions to ask:
- Is the exception new or an old one that increased?
- Is the stack trace pointing to a recently changed file?
- Is it a null/undefined, timeout, parsing, auth, DB, or network error?
- Is there a common input pattern?
If error reporting groups by signature, focus on the top 1–3 signatures.
3) Compare the deploy diff
Review the deploy for:
- recently changed exception handling
- new dependencies or upgraded packages
- config/env var changes
- schema changes
- feature flags/defaults
- changes to async logic, retries, or concurrency
- logging or serialization changes that could throw
Pay attention to:
- code paths that used to be guarded but now aren’t
- functions that may now throw where they previously returned null/false
- changed validation assumptions
- race conditions introduced by parallelization
4) Check for environment/config mismatches
A deploy can break due to differences between local/staging and prod:
- missing or renamed env vars
- incorrect secrets/credentials
- changed service endpoints
- expired certs
- incompatible runtime version
- missing permissions
- changed file paths or mount points
Common symptom: the app deploys fine, then fails at runtime on a specific operation.
5) Validate dependencies and runtime compatibility
If the spike started right after deploy, check:
- package/library upgrades
- runtime version changes
- OS/container base image changes
- serialization/deserialization compatibility
- database driver changes
- API contract changes with downstream services
Look for:
- deprecations becoming errors
- stricter parsing/validation
- changed default behavior in a library
- unsupported Node/Python/Java/.NET version combinations
6) Inspect recent schema or contract changes
If the app reads from DB, queue, cache, or external APIs:
- did the schema change without a backward-compatible rollout?
- are there nulls where code expects values?
- did a new field start arriving or disappear?
- did an enum/value format change?
- did a downstream service alter response shape or status codes?
A common issue is deploying app code before data migration is complete, or vice versa.
7) Reproduce with real inputs
Use production-like data and request shapes:
- replay a failing request from logs
- use the same user/tenant/session context
- feed the same payload into a staging replica
- compare response between old and new versions
If you can’t reproduce locally:
- add temporary logging around the failing line
- capture sanitized input
- compare execution path to the previous release
8) Check for masked exceptions or missing handling
Sometimes a deploy doesn’t create the bug; it exposes it by removing a catch, changing async flow, or causing a previously handled error to escape.
Look for:
- missing
try/catcharound new code - promises/async tasks not awaited
- exceptions thrown inside callbacks/threads/background jobs
- error handlers that rethrow incorrectly
- swallowed errors leading to later uncaught exceptions
9) Roll back or disable the risky change
If the spike is severe and tied to the deploy:
- rollback the release
- disable the feature flag
- revert the config change
- pause rollout if still in progress
This is often the fastest way to stop the bleeding while you investigate.
10) Add targeted observability
If you need more signal:
- log exception class, message, stack, request ID, version, feature flag state
- include route, tenant, region, and dependency response codes
- add tracing around the failure point
- capture a sample payload, safely sanitized
Be careful not to log secrets or personal data.
11) Typical root causes to look for
Very common post-deploy exception causes:
- null/undefined access from changed data shape
- missing env var
- bad JSON parsing from a new response format
- database constraint violation
- timeout due to slower query or API
- auth/permission failure after secret change
- date/time parsing/locale issues
- unhandled promise rejection / async exception
- feature flag enabled for a broader audience than intended
12) Use a simple incident checklist
A quick incident flow:
- Confirm the spike timing vs deploy.
- Identify the top exception signature.
- Find the exact version and code path.
- Compare changed code/config/dependencies.
- Reproduce or isolate with real inputs.
- Roll back or disable feature if needed.
- Fix root cause and add a regression test.
- Postmortem: why wasn’t it caught in staging/canary?
Prevention for next time
- canary deploys with automated error-budget checks
- alerting on uncaught exceptions by version
- contract tests for upstream/downstream payloads
- feature flags with gradual rollout
- synthetic checks on critical paths
- better structured logging with request/version correlation
- runtime/config validation at startup
- backward-compatible schema migrations
If you want, I can also give you:
- a step-by-step incident runbook
- a checklist for Node/Python/Java/.NET
- or a template for investigating from logs/APM