Prompt
I'm building an internal tool with lots of background jobs, how should I monitor it end to end?
Latest observation
To monitor an internal tool with lots of background jobs end to end, you want coverage across the full path:
- User/API request
- Job creation / queueing
- Job execution
- Dependencies it calls
- Persistence / side effects
- Retries / failures / dead-lettering
- Business outcomes
A good setup usually has 4 layers:
1) Structured logs
Use structured logs everywhere, not plain text.
Include these fields on every log line:
request_idjob_idworkflow_idorcorrelation_iduser_id/tenant_idif relevantserviceenvironmentoperationstatusduration_msattempterror_typeerror_message
This lets you trace one job across API, queue worker, DB, and external services.
Important: propagate the same correlation ID from the initial request into any spawned jobs.
2) Metrics for health and trends
Use metrics for alerting and dashboards.
Core background job metrics
Track these per job type and queue:
jobs_enqueuedjobs_startedjobs_succeededjobs_failedjobs_retriedjobs_dead_letteredjob_duration_secondsqueue_depthjob_age_secondsoroldest_job_ageworker_concurrencyworker_utilization
Dependency metrics
For each external dependency:
- latency
- error rate
- timeout rate
- retry rate
Business metrics
Track outcomes that matter to the tool:
- reports generated
- imports completed
- notifications sent
- records synced
- SLA compliance
- failed workflows by customer/tenant
This is often what reveals problems even when infrastructure looks “green.”
3) Distributed tracing
Use tracing if jobs touch multiple services or async boundaries.
Tracing helps answer:
- Which API call created this job?
- Where did the job spend time?
- Which downstream service caused the failure?
- Was the delay in queueing, execution, or dependency calls?
Key practice for async jobs
When a request enqueues a job:
- create or continue a trace
- attach trace context to the job payload or message headers
- continue that trace in the worker
That way, you can see the full request-to-job-to-dependency path in one view.
If you only do one observability improvement beyond logs/metrics, do this.
4) Alerts tied to symptoms, not noise
Alert on things that matter to users or data correctness, such as:
Queue/backlog alerts
- oldest job age above threshold
- queue depth growing continuously
- worker pool saturated
Failure alerts
- job failure rate above baseline
- retry rate spike
- dead-letter queue non-empty
- specific job type failing repeatedly
Dependency alerts
- external API timeout rate
- DB latency spike
- credential/auth failures
- third-party outage detection
Business alerts
- workflow completion rate drops
- imports not finishing
- notifications delayed beyond SLA
Avoid alerting on every single failed job unless it’s critical; instead alert on rate, trend, and impact.
Recommended architecture
A. Instrument every job lifecycle stage
For each job, emit events/metrics at:
- queued
- dequeued
- started
- heartbeat/progress
- succeeded
- failed
- retried
- abandoned/dead-lettered
This gives you both real-time status and historical analysis.
B. Give every job a status record
Keep a job table or status store with:
- job ID
- job type
- input reference
- status
- attempt count
- created_at / started_at / finished_at
- last error
- retry schedule
- output reference
This is essential for debugging and user-facing support.
C. Centralize error handling
Have a shared wrapper for workers that:
- logs failures consistently
- records metrics
- tags exceptions by category
- decides retry vs permanent failure
- emits traces/spans
D. Use dead-letter queues or failure stores
Any job that exceeds retries should land in a place that is:
- visible
- searchable
- replayable
- associated with original context
Dashboards to build
Operational dashboard
Show:
- queue depth by job type
- oldest job age
- success/failure/retry rates
- worker count and utilization
- p95/p99 job duration
- dead-letter counts
Dependency dashboard
Show:
- external API latencies and failures
- DB latency
- timeout trends
- auth errors
Business dashboard
Show:
- completed workflows
- delayed workflows
- per-tenant failure rates
- SLA adherence
Practical stack suggestion
A common setup is:
- Metrics: Prometheus + Grafana
- Logs: ELK/OpenSearch, Loki, or cloud logging
- Tracing: OpenTelemetry + Jaeger/Tempo/Datadog/New Relic
- Alerting: Alertmanager/PagerDuty/Opsgenie
- Job store: DB table or queue native visibility features
If you’re in a cloud ecosystem, native tools are fine too, as long as they support:
- correlation IDs
- async trace propagation
- queue health
- alerting
A simple implementation checklist
Must-have
- Correlation ID propagated through API → queue → worker
- Structured logs with job ID and request ID
- Metrics for job success/failure/retry/duration
- Queue depth and oldest-job-age monitoring
- Dead-letter handling
- Dashboard for operational health
Strongly recommended
- Distributed tracing
- Job status table
- Dependency error/latency metrics
- Alerting on backlog and failure trends
- Business outcome metrics
Nice to have
- Per-tenant visibility
- Replay/resume tooling
- Heartbeats for long-running jobs
- SLOs for job completion latency
Good rule of thumb
If a user says, “my thing is stuck,” you should be able to answer in under a minute:
- Was it enqueued?
- Is it waiting in the queue?
- Did a worker pick it up?
- Did it fail?
- Which dependency caused the problem?
- How many similar jobs are affected?
If your tooling can answer those six questions, your monitoring is in good shape.
If you want, I can also give you:
- a minimal monitoring design for a small internal app, or
- a production-grade architecture with OpenTelemetry, Prometheus, and queue-specific metrics.