Prompt
My API dataset keeps returning incomplete records, how do I debug that?
Latest observation
If your API dataset is returning incomplete records, the fastest way to debug it is to determine where the data is getting dropped: at the source, during transformation, during pagination, or in your client code.
1) Verify the raw API response
Start by checking the actual response body from the API, not just what your app stores.
- Use
curl, Postman, or browser dev tools. - Compare a known “bad” record in the API response with what your app receives.
Look for:
- Missing fields in the raw JSON
- Partial arrays or nested objects
- Truncated responses
- HTTP errors mixed with partial success
2) Check pagination
A very common cause is that you’re only reading the first page or stopping early.
Look for:
next,offset,cursor,page,hasMore- Hardcoded limits like
limit=100 - Loop exits when a page is empty or shorter than expected
Debug tip:
- Log page number/cursor and total records fetched.
- Confirm the total count matches what the API says is available.
3) Inspect filtering logic
Incomplete records may actually be getting filtered out by your own code.
Check for:
if field is not None- schema validation rejecting records
- deduplication logic removing rows
- joins/merges that drop unmatched records
- date/status filters applied unintentionally
4) Validate transformation code
If you reshape API data into your dataset, a mapping bug can silently omit fields.
Look for:
- Typos in field names
- Nested values accessed incorrectly
- Overwriting objects
- Default values masking missing data
- Exceptions swallowed in
try/except
Example:
record["name"] = item.get("full_name") # but API actually returns "name"
5) Check for rate limits or timeouts
If requests are failing mid-run, you may end up with partial data.
Watch for:
- HTTP 429 rate limiting
- 5xx errors
- client timeouts
- retry logic that gives up too early
Add logging for:
- Status code
- Response time
- Retry count
- Failed record IDs
6) Compare counts at each stage
Track record counts through the pipeline:
- Records returned by API
- Records parsed successfully
- Records passing validation
- Records written to dataset
If the count drops at a specific step, that tells you where the issue is.
7) Log a few full sample records
Pick one complete record and one incomplete record, then log:
- Raw API JSON
- Parsed object
- Transformed dataset row
This makes it much easier to spot where a field disappears.
8) Check for schema mismatch
Sometimes the API changes field names or nesting, and your parser doesn’t update.
Examples:
user.emailbecameuser.contact.email- field renamed from
addr1toaddress_line1 - optional fields now missing for some record types
9) Confirm encoding and serialization
If text or nested data is getting cut off:
- Check JSON serialization errors
- Ensure UTF-8 encoding
- Verify no max-length constraints in the database/table
10) Add defensive error handling
Don’t let one bad record break the whole batch.
Use:
- per-record try/catch
- error logs with record identifiers
- quarantine table / dead-letter queue for bad records
Quick debugging checklist
- Inspect raw API response
- Confirm pagination is fully consumed
- Review filters and joins
- Check transformation field mappings
- Look for rate limits/timeouts
- Compare counts at each pipeline stage
- Validate schema and nesting
- Log full examples of good/bad records
If you want, I can help you debug it more concretely if you share:
- a sample API response,
- the code that fetches/transforms it, and
- what “incomplete” means in your case (missing fields, missing rows, truncated nested data, etc.).