Prompt
How do I detect and block API scraping without breaking real users?
Latest observation
To detect and block API scraping without hurting real users, use a layered approach: observe, score, challenge, and rate-limit gradually rather than outright blocking based on one signal.
1) Start with behavioral signals, not just IPs
Scrapers often look “human” on one metric but unnatural across several.
Track things like:
- Request velocity: bursts, constant high rate, or perfectly regular intervals
- Access patterns: sequential IDs, full-table scans, repeated pagination from page 1 onward
- Coverage: hitting large portions of your dataset with little variation
- Session behavior: many accounts from one device/IP, new accounts immediately scraping
- User-agent anomalies: missing headers, generic automation tool signatures
- Timing patterns: 24/7 activity, no diurnal pattern, no think time
- Error patterns: lots of 404/403/429 or schema-probing requests
- Endpoint mix: only “data export” endpoints, never normal app flows
A simple fraud-style risk score works well: multiple weak signals combined are more reliable than one hard rule.
2) Use progressive enforcement
Don’t jump straight to bans unless you’re sure.
A good escalation ladder:
- Log only
- Soft throttle (slower responses, lower burst capacity)
- 429 rate limit
- Require re-auth / token refresh
- CAPTCHA or proof-of-work for suspicious flows
- Temporary block
- Permanent block only for clear abuse
This helps protect legitimate users who may have occasional spikes or unusual behavior.
3) Rate-limit by identity, not just IP
IP-only blocking breaks mobile users, offices, NATs, and VPNs.
Prefer rate limits on:
- API key / OAuth client
- User account
- Device/session
- IP as a supporting signal
- ASN / datacenter reputation as a weaker signal
Use:
- Global limit per account/app
- Endpoint-specific limits
- Burst + sustained limits
- Sliding window or token bucket algorithms
4) Protect high-value endpoints differently
Not all endpoints need the same treatment.
Examples:
- Public metadata: lighter limits
- Search/filter endpoints: medium
- Bulk exports, list pages, detailed records: stricter
- Authentication, password reset, invite, pricing, and account endpoints: very strict
For sensitive or high-value data:
- Require auth
- Limit page size
- Disallow unbounded queries
- Add cursor-based pagination
- Cap result sets
- Cache responses where possible
5) Make scraping harder with API design
Reduce the usefulness of automation without blocking legit clients.
Good patterns:
- Cursor pagination instead of offset pagination
- Stable but expiring cursors
- Maximum page size
- Field-level access control
- Selective redaction
- Signed requests for trusted clients
- Short-lived tokens
- Per-client quotas
- Schema changes and versioning to slow brittle scrapers
Avoid:
- Predictable sequential IDs exposed without controls
- Unlimited list endpoints
- Infinite scroll APIs with no throttling
6) Use bot/challenge friction only when risk is high
CAPTCHAs and challenges can stop simple automation, but they hurt UX if overused.
Best practice:
- Trigger only on risk spikes
- Prefer step-up verification only for suspicious actions
- Use device binding or re-authentication
- Avoid CAPTCHA for normal API flows unless absolutely necessary
For APIs, a better pattern than CAPTCHA is often:
- temporary token refresh,
- secondary verification,
- or a limited “cooldown” response.
7) Detect abuse using anomaly baselines
What’s “normal” varies by customer and endpoint.
Build baselines for:
- Requests per minute
- Unique records fetched per hour
- Distinct pages/filters used
- Error rates
- Time of day
- Session length
- Ratio of reads to writes
Then alert on deviations:
- A brand-new account suddenly accesses thousands of records
- A client that normally reads 10/min jumps to 500/min
- The same pattern repeats across many accounts from one device fingerprint
8) Separate humans from automation by workflow
Real users usually follow workflows:
- login → browse → filter → open details → maybe export
Scrapers often:
- jump directly to data endpoints,
- enumerate systematically,
- avoid UI-like behavior,
- and make no “navigation” requests.
If you have an app and API, compare:
- Web UI navigation patterns
- API-only access
- Presence of assets, JS, and normal browser behavior
But don’t assume API-only is bad—many legitimate integrators are API-only.
9) Use allowlists and partner tiers
Some “scraping-like” behavior is legitimate:
- internal tools
- partner integrations
- customers exporting their own data
- monitoring systems
- mobile apps with aggressive polling
Create:
- documented partner API tiers,
- higher quotas for trusted clients,
- allowlists for known good service accounts,
- and separate terms for bulk access.
This reduces false positives.
10) Add logging that helps you explain decisions
If you block someone, you should know why.
Log:
- request metadata
- auth identity
- quota state
- risk score components
- triggered rule(s)
- action taken
- correlation IDs
This helps you:
- tune thresholds,
- reverse false positives,
- and support legitimate users who get caught.
11) Test for false positives before enforcing
Replay real traffic:
- support tickets
- mobile app usage
- partner integrations
- power users
- batch jobs
Measure:
- how many legitimate sessions would be throttled,
- which endpoints are most affected,
- and what thresholds preserve UX.
A/B test conservative controls first, then tighten.
12) A practical starter policy
If you want something simple to implement:
- Per API key:
- burst: 20 requests / 10 sec
- sustained: 600 requests / hour
- Per endpoint:
- search/list endpoints: stricter than detail endpoints
- Risk score triggers:
- sequential ID enumeration
-
X unique records/minute
- high 404 rate
- datacenter ASN + no browser-like headers
- Responses:
- first offense: 429 with retry-after
- repeated suspicious behavior: cooldown
- clear abuse: temporary suspension
13) What to avoid
- Blocking purely by IP
- Using only user-agent checks
- Hard banning on one burst
- CAPTCHA on every request
- Relying on hidden endpoints as “security”
- Introducing controls that break accessibility tools or legitimate automation
14) Good rule of thumb
Optimize for “make scraping expensive” instead of “make scraping impossible.”
You usually want to:
- reduce scale,
- increase maintenance cost,
- detect quickly,
- and keep legitimate traffic smooth.
If you want, I can help you design a specific anti-scraping policy for your API stack (e.g. REST/GraphQL, Cloudflare/Akamai, AWS API Gateway, FastAPI, Express, etc.) or propose exact rate-limit and risk-score rules.