Prompt

How do I detect and block API scraping without breaking real users?

Technology / Cybersecurity1 observationLast seen Oct 29, 2026

Latest observation

Oct 29, 2026OpenAI APIWeb search: off

To detect and block API scraping without hurting real users, use a layered approach: observe, score, challenge, and rate-limit gradually rather than outright blocking based on one signal.

1) Start with behavioral signals, not just IPs

Scrapers often look “human” on one metric but unnatural across several.

Track things like:

  • Request velocity: bursts, constant high rate, or perfectly regular intervals
  • Access patterns: sequential IDs, full-table scans, repeated pagination from page 1 onward
  • Coverage: hitting large portions of your dataset with little variation
  • Session behavior: many accounts from one device/IP, new accounts immediately scraping
  • User-agent anomalies: missing headers, generic automation tool signatures
  • Timing patterns: 24/7 activity, no diurnal pattern, no think time
  • Error patterns: lots of 404/403/429 or schema-probing requests
  • Endpoint mix: only “data export” endpoints, never normal app flows

A simple fraud-style risk score works well: multiple weak signals combined are more reliable than one hard rule.


2) Use progressive enforcement

Don’t jump straight to bans unless you’re sure.

A good escalation ladder:

  1. Log only
  2. Soft throttle (slower responses, lower burst capacity)
  3. 429 rate limit
  4. Require re-auth / token refresh
  5. CAPTCHA or proof-of-work for suspicious flows
  6. Temporary block
  7. Permanent block only for clear abuse

This helps protect legitimate users who may have occasional spikes or unusual behavior.


3) Rate-limit by identity, not just IP

IP-only blocking breaks mobile users, offices, NATs, and VPNs.

Prefer rate limits on:

  • API key / OAuth client
  • User account
  • Device/session
  • IP as a supporting signal
  • ASN / datacenter reputation as a weaker signal

Use:

  • Global limit per account/app
  • Endpoint-specific limits
  • Burst + sustained limits
  • Sliding window or token bucket algorithms

4) Protect high-value endpoints differently

Not all endpoints need the same treatment.

Examples:

  • Public metadata: lighter limits
  • Search/filter endpoints: medium
  • Bulk exports, list pages, detailed records: stricter
  • Authentication, password reset, invite, pricing, and account endpoints: very strict

For sensitive or high-value data:

  • Require auth
  • Limit page size
  • Disallow unbounded queries
  • Add cursor-based pagination
  • Cap result sets
  • Cache responses where possible

5) Make scraping harder with API design

Reduce the usefulness of automation without blocking legit clients.

Good patterns:

  • Cursor pagination instead of offset pagination
  • Stable but expiring cursors
  • Maximum page size
  • Field-level access control
  • Selective redaction
  • Signed requests for trusted clients
  • Short-lived tokens
  • Per-client quotas
  • Schema changes and versioning to slow brittle scrapers

Avoid:

  • Predictable sequential IDs exposed without controls
  • Unlimited list endpoints
  • Infinite scroll APIs with no throttling

6) Use bot/challenge friction only when risk is high

CAPTCHAs and challenges can stop simple automation, but they hurt UX if overused.

Best practice:

  • Trigger only on risk spikes
  • Prefer step-up verification only for suspicious actions
  • Use device binding or re-authentication
  • Avoid CAPTCHA for normal API flows unless absolutely necessary

For APIs, a better pattern than CAPTCHA is often:

  • temporary token refresh,
  • secondary verification,
  • or a limited “cooldown” response.

7) Detect abuse using anomaly baselines

What’s “normal” varies by customer and endpoint.

Build baselines for:

  • Requests per minute
  • Unique records fetched per hour
  • Distinct pages/filters used
  • Error rates
  • Time of day
  • Session length
  • Ratio of reads to writes

Then alert on deviations:

  • A brand-new account suddenly accesses thousands of records
  • A client that normally reads 10/min jumps to 500/min
  • The same pattern repeats across many accounts from one device fingerprint

8) Separate humans from automation by workflow

Real users usually follow workflows:

  • login → browse → filter → open details → maybe export

Scrapers often:

  • jump directly to data endpoints,
  • enumerate systematically,
  • avoid UI-like behavior,
  • and make no “navigation” requests.

If you have an app and API, compare:

  • Web UI navigation patterns
  • API-only access
  • Presence of assets, JS, and normal browser behavior

But don’t assume API-only is bad—many legitimate integrators are API-only.


9) Use allowlists and partner tiers

Some “scraping-like” behavior is legitimate:

  • internal tools
  • partner integrations
  • customers exporting their own data
  • monitoring systems
  • mobile apps with aggressive polling

Create:

  • documented partner API tiers,
  • higher quotas for trusted clients,
  • allowlists for known good service accounts,
  • and separate terms for bulk access.

This reduces false positives.


10) Add logging that helps you explain decisions

If you block someone, you should know why.

Log:

  • request metadata
  • auth identity
  • quota state
  • risk score components
  • triggered rule(s)
  • action taken
  • correlation IDs

This helps you:

  • tune thresholds,
  • reverse false positives,
  • and support legitimate users who get caught.

11) Test for false positives before enforcing

Replay real traffic:

  • support tickets
  • mobile app usage
  • partner integrations
  • power users
  • batch jobs

Measure:

  • how many legitimate sessions would be throttled,
  • which endpoints are most affected,
  • and what thresholds preserve UX.

A/B test conservative controls first, then tighten.


12) A practical starter policy

If you want something simple to implement:

  • Per API key:
    • burst: 20 requests / 10 sec
    • sustained: 600 requests / hour
  • Per endpoint:
    • search/list endpoints: stricter than detail endpoints
  • Risk score triggers:
    • sequential ID enumeration
    • X unique records/minute

    • high 404 rate
    • datacenter ASN + no browser-like headers
  • Responses:
    • first offense: 429 with retry-after
    • repeated suspicious behavior: cooldown
    • clear abuse: temporary suspension

13) What to avoid

  • Blocking purely by IP
  • Using only user-agent checks
  • Hard banning on one burst
  • CAPTCHA on every request
  • Relying on hidden endpoints as “security”
  • Introducing controls that break accessibility tools or legitimate automation

14) Good rule of thumb

Optimize for “make scraping expensive” instead of “make scraping impossible.”
You usually want to:

  • reduce scale,
  • increase maintenance cost,
  • detect quickly,
  • and keep legitimate traffic smooth.

If you want, I can help you design a specific anti-scraping policy for your API stack (e.g. REST/GraphQL, Cloudflare/Akamai, AWS API Gateway, FastAPI, Express, etc.) or propose exact rate-limit and risk-score rules.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.