How do I set up a task runner for submitting online requests on cookie-heavy sites?

To set up a task runner for legitimate, cookie-heavy web workflows (for example: repeated form submissions, account actions you’re authorized to automate, internal tools, or your own site), the key is to manage sessions, cookies, retries, and rate limits reliably.

I can help with a safe, generic architecture for that. I won’t help with bypassing anti-bot systems, CAPTCHAs, or access controls, but I can show you how to build a robust automation runner for permitted requests.

Recommended architecture

1. Use a real browser session when cookies matter

For sites with heavy JS and lots of cookie/session state, use a browser automation library such as:

• Playwright (recommended)
• Selenium
• Puppeteer

Playwright is often easiest because it handles:

• persistent cookies
• browser contexts
• request/response inspection
• storage state export/import

2. Keep sessions isolated per account/job

Use one browser context per account or task:

• separate cookie jar
• separate localStorage/sessionStorage
• easier cleanup and debugging

3. Persist login state

If the workflow needs authenticated requests:

• log in once manually or through the site’s normal flow
• save the browser context storage state
• reload that state for future runs

Example in Playwright:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();

const page = await context.newPage();
await page.goto('https://example.com/login');

// perform normal login steps here
// ...

await context.storageState({ path: 'storage-state.json' });
await browser.close();

Then reuse it later:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({
  storageState: 'storage-state.json'
});

const page = await context.newPage();
await page.goto('https://example.com/dashboard');

4. Build a task queue

If you’re submitting many requests, use a queue so you can control:

• concurrency
• delays
• retries
• per-domain rate limiting

Options:

• Node.js: BullMQ, Bee-Queue, p-queue
• Python: Celery, RQ, asyncio queue

A simple pattern:

• fetch job
• open browser context
• run action
• validate response
• close context
• record result

5. Add retry logic with backoff

Requests can fail due to:

• stale session
• timeouts
• transient server errors
• expired cookies

Use:

• exponential backoff
• max retry count
• re-authentication on session expiration

6. Detect session expiry cleanly

Check for:

• redirects to login
• HTTP 401/403
• missing expected page elements
• unexpected empty responses

When detected:

• refresh login
• reload storage state
• retry once or twice

7. Respect site rules and limits

For legitimate automation:

• stay within rate limits
• use official APIs if available
• avoid unnecessary parallelism
• identify your automation if the site requires it

Example task runner skeleton in Node.js

import pLimit from 'p-limit';
import { chromium } from 'playwright';

const limit = pLimit(2); // max 2 concurrent tasks

async function runTask(task) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    storageState: 'storage-state.json'
  });
  const page = await context.newPage();

  try {
    await page.goto(task.url, { waitUntil: 'networkidle' });

    // perform allowed form actions here
    // await page.fill(...)
    // await page.click(...)

    return { ok: true };
  } catch (err) {
    return { ok: false, error: err.message };
  } finally {
    await browser.close();
  }
}

const tasks = [
  { url: 'https://example.com/form1' },
  { url: 'https://example.com/form2' },
];

const results = await Promise.all(tasks.map(task => limit(() => runTask(task))));
console.log(results);

If you only need HTTP requests

If the site exposes endpoints you are authorized to call, use a normal HTTP client and cookie jar:

• Node.js: axios + cookie jar
• Python: requests.Session()

Example in Python:

import requests

s = requests.Session()
s.headers.update({"User-Agent": "MyApp/1.0"})

r = s.get("https://example.com")
print(r.status_code)

But note: on cookie-heavy, JS-driven sites, browser automation is usually more reliable.

Operational tips

• Use separate config for credentials, cookies, and job parameters
• Log request IDs, timestamps, and response codes
• Store screenshots or HTML snapshots on failures
• Keep secrets in a vault or environment variables
• Rotate browser profiles only for clean separation, not to evade site protections

Best practice choice

• Simple authenticated web flows: Playwright
• High-volume authorized API calls: HTTP client + session cookies
• Distributed scheduled jobs: queue + worker + browser automation

If you want, I can help you with one of these next:

1. a Playwright starter project
2. a Python requests.Session version
3. a job queue/worker design
4. a cookie/session persistence example