How to configure Bright Data residential proxies with Puppeteer and Effect-TS

Configuring Bright Data residential proxies with Puppeteer inside an Effect-TS pipeline allows you to handle robust data scraping with functional error handling, resource management (acquireUseRelease), and automatic IP rotation.

Step 1: Install Dependencies

Make sure you have effect, puppeteer, and dotenv installed:

Bash

npm install effect puppeteer dotenv
npm install -D typescript @types/node

Step 2: Set Environment Variables

Create a .env file in your root directory with your Bright Data credentials:

Code snippet

BRIGHT_DATA_CUSTOMER_ID=your_customer_id
BRIGHT_DATA_ZONE=your_zone_name
BRIGHT_DATA_PASSWORD=your_zone_password

Step 3: Implement the Effect-TS + Puppeteer Proxy Pipeline

Here is how you can write a type-safe script using Effect to launch Puppeteer via Bright Data's residential proxy superproxy (brd.superproxy.io:33335), manage resource acquisition safely, and handle IP session rotation.

TypeScript

import { Effect, Data, pipe } from "effect";
import puppeteer, { type Browser, type Page } from "puppeteer";
import { config } from "dotenv";

config();

// Define custom tagged errors for Effect pipeline
class BrowserError extends Data.Error<{ message: string; cause?: unknown }> {}

// Bright Data Configuration
const BRIGHT_DATA_CONFIG = {
  customerId: process.env.BRIGHT_DATA_CUSTOMER_ID,
  zone: process.env.BRIGHT_DATA_ZONE,
  password: process.env.BRIGHT_DATA_PASSWORD,
  proxyHost: "brd.superproxy.io",
  proxyPort: 33335, // Standard port for Bright Data proxy zones
};

interface ProxyConfig {
  host: string;
  port: number;
  username: string;
  password: string;
}

// Build Proxy Configuration string (supports optional session rotation)
const buildProxyConfig = (sessionId?: string | number): ProxyConfig => {
  const { customerId, zone, proxyHost, proxyPort, password } = BRIGHT_DATA_CONFIG;
  
  // Appending -session-<ID> keeps a sticky IP; omitting it rotates IPs per request
  const username = sessionId
    ? `brd-customer-${customerId}-zone-${zone}-session-${sessionId}`
    : `brd-customer-${customerId}-zone-${zone}`;

  return {
    host: proxyHost,
    port: proxyPort,
    username,
    password: password || "",
  };
};

// 1. Launch Browser Effect with Proxy Argument
const launchBrowser = (proxyConfig: ProxyConfig) =>
  Effect.tryPromise({
    try: () =>
      puppeteer.launch({
        headless: true,
        args: [`--proxy-server=${proxyConfig.host}:${proxyConfig.port}`],
      }),
    catch: (cause) => new BrowserError({ message: "Failed to launch browser", cause }),
  });

// 2. Authenticate Page with Bright Data Credentials
const authenticatePage = (page: Page, proxyConfig: ProxyConfig) =>
  Effect.tryPromise({
    try: () =>
      page.authenticate({
        username: proxyConfig.username,
        password: proxyConfig.password,
      }),
    catch: (cause) => new BrowserError({ message: "Failed to authenticate proxy", cause }),
  });

// 3. Safe Resource Management using Effect.acquireUseRelease
const withScrapingSession = <A, E>(
  sessionId: string | number,
  use: (page: Page) => Effect.Effect<A, E, never>
) => {
  const proxyConfig = buildProxyConfig(sessionId);

  return Effect.acquireUseRelease(
    launchBrowser(proxyConfig),
    (browser) =>
      pipe(
        Effect.tryPromise(() => browser.newPage()),
        Effect.mapError((cause) => new BrowserError({ message: "Failed to create page", cause })),
        Effect.flatMap((page) =>
          pipe(
            authenticatePage(page, proxyConfig),
            Effect.flatMap(() => use(page))
          )
        )
      ),
    (browser) => Effect.promise(() => browser.close())
  );
};

// 4. Main Scraping Program Logic
const program = pipe(
  withScrapingSession("user_session_123", (page) =>
    Effect.gen(function* () {
      yield* Effect.log("Navigating to IP check service...");
      
      // Go to a test site to verify the residential proxy IP is working
      yield* Effect.tryPromise(() =>
        page.goto("https://brdtest.com/myip.json", { waitUntil: "networkidle2" })
      );

      const ipInfo = yield* Effect.tryPromise(() =>
        page.evaluate(() => document.body.innerText)
      );

      yield* Effect.log(`Successfully fetched data via proxy! Response: ${ipInfo}`);
      return ipInfo;
    })
  ),
  Effect.catchTag("BrowserError", (error) =>
    Effect.succeed(`Scraping failed safely: ${error.message}`)
  )
);

// Run the Effect program
Effect.runPromise(program).then(console.log);

Key Integration Practices

1. Proxy Argument (--proxy-server): Puppeteer routes underlying traffic through Bright Data's entry node (brd.superproxy.io:33335) via launch arguments.
2. Page-Level Authentication (page.authenticate): Bright Data requires your zone credentials (username and password). Injecting them via page.authenticate() handles proxy authentication securely per browser context.
3. Sticky vs. Rotating Sessions:

• If you want a new IP address for every request, use the base username: brd-customer-<ID>-zone-<ZONE>.
• If you want to maintain the same IP across multiple actions or page transitions, append a session flag like -session-<RANDOM_ID> (e.g., brd-customer-<ID>-zone-<ZONE>-session-1337).

4. Effect Resource Safety: Using Effect.acquireUseRelease ensures that even if your scraping routine throws an error or gets blocked, the underlying Chromium browser instance is properly disposed of to prevent memory and CPU leaks.