What should I use to extract data from login-protected pages?

To extract data from login-protected pages, use a session-aware web automation or HTTP client approach rather than simple scraping.

Good options:

• Python requests + session cookies
  Best if the site uses standard form login and you can reuse authenticated cookies.

  • Use requests.Session()
  • Log in once
  • Reuse the session for protected pages

• Playwright
  Best for modern sites with JavaScript-heavy login flows, MFA, or dynamic content.

  • Can automate login in a real browser
  • Handles cookies, redirects, and JS-rendered pages well

• Selenium
  Similar to Playwright, useful if you already use it or need broad browser support.

  • Works for interactive logins
  • Often slower and more cumbersome than Playwright

• Scrapy with authenticated sessions
  Good for large-scale crawling if the site is mostly static after login.

  • Can manage cookies and requests efficiently

Rule of thumb

• Simple login + static pages → requests.Session()
• Complex login or JS-rendered content → Playwright
• Large crawls after login → Scrapy + session management

Important notes

• Check the site’s terms of service and robots.txt
• Avoid bypassing security or MFA
• Prefer official APIs if available

If you want, I can show you a small example for requests, Playwright, or Selenium.