Prompt
How do I click through a logged-in web app and pull the data out?
Latest observation
You can do this, but the best method depends on the app’s architecture and your permission to access the data.
First: the legitimate / practical routes
If you’re authorized to access the app, the usual options are:
-
Use an official API
Check whether the app has a public or private API. This is usually the cleanest way to pull data. -
Use browser automation
If the data only appears after clicking around in the UI, tools like Playwright or Selenium can:- log in
- navigate pages
- click buttons / fill forms
- extract table contents or network responses
-
Inspect network calls Many web apps load data via XHR/fetch requests. If you can identify the backend endpoints, you may be able to retrieve the data directly after authenticating.
-
Export features Some apps already have CSV/XLSX export buttons or report downloads. Use those if available.
A typical browser automation approach
High-level workflow:
- Open the site in an automated browser
- Sign in
- Wait for the target page to load
- Click through menus/buttons
- Read the rendered DOM, or capture the API response
- Save to JSON/CSV
Example with Playwright (Python)
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://example.com/login")
page.fill("#username", "your_username")
page.fill("#password", "your_password")
page.click("button[type=submit]")
page.wait_for_load_state("networkidle")
page.goto("https://example.com/app/data")
page.wait_for_selector("table")
rows = page.locator("table tbody tr")
data = []
for i in range(rows.count()):
row = rows.nth(i)
data.append(row.inner_text())
print(data)
browser.close()
If the data is fetched by network requests
Often the page makes requests like:
/api/orders/graphql/reports/search
You can use the browser’s devtools Network tab to see these. Then:
- reuse your authenticated session/cookies
- send the same request programmatically
- parse JSON or HTML response
Important cautions
- Don’t bypass access controls or scrape data you aren’t authorized to access.
- Respect terms of service, robots rules where applicable, and rate limits.
- If the site uses MFA/SSO, automation may still be possible, but you’ll want to use compliant methods and secure handling of credentials.
If you want, I can help you with a specific stack:
- Playwright
- Selenium
- requests + cookies
- capturing network calls
- logging in to an SSO/MFA app
- extracting tables to CSV
Brands