Prompt
How do I automate downloads from a web portal on a schedule?
Latest observation
Oct 26, 2026
OpenAI APIWeb search: off
Automating downloads from a web portal on a schedule usually comes down to one of these approaches, from simplest to most robust:
1) Use the portal’s API or built-in export
If the portal offers:
- an API
- a scheduled report/export
- email delivery of files
- an SFTP/FTP drop
…that’s the best option. It’s more reliable than browser automation and usually less fragile.
2) Script the download with browser automation
If there’s no API, use a script that logs in, navigates the portal, and clicks the download link/button.
Common tools:
- Python + Playwright (recommended)
- Python + Selenium
- Node.js + Playwright
Typical flow:
- Launch browser
- Go to login page
- Enter credentials
- Handle MFA if needed
- Navigate to the report/download page
- Trigger the download
- Save the file to a known directory
- Rename/archive it
Example with Playwright (Python)
from playwright.sync_api import sync_playwright
from pathlib import Path
DOWNLOAD_DIR = Path("downloads")
DOWNLOAD_DIR.mkdir(exist_ok=True)
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page(accept_downloads=True)
page.goto("https://portal.example.com/login")
page.fill("#username", "your_username")
page.fill("#password", "your_password")
page.click("button[type=submit]")
page.wait_for_load_state("networkidle")
page.goto("https://portal.example.com/reports")
with page.expect_download() as download_info:
page.click("text=Download CSV")
download = download_info.value
download.save_as(DOWNLOAD_DIR / download.suggested_filename)
browser.close()
3) Schedule it with cron / Task Scheduler / a job runner
Once the script works, schedule it:
Linux/macOS: cron
Run every day at 7 AM:
0 7 * * * /usr/bin/python3 /path/to/download_script.py >> /path/to/logs/download.log 2>&1
Windows: Task Scheduler
- Create a basic task
- Trigger: daily/weekly
- Action: start a program
- Program:
python.exe - Arguments:
C:\path\to\download_script.py
Cloud options
If you want it running server-side:
- GitHub Actions
- AWS Lambda / ECS / EC2
- Azure Automation
- Google Cloud Run / Scheduler
4) Make it robust
Automations break when portals change, so add:
- retry logic
- timeouts
- logging
- file validation
- alerts on failure
- secure credential handling
Use:
- environment variables
- secret managers
- encrypted config files
Avoid hardcoding passwords in scripts.
5) If MFA or CAPTCHA is involved
- MFA: some portals support app passwords, service accounts, or API tokens; otherwise browser automation may need a human-in-the-loop step.
- CAPTCHA: don’t try to bypass it; use official access methods or request an API / whitelisted automation account.
Recommended stack
If you’re starting from scratch:
- Python
- Playwright
- cron/Task Scheduler
- logging + alerts
Practical example architecture
- Script downloads file daily
- Saves to a timestamped folder
- Verifies file size/content
- Uploads to storage or sends to downstream system
- Sends success/failure email or Slack message
If you want, I can help you with:
- a Python Playwright starter script
- a Selenium version
- a cron/Task Scheduler setup
- or a more specific solution if you share the portal’s login/download flow.