How do I set up a deployment pipeline for environment promotion with self-hosted runners?

To set up a deployment pipeline for environment promotion (for example, development → staging → production) that uses self-hosted runners, combine labeled self-hosted runners with environment definitions and sequential jobs that enforce promotion gates. The pattern works in GitHub Actions, GitLab CI, and similar systems. Below is a practical, platform-agnostic approach focused on the most common tools, with concrete examples for GitHub Actions (the principles map directly to GitLab).

1. Prepare and register self-hosted runners

Provision machines (VMs, bare metal, or containers) that can reach your target environments and have the necessary tools (Docker, kubectl, cloud CLIs, etc.). Register each runner with your Git provider: GitHub: Go to repository/organization Settings → Actions → Runners → New self-hosted runner, then run the provided config.sh (or equivalent) commands. GitLab: Install GitLab Runner and register it with a registration token, assigning tags.

Apply useful labels so jobs can target the right runners: Common labels: self-hosted, linux, x64. Environment-specific or role-specific labels: deploy-dev, deploy-staging, deploy-prod, or gpu if needed.

Prefer ephemeral runners (one job per runner lifetime) for better isolation and security, especially for production deploys. Configure them as a systemd service or use autoscaling solutions (Actions Runner Controller on Kubernetes, or similar). Place runners in network segments that can reach only the environments they are allowed to deploy to, and restrict outbound access to the minimum required.

2. Define environments and protection rules

In GitHub: Repository Settings → Environments → create development, staging, and production. For staging and production add protection rules: required reviewers, wait timers, and deployment branch restrictions. Attach environment-specific secrets and variables (API keys, kubeconfigs, cloud credentials).

In GitLab: Use the environment: keyword in jobs; protected environments and approval rules provide the same gating. These environments give you a clear promotion path, deployment history, and the ability to require human approval before higher stages run.

3. Write the promotion pipeline Create a workflow (.github/workflows/deploy.yml or .gitlab-ci.yml) with sequential jobs that depend on one another and target the correct runners and environments. Example structure for GitHub Actions:

A build/test job (can run on any suitable self-hosted or hosted runner). A deploy-dev job that runs only after the build succeeds, uses a runner labeled for development, and targets the development environment. A deploy-staging job that needs the successful completion of deploy-dev, uses a staging-labeled runner, and targets the staging environment (with optional required reviewers). A deploy-production job that needs deploy-staging, uses a production-labeled runner, targets the production environment, and is protected by required reviewers or a wait timer.

Key practices:

Use needs: (GitHub) or stage dependencies (GitLab) so each environment only deploys after the previous one succeeds. Reference the environment in the job so secrets and protection rules apply automatically. Make production (and often staging) jobs manual or approval-gated. Pass the same artifact/image tag through the promotion chain so you deploy the exact same build that was tested.

4. Secure the runners and pipeline

Run jobs inside containers where possible for isolation. Use short-lived credentials or OIDC federation instead of long-lived secrets on the runners. Limit which repositories or branches can use production-labeled runners (via runner groups in GitHub or protected runners/tags in GitLab). Keep runner software and host OS patched; monitor runner health and job queue depth. Store environment-specific configuration as secrets or variables scoped to the environment, never in the workflow file.

5. Operational tips for reliable promotion

Tag every build with a unique version or commit SHA and promote that exact artifact. Add smoke or integration tests after each environment deploy before allowing the next promotion. Log deployment history so you always know what is running where. For Kubernetes or GitOps setups, the pipeline can update a Git repository (image tag or manifest) and let a GitOps controller (Argo CD, Flux) perform the actual sync; the self-hosted runner only needs to open the PR or commit the change. Start with a simple three-stage pipeline and add canary/blue-green steps later if needed.

Once the runners are registered with clear labels and the environments are protected, a push (or manual trigger) will build the artifact once and then promote it safely through the environments under the control of the self-hosted runners and the approval gates you configured.