How do I set up a shell script tool for automating repeatable ops tasks across our SRE team?

A good way to set up a shell-script tool for repeatable SRE ops work is to treat it like a small product: predictable interface, safe defaults, versioned distribution, and a little discipline around testing and change control.

1) Start with the right scope

Use a shell script tool when the task is:

• short-lived or repeatable
• mostly gluing together existing CLIs
• easy to express as linear steps
• okay to run from a terminal or CI job

If it needs complex parsing, concurrency, rich state, or long-term maintainability, consider Python/Go instead.

2) Define the tool contract

Before coding, specify:

• Purpose: what operational task it automates
• Inputs: flags, env vars, config file
• Outputs: logs, artifacts, exit codes
• Safety model: dry-run, confirmation, idempotency
• Dependencies: bash, jq, kubectl, aws, etc.
• Permissions: who can run it and with what credentials

A simple standard helps a lot:

• --dry-run
• --force
• --verbose
• --env <name>
• --config <file>

3) Use a robust shell pattern

Prefer Bash with strict mode:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Then add:

• getopts or a CLI parser for arguments
• functions for each logical step
• explicit error handling
• traps for cleanup

Example skeleton:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

readonly SCRIPT_NAME="$(basename "$0")"

usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [--dry-run] [--env ENV] <command>

Commands:
  rotate-cert   Rotate service certificate
  cleanup       Remove stale resources
EOF
}

log() { printf '[%s] %s\n' "$(date -Is)" "$*" >&2; }
die() { printf '[%s] ERROR: %s\n' "$(date -Is)" "$*" >&2; exit 1; }

DRY_RUN=0
ENVIRONMENT=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run) DRY_RUN=1 ;;
    --env) ENVIRONMENT="${2:-}"; shift ;;
    -h|--help) usage; exit 0 ;;
    *) break ;;
  esac
  shift
done

[[ -n "$ENVIRONMENT" ]] || die "--env is required"
[[ $# -gt 0 ]] || die "missing command"

run() {
  if [[ "$DRY_RUN" -eq 1 ]]; then
    log "DRY-RUN: $*"
  else
    log "Running: $*"
    "$@"
  fi
}

4) Make it safe by default

For SRE tooling, safety matters more than cleverness.

Recommended protections:

• default to dry-run where possible
• require confirmation for destructive actions
• verify target environment before acting
• check preconditions early
• make operations idempotent
• use explicit timeouts on network calls
• avoid rm -rf-style patterns unless tightly scoped

Add guardrails like:

[[ "$ENVIRONMENT" =~ ^(dev|staging|prod)$ ]] || die "invalid environment"

For destructive actions:

read -r -p "Proceed with deleting 12 resources in prod? (yes/no) " answer
[[ "$answer" == "yes" ]] || die "aborted"

5) Make dependencies explicit

Use a dependency check at startup:

require_cmd() {
  command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"
}

require_cmd jq
require_cmd kubectl

If you need specific versions, validate them too.

6) Centralize configuration

Avoid hardcoding cluster names, regions, or endpoints in the script.

Use one of:

• environment variables
• a config file (.env, YAML, or simple key/value)
• a repo-managed defaults file

Example:

: "${AWS_REGION:=us-east-1}"
: "${KUBE_CONTEXT:=prod-us-east-1}"

For team use, it’s often useful to keep:

• defaults.sh
• config/<env>.env
• README.md with examples

7) Build it for team distribution

Put the script in a shared repo:

• ops-tools/
• versioned with Git tags/releases
• reviewed via PR
• changelog maintained

A practical structure:

ops-tools/
  bin/
    incident-restart.sh
    rotate-cert.sh
  lib/
    common.sh
  test/
    test_incident_restart.bats
  README.md
  Makefile

If multiple scripts share helpers, put common functions in lib/common.sh and source them.

8) Add testing

Even shell tools benefit from tests.

At minimum:

• syntax check: bash -n script.sh
• shell lint: shellcheck
• formatting: shfmt

For behavior:

• use bats for unit/integration tests
• mock external commands when possible
• test dry-run mode
• test invalid inputs and failure paths

CI pipeline example:

• shellcheck
• shfmt -d
• bats test/
• package artifact or publish release

9) Improve observability

Scripts should log clearly:

• start/end
• target environment
• action being taken
• command outputs when relevant
• failure reason

Prefer structured-ish logs if possible:

log() {
  printf 'ts=%s level=info msg="%s"\n' "$(date -Is)" "$*" >&2
}

For long-running actions, emit step markers:

log "Step 1/4: validating cluster access"
log "Step 2/4: collecting resources"

10) Make execution easy

Your team will use it more if running it is simple.

Good patterns:

• ./bin/rotate-cert.sh --env prod --dry-run
• make rotate-cert ENV=prod
• docker run ... if dependencies are annoying
• a wrapper in CI for approvals

If the environment is heterogeneous, a containerized runner can reduce “works on my laptop” problems.

11) Manage credentials carefully

Avoid embedding secrets in scripts.

Use:

• cloud identity federation / SSO
• short-lived credentials
• kube contexts / IAM roles
• secret stores
• CI-managed secrets

Also:

• never set -x around sensitive commands
• redact tokens from logs
• keep secrets out of command-line args when possible

12) Document operational runbooks

For each tool, document:

• what it does
• when to use it
• prerequisites
• examples
• rollback/undo steps
• known failure modes
• owner/team

A good README snippet:

## Usage
Rotate cert in staging:
./bin/rotate-cert.sh --env staging --dry-run
./bin/rotate-cert.sh --env staging

## Safety
- Dry-run supported
- Requires kubectl access to target cluster
- Exits non-zero on any failed subcommand

13) Keep the interface stable

Treat flags and behavior as an API:

• avoid breaking changes
• deprecate gradually
• version releases
• announce changes to the team

14) A practical team workflow

1. Engineer writes or updates script in PR
2. Reviewer checks safety, idempotency, and logging
3. CI runs lint/tests
4. Merge and tag a release
5. Publish usage notes in the runbook
6. Optional: add a small wrapper in Slack/ChatOps or CI

15) Recommended minimum standard

If you want a simple baseline, require:

• Bash strict mode
• shellcheck
• --dry-run
• input validation
• dependency checks
• clear logging
• README usage examples
• PR review + CI

---

If you want, I can also provide:

1. a production-ready Bash script template,
2. a Makefile + CI pipeline for shell scripts, or
3. a shared common.sh library pattern for your SRE team.