SkyHub’s container images are built and deployed by AWS CodeBuild, not GitHub Actions. Every backend service, the frontend, and the on-drone (Core) images each get a dedicated aws_codebuild_project defined in Terraform under skyhub_terraform/modules/*/…_build_deploy.tf. A GitHub push-webhook triggers the build; the build produces an image (or a static bundle), pushes it, and rolls the target.
The gateway repo’s own GitHub Actions (.github/workflows/pytest_unit_tests.yml, ruff.yml) only run pytest + Ruff — they never build or push an image. All image build/push and deploy happens in CodeBuild. For the gateway’s Docker build itself (Dockerfile.ecr, gunicorn runtime) see /deployment/gateway-build-runtime.

​
The universal pipeline shape

Almost every service pipeline follows the same four-step recipe: build Dockerfile.ecr → push to ECR :latest → aws ecs update-service --force-new-deployment. The pipeline runs a docker-in-docker daemon inside a privileged CodeBuild container, logs in to ECR, builds with layer caching, pushes, and forces the Fargate service to pull the new :latest and replace its task. The canonical example is the gateway (api) pipeline in skyhub_terraform/modules/api/api_build_deploy.tf:
modules/api/api_build_deploy.tf (buildspec, abridged)
version: 0.2
phases:
  install:
    commands:
      - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 &
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
      - export ECR_URI="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ECR_URI
  build:
    commands:
      - docker build -f Dockerfile.ecr --cache-from $IMAGE_NAME:latest -t $IMAGE_NAME:latest .
  post_build:
    commands:
      - docker push -a $IMAGE_NAME
      - aws ecs update-service --service skyhub-prod-api-service --force-new-deployment --cluster skyhub-prod-cluster --region $AWS_DEFAULT_REGION

​
Build environment

Compute

BUILD_GENERAL1_SMALL, privileged_mode = true (docker-in-docker). Most builds use the ARM image aws/codebuild/amazonlinux2-aarch64-standard:3.0 — matching the ARM64 Fargate tasks. Janus is the exception: it builds x86 on amazonlinux2-x86_64-standard:5.0.

Caching & logs

cache { type = LOCAL, modes = [LOCAL_DOCKER_LAYER_CACHE, LOCAL_SOURCE_CACHE] } plus --cache-from :latest. Logs go to CloudWatch group /aws/codebuild/<project> with 7-day retention.
Most build projects run outside the VPC on purpose — a comment in api_build_deploy.tf explains it avoids taxing the single NAT gateway with dependency downloads (ingress is free). The two exceptions that must be in-VPC are the DB migration build (needs to reach database.skyhub-prod.internal) and the user-VPN build (SSHes to the jumphost private IP).

​
Trigger model: what fires a build

The webhook filter is environment-aware; the checked-out branch (source_version) is not. The single filter is defined once per environment in environments/prod/main.tf (and dev) and passed into every module as webhook_config:
environments/prod/main.tf
locals {
  codebuild_webhook_config = {
    build_type = "BUILD"
    filter_groups = [{
      event_pattern    = "PUSH"
      head_ref_pattern = var.environment == "prod" ? "^refs/tags/v_.*$" : "^refs/heads/develop$"
    }]
  }
}
EnvironmentFires onExample
prodpush of a git tag matching ^refs/tags/v_.*$git tag v_1.4.2 && git push origin v_1.4.2
devpush to the develop branch (^refs/heads/develop$)git push origin develop
Each project also pins a source_version (e.g. develop for the API, main for ws-proxy, development for Janus/WHIP). That pin is the ref checked out for manually started builds (aws codebuild start-build); a webhook-triggered build checks out the ref that fired it. So the hard-coded source_version and the webhook’s real ref can differ — a known source of confusion. If you run a build by hand and it deploys “the wrong branch”, it built the pinned source_version.

​
The pipelines

Every project name is prefixed with the resource tag (skyhub-prod-…). Sources are all under github.com/ID-Robots/.
Pipeline (CodeBuild project)Source reposource_versionDeploy action
skyhub-prod-api-build-and-deployskyhub_gateway_servicedeveloppush :latest → force-new-deployment (gateway)
skyhub-prod-api-migrateskyhub_gateway_serviceprodbuild image, docker run … flask db upgrade (no ECS deploy; in-VPC, no webhook)
skyhub-prod-frontend-build-and-deployskyhub_dashboarddevelopment (repo_branch)yarn build → sync dist/ to skyhub-prod-ui-bucket (no ECS)
skyhub-prod-janus-build-and-deployskyhub_janusdevelopmentpush :latest → force-new-deployment (janus). x86, --no-cache
skyhub-prod-whip-build-and-deployskyhub_whipdevelopmentpush :latest → force-new-deployment (whip)
skyhub-prod-ws-proxy-build-and-deployskyhub_ws_proxymainpush :latest → force-new-deployment (ws_proxy)
skyhub-prod-vpn-build-and-deployskyhub_user_vpnmainpush :latest → SSH into jumphost, run deploy.sh (docker-compose)
skyhub-prod-drone-build + layered buildsskyhub_coremain / cowboy/videobuild Core images / template docker-compose.installer.yml → S3 (no webhooks — manual)
CodeBuild builds and ECR pushes are the source of the redeploy-loop alarms on the gateway: a service that keeps replacing tasks trips the “pending tasks / deployment count / ‘Shutting down’ log spike” CloudWatch alarms in modules/api/api.tf. See /deployment/storage-backup-alarms.

​
Shared CodeBuild primitives

modules/codebuild/codebuild.tf defines the pieces every pipeline reuses:
  • aws_codebuild_source_credential — a GitHub PERSONAL_ACCESS_TOKEN (user todor943) that authorizes CodeBuild to clone the private ID-Robots/* repos. The token itself comes from var.github_token in settings.tfvars (a committed secret — rotate and move to SSM/Secrets Manager).
  • aws_ssm_parameter /skyhub/prod/slack_webhook — the Slack incoming-webhook URL passed to builds as SLACK_WEBHOOK_URL for start/finish notifications.
  • permissive_sg — an all-traffic security group reused by in-VPC builds.

​
DB migration build (separate, manual)

Schema migrations are not part of the deploy. modules/api/api_migrations.tf defines an independent skyhub-prod-api-migrate project that builds the same Dockerfile.ecr image and then runs Alembic against the live database:
modules/api/api_migrations.tf (build/run, abridged)
- docker build -f Dockerfile.ecr --cache-from $IMAGE_NAME:latest -t migrator-image .
- docker run \
    -e FLASK_APP=/app/src/migrator \
    -e DB_USERNAME=… -e DB_PASSWORD=… -e DB_IP=database.skyhub-prod.internal -e DB_NAME=skyhub \
    --rm migrator-image python -m flask db upgrade
Key facts a future editor must preserve:
  • It runs inside the VPC (vpc_config with the API ECS security group) so it can reach database.skyhub-prod.internal:5432.
  • Its source_version is prod, and it has no webhook — you start it manually (aws codebuild start-build --project-name skyhub-prod-api-migrate) when a release includes migrations.
  • The gateway’s own dev shortcut (db.create_all() when APP_ENVIRONMENT=dev) is not used here — production always goes through Alembic. See /gateway/data/migrations-connection.

​
Frontend build (Angular → S3)

The dashboard has no runtime container — it is a static SPA. modules/frontend/frontend_build_deploy.tf clones skyhub_dashboard, builds it with yarn, and overwrites the UI bucket:
1

Install & notify

Pin Node with n 24.11.1, post a Slack “build started”, then yarn install --frozen-lockfile.
2

Build

yarn run build --configuration production (the build_configuration input; dev uses aws-dev). Produces dist/.
3

Deploy to S3

aws s3 rm s3://skyhub-prod-ui-bucket --recursive then aws s3 cp --recursive . s3://skyhub-prod-ui-bucket — the bucket is wholesale replaced, no ECS involved.
The project also wires two support buckets: skyhub-prod-ui-build-bucket (zipped build artifacts, transitioned to Glacier after 7 days) and skyhub-prod-frontend-build-cache (S3 cache, expired after 7 days). The prod webhook still applies, so a v_* tag on skyhub_dashboard rebuilds and republishes the SPA. Browsers reach that bucket through the jumphost nginx / CloudFront — see /deployment/networking-jumphost. For the Angular build configs themselves, see /dashboard/build-and-config.

​
Drone (Core) image pipelines

The on-drone images are built from skyhub_core as a layered chain, each stage a separate CodeBuild project writing to its own ECR repo: The final skyhub-prod-drone-build project doesn’t build an app image at all — it sed-templates the ECR image URLs, WHIP server, and Redis host into docker-compose.installer.yml and uploads it to s3://skyhubcore/ (prod writes docker-compose.prod.yml, dev writes docker-compose.yml, chosen by var.stage). That compose file is exactly what the gateway hands a physical drone on activation via an S3 presigned URL. See the activation flow in /gateway/api/drones-and-actions and drone-side pull auth in /gateway/security/vpn-middleware-jumphost.
All drone pipeline webhooks are commented out (in drone_build.tf, drone_base_build.tf, drone_agent_cicd.tf, drone_rtsp_cicd.tf). Core images are rebuilt manually with aws codebuild start-build, in dependency order (base → basebuild → ros2/rtsp/agent → drone-build). The layered ECR repos are also marked # TODO:FIXME: This is currently being filled manually.

​
user-VPN build (build + remote deploy)

The VPN service doesn’t run on ECS either — it runs as a container on the jumphost. modules/user_vpn/user_vpn_build_deploy.tf builds and pushes the image, templates docker-compose.aws.yml/deploy.sh (DB creds, VPN bucket, external IP), uploads them to s3://skyhub-prod-user-vpn/, then SSHes in and runs the deploy:
modules/user_vpn/user_vpn_build_deploy.tf (post_build, abridged)
- export SSH_KEY=$(aws ssm get-parameter --name "/skyhub-prod/wireguard_ssh_key" --with-decryption --query "Parameter.Value" --output text)
- ssh -p 3377 -i /tmp/ssh_key ubuntu@$WG_PRIVATE_IP "aws s3 cp s3://$VPN_BUCKET/deploy.sh /home/ubuntu/deploy.sh"
- ssh -p 3377 -i /tmp/ssh_key ubuntu@$WG_PRIVATE_IP "bash /home/ubuntu/deploy.sh"
This build is in-VPC (uses the WireGuard security group) so it can reach the jumphost’s private IP over the custom SSH port 3377, pulling the SSH key from SSM. See /ecosystem/user-vpn.

​
ECR registry conventions

All images land in the private registry <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com.
PropertyValue
App reposskyhub-prod-{api,janus,whip,ws-proxy,vpn}-image
Drone reposskyhub-prod-drone-{base,basebuild,ros2,rtsp,mavp2p,ws}-image
Tag:latest (mutable) — every build overwrites it; ECS pulls :latest
Scanningscan_on_push = true
Lifecycleuntagged images expire after 7 days
Repo policybroad (Principal: "*", ecr:*) — see security notes below
Because deploys reuse the mutable :latest tag, there is no image-digest history to roll back to in ECR itself — a rollback means rebuilding a previous ref. force-new-deployment is what makes ECS re-pull the tag (the tag string doesn’t change, so without the force flag Fargate would not know to redeploy).

​
Gotchas a future editor must preserve

The gateway’s .github/workflows/* only run pytest + Ruff. If you expect a merge to auto-deploy via GitHub Actions, it won’t — deploys come from CodeBuild webhooks on tags (prod) or develop (dev).
Hard-coded source_version (develop/main/development/prod) only governs manually-started builds. Don’t assume a manual start-build deploys the same commit a tag push would.
skyhub-prod-api-migrate and every drone pipeline have no webhook; the VPN build deploys by SSH, not ECS. Removing the manual step from a release runbook will ship code against an un-migrated DB or stale drone compose file.
The GitHub PAT and Slack webhook live in settings.tfvars; JWT_SECRET_KEY/MAIL_PASSWORD are literals in modules/api/api.tf. Build IAM roles and ECR repo policies are wildcard (ecs:*, ecr:*, ec2:*, Principal: "*"). Any hardening pass should move these to SSM/Secrets Manager and scope the roles. See /deployment/production-config.

Gateway Build & Runtime

Dockerfile.ecr, gunicorn/gevent worker model, entrypoint SSH tunnel.

ECS Fargate Services

What force-new-deployment rolls: task defs, CPU/mem, Cloud Map DNS.

Environments & Terraform State

Where webhook_config and settings.tfvars come from; init/plan/apply.

Storage, Backups & Alerting

Build/artifact buckets and the redeploy-loop CloudWatch alarms.