This page covers where SkyHub’s data and build artifacts physically live, what is actually backed up (and what is not), and how operational alarms reach a human. Everything here is defined in the skyhub_terraform repo and applies to the skyhub-prod environment in eu-central-1 (account <aws-account-id>). Three facts frame the whole page:
  • Postgres is the only durable datastore with backups. It runs as an ECS Fargate task on an EFS volume (not RDS) and is protected by an AWS Backup daily plan. See ECS Fargate Services for the task definition.
  • Redis has no persistence and no backup — it is a pure cache / pub-sub relay. Treat it as disposable.
  • S3 holds everything else: the built frontend, user/drone assets, per-user VPN configs, Terraform state, and CI build artifacts. There is no cross-region replication.

​
Storage map

​
S3 buckets

Every bucket below is created in eu-central-1 with resources_tag = skyhub-prod. Bucket names are infrastructure facts (not secrets).
BucketDefined inAccessVersioning / lifecyclePurpose
skyhub-prod-ui-bucketmodules/frontend/frontend.tfPublic read (bucket policy s3:GetObject / Principal: *, all public-access-blocks off)Versioning SuspendedStatic Angular SPA; website config with index.html + 404→index.html. Fronted by CloudFront and by the jumphost nginx.
skyhub-prod-ui-build-bucketmodules/frontend/frontend_build_deploy.tfPrivateLifecycle stash-old-builds: transition to GLACIER after 7 daysCodeBuild frontend build artifacts (frontend-build.zip).
skyhub-prod-frontend-build-cachemodules/frontend/frontend_build_deploy.tfPrivateLifecycle cleanup-old-cache: expire after 7 daysCodeBuild S3 cache (/root/.cache/pip, .angular/).
skyhub-prod-assetsmodules/assets/assets.tfPrivate (ACL private, ownership BucketOwnerPreferred)None; CORS allows GET/PUT/POST/HEAD from any originDrone/user/mission assets. Read by the gateway’s AssetService via the ASSET_BUCKET env var; the UI reads from skyhub-prod-assets.s3.eu-central-1.amazonaws.com/drone.
skyhub-prod-user-vpnmodules/user_vpn/user_vpn.tfPrivateVersioning Enabled; CORS for uploadsPer-user WireGuard configs. Referenced by the gateway as VPN_BUCKET. See User VPN & Network Isolation.
skyhub-terraform-environment-statesbackend block (environments/prod/provider_config.tf)Private—Terraform remote state; prod key skyhub-prod/terraform.tfstate. Detailed in Environments, State & Terraform Workflow.
skyhubcoreexternal (written to by modules/drone/drone_build.tf:190)Private—Target for the generated drone docker-compose.installer.yml (prod: docker-compose.prod.yml). Not created by this repo. See CI/CD: CodeBuild, ECR & Frontend Deploy.
Live prod also contains buckets that are not managed by this Terraform, seen only in an AWS inventory: skyhub-dashboard-assets, skyhub-3d-assets (3D models), and skyhub.ai. Treat them as out-of-band — a terraform apply will neither create nor track them.
skyhub-prod-ui-bucket is intentionally world-readable so the static site can be served directly. Never place anything other than compiled frontend assets in it. Private data belongs in skyhub-prod-assets (served via presigned URLs by AssetService) — see S3 Assets, HLS Video & Execution Archives.
The gateway consumes ASSET_BUCKET and VPN_BUCKET from its task environment; see Gateway Environment Variables for the full list.

​
EFS-backed Postgres & AWS Backup

Postgres persistence is an EFS file system (modules/database/postgre.tf), not an EBS volume or RDS:
  • File system creation_token = skyhub-prod-postgres-efs, tagged Name = skyhub-prod-PostgresEFS.
  • A mount target in the first private subnet; an access point rooted at /var/lib/postgresql/data (uid/gid 0).
  • The ECS task mounts it as the postgres-data volume with transit encryption enabled.
  • desired_count = 1, no read replicas — durability rests entirely on EFS + AWS Backup.
Backups are defined in modules/database/backup.tf. A single backup vault (skyhub-prod-postgres-backup-vault, KMS key aws/backup) holds all recovery points, and one backup plan (skyhub-prod-postgres-backup-plan) carries two rules:
RuleScheduleRetention
daily-efs-backupscron(0 0 * * ? *) — daily 00:00 UTC14 days
daily-ec2-backupscron(0 1 * * ? *) — daily 01:00 UTC7 days
Two selections attach resources to that plan by tag:
  • postgres_efs_backup_selection → resources tagged Name = skyhub-prod-PostgresEFS (the EFS file system).
  • jumphost_ec2_backup_selection → resources tagged Backup = daily (the WireGuard jumphost EC2 — see VPC, WireGuard Jumphost & nginx Routing).
The rule names imply “EFS gets 14 days, EC2 gets 7 days,” but that is not how AWS Backup works. A plan applies every rule to every selected resource. Because both selections point at the same plan, the EFS and the jumphost each get two recovery points per day — one retained 14 days (00:00) and one retained 7 days (01:00). Keep this in mind before “fixing” retention: editing a rule changes it for both resources.
The AWS Backup selections use db_task_role as their iam_role_arn — the same role the Postgres ECS task runs under. Its trust policy includes backup.amazonaws.com, and its inline policy is Action: "*" / Resource: "*". That works, but it is far broader than AWS Backup needs; a least-privilege pass should give backup its own role.
What is not backed up: Redis (ephemeral by design), the S3 buckets (no replication or backup plans — the user-vpn and ui buckets rely only on S3 versioning where enabled), and CloudWatch logs (the API and CodeBuild log groups set retention_in_days = 7).

​
Durability & recovery matrix

StorePersistenceBackupRecovery window
Postgres (database.skyhub-prod.internal)EFS, transit-encryptedAWS Backup (EFS)14 days (+7-day 2nd point)
Jumphost EC2EBS root (25 GB gp3)AWS Backup (EC2, tag Backup=daily)7 days (+14-day 2nd point)
RedisNoneNoneNot recoverable
skyhub-prod-assetsS3NoneLive objects only
skyhub-prod-user-vpnS3, versionedNonePrior object versions
skyhub-prod-ui-bucketS3, versioning suspendedNoneLive objects only (rebuildable from CI)

​
S3 gateway VPC endpoint

modules/vpc_endpoints/main.tf provisions a single S3 Gateway VPC endpoint (com.amazonaws.eu-central-1.s3, type Gateway) so S3 traffic from the private subnet does not egress through the single NAT gateway — a deliberate cost optimization for a one-NAT topology. 
resource "aws_vpc_endpoint" "s3" {
    vpc_id            = var.vpc_id
    service_name      = "com.amazonaws.${var.aws_region}.s3"
    vpc_endpoint_type = "Gateway"
    tags              = var.common_tags
}
The ECR interface endpoints (ecr.dkr, ecr.api) are commented out — the code note explains they are only worth their per-hour cost under frequent image pulls; today ECR pulls still traverse the NAT gateway.
The module receives subnet_ids but the endpoint resource never uses it, and no route_table_ids are wired. A gateway endpoint only routes traffic once it is associated with the route tables serving those subnets — confirm that association exists before assuming S3 traffic is actually bypassing NAT. This is a good candidate for a future cleanup.

​
Alerting: SNS + Slack + email

Operational alarms funnel through one SNS topic and fan out to email inboxes and a Slack channel. The pipeline lives in modules/alarms/main.tf; the topic ARN is exported as alarm_sns_topic and consumed by the API module. Fan-out targets (all subscribe to skyhub-prod-ecs-alerts-topic):
SubscriptionProtocolEndpoint
todor_alerts_mailemail<engineer-email>
yanko_alerts_mailemail[email protected]
karthi_alerts_mailemail<engineer-email>
slack_lambda_subscriptionlambdaskyhub-prod-slack_alert_lambda
The Lambda (slack_notifier.js, packaged as modules/alarms/slack_notifier.zip) reads SLACK_WEBHOOK_URL from its env (set to the checked-in slack_webhook var), then for each SNS record POSTs a :rotating_light: *ALARM* block carrying the alarm subject and message to Slack, tagging the on-call trio. Its role is the AWS-managed AWSLambdaBasicExecutionRole plus an SNS invoke permission. SMS subscriptions are present but commented out (sandbox/verification friction).
slack_webhook is committed in plaintext in environments/prod/settings.tfvars and flows into the Lambda’s environment. Rotating it means editing the tfvars and re-applying. A refactor should move it to SSM/Secrets Manager.

​
Current alarm consumers

Today only the API (gateway) module publishes alarms, and all three target the SNS topic (alarm_actions = [var.alarm_topic_arn]). They exist to catch redeploy/restart loops — a real risk because ECS runs at 100% Fargate Spot (see ECS Fargate Services).
AlarmMetricFires whenDefined in
skyhub-prod-API-Pending-Tasks-HighPendingTaskCount (ECS/ContainerInsights)> 0.3 avg over 3×5 min (~15 min)modules/api/api.tf:367
skyhub-prod-API-Multiple-DeploymentsDeploymentCount (ECS/ContainerInsights)> 1.4 avg over 4×2 min (~8 min)modules/api/api.tf:386
skyhub-prod-API-Shutting-Down-Log-SpikeAPI-ShuttingDownCount (log metric filter on Shutting down)Sum > 4 over 3×4 min (~12 min)modules/api/api.tf:421
No other ECS service (janus, whip, ws_proxy, redis, postgres) has alarms wired. A Spot interruption or crash loop on any of those is currently silent. Extending coverage is the most valuable near-term reliability work here.

​
Wiring up a new alarm

1

Define the metric or log filter

Add an aws_cloudwatch_metric_alarm in the relevant service module. For log-derived signals, first add an aws_cloudwatch_log_metric_filter against that service’s log group (the Shutting down filter in modules/api/api.tf:406 is the pattern to copy), then alarm on the emitted metric.
2

Route it to the SNS topic

Set alarm_actions = [var.alarm_topic_arn] on the alarm so it publishes to skyhub-prod-ecs-alerts-topic and reaches both email and Slack.
3

Pass the topic ARN into the module

Thread module.our_alarms.alarm_sns_topic into the module as an input (the API module does this via alarm_topic_arn = module.our_alarms.alarm_sns_topic in environments/prod/main.tf). Declare a matching variable in the module’s inputs.tf.
4

Add a recipient (optional)

To page a new person, add an aws_sns_topic_subscription with protocol = "email" in modules/alarms/main.tf. Email subscriptions require the recipient to click the AWS confirmation link before they receive anything.
5

Apply

Run terraform plan --var-file=settings.tfvars then apply from environments/prod. See Environments, State & Terraform Workflow.

ECS Fargate Services

The Postgres/Redis Fargate tasks, EFS mount, and Spot capacity model behind these backups and alarms.

Environments, State & Terraform

The skyhub-terraform-environment-states bucket, state keys, and the apply workflow.

CI/CD: CodeBuild, ECR & Frontend Deploy

How the build/cache buckets and the skyhubcore installer bucket get populated.

S3 Assets, HLS Video & Archives

How the gateway’s AssetService reads and writes skyhub-prod-assets.