## Postgres

For non-RDS Postgres databases (i.e. SiteDB/justintv_prod), we use WAL logs via [WAL-E](https://github.com/wal-e/wal-e)
to spin up a replica on an ec2 instance.

SiteDB is using WAL-E to continuously write base backups and incremental WAL logs to s3. To spin up the snapshot, a worker:
1. Attaches an EBS volume ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L392))
1. Locates WAL-E records on s3 ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L418))
1. Copies the records from s3 to the EBS volume ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L420))
1. Detaches the EBS volume ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L426))
1. Spins up an instance from a [specified AMI](https://git-aws.internal.justin.tv/systems/puppet/blob/efe8a85bc2a0adda3ae00b4364fcfb177ae031d1/modules/zenyatta/manifests/params.pp#L10)
   ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L473))
1. Attaches the EBS volume to the new instance ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L483))
1. Runs postgres in a [Docker container](https://git-aws.internal.justin.tv/systems/puppet/blob/62a6d664ed2ff1fe7f9f0f6452811d609b9dbfe4/hiera/cluster/zenyatta-worker.yaml#L87)
   on that instance ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/docker.py#L200))
1. Waits for Postgres to pause its recovery ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L675))
1. Selects from various tables into CSVs and uploads them to s3 ([code](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/zenyatta/tasks.py#L296))

To change the Docker image, you can update the [Dockerfile](https://git-aws.internal.justin.tv/d8a/zenyatta/blob/5eed4427a6b12aa035626bc0a573fb759feab7ed/docker/wal-pitr/Dockerfile)
and then push the container to the tag in puppet (currently `641044725657.dkr.ecr.us-west-2.amazonaws.com/d8a/postgres-pitr:latest`).
If you don't have permission to do this, you can do `sudo $(aws ecr get-login --region us-west-2)` on a zenyatta worker
to log into ECR and then push the image from there.
