# Infrastructure
Houses the infrastructure for the eventbus dashboard &amp; controlplane services

## Setup
You'll need a few things before you get started contributing to the eventbus infrastructure.

1. Make sure you have [Docker](https://www.docker.com/) and [Terraform](https://www.terraform.io/) configured on your machine. While we will be using docker for running terraform commands, you should still have the application version of terraform running locally in case you ever need to run local terraform.

2. Make sure you have followed the setup instructions for [Teleport Bastion](https://wiki.twitch.com/display/SEC/Teleport+Bastion). You'll need to run `teleport-bastion enable` if you do not have bastion enabled to interact with some of the terraform modules.

3. Make sure you have followed the setup instructions for [Twinit](https://wiki.twitch.com/display/DTA/Twinit+Services+Proxy+User+Guide). Since we use docker for our terraform commands, we will need a way to ensure that ssh is compatible with bastion within docker containers (or else your commands may hang or just not work). The directions for installing Twinit are not the best, so we will attempt to elucidate them as best as possible here:
    - `brew install twinit` or equivalent
    - `mkdir ~/.twinit` if you do not have that directory. If you do not create this, it may try to bind a socket to a directory that does not exist.
    - `twinit configuretools`
    - Follow the instructions from the configure command above. You will need to set the output url into your Docker Proxies setting in the Docker client. Then, source the `~/.bashrc-twinit`.
    - Run `twinit shell` to copy your shell settings before doing any docker work with terraform! Make sure you have bastion enabled as well (`teleport-bastion status` will tell you if it is enabled or not).
    - You may `twinit disable` or exit the shell when you are done, but be mindful that you will need to re-enable and source your shell again the next time.

**Note**: If you have problems with twinit, you may alternatively `cd` into the directories you want to run terraform in and run the commands without docker

4. Don't forget to grab your [Isengard](https://isengard.amazon.com/console-access) credentials from the console for the correct account.

**Note**: If your connection to twinit goes down, you will be unable to recreate it in the same shell. Furthermore, do NOT source in your default shell as it will try to force every application to use the proxy.

## Structure
The terraform folder is broken down into `/accounts/{staging,production}`, `/environments/{staging,production}`, and `/modules/{app,ecs,pagerduty,twitch_networking}`. The key idea of the flow here is: `environment` -> `app` -> `ecs`.

- `/accounts` relates to account specific infrastructure such as ecr repositories, etc.

- `/environments` houses environment specific configurations. Here, we instantiate the eventbus application module and control which settings (i.e. controlplane instance task memory, dashboard instance task cpu) each environment utilizes.

- `/modules` holds the key components of the infrastructure
    - `/app` manages the creation of the ecs cluster with specific application level configurations (i.e. controlplane traffic port).
    - `/ecs` manages the cluster, which consists of instances made up of `{controlplane, dashboard, dns}` tuple. Metrics for the cluster, SSL certs, and ecs roles are also defined here.
    - `/pagerduty` manages the eventbus {high,low}-{production,staging} sns topics.
    - `/twitch-networking` was added by systems to manage twitch account related things, you should not need to develop in here!

## Usage
Ensure that you are able to sync the remote state with your local state. In order to do this, you will need to init four states: `make terraform-init DIR=terraform/{accounts,environments}/{staging,production}`. Once you successfully init these states, you should be ready to manage infrastructure changes!

If you are experiencing problems with init, make sure you have both bastion and twinit enabled and have run `twinit shell`. Remember that you must also have your Isengard credentials in your shell's environment. You may also `cd` into the respective directories and run terraform from them without docker. You may need to restart or open a new shell.

Running `make terraform-plan DIR=terraform/X/Y` will generate an output plan file in `DIR`. Furthermore, running `make terraform-apply DIR=terraform/X/Y` will utilize the plan file in the directory and attempt to apply it. **Make sure that the plan you have previously run is what you want to apply!** After an apply has complete, the directory will be cleaned as not to leave leftover output plans.

## Miscellaneous
- `make terraform-fmt DIR=terraform` will format each directory with terraform's own format tool.
- `make terraform-clean DIR=terraform/X/Y` will rid the `DIR` of any output files leftover.
- Ensure that you have both [jq](https://stedolan.github.io/jq/) and [aws cli](https://aws.amazon.com/cli/) downloaded! These are required for running our script to obtain the correct task definition and image on a `terraform plan` run.

    `brew install jq` for Mac
    `apt-get install jq` for Linux

    If you do not have aws cli configured locally, follow the [aws cli installation guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)

## Versioning the Cross-Account CloudFormation

Our customers run the cross-account CloudFormation template during the [getting started](https://git-aws.internal.justin.tv/pages/eventbus/docs/getting_started/) process. This CloudFormation template is defined [here](https://git-aws.internal.justin.tv/eventbus/infrastructure/blob/master/terraform/modules/cross_account_setup/cloudformation/cloudformation.yaml.tpl). We also provide a Terraform module for customers that simply wraps this CloudFormation template.

To support semantic versions for this CloudFormation template, we leverage the automatic versioning of S3. To add a new version update the [versions.tf](https://git-aws.internal.justin.tv/eventbus/infrastructure/blob/master/terraform/modules/cross_account_setup/versions.tf) file.

Here's how versioning works:

    for version in versions.tf:
        download the cloudformation file from s3 with tag ${version.s3_tag}
        reupload this file to s3 with name ${version.name}

### Version Update Process

1. Make desired changes in `cloudformation_release_candidate.yaml.tpl`.
2. Run terraform in staging to deploy the release candidate YAML.
3. Verify that all changes work as expected. Point a test CloudFormation stack at `https://eventbus-staging-setup.s3-us-west-2.amazonaws.com/cloudformation_release_candidate.yaml` to apply release candidate changes to a test environment (e.g. clock-dev, etc)
4. Do steps 2 and 3 in production, verify changes were successful.
5. Make a copy of `cloudformation_release_candidate.yaml.tpl` into `cloudformation_<new_version>.yaml.tpl`.
6. Add `<new_version>` to the list of versions in `versions.tf`.
7. Update staging and production `environment` terraform variable `module.cross_account_setup.latest_version` to `<new_version>`.
8. Run terraform in staging and production to update the content of S3's `cloudformation.yaml` (latest) to the content of `<new_version>`.