# Automated Infrastructure Deployment (R.A.I.D)
Automating infrastructure development with Cloudformation and a little terraform.

```
Setting up a pipeline that will read cloudformation templates and configs from codecommit, setup change sets, audit and apply them
The pipeline assumes this structure exists within your source code respository:

     --> README.md
    |
     --> cloudformation
               |
         mycloudformation.yml ({TemplateFileName})
               |
                --> configs
                      |
                    us-east-1-staging.json ({AWS::region}-{StagingStackConfig})
    |
     --> buildspec.yml (Required to describe build steps for CodeBuild)
    |
     --> .ebextensions (Pipeling assumes use of Elastic Beanstalk)
               |
                --> 00-config ...
    |
     --> (REST OF SOURCE)
```

# Getting Started
Terminology:
* primary region - The region that you will be responsible for building the artifact and receiving the first deployment
of your code before fanning out into the secondary regions.
* seconday region - Regions that receive the artifact from S3, but otherwise mirror the environments setup in the primary region.

# Deploying Changes
1. Merge to master
2. Start with https://us-west-2.console.aws.amazon.com/codepipeline/home?region=us-west-2#/view/base-pipeline in twitch-web-aws
3. Approve changes to the pipeline itself and the lanmbdas
4. Switch over to twitch-web-dev and approve any changes in both us-west-2 and us-east-1
5. Switch back to twitch-web-aws us-west-2, approve changes to the other stacks
6. Switch over to us-east-1 and approve changes there

## Networking
**DISCLAIMER** \
You probably don't need to set up the actual networking pieces, those should already be set up from the account creation process.

In addition to the `networking.yaml` template, which would create a new VPC, there is a `constants_stack.yaml` that can be used to make some networking names/IDs
available to `!ImportValue` in other cloudformation stacks. The easiest way to do this is by writing down those constants in a
`parameters.json` as follows:
* Add a `parameters.json` to the appropriate folder, `networking/{account}/{region}`
* `cd networking`
* `./run.sh {account}/{region}`

The terraform pieces are used for importing values into Cloudformation so that they can be referenced with !ImportValue.

## Github Enterprise to CodeCommit
Use the [vod/github-import](git-aws.internal.justin.tv/vod/github-import) project to set up forwarding of commits from
GHE into CodeCommit. CodePipeline doesn't have VPC integration, so it can't query our GHE to see when new commits are added.

## Custom Docker Images for CodeBuild
Run `docker/go/docker2ecr.sh` to upload a docker image for a new version of go to ECR. Will use a `golang` image off of
[dockerhub](https://hub.docker.com/_/golang/). Make sure you have aws credentials available in your environment
(e.g. AWS_PROFILE or the temporary credentials you copied from Isengard).

## Common Pipeline Essentials
The referenced source files here are `base-pipeline.yaml` and `base-pipeline-lambdas.yaml`, which will create shared
resources that you can use to power your own service's deployment pipeline.

* [AWS Resource] - [CloudFormation Reference Alias] - [Description]
* S3 Bucket - ArtifactsBucket - Use this as the `ArtifactStore` for your CodePipelines.
* IAM Role - BuildRole - This role has common permissions for CodeBuild, including access to set up VPC access in your build.
Use this as the `ServiceRole` for your `AWS::CodeBuild::Project` resources.
* IAM Role - CFRole - This role has access to change the kinds of resources we are managing with CloudFormation. Use this
role to execute CloudFormation changes.
* ECR Repository - ContainerRegistry - We store a mirror of `golang` images here. Specific versions are uploaded manually.
* Lambda Function - CrossRegionLambda - Copies the build artifact from one S3 bucket to another. Your primary pipeline
will trigger based on CodeCommit changes, and then copy that artifact to other buckets with this lambda to trigger any
secondary or dev pipelines you have.
* Lambda Function - GlobalViewLambda - Monitors the execution status of another CodePipeline and attempts to assume the
same status. Use this to fail your primary pipeline if a secondary or dev pipeline fails after using the CrossRegionLambda to trigger it. 
* IAM Role - LambdaRole - We use this role when we set up the `CrossRegionLambda` and `GlobalViewLambda`. If you have a
lambda that uses similar permissions, feel free to use this role there as well.
* IAM Role - PipelineRole - Configure your `AWS::CodePipeline::Pipeline` to use this role for pipeline execution.

## Deployment Pipeline Structure
A cross region, cross account deployment pipeline involves a lot of moving pieces because CloudFormation doesn't really
support this kind of heterogeneous setup.

### Primary Pipeline
The primary pipeline is the one that is triggered first by the presence of new commits in CodeCommit. The primary pipeline
for this repo is located in `pipeline/primary.yaml`. The job of the primary pipeline is to orchestrate the whole process
of deploying code to every environment/region.

The flow of deploying code is usually pretty similar:
1. Merge to master
2. Build + Test
3. Deploy to staging (BETA) (-> as many regions as applicable)
4. Deploy to canary (GAMMA) (-> as many regions as applicable)
5. Deploy to prod (PROD) (-> as many regions as applicable)

Some steps may be omitted depending on which environments are available. Not every service has a BETA or GAMMA environment.
Sometimes manual and/or automated tests are run against a stage before the code is promoted to the next stage.

Let's try to visualize the pipeline we created for Visage in terms of these steps. In the actual pipeline there are some
manual approval steps we've added to cover some automation gaps, but we've omitted those here for simplicity.

Note each column ( || to || ) represents a CodePipeline Stage and the rows represent sequential actions within that stage.
Parallel actions within a stage are separated by a single | and don't have an arrow like `--->`.
```
  || 1) Source (CodeCommit) || ---> || 2) Build (CodeBuild) || ---> || 3a) CrossRegionLambda to BETA primary region | 3b) CrossRegionLambda to BETA secondary region || ---> || 4) deploy GAMMA primary region || ---> || 5a) deploy PROD primary region || ---> || 5b) CrossRegionLambda to PROD secondary region ||
  ||                        ||      ||                      ||      ||                          |                   |                                                ||      ||                                ||      ||                                ||      ||                          |                     ||
  ||                        ||      ||                      ||      ||                          v                   |                                                ||      ||                                ||      ||                                ||      ||                          v                     ||
  ||                        ||      ||                      ||      || 3c) GlobalViewLambda BETA primary region     | 3d) GlobalViewLambda BETA secondary region     ||      ||                                ||      ||                                ||      || 5c) GlobalViewLambda PROD secondary region     ||
```

- We can see that BETA is deployed to every region at once. This is to parallelize the work and reduce waiting.
- GAMMA only exists in the PROD account primary region.
- PROD is deployed to the primary region before the secondary region. For each step, the scope grows a bit larger:
  - GAMMA < PROD primary region < PROD everywhere else.

### Secondary Pipeline
The secondary pipeline is much simpler, it takes in a ready-to-deploy build artifact and deploys it to one account/region.
These pipelines are not attached to CodeCommit, but instead watch for the artifacts copied over by the CrossRegionLambda.

```
  || Source (S3 Bucket in Region) || ---> || deploy here ||
```

### Dev Pipeline
This repo uses **dev pipeline** as a mix between the primary and secondary pipelines. Its name comes from the "primary pipeline for the dev account".
The dev pipeline adds the functionality of triggering a secondary pipeline in another region (but the same account).

The steps are a bit different here, partially as an experiment:

1. Merge to master
1. Update the pipeline itself based on master (PROD primary region)
1. Update the lambdas that power the pipeline (PROD primary region)
1. Deploy to staging (BETA primary region) (dev pipeline)
    2. Update the pipeline itself (BETA primary region)
    2. Update the lambdas that power the pipeline (BETA primary region)
    2. Update the networking and base-pipeline stacks (BETA primary region)
    2. Deploy to BETA secondary region (secondary pipeline)
1. Update the networking and base-pipeline stacks (PROD primary region)
1. Deploy to PROD secondary region (secondary pipeline)

Primary:
```
  || 1. Source (CodeCommit) || ---> || 2. Update Pipeline PROD primary region || ---> || 3. Update Lambdas PROD primary region || ---> || 4a) CrossRegionLambda to BETA primary region || ---> || 5a) update networking stack PROD primary region | 5b) update base-pipeline stack PROD primary region || ---> || 6a) CrossRegionLambda to PROD secondary region ||
  ||                        ||      ||                                        ||      ||                                       ||      ||                           |                  ||      ||                                                                                                      ||      ||                          |                     ||
  ||                        ||      ||                                        ||      ||                                       ||      ||                           v                  ||      ||                                                                                                      ||      ||                          v                     ||
  ||                        ||      ||                                        ||      ||                                       ||      || 4b) GlobalViewLambda BETA primary region     ||      ||                                                                                                      ||      || 5c) GlobalViewLambda PROD secondary region     ||
```

Dev:
```
  || 1. Source (S3) || ---> || 2. Update Pipeline BETA primary region || ---> || 3. Update Lambdas BETA primary region || ---> || 5a) update networking stack BETA primary region | 5b) update base-pipeline stack BETA primary region || ---> || 6a) CrossRegionLambda to BETA secondary region ||
  ||                ||      ||                                        ||      ||                                       ||      ||                                                                                                      ||      ||                          |                     ||
  ||                ||      ||                                        ||      ||                                       ||      ||                                                                                                      ||      ||                          v                     ||
  ||                ||      ||                                        ||      ||                                       ||      ||                                                                                                      ||      || 5c) GlobalViewLambda BETA secondary region     ||
```

- The pipelines update themselves! This is something the Visage pipeline isn't configured to do, but is also kind of mind-bending.
  - I would strongly recommend the `RestartExecutionOnUpdate: true` setting on any pipelines that do this!
- Some parts of PROD are deployed before BETA (the pipeline and the lambdas).
  - this is to update the pieces that do the deployment to BETA to make sure we can always fix bugs
- BETA deploys one region at a time, using a new and specialized type of pipeline.
  - less complexity in the primary pipeline
  - requires a 3rd type of pipeline
  - takes longer
