# `central-elk`

## Central Logging Overview

This is a component in a larger system made up of a few generic terraform modules and any number of implementation
modules which take advantage of the generic modules.

This module is the core of the central stack consisting of Elasticsearch and Kibana seen in the lower left side of the
section marked `Systems Team ownership`.

This diagram's current version is in twitch version control under [`terraform-modules/central-elk`](http://git-aws.internal.justin.tv/terraform-modules/central-elk).

```
+---------------------------------------------------------------------------------------------------------------------+
|                                                                                                                     |
|  Team A's ownership           This infrastructure will be repeated for each team                                    |
|                                                                                                                     |
| +---------------------------------------------------------------------------------------------------------------+   |
| |                                                                                                               |   |
| |  Team A's VPC/Network                                                                                         |   |
| |                                                                                                               |   |
| | +-----------------------------------------------+    +------------------------------------------------------+ |   |
| | |                                               |    |                                                      | |   |
| | | application host                              |    | team's logging endpoint                              | |   |
| | |                                               |    |                                                      | |   |
| | +-------------------------+----------+----------+    +--------------------------+---------------------------+ |   |
| | |                         |          |          |    |                          |                           | |   |
| | | application a           | local    | rsyslog  +--> | rsyslog condenser        | s3 bulk shipper           | |   |
| | | logs directly to syslog | socket   |          |    | (if grok is unnecessary) | omprog module integration +---+ |
| | |                         |          |          |    |                          |                           | | | |
| | +-----------------------------------------------+    +------------------------------------------------------+ | | |
| | |                         |          |          |    |                          |                           | | | |
| | | application b           | /^ar/log | filebeat +--> | logstash filebeat ingest | s3 bulk shipper           | | | |
| | | can't log to syslog     |          |          |    | syslog ingest            | pipe plugin integration   | | | |
| | |                         |          |          |    | (for logstash modules)   |                           | | | |
| | +-------------------------+----------+----------+    |                          |                           | | | |
| |                                                      +--------------------------+------------------------+--+ | | |
| |                                                                                                          |    | | |
| +---------------------------------------------------------------------------------------------------------------+ | |
|                                                                                                            |      | |
|                                                                                                            v      v |
|                                                                                                                     |
|                                                                                  +----------------------------------+
| +-------------------------------------+                                          |                                  |
| |                                     |                                          | team a's logging s3 bucket       | <-+
| | sqs queue                           |                                          |                                  |   |
| |                                     |                                          +---------------------------+------+   |
| | permissions granted to systems team |                                          |                           |      |   |
| | to read/delete from queue           | <----------------------------------------+ sqs trigger               |      |   |
| |                                     |                                          |                           |      |   |
| |                                     |                                          +---------------------------+      |   |
| |       [integration point]           |                                                                             |   |
+-+-------------------------------------+-----------------------------------------------------------------------------+   |
                                                                                                                          |
                                    ^                                                                                     |
                                    |                                                                                     |
+--------------------------------------------------------------------------------------------------------+                |
|                                   |                                                                    |                |
| Systems Team ownership            |                                                                    |                |
|                                   |                                                                    |                |
|                                   |  +---------------------------------------------------------------+ |                |
|                                   |  |                                                               | |                |
|                                   |  | Lambda Function  (one per sqs queue)                          | |                |
| +---------------------+           +--+                                                               +------------------+
| |                     |              | Reads from SQS queue                                          | |
| | Cloudwatch Trigger  |              | Upserts to ES ^ia bulk                                        | |
| |                     +------------^ | Documents named deterministically to enable stateless retry   | |
| | * * * * *           |        +---^ |                                                               | |
| |                     |        |     |                                                               | |
| +---------------------+        |     |                                                               | |
|                                |     | Schedules itself immediately ^ia async call if SQS queue      | |
|                                +-----+ isn't empty.                                                  | |
|                                      |                                                               | |
| +------------------------------+     | On error of any kind, exits immediately.                      | |
| |                              |     |                                                               | |
| | Elasticsearch                |     | If SQS queue has enough entries, this function will end up    | |
| |                              | <---+ running twice due to cloudwatch trigger and self+calling.     | |
| | ASG can increase capacity    |     |                                                               | |
| |                              |     | On 429, all running functions are expected to exit and be     | |
| +------------------------------+     | rerun at a single concurrent call on the next minute.         | |
| |                              |     |                                                               | |
| | Kibana (open to twitch VPCs) |     +---------------------------------------------------------------+ |
| |                              |                                                                       |
+-+------------------------------+-----------------------------------------------------------------------+

```

## USAGE

To deploy a new central stack, you will want to create a new terraform project which uses this repo as a
module.

```
variable "service"     {default = "my-central-elk"}
variable "environment" {default = "dev"}
variable "owner"       {default = "my-team"}
variable "owner_email" {default = "my-team@justin.tv"}

provider "aws" {
    # This will need to be in your .aws/credentials file
    profile = "twitch-central-dev"
    region  = "us-west-2"
}

data "aws_security_group" "sg" {
    # You can also just hard code a sg id by typing 'id = "sg-abcd1234"'
    name = "twitch_subnets"
}

# Discover your subnets as well.  Remember, you need three.
data "aws_subnet" "subnet_a" { tags { "Name" = "Private - A" } }
data "aws_subnet" "subnet_b" { tags { "Name" = "Private - B" } }
data "aws_subnet" "subnet_c" { tags { "Name" = "Private - C" } }

module "central" {
    # If you're using the Rakefile in systems/systems-terraform, it the module will end up in vendor/modules.
    # If you're doing this by hand, mkdir -p vendor/modules; cd vendor/modules; git clone <thisrepourl>
    source = "vendor/modules/central-elk"

    # Everything below here is set via the data and variables above.
    environment = "${var.environment}"
    security_groups = ["${data.aws_security_group.sg.id}"]
    subnet_ids = ["${data.aws_subnet.subnet_a.id}", "${data.aws_subnet.subnet_b.id}", "${data.aws_subnet.subnet_c.id}"]
    service = "${var.service}"
    owner = "${var.owner}"
    owner_email = "${var.owner_email}"
    azs = ["${data.aws_subnet.subnet_a.availability_zone}", "${data.aws_subnet.subnet_b.availability_zone}", "${data.aws_subnet.subnet_c.availability_zone}"]
}
```

Using this, you will deploy one third of the syslog stack; specifically, you will deploy the data storage and visualization section.

View kibana at `my-central-elk-kibana.dev.us-west2.justin.tv`.  Replace my-central-elk with whatever you set `service` to.

## CAVEATS

This module requires that the account and region have exactly three AZs to deploy into.  It will not work
with any other number.
