# `central-elk-lambda-bridge`

## Central Logging Overview

This is a component in a larger system made up of a few generic terraform
modules and any number of implementation modules which take advantage of the
generic modules.

This module is the bridge which brings items out of the team-owned stack into
the data storage stack which is owned by the Systems team.  This module can be
viewed in the diagram as the `Lambda Function` and `Cloudwatch Trigger` in the
`Systems Team ownership` section of the diagram in the
[`central-elk README.md`][1]

## Lambda function details

The lambda function deployed by this module runs once a minute for up to one minute.
It processes any new items in the S3 bucket exposed to the central stack by the team-specific
logging stack via the SQS queue.

### Lambda function step-by-step

1. reserve item from SQS queue
2. download batch in queue message from S3
3. upload batch to ES via bulk call
4. if item failed to upload because of a 429, exit function.
5. if item failed to upload for any other reason, goto 1
6. if item was successfully uploaded, delete reserved item from SQS
7. if function runtime is less than one minute, goto 1
8. if SQS queue contains more items, use lambda invoke-async to call function again

Of particular interest:

* If the ES cluster is overloaded, the lambda function will back off
  and allow the queue to fill for another minute before trying to process more items
* If there are enough items on the queue to occupy the function for a full minute,
  the function will call itself *in addition to* the cloudwatch event causing another
  function to run resulting in the function scaling to meet peaks.

### Cloudwatch Event

The Cloudwatch event causes the lambda function to run once a minute regardless
whether the function is already invoked allowing for scaling.  Because it runs once
a minute, during times of low load, it's possible that events can be delayed by a minute
or more delays should be limited to less than two minutes under normal conditions.

[1]: https://git-aws.internal.justin.tv/terraform-modules/central-elk/blob/master/README.md
