# `TwitchTelemetry`

This package represents the core of the TwitchTelemetry packages. It defines what samples actually are, as well as 
standardizing dimension and metric names. It additionally provides an interface for observing metrics and aggregation
functionality (including the main definition of SEH1 histograms).

We strongly recommend reading the [Fulton Metrics Doc](https://docs.fulton.twitch.a2z.com/docs/metrics.html) for more
information on this package and for diagrams on how it relates to the other TwitchTelemetry packages.

## Referring to this package
To refer to this dependency, refer to its alias of `code.justin.tv/amzn/TwitchTelemetry`

## Help
For help using this package, please ping the [#fulton](https://twitch.slack.com/messages/C9BUPDUC8) Slack channel

## Types in this package

* _Sample_: A single value with some metadata.

* _MetricID_: Uniquely identifies a metric.  Name plus dimensions. 

* _SampleObserver_: An interface for accepting individual samples.

* _Distribution_: Multiple samples for the same metric and timestamp.
  It doesn't store every value.  To save space, it stores the samples similar to CloudWatch.

* _SampleUnbufferedObserver_: Probably should have been named DistributionBatchObserver.
  An interface for accepting a batch of distributions.

* _Aggregator_: Accepts samples.  Distributions can be pulled out of it.

* _BufferedAggregator_: Accepts samples.  Periodically pushes distributions to an observer.

* _SampleBuilder_: A convenience type for business logic that wants to convert a value to a
  sample, and cares where the sample goes, but doesn't care about other details.
  Each call to SampleBuilder includes a value and a minimal amount of additional information
  that is likely to be unique to that value.  The return value is a complete sample.
  SampleBuilder uses its fields to fill in the details.  It's a good way of passing the
  pile of details around to where they're needed.  If a tweak is needed, clone the
  SampleBuilder and mutate the config before calling or handing off to another chunk of code.

* _SampleReporter_: A convenience type for business logic that wants to report a value but
  doesn't care about any details including where it goes.  It's a SampleBuilder plus
  a SampleObserver.  Most business logic will likely be using this instead of a SampleBuilder.

* _SampleReportingWriter_: Wraps an io.Writer and reports metrics for everything it handles.
  It's intended for monitoring stdout for alarming if it becomes a bottleneck.

* _SEH1_: See section below.

## An explanation of why this package is the way it is 
The biggest lesson learned (and re-learned) throughout development of this package is that
metrics sound easy, but they're actually very hard to implement.  We learned a lot building
this library, and as we learned we ended up needing to scrap what we had and re-write it
from scratch.  This is the third or fourth from-scratch rewrite of this library.
The following sections will attempt to answer the "why" of this library.

### Design goals during intitial development
The main areas of focus for TwitchTelemetry are correctness, AWS costs, and trying to
avoid one-way doors.  Efficiency and performance intentionally was thrown to the back seat as
something that just needed to be "good enough" at the start of the project.
As with many areas of Fulton, we targeted having the first release of Fulton address the needs
of 80% of customers, and tradeoff the long tail for faster time to having something running.
As such, there's a lot of room for improvement, but because of our efforts, that improvement
can mostly be done over time as technology advances and prioritization allows.

TwitchTelemetry was a large undertaking and was very much an investment point for Fulton.
Work on TwitchTelemetry was started a week or two before the team officially formed and moved
desks together.  It was started from scratch to be able to include domain knowledge from
multiple sources as well as incorporate upcoming CloudWatch functionality that hadn't even
been built yet.  This is very different from the rest of Fulton which aimed to leverage
existing technology and fill in the gaps where needed.

##### Correctness
This is the obvious top priority because correct telemetry is critical to the operation of
a service.  We tried to keep the codebase small-ish and straight forward to decrease
the chance of defects, and deviated into complexity when there was a need for it.

##### CloudWatch and AWS Cost
TwitchTelemetry started out focused on using the Amazon internal metrics stack instead of
CloudWatch, because CloudWatch was lacking functionality.  The Fulton team coordinated with
the CloudWatch team.  In order to align with the future of CloudWatch and
Amazon internal metrics, we planned on switching to CloudWatch once specific critical features
became available.  Later on we did switch the default to CloudWatch.

Under the hood, some time ago, CloudWatch became the backend for the Amazon internal metrics
stack.  The internal stack is a wrapper, and the backend that used to be known as "PMET"
no longer exists even though the name lives on.  Their overall future plan, at least when
TwitchTelemetry was being built, is to maintain the internal Amazon wrapper because it's too
entrenched to get rid of it, and instead strongly encourage people to move to the external
CloudWatch offering by investing heavily in making that more attractive than the internal one.
It's a very long term plan.

Since CloudWatch bills on API calls, client-side aggregation is very important.
TwitchTelemetry was the first Twitch Golang CloudWatch client to offer client-side aggregation
because we built it to be able to do the aggregation months before the public CloudWatch API
actually supported accepting pre-aggregated data.  Once that feature was released, we only had
to make some minor adjustments to fit the real API, and several production services adopted it
quickly to cut their AWS costs down.

##### API Design and One-Way Doors
It felt like everything we did involved more tradeoffs than we could keep track of.
It was very obvious that some parts of the library were going to end up wrong no matter what
we did.  We were starting a new core library from scratch with developers that had extensive
metrics experience, little to no GoLang experience, and of course deadlines to meet.
So, we needed to build something that was functionally correct, was superior to other clients
by leveraging cutting edge CloudWatch features, and could later have its guts ripped out and
replaced by a more efficient implementation once GoLang experts got ahold of it.

The first thing we designed was the API used by the service.  After multiple iterations of
learning and restarting from scratch, we decided that the best API that we could hope to have
was currently being built over in Video:
[Twitch-Video-MetricsMiddleware](https://code.amazon.com/packages/Twitch-Video-MetricsMiddleware-V2/trees/mainline).

The main things to know about Twitch-Video-MetricsMiddleware:
* It provides a way to observe what the service is doing.
* It's not intrusive to the service's business logic.  It hooks in as middleware to watch
  what the business logic does.  It doesn't require the business logic to tell it, so there's
  less boiler plate and room for mistakes.
* It supports much more functionality than just metrics, a lot of which is good stuff that
  we want long term.
* It fit how GoLang and Twirp services work far better than anything we could hope to design
  given our lack of GoLang experience at the time.

Twitch-Video-MetricsMiddleware is highly recommended regardless of if you're using Fulton or not.
It allows service owners to easily add/remove monitoring solutions with minimal impact to the
business code.  Having this meant that we could later on make improvements to the majority of
how TwitchTelemetry handles metrics and only custom metrics would potentially be impacted by
API changes or need some sort of adapter.

One of our goals was to standardize metrics across the company.  That fits nicely with being able
to observe what the service is doing.  We can easily convert observations to standardized metrics.
The remaining gap is custom metrics that can't simply be observed from middleware.  Business logic
usually wants to just throw metrics numbers over the wall and not worry about the details.
SampleBuilder/SampleReporter provide a hands-off way to do that.

### Rollups
I found rollups to be one of the harder things to build in TwitchTelemetry.  There's a lot to
keep track of and a lot of tradeoffs.  There was also guessing at what customers might do with
their metrics and how rollups would interact with that. 

Rollups are expensive.  They're calculated locally and shipped to CloudWatch.  Each rollup adds
a pile of new metrics that is usually proportional to the number of metrics that are being
reported.  Lots of things scale with the number of unique metrics:
* CW charges directly on this.
* CPU/Network usage due to larger payloads to CW.  If the payload gets too large it starts
   using multiple CW calls.  CW charges for API calls.
* TwitchTelemetry's memory footprint.  It has to store the aggregations.

Every default rollup that we have compounds the cost for all customers.  Also, once it's added
we can't easily get rid of it because someone probably has started relying on it.  Now that both
dashboards and alarms support metric math, there's no longer a strict need to do rollups across
dimensions with small cardinality and a fixed set of values that rarely changes.  Rolling up in
dashboards/alarms is cheaper, but has the risk of missing new values.  OpsGen could address some
of the mess associated with this.

In theory, adding a new rollup could crash a customer service if it's already running hot and the
rollup somehow interacts badly with their set of metrics.  Also, the rollup code is almost in the
critical path.  It has to run for every single reported metric which is many times per request.
It's isolated from the critical path by a buffered channel, so it won't delay requests, but CPU
and memory allocations are shared resources and do directly scale with requests.

The original set of rollups were decided before CW alarms supported metric math, which meant we
needed rollups to cover most of the alarm configs that we expected.  I remember we looked at what
MWS does for most internal services, but I don't remember how much influence that had.  I remember
running numbers on what different rollups would do in terms of number of unique metrics and it's
very easy for the numbers to grow to absurd scale.  It'd be nice to just have all the metrics in
case you want them later, but the reality of the situation is that is really expensive.

The original implementation of TwitchTelemetry has hardcoded rollups in SampleBuilder.  It would
be good to make that easier to globally configure if needed.  High cardinality dimensions still
have to have rollups because dashboards/alarms can't do it.  If a team needs new rollups, the
best option is to either manually build samples with custom rollups or modify the rollups after
calling SampleBuilder.

### SEH1

SEH1 is a specific implementation of SEH, and is a technology owned by Amazon.  The CloudWatch
backend uses it.  MWS also does because it's backed by CloudWatch.  TwitchTelemetry uses it to
aggregate the data locally.  There is some data loss there, but it usually won't be noticed
since it only affects percentile estimates and the loss rate is low.  The aggregations are
sent to CloudWatch as value/count pairs because the CloudWatch API doesn't expose how data is
stored internally.  CloudWatch then takes the incoming data and fits it into it's own SEH1
data storage.  Because the two algorithms match bucket-for-bucket, there's no data loss on
the CloudWatch side and this is the best we can do for accuracy.

### Buffering

The original implementation did aggregation in a blocking way.  The senders synchronously
aggregated the data into a buffer that was implemented as a nested structure and then returned.
Network calls (flushing) was done in the background.  It wasn't possible to drop metrics.
If there was too much locking on the structure then business logic would be impaired.

It was modified to accept metrics into a buffered channel to ensure that business logic is
not blocked.  This change led to some confusing naming in the senders.  The Buffered sender
is the original one that does aggregation inside the sender.  The unbuffered sender is the
new one that receives pre-aggregated data and doesn't need to aggregate/buffer anything
because that was already handled for it.

The statsd sender flushes very frequently compared to the others because metric timestamps
are recorded by the server.  The client doesn't send timestamps.

Lambda was implemented to synchronously flush at the end of every request, so it's relatively
inefficient.  Definitely should migrate to a better solution once it's available. 

### Aggregation

The aggregator buckets incoming samples the same way that CloudWatch/MWS does.  Separate metric
names/dimensions get different buckets, and separate data points within those are kept separate.
Because of this bucketing by time, the aggregator needs to know the aggregation period to use.

The original implementation flushes what it has every half minute.  It can be made more efficient
by flushing buckets for minutes that are complete and won't accumulate any more data, and keeping
buckets for the current minute.

Another improvement that would likely make a measurable difference in customer service performance
is to stop using strings for metric names and dimensions in the Sample and Dimension structs.
If I remember right, these strings are actually one of the largest things that shows up on
benchmarks of a newly generated service that just returns data from in-memory cache.
Most services probably only have a handful of values and combinations of values.  Map those
to small numbers.  A low cardinality of small numbers can either be mapped to sequence IDs
and/or bit packed into a 64 bit field to make the data structures much more efficient.
Then unpack before handing off to senders.  This implementation probably can be concealed
within TwitchTelemetry.

### Backdating samples to the request start time

The TwitchTelemetryMetricsMiddleware OperationMonitor reports all standardized metrics with
the timestamp for when the server started processing the request.  This choice actually makes
quite a difference.  Reporting the request start time is in line with how Amazon internal
tools handle it.

An example where it makes sense to lump all request metrics into the same data point:
If a request fails because of a call to a dependency, it's good to have those two metrics
lined up together when they are plotted on the same graph to show the correlation,
even if the dependency failed a fraction of a second before the minute changed and the
request failure happened after the minute change.

Given that we're grouping related metrics to a single timestamp for our sanity,
it should be consistent across systems.  Since Amazon uses request start time,
it makes sense to go with that.

A possibly not very good example of why request start time makes sense:
Requests are normally handled quickly by a system.  Something breaks momentarily and makes
some requests get stuck for a long time, specifically one or more data points worth of time.
When they finish, backdating those metrics to the request start time will make them align to
when something broke.  In this example it's not really possible for the original breakage to
occur at the end of the request because requests finish quickly in normal conditions.

So when reading graphs, just keep in mind that each data point is representing what happened
to requests that started at that point in time.

Custom metrics will also do this timestamp backdating if the business logic uses a reporter
that was obtained from SampleReporterWithContext.

### Reference
* Fulton Docs
    * [Metrics](https://docs.fulton.twitch.a2z.com/docs/metrics.html).  There's also a [middleware](https://docs.fulton.twitch.a2z.com/docs/metrics.html#how-metricsmiddleware-works) section with a picture.
    * [Alarming](https://docs.fulton.twitch.a2z.com/docs/alarms.html)
    * [Dashboards](https://docs.fulton.twitch.a2z.com/docs/dashboards.html)
    * [TwitchTelemetryOpsGenerator](https://docs.fulton.twitch.a2z.com/docs/twitchtelemetryopsgenerator.html)
* [Twitch Telemetry Naming Conventions](https://docs.google.com/document/d/1-1z7B9HS8_YRYne09DvIwsN_jvYSGw77SjX0u4GWetM/edit) was used to figure out the naming conventions and rollups.
* Results of using client side aggregation.
    * [TwitchTelemetry Savings](https://docs.google.com/document/d/16t_Ot5_n9YYTULTkZMLO1JDx7C-ByvekuetaJY4xTF0/edit)
    * [Fortuna TwitchTelemetry](https://docs.google.com/document/d/18Xkdd8IWBFyR3AUE6XkW3HqVMAjss5yk_Y_tLG4gNzQ/edit#heading=h.5quiz6jt4hef)
    * [Fortuna TwitchTelemetry Transition Results](https://docs.google.com/presentation/d/1mZVGdjdEzBWb1fVi6o-pklSApmCvTcB5nXRgiTjLAtM/edit#slide=id.p)
* Metrics/Dashboard generation. 
    * [TwitchTelemetryOpsGenerator PR FAQ](https://docs.google.com/document/d/1-xUacfy3GBEgXylN-sx502WSYsiD-OMnZQyauxYx_gk/edit)
    * [TwitchTelemetryOpsGen Auto-Generated Alarms/Dashboards](https://docs.google.com/document/d/1yI2zHwhhf8ZmL_8vbShSHRMgXo1QHA56ks_FEHjozQY/edit#heading=h.et2rip27pqin)
* [TwitchTelemetry Load Test](https://docs.google.com/spreadsheets/d/1kZ_SYNEVxLLm22SxZ5lSYL6OZuEfOq3s7m7imhFFEUk/edit#gid=0) is a progression of tests and changes.  See notes column on right.  After some fixes to TwitchTelemetry, it worked well enough and the changes were checked in.
* Original decision to start with MWS.
    * [MWS Migration PR FAQ
](https://docs.google.com/document/d/1gEvjqX8MniAwIly9Ta8M3d9eGY3qxILZxQZb57UlKIM/edit#)
    * [MWS Proposal and Transition Plan](https://docs.google.com/document/d/1OPeAq-DeGr70Vc4mzk8J1OVfneJ9Av0hLtnDZZMeoRY/edit#heading=h.old45nkm0des)
    * [CloudWatch Transition Plan](https://docs.google.com/document/d/1c-g17zSo0KhpT3TZijs_8WfzJUhiloYJAKGMCRInqyE/edit#heading=h.3g52f7zddggz) We first looked into CloudWatch, but after a conference call with key people across Twitch, the consensus was MWS.
* SEH1
    * [MWS Docs](https://w.amazon.com/bin/view/Monitoring/Documentation/MonitoringWebService/PutMetricDataForAggregation/#Distribution_Input)
    * [Reference implementation](https://code.amazon.com/packages/MetricAggregation/blobs/mainline/--/src/amazon/monitoring/metric/aggregation/distribution/SEH1Distribution.java)
