TwitchTelemetrySystemMetricsAgent
===

A GoLang library that can be used to collect and submit system telemetry samples.

Documentation
---

The following represents a sample integration of how to uses the System Metrics agent. The integration assumes you are
adding the library to your server's `main.go` (likely somewhere in your `cmd/` folder). The `bs.` variables below
are part of the Fulton Bootstrap, but do not require usage of Fulton (and can be manually constructed).

Sample integration to initiate polling for system metrics:
```
// Somewhere below your server definition...

// Create a poller for system level metrics
systemMetricsConfig := agent.SystemMetricsCollectorConfig{
    Collectors: []agent.SystemMetricsCollector{agent.CPUCollector, agent.NetCollector, agent.DiskCollector, agent.DiskIOCollector, agent.MemCollector, agent.LoadCollector},
    PerCPU:     true,
}
systemMetricsPoller := agent.NewSystemMetricsPollingCollector(systemMetricsConfig, 10*time.Second, bs.ProcessIdentifier, bs.SampleReporter.SampleObserver, bs.Logger)
systemMetricsPoller.Start()
```
and then to gracefully end polling...

```
// Somewhere in your server's shut down function
systemMetricsPoller.Stop()
```

As you can see, we pass an `agent.SystemMetricsCollectorConfig` into `NewSystemMetricsPollingCollector`. This config
has two components.

`Collectors` represents a list of all system metric agents you wish to use. This should be one or more of the following:

| Agent Collector | Meaning | Metrics Emitted |
| ----------------|:-------:|:---------------:|
|`agent.CPUCollector` | CPU usage metrics overall and per-CPU if desired | `CpuUser`, `CpuSystem`, `CpuIoWait` |
|`agent.NetCollector` | Network metrics| `NetBytesSent`, `NetBytesRecv`, `NetPacketsSent`, `NetPacketsRecv` |
|`agent.DiskCollector` | Disk space metrics in Bytes| `DiskTotal`, `DiskFree` |
|`agent.DiskIOCollector` | Disk IO metrics | `DiskIOReadBytes`, `DiskIOWriteBytes`, `DiskIOReadTime`, `DiskIOWriteTime`, `DiskIOIoTime` |
|`agent.MemCollector` | Memory (RAM) metrics in Bytes | `MemTotal`, `MemAvailable`, `MemUsed` |
|`agent.LoadCollector` | Load metrics, represents load over 1, 5, and 15 minutes | `Load1`, `Load5`, `Load15` |

`PerCPU` represents whether CPU metrics should be obtained for every CPU. If this is set to `false` (the default), then
only a `cpu-total` metric will be emitted for CPU metrics (instead of separate metrics for `cpu0`, `cpu1`, etc.).
Obviously, setting this to `true` will increase the number of CPU metrics you're emitting by N, where N is the number
of CPUs your instance/ container is using.

When using TwitchTelemetry, we do not utilize the `Substage` dimensions but do utilize the `ProcessAddress` dimension,
which rolls up. This means you will see the above metrics for your service's hosts (e.g. your service's docker tasks
or bare metal or whatever the thing running your service is) as well as a rollup so you can alarm across all hosts 
(e.g. alarm on the avg, p99, max etc.).