[![GoDoc](https://godoc.internal.justin.tv/code.justin.tv/release/trace?status.svg)](https://godoc.internal.justin.tv/code.justin.tv/release/trace)


[Trace](https://trace.internal.justin.tv/)
===

[Access problem? See FAQ](FAQ.md)
---

Overview
---
Trace builds cross-service callgraphs.  It observes relationships between
internal services as they are used to satisfy user requests.  As a public-
facing service gets help from other internal services or databases to
construct a response for the user, each interaction is noted and attributed to
the initial user request.

It uses a unique Transaction ID for each incoming user request, which is
propagated through to each internal service that does work as a result of the
user's request.  Services emit Event records which are collected for analysis.
A set of Events forms a callgraph for the involved distributed system, in the
form of a tree of remote procedure calls.  Each Event includes the Transaction
ID identifying the tree and a Path to the node it describes within the tree.

Each Event also includes additional information, such as the Kind of event
observed, the current time, and the name of the service, hostname, and process
id where the event was observed.  Events may also include information specific
to the RPC, such as the identity of the network peer, the HTTP method and
status code, or the normalized SQL query executed.  Event message format and
semantics are thoroughly documented in the [protobuf description
file](pbmsg/event.proto).


Go Version Policy
---

The Trace service aims to use _very_ recent versions of Go.
We try to use a version of Go that's been committed within the last two weeks,
which is much more aggressive than other projects.

We do this because the Trace service is in a good position to test changes to
the Go compiler, runtime, and core packages.
It contains a variety of programs that are largely representative of Go usage
at Twitch, so many issues that would affect other Twitch services can likely
be observed in Trace's programs first.
Finding these problems and reporting them to the Go project as soon as
possible means they're more likely to be fixed in official Go releases, and
that Go will work better for Twitch services.
Furthermore since Trace is not a user-facing service, the impact of poorly-
behaved Go versions is relatively small.


To this end, the Trace service aims to use a version of Go that's been
committed within the last two weeks.
The service may fall behind Go development if bugs in Go make recent commits
unstable or otherwise unusable, in which case those bugs should be
[reported to the Go project](https://golang.org/issue/new).
Trace will often use Go's master branch, though it may opt to use relevant
`dev.*` branches, or `release-branch.*` branches for upcoming releases.

Fresh Go commits are tested on Trace's canary environment before moving on to
its production environment.
Trace is also used to test Go changes that are still under review.

During the Go 1.7 development cycle, the Trace project remained on the stable
1.6 release until go1.7beta2.
Using the beta versions uncovered several bugs in Go, most of which were easy
to isolate and were quickly fixed.
At least [one of the bugs](https://golang.org/issue/16528) was found too late
in the release cycle so will likely not be fixed until Go 1.8, a delay of six
months.

Code in Trace's repository must continue to successfully compile with the most
recent stable release of Go's compiler, runtime, and core packages.


Architecture and Operations
---

![Data flow diagram](https://git-aws.internal.justin.tv/release/trace/wiki/static/dataflo2.jpg)

Trace's data pipeline is designed with three goals in mind:
- minimal impact on instrumented services
- scalability up through handling millions of events per second without resorting to sampling
- security of data throughout the pipeline, primarily in terms of confidentiality

We achieve these, respectively, by
- accepting data over UDP via a host-local process
- dividing collection and aggregation of data into separate steps, shuttling the data around with [Kafka](http://kafka.apache.org/)
- running most of the pipeline in a locked-down AWS security group, restricting SSH access to the hosts, and only presenting data that might contain user information to those with a need to know

Instrumented processes emit Trace events over UDP to a local process,
[code.justin.tv/release/villagers/cmd/barrel](https://git-aws.internal.justin.tv/release/villagers/tree/master/cmd/barrel),
listening on localhost:8943. Barrel forwards the events to collection processes,
[cmd/pbcollect](cmd/pbcollect), it discovers
[via consul](http://consul.internal.justin.tv/ui/dist/#/us-west2/services/code.justin.tv/release/trace/cmd/pbcollect?filter=col).

The collection processes send events to the trace-events Kafka topic. This
topic is consumed by [cmd/statsdsink](cmd/statsdsink) which sends server-
observed request durations to statsd for creating
[per-repo service dashboards](http://code.justin.tv/video/usher). The trace-
events topic is also consumed by [cmd/aggregate](cmd/aggregate), which sends
complete transactions to the trace- transactions topic after a short delay.

The trace-transactions Kafka topic is consumed by the
[Trace gRPC API server](https://godoc.internal.justin.tv/code.justin.tv/release/trace/pbmsg#TraceClient),
[cmd/api](cmd/api).

[Ganglia host list](https://ganglia-ec2.internal.justin.tv/?r=hour&cs=&ce=&c=unclassified&h=&tab=m&vn=&hide-hf=false&m=load_one&sh=2&z=small&hc=0&host_regex=%5Etrace-&max_graphs=0&s=by+name)

[Kafka dashboard](http://grafana.prod.us-west2.justin.tv/dashboard/db/trace-kafka)

[Kafka consumer group dashboard](http://grafana.prod.us-west2.justin.tv/dashboard/db/trace-kafka-consumer-group-lag)

[Dashboard for cmd/pbcollect](http://grafana.prod.us-west2.justin.tv/dashboard/db/trace-collector)

[Dashboard for cmd/statsdsink](http://grafana.prod.us-west2.justin.tv/dashboard/db/trace-statsdsink)

[Dashboard for cmd/aggregate](http://grafana.prod.us-west2.justin.tv/dashboard/db/trace-aggregator)
