# Chitin (KY-tin): Process Exoskeleton

[![GoDoc](http://godoc.internal.justin.tv/code.justin.tv/common/chitin?status.svg)](http://godoc.internal.justin.tv/code.justin.tv/common/chitin)

We've got a lot of tiny services. Some big ones too. We're going to have even
more in the future, so we may as well get ready now.

There are a bunch of common problems that each service runs into. Some
problems are easy, so the author comes up with a solution on the spot that
works for them. Examples include  deciding which port to listen to and
handling signals (often by "do nothing"). The more challenging problems are
often left unsolved. Examples of this are context-enhanced logs and cross-
service request tracing.

Chitin will make it easy for service authors to create daemons that comply
with our best practices. More consistent daemons will lead to more reliable
operations. Standard calling conventions will enable more general debugging
tools.

## A Quick Reminder

This feature list is only partly completed.

This feature list has not yet been ratified.

## The (Cross-Platform) Specifics

### Serving HTTP

When serving inbound HTTP requests, Chitin manipulates some headers and
instruments various parts of the HTTP call cycle.

*   The `Trace-ID` and `Trace-Span` headers are read. If the `Trace-ID` header
is absent, a new trace id is generated for the request. These values follow
the request through the daemon, appearing in every related log line and
external service call.

    The `Trace-ID` header is interpreted as a base-10 integer.

*   Headers with the name `Chitin`, or beginning with `Chitin-` will be
removed from the request. When an HTTP handler reads a request with a header
in the `Chitin` namespace, the handler author can be confident that the header
was set by the Chitin runtime.

    There are currently no headers defined in the `Chitin` namespace.

*   Headers in the `Chitin` namespace are removed from the response. They'd be
removed by a Chitin-compliant client anyway.

*   Chitin exposes a few hooks for instrumenting each HTTP request's
lifecycle.

    1. Request headers have been read
    2. Request body is consumed (including its byte count)
    3. Response headers are ready to be written
    4. Response body is written to kernel memory
    5. Handler has returned
    6. The client disconnected prematurely
    7. The request body is closed

*   HTTP verbs are restricted to `GET`, `POST`, `PUT`, `HEAD`, and `DELETE`.
There's no technical reason for this, but enforces some level of sanity. There
are currently no plans to support the `SPACEJUMP` verb.

### Consuming HTTP

When making outbound HTTP requests, Chitin manipulates some headers and
instruments various parts of the client-side HTTP call cycle.

*   The `Trace-ID` and `Trace-Span` headers are written. The `Trace-Span`
value is modified to reflect the depth of the call.

*   Headers in the `Chitin` namespace are removed, and a warning is printed in
the log. If the headers were sent to a remote Chitin service, that remote
service would have removed the headers anyway.

*   Headers in the `Chitin` namespace are removed from the response. When an
HTTP client reads a response with a header in the `Chitin` namespace, the user
can be confident that the header was set by the Chitin runtime.

*   The connection pool is periodically cycled to allow rebalancing.

*   HTTP verbs are restricted to `GET`, `POST`, `PUT`, `HEAD`, and `DELETE`.

### Logging

Chitin enables context-enhanced logs. Log lines get a
[`logfmt`](http://godoc.org/github.com/kr/logfmt)-formatted prefix containing
`trace-id` and `trace-span` keys.

### Signals

The Chitin runtime decouples operator intent from the specifics of POSIX
signals.

Users of the Chitin runtime can indicate their ability to:

1. Shut down gracefully, allowing existing clients to execute one last request
before being disconnected

2. Shut down quickly, cleaning up state as required and waiting for sub-
processes to exit, but not waiting for client requests to complete

3. Shut down extremely quickly, performing only the minimum required cleanup
and dying as soon as possible (every process for itself)

4. Reload their config from disk and continue running.

The Chitin package will convert POSIX signals into requests for those
particular actions with the following mapping:

1. `SIGUSR2` and `SIGINT` trigger a graceful shutdown
2. `SIGTERM` triggers a prompt shutdown
3. `SIGQUIT` triggers a very fast shutdown
4. `SIGHUP` triggers a config reload

### Other Freebies

TCP keepalive is enabled for sockets managed by Chitin.

## Specifics of the Go API

### Serving HTTP

*   When a request handler panics, the stack trace is wrapped in
`logfmt`-format is context-enhanced.

### Profiling and Process Visibility

Chitin automatically enables the `net/http/pprof` and `expvar` handlers.

In addition, chitin provides tools to detect and diagnose misuse of the Go
http client that results in leaking network connections.

    func buggy() {
        http.Get("http://example.com")
    }

    func good(ctx context.Context) error {
        resp, err := chitin.Client(ctx).Get("http://example.com")
        if err != nil {
            return err
        }
        defer resp.Body.Close()
        return nil
    }

When the context is canceled, chitin will cancel the outbound request
automatically. Chitin uses the runtime/pprof package to keep track of the
current outbound requests, so an engineer can see the stacks of callers that
haven't correctly disposed of their requests' response bodies.

A command like the following, with the help of the [Go pprof blog
post](https://blog.golang.org/profiling-go-programs) can help debug http
connection leaks:

    $ go tool pprof http://localhost:8080/debug/pprof/code_justin_tv_common_chitin_httpclient
