# rediczar

rediczar is a helper library around [github.com/go-redis/redis](https://github.com/go-redis/redis). It provides many functionalities to supplement redis usage:

* Context-aware commands and pipelining
* Caching
* Prefixing command key arguments for namespacing
* Rate limiting
* Command stat tracking
* Pool stat tracking
* Creating a go-redis cluster client with defaults that are load-tested, and rate-limits connection creation to keep a redis cluster from death spiraling

A mirror of `github.com/go-redis/redis` is maintained at [code.justin.tv/chat/goredis](https://git-aws.internal.justin.tv/chat/goredis) that is compatible with `dep`. Additionally, its connection pooling is patched to improve availability and latency during memory pressure (evictions). It's recommended to use this mirror, and installation instructions are below.

## Improvements over identity/rediser:
* Supports context cancellations and deadlines. rediczar uses go-redis v7, which supports context. rediser depends on go-redis v6, which does not support contexts. The rediser client accepts contexts, so it gives a false sense that context is supported.
* Continuous unit and load testing. Built from the ground up with full test coverage
* Key prefixing support all pipelined commands
* Supports all redis commands; parity with go-redis command client

## Installation

### Go Modules

1. Install `go get code.justin.tv/chat/rediczar`

2. Replace `github.com/go-redis/redis/v7` with internal mirror

    `go mod edit -replace github.com/go-redis/redis/v7=code.justin.tv/chat/goredis/v7@v7.0.0-beta.4`

`go.mod` should resemble

```go
require (
  // ...
  github.com/go-redis/redis/v7 v7.0.0-beta.4
  // ...
)

replace github.com/go-redis/redis/v7 => code.justin.tv/chat/goredis/v7 v7.0.0-beta.4
```

### dep

1. Update Gopkg.toml

    ```toml
    [[constraint]]
      name = "code.justin.tv/chat/rediczar"
      version = "1.8.0"

    [[override]]
      name = "github.com/go-redis/redis"
      source = "code.justin.tv/chat/goredis"
      version = "v7.0.0-beta.4"
    ```
2. Run `dep ensure`

## Usage

The [examples](examples/) directory contains compilable Go examples. Keep in mind rediczar depends on v7 of go-redis, so the import path is `github.com/go-redis/redis/v7`.

It's recommended to use `redefault.NewClusterClient`. It will create a redis cluster client with [defaults](redefault/cluster.go) (timeouts, redirects/retries, pool size) that are load-tested, and rate-limits connection creation to keep a redis cluster from death spiraling.

### Creating the wrapper client

The wrapper client provides context-aware methods, prefixing, caching, and rate limiting helpers. See the [rediczar.ThickClient](/client.go) Go interface for all
available wrapper client methods.

`rediczar.PrefixedCommands.Redis` accepts any Redis client that implements [github.com/go-redis/redis/v7](https://github.com/go-redis/redis/blob/3e1f1aba0eebe8ee37d4f49914b4ce7c57f1ad00/commands.go#L49)

Single client. [Full single client example](examples/singleclient/main.go)
```go
c := redis.NewClient(&redis.Options{
  Addr: "localhost:6379",
})

client := &rediczar.Client{
  Commands: &rediczar.PrefixedCommands{
    Redis:     c,
    KeyPrefix: "prefix", // optional prefix prepended to all keys
    RunMetrics: &statsdrunmetrics.StatTracker{ // optional command metric tracking
      Stats: stats.NewSubStatter("redis.commands"),
    },
  },
  Logger:    logger,   // optional logger for internal errors
}
```

Cluster client. [Full cluster client example](examples/clusterclient/main.go) including tracking pool stats
```go
c := redefault.NewClusterClient("localhost:30001", nil)

// Optional: track pool stats (see examples/clusterclient/main.go)

client := &rediczar.Client{
  Commands: &rediczar.PrefixedCommands{
    Redis:     c,
    KeyPrefix: "prefix", // optional prefix prepended to all keys
    RunMetrics: &statsdrunmetrics.StatTracker{ // optional command metric tracking
      Stats: stats.NewSubStatter("redis.commands"),
    },
  },
  Logger:    logger,   // optional logger for internal errors
}
```

### Caching

`Cached*` methods can be used for caching data from other services or databases. The `Cached` method can cache any Go struct that serializes to JSON. To invalidate a cache entry, use `Del`.

It's recommended to use type-safe cache methods for primitives. Type-safe cache methods:
* `CachedBool`
* `CachedBoolDifferentTTLs`
* `CachedInt`
* `CachedInt64`
* `CachedString`
* `CachedIntSlice`
* `CachedInt64Slice`
* `CachedStringSlice`

`CachedSet` can be used to set and serialize an object. `CachedGet` can be used to fetch a key from the cache and deserialize. These methods are useful if you already have an object loaded from a data source, or want to fetch existing cache values without loading from a data source.

```go
type UserLoader struct {
  Redis        rediczar.ThickClient
  Database     database.Client
}

func (u *UserLoader) GetUserByID(ctx context.Context, userID string) (User, error) {
  var u User
  // Cache database user fetches for 1 hour
  err := s.Client.Cached(ctx, fmt.Sprintf("user:%d", userID), time.Hour, &u, func() (interface{}, error) {
    return s.Database.GetUserByID(ctx, userID) // GetUserByID returns (*User, error)
  })
  return u, err
}
```

### Rate Limiting

#### Tumbling window

[Tumbling window](https://stackoverflow.com/a/40599361) rate limiting can be done with `RateIncr`. [Full tumbling window rate limiting example](examples/tumblingwindowratelimit/main.go)

Example: Rate limiting that allows 10 operations per 10 second non-overlapping windows
```go
count, err := client.RateIncr(ctx, "key1", 10*time.Second)
if err != nil {
  return err
}

if count > 10 {
  // Rate limit if there are more than 10 operations in the last non-overlapping window
  continue
}

// Allow operation
```

#### Sliding window

[Sliding window](https://stackoverflow.com/a/40599361) rate limiting can be done with `SlidingWindowRateLimit`. [Full sliding window rate limiting example](examples/slidingwindowratelimit/main.go)

Example: Rate limiting that allows 10 operations per 10 second overlapping windows
```go
allow, err := client.SlidingWindowRateLimit(ctx, "key1", 10*time.Second, 10)
if err != nil {
  return err
}

if !allow {
  return nil
}

// Allow operation
```

### Pipelined Commands

Prebuilt pipelined commands for convenience:

* `PipelinedDel` This is useful for issuing multiple DEL commands across a cluster
* `PipelinedGet` This is useful for issuing multiple GET commands across a cluster
* `MSetWithTTL` MSet alone doesn't support TTL arguments
* `MSetNXWithTTL` MSet alone doesn't support NX and TTL arguments

### Mocks

[redimock](/redimock) contains mocks for unit testing
* `redimock.Client` is a [counterfeiter](https://github.com/maxbrunsfeld/counterfeiter) generated mock for `rediczar.ThickClient`.
* `redimock.CacheStubbedClient` stubs out all `Cache*` helper functions so they always cache miss. This is useful for tests that don't want to test caching behavior.

### Circuits

[circuitgen](https://github.com/twitchtv/circuitgen) can be used to generate a circuit wrapper to encapsulate calling circuits. It is recommended
to generate a circuit wrapper on the `Commands` interface because its methods are direct calls to redis. Other methods on `ThickClient`, like
`Cached*`, use multiple redis commands, and can include other service calls.

Example: Generating circuit wrapper into package internal/wrappers
```bash
circuitgen --pkg code.justin.tv/chat/rediczar --name Commands --alias RediczarCommands --out internal/wrappers
```

Example: Setup cluster client with the above circuit wrapper
```go
c := redefault.NewClusterClient("localhost:30001", nil)

commands := &rediczar.PrefixedCommands{
  Redis:     c,
  KeyPrefix: "prefix", // optional prefix prepended to all keys
}

wrappedCommands, err := wrappers.NewCircuitWrapperRediczarCommands(manager, commands, wrappers.CircuitWrapperRediczarCommandsConfig{
  ShouldSkipError: func(err error) bool {
    return err == redis.Nil // Keys that do not exist should be treated as successful requests
  },
})
if err != nil {
  return err
}

client := &rediczar.Client{
  Commands: wrappedCommands,
}
```

## Migrating from identity/rediser

1. Change imports from `github.com/go-redis/redis` to `github.com/go-redis/redis/v7`. This is especially important if using Go modules when checking if a key does not exist by comparing
`err == redis.Nil`. If you are still using `dep`, then it is not as important because `github.com/go-redis/redis.Nil` is aliased to `github.com/go-redis/redis/v7.Nil`
    * Option 1: If the project is using Go modules, use the [mod](https://github.com/marwan-at-work/mod) tool to change import paths. `mod upgrade --mod-name=github.com/go-redis-redis -t 7`
    * Option 2: Find, replace, and rerun goimports
        ```bash
        git grep '"github.com/go-redis/redis"' | cut -f1 -d: | grep '\.go$' | sort | uniq | grep -v vendor | xargs sed -i '' -e 's#"github.com/go-redis/redis"#"github.com/go-redis/redis/v7"#g'
        find . -iname "*.go" -not \( -path '*/vendor/*' \) -not \( -path '*/_tools/*' \) -print0 | xargs -0 goimports -w
        ```
1. Replace `rediser.NewClient` by [creating a wrapper client](#creating-the-wrapper-client). Decide if you want to create a single node, cluster, or ring client
    * Map `code.justin.tv/identity/rediser/common.Options` to go-redis options
      * [Cluster option mapping](https://git-aws.internal.justin.tv/identity/rediser/blob/f06ab07bfb21e6702260ff889344c5ebaa1bbdc4/common/options.go#L86-L101)
      * [Ring option mapping](https://git-aws.internal.justin.tv/identity/rediser/blob/f06ab07bfb21e6702260ff889344c5ebaa1bbdc4/common/options.go#L105-L123)
    * If `code.justin.tv/identity/rediser/common.Options.KeyPrefix` is set, copy it to `rediczar.PrefixedCommands.KeyPrefix`
    * If `code.justin.tv/identity/rediser/common.Options.{MarshalFunc,UnmarshalFunc}` are set, copy them to `rediczar.Client.{Encode,Decode}` respectively
1. If you are using the `rediser.Handler` Go interface, replace it with `rediczar.ThickClient`

    Find, replace, and rerun goimports
    ```bash
    git grep 'rediser.Handler' | grep -v vendor | cut -f1 -d: | sort | uniq | xargs sed -i '' -e "s#rediser.Handler#rediczar.ThickClient#g"
    find . -iname "*.go" -not \( -path '*/vendor/*' \) -not \( -path '*/_tools/*' \) -print0 | xargs -0 goimports -w
    ```

1. `Invalidate` has been removed since it's a straight pass-through to Del. Use `Del` instead
1. `MSetNxWithTTL` has been renamed to `MSetNXWithTTL`. It also returns a bool for each key indicating whether it was set.
1. `CachedStringNative` has been removed. Use `CachedString` instead.
1. `Eval` has been changed to pass arguments from `args []interface{}` to `args ...interface{}` to match go-redis. To prevent errors during migration, if `args[0]` is an `[]interface{}`, then rediczar will flatten the args
1. `code.justin.tv/identity/rediser/client.MockHandlerWithStubbedCache` (mockery mock) has been replaced with `code.justin.tv/chat/rediczar/redimock.CacheStubbedClient` (counterfeiter mock)

    Find, replace, and rerun goimports
    ```bash
      git grep 'rediser.MockHandlerWithStubbedCache' | grep -v vendor | cut -f1 -d: | sort | uniq | xargs sed -i '' -e "s#rediser.MockHandlerWithStubbedCache#redimock.CacheStubbedClient#g"
      find . -iname "*.go" -not \( -path '*/vendor/*' \) -not \( -path '*/_tools/*' \) -print0 | xargs -0 goimports -w
    ```

1. `code.justin.tv/identity/rediser/client.MockHandler` (mockery mock) has been replaced with `code.justin.tv/chat/rediczar/redimock.Client` (counterfeiter mock)

    Find, replace, and rerun goimports
    ```bash
    git grep 'rediser.MockHandler' | grep -v vendor | cut -f1 -d: | sort | uniq | xargs sed -i '' -e "s#rediser.MockHandler#redimock.Client#g"
    find . -iname "*.go" -not \( -path '*/vendor/*' \) -not \( -path '*/_tools/*' \) -print0 | xargs -0 goimports -w
    ```

1. `PipelinedGet` has been changed to return `nil` for non-existent keys instead of empty strings

1. `code.justin.tv/identity/rediser/cache.Handler` has equivalents in `rediczar.ThickClient`. Mapping from `rediser/cache.Handler` to `rediczar.ThickClient`

    * `Fetch` -> `Cached`
    * `FetchBool` -> `CachedBool`
    * `Delete` -> `Del`
    * `Get` -> `CachedGet`
    * `Set` -> `CachedSet`

1. Command and pool statsd metrics are in the `commands.` and `pool.` namespaces respectively. Grafana graphs will need to be updated

## Development

`make` to run all checks. This is also automatically ran in CI in a Docker image.

`make gen` to generate code. All generates files end with `.gen.go`. Code generation is used to generate context-aware and prefixing code for every Redis command.

`make build` to verify dependnecies and build the Go code.

`make test` to run Go tests natively.

`make lint` to run Go linting.

### redefault Integration tests

Package `redefault` contains integration tests. These tests are ran manually on a local development machine

Create a local redis cluster. redis-cli 5.0.0+ is required.

    $ cd redefault
    $ ./create-cluster start
    $ ./create-cluster create

`make integration_test` to run the Go integration tests

## Contributing

Contributions are extremely welcome! Please create a pull request with any improvements or bug fixes, and include updates to tests too.

## Unsupported features

WATCH wrapper for Redis transactions has not been implemented. Use a Lua script instead to execute many commands atomically.
