# SQS Poller Design

*NOTE* While this is developed 'in-situ' in the Event Bus client library, we want
to evolve this into something worthy of being pulled into its own library, hence
the current separation into its own package.

The SQS poller is designed to be a resilient concurrent poller that pulls messages
out of SQS to be processed by consumer code. It indends to poll the queue 
sufficiently fast to saturate your worker, but uses backpressure to help prevent
over-committing to work.


## Goroutines

This design compreses several groups of goroutines each doing independent work:

1. **Pollers** (1 or more) run an infinite loop of polling SQS and forwarding 
   messages to the messages channel.
2. **Acking** Batch finished message IDs into DeleteMessageBatch calls.
3. (coming in EDGEPLAT-1900) **Stats Hub** (always 1 goroutine) gets stats from the pollers about how 
   long was spent polling and how many messages were received and decides 
   whether to add or remove pollers.

## Scaling

(Coming in EDGEPLAT-1900)

For each loop of the poller, it sends stats to the stats hub about the 
following things:

1. How long spent waiting for ReceiveMessage
2. How many messages were received
3. How long spent enqueueing these messages (hint about blocking)
4. If we got any rate-limit errors from AWS.

With this information the stats hub looks at whether the ReceiveMessage
returned early, or waited the entire duration. Furthermore, it checks
primarily on whether it returned 0, 1, or more than 1 message.

If we got more than 1 message or especially the maximum 10 messages, then
it is a signal that we should be adding more poller goroutines.

However, to prevent over-fetch scenarios where we pull a lot of messages off 
the queue and don't have enough CPU to process them, we monitor time spent
blocking as well and will not scale up if we're blocking a lot on enqueueing.

## Acking

An important part of working with SQS is making sure we delete messages once
processing work is complete. If we get a non-error response from the 
dispatcher, we will delete that messages' ID off the queue so it does not 
get re-delivered again after the delivery window.

SQS provides a batch delete operation, so we will optimistically delete 
things in batches, but also have a min frequency of deletions set so that
in general, deletions still happen within the receive window.

## Shutdown

We try to take the most safe route to shutdown, even though it may take longer.
Essentially, we're trying to shut down with no messages that have been pulled
from SQS without being processed. The shutdown sequence looks like this:

1. Stop all pollers to stop filling channels with messages.
2. Wait for all in-flight calls to finish
3. Perform any left-over DeleteMessage if any.

However, we know this could take a while so we offer one parameter which is 
effectively a hara-kiri timeout: If this interval is reached, we cancel the
context on all remaining outgoing requests. This acts as a way to set up a
maximum time to shutdown parameter.

If this timeout is reached, then it is possible that some messages are going
to be re-delivered after the VisibilityTimeout to another poller, which is
probably not a big deal for most use cases, but it does underscore the need
for idempotency.
