---
title: SQS Queues
weight: 2
---

## Create an SQS Queue in Your Account

To create an Event Bus compatible SQS queue, it must:

1. Use the Event Bus KMS key so messages are encrypted at rest and in transit
1. Use the Event Bus SQS policy so messages can be properly delivered

To make things easy, these values are included in the CloudFormation stack you ran in the [Getting Started]({{< ref "getting_started/_index.md" >}}) step. Take a look at the following examples to see how to integrate them:

{{% expand_markdown "Terraform" %}}

```tf
resource "aws_sqs_queue" "my_queue" {
  name              = "eventbus-my-queue"
  policy            = "${aws_cloudformation_stack.eventbus.outputs["EventBusSQSPolicyDocument"]}"
  kms_master_key_id = "${aws_cloudformation_stack.eventbus.outputs["EventBusKMSMasterKeyARN"]}"
  redrive_policy    = "{\"deadLetterTargetArn\":\"${aws_sqs_queue.my_deadletter.arn}\",\"maxReceiveCount\":5}"
}

resource "aws_sqs_queue" "my_deadletter" {
  name              = "eventbus-my-queue-deadletter"
  kms_master_key_id = "${aws_cloudformation_stack.eventbus.outputs["EventBusKMSMasterKeyARN"]}"
}
```
{{% /expand_markdown %}}

{{% expand_markdown "Cloudformation" %}}

```yaml
  Queue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: "eventbus-my-queue"
      KmsMasterKeyId: !GetAtt EventBus.Outputs.EventBusKMSMasterKeyARN
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt DeadletterQueue.Arn
        maxReceiveCount: 5

  DeadletterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: "eventbus-my-queue-deadletter"
      KmsMasterKeyId: !GetAtt EventBus.Outputs.EventBusKMSMasterKeyARN

  QueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument: !GetAtt EventBus.Outputs.EventBusSQSPolicyDocument
      Queues:
        - !Ref Queue
```
{{% /expand_markdown %}}

{{% expand_markdown "Cloudformation (Fulton SAM stacks)" %}}

This differs from the previous CloudFormation template in two ways:

1. Fulton will likely have you deploy multiple stacks, one per developer in the same account. For this reason, you need to run the EventBus stack separately and use the exported [cross-stack references](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html) to our values.
1. SAM and stage stacks will automatically assign generated names to resources based on the top-level resource name in the YAML. EventBus expects SQS queues with `eventbus` in the name, which is why resource names below are specifically provided in lower camel case (typically upper camel case).

```yaml
  eventbusQueue:
    Type: AWS::SQS::Queue
    Properties:
      KmsMasterKeyId:
        Fn::ImportValue: "eventbus-kms-master-key-arn"
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt eventbusDeadletterQueue.Arn
        maxReceiveCount: 5
  eventbusDeadletterQueue:
    Type: AWS::SQS::Queue
    Properties:
      KmsMasterKeyId:
        Fn::ImportValue: "eventbus-kms-master-key-arn"
  EventBusQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Fn::ImportValue: "eventbus-sqs-policy-document"
      Queues:
        - !Ref eventbusQueue
```
{{% /expand_markdown %}}

{{% notice note %}}
Our cross-account permissions are scoped only to queues with **lowercase** "eventbus" somewhere in the name. It can be anywhere in the name -- beginning, end, middle -- but that lowercase string has to be _somewhere_ or we won't be able to validate or send to your queue.
{{% /notice %}}

{{% notice note %}}
The Event Bus will peridiocally verify that your SQS queue meets the requirements listed. For example, if the KMS key is ever removed from your SQS queue, your service no longer meets Twitch's data handling requirements and alerts will trigger.
{{% /notice %}}

## Dead-Letter Queues {#dead-letter}

When you create a new queue, we highly recommend you also create a dead-letter queue.

When a message fails more than N times, it'll be put on the dead-letter queue, preventing failing messages from building up forever. For example, if you set `maxReceiveCount` to `5`, it means that if your worker doesn't succeed 5 different times on a message, then the message will be sent to your dead-letter queue.

We also recommend you set up alarms on `ApproximateNumberOfMessagesVisible` on the dead-letter queue so your on-call can know if messages are failing. Afterwards you can inspect the messages in the AWS console or with command-line tools to figure out why they may have failed.

## Advanced Tuning {#fine-tuning}

Our examples include suggested defaults that should work for a wide variety of applications. However there are special scenarios where you may want to modify these values. Following are some recommendations to consider:

### Visibility Timeout

This controls the amount of time that a message is exclusive to a single worker instance. After this timeout, if the worker hasn't flagged the message as completed, it will show up on the queue again for another worker to pick up.

If you set this value too low, there are two implications:

1. The worker might not have enough time to process the message, causing another worker to pick it up. This will potentially exacerbate any heavy load as you're processing the same message more than once.
1. During outages the retry count will increment quickly, causing the message to be dead-lettered sooner.

### Max Receive Count {#dead-letter-receive-count}

This is the number of failures after which the message is moved to the dead-letter queue.

To determine this value, we recommend looking at your expected downtimes or blips and calculate `visibility timeout * max receive count` seconds. Make sure this value gives you a large enough window to still process messages during such downtimes.

As a concrete example, the WebSub service recently had an RDS master failover, which takes 1-2 minutes for the new master to come online. In one incident, it took 92 seconds for the failover to complete, and the max receive count was set to 3. As you see, `3 * 30 = 90 seconds`, which was too short and some of the messages were dead-lettered. A safer value for this would have been `5`, which would allow messages to still be serviced even with a slightly longer database downtime.

### Message Retention Period

This determines how long a message can stay in the queue. For example, if your service was offline for an extended period of time, old messages would eventually be removed from the queue.

We recommend keeping this value high and instead relying on your dead-letter queue to handle failing messages. However if you need to shed load during high throughput, you may consider setting this value to something smaller.

### Buffering for Spikes in Traffic

Using SQS should shield your service from big spikes of traffic. However, this assumes that your service is not pulling messages off of the queue faster than it can handle. If your service is negatively impacted by such spikes consider lowering the threshold for [number of pollers and deliver concurrency](https://git.xarth.tv/eventbus/client/blob/master/subscriber/sqsclient/config.go#L19-L20).

Our recommendation is for your service to create an alarm for the [number of visible messages](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) on the queue. This will allow you to determine if your service is not processing the events quickly enough.
