Questions tagged [amazon-sqs]

Amazon Simple Queue Service (SQS) is part of Amazon Web Services, providing a fully-managed message queuing service for reliably communicating among distributed software components and microservices. Use this tag for questions that describe Amazon's Simple Queuing services.

Introduction

Amazon Simple Queue Service (SQS) is part of Amazon Web Services, providing a fully-managed message queuing service for reliably communicating among distributed software components and microservices. Use this tag for questions that describe Amazon's Simple Queuing services.

SQS supports two different types of queues:

  • Standard Queues - Amazon SQS offers standard as the default queue type. A standard queue lets you have a nearly unlimited number of transactions per second. Standard queues guarantee that a message is delivered at least once. However, occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order. Standard queues provide best-effort ordering which ensures that messages are delivered in the same order as they are sent.

  • FIFI Queues - The FIFO queue complements the standard queue. The most important features of this queue type are FIFO (first-in-first-out) delivery and exactly-once processing: The order in which messages are sent and received is strictly preserved, and a message is delivered once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue. FIFO queues also support message groups that allow multiple ordered message groups within a single queue. FIFO queues are limited to 300 transactions per second (TPS), but have all the capabilities of standard queues.

13 questions
23
votes
2 answers

Is it possible to set CloudWatch alarm for a combination of metrics?

For example, I want to trigger a CloudWatch alarm for the following: Number of messages sent - Number of messages deleted >= 6 The above is for AWS SQS metrics. And I want that metric to trigger AWS Lambda if that alarm is set off. Is there any way…
Dawny33
  • 2,816
  • 3
  • 24
  • 62
23
votes
1 answer

How do I work around the fact that AWS SQS is not HIPAA compliant?

I have a use case where data from S3 is queued into AWS SQS, which is in turn connected to CloudWatch, whose metrics will be triggering AWS Lambda. However, I want the architecture to be HIPAA compliant. So, I have come up with this idea: Once my…
Dawny33
  • 2,816
  • 3
  • 24
  • 62
13
votes
1 answer

How to communicate queue-based processing delays to non-technical team members?

I am responsible for a set of SQS queue-processing jobs with a scaling policy on the ApproximateNumberOfMessagesVisible CloudWatch metric. These jobs can fail to keep up with the amount of messages sent for any number of reasons: Service…
13
votes
2 answers

Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm?

Currently, for triggering a lambda function in response to a CloudWatch alarm, I need to route the alarm via SNS, like this: AWS CloudWatch Alarm --> Send to a topic in SNS --> SNS topic triggers the lambda function So, is there a way I can do it…
Dawny33
  • 2,816
  • 3
  • 24
  • 62
7
votes
3 answers

How to ensure equitable distribution of SQS messages in a distributed systems setting?

I have multiple servers, each having a script polling an SQS queue [all polling the same queue]. So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there…
Dawny33
  • 2,816
  • 3
  • 24
  • 62
5
votes
1 answer

AWS SQS - use case for "short polling"

When using event driven processing of work with AWS SQS and worker clients. the use of long polling is often recommended. AWS documents this here Essentially when there are no message in the queue, the worker does not need to keep making requests to…
jdog
  • 650
  • 3
  • 10
4
votes
2 answers

Is it better to perform delayed jobs using queues or at app level?

I'm trying to scale up my push notification infrastructure. We currently use laravel app running on 2 ec2 instances horizontally scaled behind a load balancer, and each instance has a beanstalk queue engine that processes jobs created by laravel…
abbood
  • 473
  • 4
  • 13
2
votes
1 answer

AWS SNS-SQS: Checking a subscription already exists

I've found it's a useful strategy to create relevant queues whenever a queue worker service boots. For instance, on service boot, it checks whether its queues exist already in AWS and creates them if they don't. Something I'm struggling with now is…
Trout
  • 31
  • 3
2
votes
2 answers

Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1

I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process. Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but…
jdog
  • 650
  • 3
  • 10
1
vote
1 answer

How to programmatically enable and disable SQS triggering AWS Lambda?

I have a requirement where I want to stop my lambda from receiving messages from SQS. And I want to implement this like an API which can toggle between the disabling and enabling of the AWS Lambda getting triggered by SQS. I tried looking for…
0
votes
1 answer

Processing message queues quickly without wasting money on AWS Lambda

What are some strategies and patterns for processing message queues quickly? I have a Lambda function that is invoked on a schedule every x minutes to process a number of messages in the queue. I could reduce the interval, but a lot of the time…
Simian
  • 103
  • 3
0
votes
1 answer

Filtering AWS SQS Tags using JQ

I'm having doubts when trying to run a query with JQ. I have an SQS queue with Tags applied and when I run aws sqs list-queue-tags --region sa-east-1 --queue-url --output json | jq returns the output: { "Tags": { "owner":…
Vader
  • 466
  • 3
  • 15
0
votes
1 answer

SQS: What could high number of visible messages mean if messages received & deleted are normal?

I'm new to SQS. I recently noticed the ApproximateNumberOfMessagesVisible in my SQS queue started rising. What was interesting though is that there was no sudden surge in the number of messages received or depression in number of messaged being…