Pretty much the title.

My lambda has failed to process some messages coming from my FIFO SQS queue. The visibility timeout is set to 18 minutes, and Message retention period is set to 4 days.

To my understanding, my consuming lambda will try to process the message every 18 minutes for 4 days. Yet, just some of these failing messages have been moved to the DLQ way earlier than 4 days.

Is there something that I am missing here?

Also, since it is a FIFO queue: assuming there is a series - [1,2,3,4], of messages incoming to my queue having the same GroupID. If message 1 fails, then messages 2,3,4 are not processed. Is this correct? Assuming than that message 1 is moved to DLQ after 4 days, then message 2 will try to get consumed, so on and so forth. Is my understanding correct?

Thanks in advance.

CodePudding user response：

You also need to configure the maxReceiveCount setting on the queue. This determines how many times a single message will be retried before it is moved to the DLQ. I think the default value is 1, so after a single failure a message will be moved to the DLQ.

CodePudding user response：

Message retention period - This is the max time that the message will remain in your queue if you don't process it. So, it doesn't mean that it will be processed for 4 days, every 18 minutes. This is because you can have your own workers, running on servers, you don't have to use Lambda functions. If you use Lambda, it's pretty much instantaneous, but if you use your own servers, maybe you'll have downtime and workers are not available, so in that case, you want to retain messages for a certain time to be able to process them later. The message retention period can be up to 14 days.

You are using a FIFO queue, which had an "exactly once delivery" policy. That means that you can process the message only once when you use the FIFO queue. If you want to leave failed messages in your queue and re-process them, you have to use a regular queue. When you use the standard queue, you can set the number of processing attempts before moving a message to the DLQ (usually it's 3-4, but you can set your own value).

In your example, what happens is that messages will be sent in the exact order in which they were received by the SQS. It will not wait to check if message 1 was processed or not, because FIFO queue has "exactly once delivery" and you can use multiple workers to process the messages. Since you're using Lambda functions, they will be triggered pretty quickly by SQS.