Every SaaS has background workers.  Processes that sync data between platforms, aggregate stats, run billing, send alerts, and a million other things that don’t happen through direct user interaction.  Every SaaS also has limited resources; servers, databases, caches, lambdas, and other infrastructure all cost money.  

Most SaaS go through three main phases as they mature and discover that queues are harder than they seem:

On Demand -> Homegrown database as a Queue -> 3rd party queue software

Whether driven by a database or proper queue, this high level system emerges:

Enter The Dream Of Priority

Because resources are limited and some jobs, and some customers, are more important than others, the idea of a Priority Queue will emerge.  

“There are hours of work on the queue, and an important batch of jobs needs to be done now!  If only some jobs can move to the front of the line.”

A Priority Queue seems like a great solution.  The processes that create work can set a priority, add it to the queue, and sorting will take care of everything!

Unfortunately, what happens is that only high priority work gets processed.  This is known as resource starvation. Low priority jobs sit at the back of the queue and wait until there are no high priority jobs.  

Priority Queues only give you two options: add enough workers to prevent queue starvation, or make the priority algorithm more complex.  Since resources are limited, engineers start getting creative and work on algorithms involving priority and age.

There is a much simpler solution.

Multiple Queues Have Priority Without Starvation

A Multiple Queue system is a prioritized collection of simple fifo queues.  Each queue also has a dedicated worker pool, which prevents starvation.  The key difference is that the workers will check the other queues when theirs is empty.

When all the queues have work, they behave like independent queues:

When the priority queue is empty, the priority workers pick up low priority tasks.

Multiple Queues solve priority in SaaS friendly ways:

• No resource starvation.  Some customers may be more important, but no customer ever gets ignored.
• No wasted resources.  High priority workers never sit idle waiting for high priority work.

Multiple Queues Push Complexity To Workers

Instead of having a complex Priority Queue, Multiple Queue systems push some complexity to the workers.  Workers have to know which queue is their primary, and the order of the remaining queues.

Multiple Queues Have More Adjustment Options

Priority Queues only had 2 adjustment options: add more workers or adjust the algorithm.  Multiple Queues allow much finer grained controls:

• The number of different queues.  Adding a new, super duper higher priority pool would not require a code change.
• The size of the worker pool for each queue.  Queues do not need to have the same size pools, and can be adjusted dynamically.
• The relative priority of the queues.  Priority becomes config, not an algorithm.

Conclusion – For SaaS Workers, Use Multiple Queues

Background workers are a common, critical, feature for most SaaS companies.  Resource constraints make it impossible to run all background jobs as soon as they are created.  Some jobs will have different priorities, which will require implementing either Priority Queues or Multiple Queues.  Priority Queues sound like the correct answer because they describe the problem, but create resource starvation and ever increasing complexity.  Multiple Queues are simple, safe from starvation, and much more effective for SaaS use cases.