Continuing the discussion of the data loading and messaging, this post is going to cover the tradeoffs to consider when working with Topic and Broadcast based systems.

Directly Consuming Topics Isn’t Really A Thing

Topics, Pub/Sub, and other one-to-many messaging systems are an extremely interesting and important abstraction, but they don’t overlap with a discussion of data loading.  The message broker will either use a queue for each client, which is covered here (link), or it is a broadcast, without guarantees. 

Broadcast 

Broadcast based systems come without guarantees.  The broadcast server sends the message with no idea about how many listeners want the message, or how many that want it, get it.  There is also no guarantee that messages will arrive in order. 

These systems are designed to minimize the latency and push all of the queuing onto the receiver.  This is when you need to understand your processor’s OSI Layer, implementation.  It’s the rare time that the amount of memory on the network card might be important!

But, for the purposes of message processing and data loading, all of those details will be abstracted away.

To the application, things look very similar to running off a queue.  The application has an ordered list of messages, and it needs to process them as quickly and consistently as possible.

Broadcast doesn’t have message visibility or the risk of double processing.  Instead the pressure comes from internal queue and buffer management.  The messages are on the processor, if they aren’t processed fast enough you will run out of memory.

Conclusion

Topic and Broadcast based messaging systems are extremely powerful systems.  When it comes to data loading however, they tend to be a subset of queue processing.

The need for consistent performance will push your design to the lower right quadrant.