Continuing the discussion of the data loading and messaging, this post is going to cover the tradeoffs to consider when working with Topic and Broadcast based systems.
Directly Consuming Topics Isn’t Really A Thing
Topics, Pub/Sub, and other one-to-many messaging systems are an extremely interesting and important abstraction, but they don’t overlap with a discussion of data loading. The message broker will either use a queue for each client, which is covered here (link), or it is a broadcast, without guarantees.
Broadcast
Broadcast based systems come without guarantees. The broadcast server sends the message with no idea about how many listeners want the message, or how many that want it, get it. There is also no guarantee that messages will arrive in order.
These systems are designed to minimize the latency and push all of the queuing onto the receiver. This is when you need to understand your processor’s OSI Layer, implementation. It’s the rare time that the amount of memory on the network card might be important!
But, for the purposes of message processing and data loading, all of those details will be abstracted away.
To the application, things look very similar to running off a queue. The application has an ordered list of messages, and it needs to process them as quickly and consistently as possible.
Broadcast doesn’t have message visibility or the risk of double processing. Instead the pressure comes from internal queue and buffer management. The messages are on the processor, if they aren’t processed fast enough you will run out of memory.
Conclusion
Topic and Broadcast based messaging systems are extremely powerful systems. When it comes to data loading however, they tend to be a subset of queue processing.
The need for consistent performance will push your design to the lower right quadrant.
Messaging systems don’t change the fundamental tradeoffs involved in data loading, but they do add a lot of opinionated weight to the choices.
First, messaging and streaming have a lot of potential meanings and implementations. To overgeneralize, there are queues, where each message should be read once, topics, where each client should receive each message, and broadcast, where there are no guarantees about anything.
Queues and topics work as polling loops within your software. Broadcast messaging comes directly to the machine at the network level; the abstractions are highly dependent on implementation.
For the rest of this article I’m going to cover queues. I’ll cover topics and broadcast in future articles.
Loading Patterns with Queues
The main feature of a message queue is that you want each message to be processed exactly once. This requires clients claim messages when taking them off the queue, notifications back to the queue when messages have been processed, and a client timeout, after which a claimed message will reappear at the head of the queue.
A client has to connect to a queue, ask for some number of messages and indicate how long it will wait for the max number. For example, AWS’ SQS the defaults are 10 messages or 30 seconds. The client will block until 10 messages are available, or it has been waiting for 30 seconds.
Set the message request too big and you will wait while the queue fills. Set the timeout to long and the first few messages will get stale while the client waits. You need to balance the time cost of polling the queue against the cost of having your workers idle.
The main concern however, is visibility timeouts. If your client grabs 10 messages and has a 30 second timeout it absolutely must finish all 10 in under 30 seconds. After 30 seconds the unacked messages will be released to the next waiting worker and your message will be processed twice.
Optimizing worker counts, polling settings, and timeouts requires maximizing execution consistency. When working at scale, how long an action takes matters much less than how consistent the timing is.
If your system is drinking from a firehose, you need to push everything towards a Pre-Cache model. Pre-Cache pushes data loading out of the critical path, which gives the most consistent timing for message processing.
Having fully Pre-Cached queue processors is rarely possible, there is too much data and it changes to often. Read Through Caching is the practical alternative.
Link Tracking Example
Link Tracking, recording when someone clicks on a link, is a common activity for Marketing SaaS. I’m going to use a simplified version and walk through each of the data patterns.
At a high level, we have a queue of events with 4 pieces of data that describe the click event: Customer Name, URL, Email, and Timestamp.
We want to process each event exactly once, as quickly as possible, and we don’t care about the order messages are processed. However, we can’t insert directly, we need to normalize Customer Name, URL, and Email into customerId, urlId, and emailId.
Lazy Load
The nice things about the Lazy Load pattern is that it starts quickly and is simple to understand.
Unfortunately, every event requires 3 trips to the database to normalize the data. This adds load to the database and has highly variable processing time.
It’s a fine place to start, but a terrible place to stay.
Pre-Fetch
The Pre-Fetch pattern separates fetching the data into a separate step from execution. In this example, the worker would be attempting to normalize multiple events at the same time. Once an event has been fully normalized, it is inserted into the database.
Pre-Fetch adds a lot of complexity because the workers now have internal concurrency in addition working in parallel with each other. This might be worth it if the data was being composed from different resources in a micro-service architecture and the three calls could be done in parallel. In this example though, you would be pounding the database.
Pre-Cache
The great thing about a Pre-Cache model is that there are no reads against the database during processing. It is read-only during init, and write only from the queue. That will really help the database scale!
Pre-Cache is impractical in this use case though. We might know the full set of customers at startup, but urls and emails are open ended. Pure Pre-Cache setups require that all of the data be known before starting. That can work for things like file imports and trading systems, but not link tracking.
Read Through Cache
The Read Through Cache Pattern is likely to be the best option for our use case. It is more flexible than the Pre-Cache Pattern and much friendlier to the database than Lazy Loading or Pre-Fetching. It pushes complexity to the cache where there are lots of great solutions. Most languages have internal caching mechanics, and external caches like memcached and redis are widely supported.
Conclusion
Reading from message queues is a common problem and usually requires composing data from a database. What’s the best way to load the data? As always, the right data loading pattern depends on a your conditions and assumptions.
Lazy Loading is always a decent place to start because it is simple. Performance, cost, and scaling constraints will push your design towards Read Through Caching and Pre-Caching. Pre-Fetch models are likely to be more effort than they are worth because they add a second source of concurrency and complexity.
Remember, requirements change over time, so will the best solution to your problem!
A new project arrives, you gather the team, and start planning. You talk through the requirements, write up the work, and put it in order. Then you iterate! Repeatedly work on the first task in the queue and keep going until the queue is empty.
Iterating off of a queue sounds iterative, it’s right there in the description. But it misses the mark because it ignores everything learned from implementing each piece of work.
Instead of imagining a work queue, envision a heap. With a heap, you are only guaranteed that the top element is most important. That’s it, the top, and the rest. Critically, when you take the top piece of work, you have to sort the heap. A heap forces you to consider what you’ve learned and apply it to determine what comes next.
An iterative process requires constant re-evaluation. There’s the current most important step, and a heap of work. Iterating off a perfect work queue is a sirin song, an illusion and a trap. To iterate, learn, and apply your learning, after every step.
The Four Patterns Of Data Loading are about two main trade offs: simplicity for performance, and freshness for execution consistency.
This may seem odd because the quadrants are defined by loading and caching strategies, not simplicity, performance or execution consistency.
Simple or Performant
The decision to use caching is about trading simplicity for performance. You can simply load the data every time you need it. If you’re using MySql on AWS, a basic query will take about 2ms to return. The pattern is very simple and self contained: load data when needed.
Caching, saving data for reuse, improves performance by reducing the time it takes to use the data again. In exchange, you have to think about your code and determine:
Will I use the data again?
Is the data likely to change in the DB while I have it cached?
If the data does change, do I want to use the latest version or the version that the process has been using so far?
How much server memory will I need for the cache?
Example - Adding a Tag to a Contact
Imagine a simple operation, adding a tag to a contact. The tag is a string and the contact is represented by an email address. You need to transform the tag and email into ids and store them in a normalized database table. For simplicity's sake, let’s say all DB operations take 2ms.
There are 3 DB Operations
Load the contactId based on email
Load the tagId based on tag
Insert into contact_tags
With the On Demand access pattern, we do each action every time. This requires 3 trips to the DB for 6ms.
Similarly, with the Pre-Load pattern, we spend 2ms pre-loading the tagId, and each operation takes 4ms.
Using a Read Through Cache, we store the tagId after the first load. The first operation takes 6ms and each additional operation takes 4ms.
Finally, with the Pre-Cache pattern, we spend 2ms pre-loading the data and each operation takes 4ms.
1 Tag, 1 Contact
1 Tag, 10 Contacts
10 Tags, 10 Contacts
Init
Exec
Init
Exec
Init
Exec
On Demand
0ms
6ms
0ms
60ms
0ms
600ms
Pre-Load
2ms
4ms
20ms
40ms
200ms
400ms
Read Through Cache
0ms
6ms
0ms
42ms
0ms
420ms
Pre-Cache
2ms
4ms
2ms
40ms
20ms
400ms
Freshness or Execution Consistency
The next tradeoff to consider the value of fresh data vs execution time consistency. This goes beyond questions of caching, it also affects whether you can use the Pre-Load strategy at all. A big advantage of the Pre-Load and Pre-Cache strategies is that the execution time is lower and less variable.
Stock trading software is designed to pre-load as much data as possible and can spend minutes initializing so that the actual buying and selling happens in microseconds. Similarly, internet ad networks like Google’s demand responses in 100ms or less. Having consistent execution times in each piece of your software makes it much easier to monitor performance for signs of trouble.
Security software and reporting sit on the other end of the spectrum. It doesn’t matter if a user had permission 5 minutes ago and everyone hates waiting for report data to update. In these cases the variance for each response is much less important than getting the most recent data.
Some data never changes once it has been created. In the example above of adding a tag to a contact, both tagId and contactId will never change during your program’s execution. Generally, anything with ‘id’ in the name is safe to cache. On the other hand counts, permissions, and timestamps change all the time.
Strategies can be good for some situations and terrible for others. Sometimes it depends on expectations vs money.
Ids and static data
Permissions
Counts and Reporting
On Demand
Bad
Good
Good, until it doesn’t scale
Pre-Load
Good
It depends on time elapsed
It depends on time and money
Read Through Cache
Good
It depends on time elapsed
It depends on time and money
Pre-Cache
Best
Bad
It depends on time and money
Conclusion
The “right” data loading pattern is a moving target. Remember that in the beginning load is low and there are continuous changes. Simplicity is always a great choice when there isn’t enough scale to justify complexity.
As software matures two trade offs emerge: Simplicity vs Complexity and Freshness vs Consistency.
You’re changing the software for a reason. When you consider the tradeoffs it should become clear which patterns will help solve your problem.
This is a real piece of mail from a real dental insurance company. I blacked out the PII and company name, and added the highlighting.
I understand how this can happen.
I’m sure they started with the Short-Term disability benefits template and then spent all their time getting the dental benefit information to print correctly.
It wasn’t until it was too late that they actually paid attention to the address page. That’s when they discovered that the address page is hard coded to say SHORT - TERM DISABILITY BENEFITS.
Tech, product and legal got together and decided that “Please disregard” was the best that they could do.
Please disregard - Integration came too late in the project.
Please disregard - Our software isn’t flexible and our deadlines are tight.
Please disregard - This was the best we could do.
Explanations: Management doesn't care about quality.
Rewrites often start off with the assertion that the current code is buggy and can’t be fixed. The replacement code will fix all of the problems because the technology will be better, the devs have improved, or for any other reason.
The rewrite project will get off to a great start, but before production, it will run into The Two Clock Problem.
What is the Two Clock Problem
The Two Clock Problem occurs when you have two clocks that run at different speeds. One clock must be wrong. With only two clocks, you can’t tell which one is wrong. Even worse, the second clock can also be wrong.
Rewrites Have Two Clocks
The first clock is the original system. The second clock is the rewritten system. They produce different answers. The original system was deemed unreliable, so different answers should be a good thing. Except that different doesn’t mean correct, it means different. Both systems could be wrong. Sometimes, the original system is correct.
Two Clocks and No Confidence
The two clock problem kills rewrites because there is no way to gain confidence in the new system. Customers will see that the data has changed, but without an explanation there won’t be trust or confidence.
Finding the explanations requires finding and fixing all the differences. Fixing bugs, differing definitions, and edge case logic until the two systems return the same result.
Finally, when your two clocks agree, then you are ready to release the rewrite.
Except by that point you don’t need the rewrite anymore.
Years ago, Patrick McKenzie, wrote an article titled Falsehoods Programmers Believe About Names. The article inspired a short lived burst of other programming falsehoods. This is a very late entry in the genre, covering incorrect assumptions about software projects.
All of these assumptions are wrong. Try to make fewer of these assumptions when working on projects. Since you are always working on projects at work, always be questioning your assumptions about projects.
Projects have defined beginnings.
Maybe not formal beginnings, but there is a point when you are supposed to start work.
Your manager knows when you should start working.
You can use your priorities to determine when you should start working.
You should be working on the project because you were asked.
You should not be working on the project because you weren’t asked.
Projects have defined endings.
Successful projects have endings.
Failed projects have endings.
The project will solve the problem.
The project will solve a problem.
The project won’t make the problem worse.
Everyone on the project agrees on what problems the project is supposed to solve.
Everyone agrees about what solving the problems means.
Solving the problem will make the project a success.
Not solving the problem will make the project a failure.
There is a relationship between the project’s success and the status of the problem.
The software you are asked to write will solve the problem.
The software you are asked to write will make the project a success.
Writing the software you are asked to write means you are doing a good job.
At best, projects are best guesses by well intended people. At their worst, projects can become meaningless busywork that is completely unrelated to any problems or desires at a company. The fewer false assumptions you buy into the more effective you will be.
The hard part is convincing people that what they’re reaching for is a hot stove.
Having developers work for 6 months and then do a single release is appealing because development proceeds in the straightest line over the shortest distance. When I push for small releases and iteration, the main pushback is that the incremental releases add work. The tradeoff appears to be one release after 6 months, or 7 monthly releases.
Seven months is more than six months, so iteration is more work!
The problem is that the single release after 6 months of work is “hot stove”. It’s a plan that needs the requirements to be perfect at the start. Nothing new can be learned during the project. Customers and the market have to remain exactly the same. It’s a plan that will leave you burned.
It isn’t six months versus seven months. It is 6 months, plus 3-6 months of rework versus seven months.
Explaining the risks is ineffective. People will agree that if the plan is wrong, that if we learn new things, or if the market changes then of course a 6 month plan will end up delivering the wrong thing. Of course you will then need another 3 months to correct. That’s obvious.
After agreeing about the risk, they will then declare that those risks don’t apply to THIS project. This project isn’t at all like those “hot stoves”. This project plan is correct and everything has been considered.
Describing the danger doesn’t work if the problem doesn’t resonate with the listener. Instead of urging people not to touch hot stoves, work on helping them identify the hot stove.
Instead of talking about the risks of a single release, talk about the value of iteration.
Don’t talk about the likelihood that the new workflows aren’t perfect, talk about the value of early customer feedback.
Don’t talk about the risk that the 3rd party system doesn’t really work like their spec, talk about the value of early integration.
Don’t talk about holding back valuable features for a gigantic release, talk about delivering value to customers.
Once people see the risk and the tradeoffs, you won’t have to convince them not to burn themselves. After all, everyone understands that you shouldn’t touch hot stoves.
