Isaac and Jeffrey discuss the topic of doing full rewrites when it comes to architecture. They explore a case where a client needed to move from a single EC2 instance to a more stable system with load-balanced instances. They discuss the parallels between rewriting code and rewriting architecture, and the challenges and risks involved.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
The Four Patterns Of Data Loading are about two main trade offs: simplicity for performance, and freshness for execution consistency.
This may seem odd because the quadrants are defined by loading and caching strategies, not simplicity, performance or execution consistency.
Simple or Performant
The decision to use caching is about trading simplicity for performance. You can simply load the data every time you need it. If you’re using MySql on AWS, a basic query will take about 2ms to return. The pattern is very simple and self contained: load data when needed.
Caching, saving data for reuse, improves performance by reducing the time it takes to use the data again. In exchange, you have to think about your code and determine:
Will I use the data again?
Is the data likely to change in the DB while I have it cached?
If the data does change, do I want to use the latest version or the version that the process has been using so far?
How much server memory will I need for the cache?
Example – Adding a Tag to a Contact
Imagine a simple operation, adding a tag to a contact. The tag is a string and the contact is represented by an email address. You need to transform the tag and email into ids and store them in a normalized database table. For simplicity’s sake, let’s say all DB operations take 2ms.
There are 3 DB Operations
Load the contactId based on email
Load the tagId based on tag
Insert into contact_tags
With the On Demand access pattern, we do each action every time. This requires 3 trips to the DB for 6ms.
Similarly, with the Pre-Load pattern, we spend 2ms pre-loading the tagId, and each operation takes 4ms.
Using a Read Through Cache, we store the tagId after the first load. The first operation takes 6ms and each additional operation takes 4ms.
Finally, with the Pre-Cache pattern, we spend 2ms pre-loading the data and each operation takes 4ms.
1 Tag, 1 Contact
1 Tag, 10 Contacts
10 Tags, 10 Contacts
Init
Exec
Init
Exec
Init
Exec
On Demand
0ms
6ms
0ms
60ms
0ms
600ms
Pre-Load
2ms
4ms
20ms
40ms
200ms
400ms
Read Through Cache
0ms
6ms
0ms
42ms
0ms
420ms
Pre-Cache
2ms
4ms
2ms
40ms
20ms
400ms
Freshness or Execution Consistency
The next tradeoff to consider the value of fresh data vs execution time consistency. This goes beyond questions of caching, it also affects whether you can use the Pre-Load strategy at all. A big advantage of the Pre-Load and Pre-Cache strategies is that the execution time is lower and less variable.
Stock trading software is designed to pre-load as much data as possible and can spend minutes initializing so that the actual buying and selling happens in microseconds. Similarly, internet ad networks like Google’s demand responses in 100ms or less. Having consistent execution times in each piece of your software makes it much easier to monitor performance for signs of trouble.
Security software and reporting sit on the other end of the spectrum. It doesn’t matter if a user had permission 5 minutes ago and everyone hates waiting for report data to update. In these cases the variance for each response is much less important than getting the most recent data.
Some data never changes once it has been created. In the example above of adding a tag to a contact, both tagId and contactId will never change during your program’s execution. Generally, anything with ‘id’ in the name is safe to cache. On the other hand counts, permissions, and timestamps change all the time.
Strategies can be good for some situations and terrible for others. Sometimes it depends on expectations vs money.
Ids and static data
Permissions
Counts and Reporting
On Demand
Bad
Good
Good, until it doesn’t scale
Pre-Load
Good
It depends on time elapsed
It depends on time and money
Read Through Cache
Good
It depends on time elapsed
It depends on time and money
Pre-Cache
Best
Bad
It depends on time and money
Conclusion
The “right” data loading pattern is a moving target. Remember that in the beginning load is low and there are continuous changes. Simplicity is always a great choice when there isn’t enough scale to justify complexity.
As software matures two trade offs emerge: Simplicity vs Complexity and Freshness vs Consistency.
You’re changing the software for a reason. When you consider the tradeoffs it should become clear which patterns will help solve your problem.
This is a real piece of mail from a real dental insurance company. I blacked out the PII and company name, and added the highlighting.
I understand how this can happen.
I’m sure they started with the Short-Term disability benefits template and then spent all their time getting the dental benefit information to print correctly.
It wasn’t until it was too late that they actually paid attention to the address page. That’s when they discovered that the address page is hard coded to say SHORT – TERM DISABILITY BENEFITS.
Tech, product and legal got together and decided that “Please disregard” was the best that they could do.
Please disregard – Integration came too late in the project.
Please disregard – Our software isn’t flexible and our deadlines are tight.
Please disregard – This was the best we could do.
Explanations: Management doesn’t care about quality.
Isaac and Jeffrey discuss the ‘two clock problem’ as a metaphor for software rewrites. They explain that adding a new system doesn’t improve understanding if you don’t know how the current system is wrong. They also emphasize that rewriting a system without understanding the current system is costly and may not be worth it. Instead, they suggest spending more time understanding and documenting the current system to mitigate the need for a rewrite.
Takeaways
Adding a new system doesn’t improve understanding if you don’t know how the current system is wrong.
Rewriting a system without understanding the current system is costly and may not be worth it.
Spending more time understanding and documenting the current system can mitigate the need for a rewrite.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
Rewrites often start off with the assertion that the current code is buggy and can’t be fixed. The replacement code will fix all of the problems because the technology will be better, the devs have improved, or for any other reason.
The rewrite project will get off to a great start, but before production, it will run into The Two Clock Problem.
What is the Two Clock Problem
The Two Clock Problem occurs when you have two clocks that run at different speeds. One clock must be wrong. With only two clocks, you can’t tell which one is wrong. Even worse, the second clock can also be wrong.
Rewrites Have Two Clocks
The first clock is the original system. The second clock is the rewritten system. They produce different answers. The original system was deemed unreliable, so different answers should be a good thing. Except that different doesn’t mean correct, it means different. Both systems could be wrong. Sometimes, the original system is correct.
Two Clocks and No Confidence
The two clock problem kills rewrites because there is no way to gain confidence in the new system. Customers will see that the data has changed, but without an explanation there won’t be trust or confidence.
Finding the explanations requires finding and fixing all the differences. Fixing bugs, differing definitions, and edge case logic until the two systems return the same result.
Finally, when your two clocks agree, then you are ready to release the rewrite.
Except by that point you don’t need the rewrite anymore.
Company culture can have a significant impact on the style of code delivery. A rigid and bottlenecked code review process can slow down code delivery and create frustration. Alternatively, an open and collaborative culture can lead to faster and more efficient code delivery. It is important to strike a balance between code quality and speed of delivery. Perfect code that doesn’t solve the problem is not useful, while imperfect code that is easy to change and improve can be valuable. When considering a job, it is important to ask about the company’s code delivery style and whether there is a culture of collaboration and openness.
Takeaways:
A rigid and bottlenecked code review process can slow down code delivery
A collaborative and open culture can lead to faster and more efficient code delivery
Striking a balance between code quality and speed of delivery is important
Imperfect code that is easy to change and improve can be valuable
When considering a job, ask about the company’s code delivery style and culture
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
In our new series ‘Fulfilling Fridays’, Jeffrey shares a fulfilling software project he worked on at Guaranteed Rate, a mortgage broker company.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
Isaac and Jeffrey continue to discuss falsehoods that programmers believe about projects. They explore the misconception that the project will solve the stated problem, highlighting instances where the delivered software did not actually solve the problem. They also discuss the importance of understanding how the user will use the software and the need for clear communication and feedback loops to ensure that the project is on track. They emphasize that the success of a project is not solely determined by the software, but also by factors such as the alignment of project goals and the availability of resources. Overall, they highlight the need for critical thinking and continuous communication to avoid common pitfalls in project development.
Takeaways:
The project may not always solve the stated problem, even if the software is delivered as requested.
Understanding how the user will use the software is crucial for building a useful and successful product.
Clear communication and feedback loops are essential to ensure that the project is on track and aligned with the user’s needs.
The success of a project is not solely determined by the software, but also by factors such as project goals and available resources.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
