I help SaaS Companies retain whale clients
In this episode, Isaac and I try to answer the question "What should you do if you Never Rewrite?"
Listen today at https://www.neverrewrite.com/podcast/episode-two-iteratively-replace
Or Subscribe at:
Do you have a problem that requires a rewrite? Can't find another solution? Don't know where to begin?
Contact Isaac and I and we can help talk you through it!
I am very excited to announce, Never Rewrite, my new podcast with co-host Isaac Askew!
Each week will be discussing why you should resist the urge to start over and rewrite your software, and how you can evolve your existing system instead.
If you have a software quality, scaling or performance problem, and the only way forward seems to be starting over, we would love to have you on the show!
Listen at https://neverrewrite.com
The Ship of Theseus is a thought experiment about whether a ship that has slowly had all of its parts replaced is still the same ship.
My core tenant, Never Rewrite, raises the same question. If you replace your system, one line at a time, will you eventually have a new system?
When the last line of code from the original system is gone, is it the same one or a new one?
Does it make any difference to your customers?
Unlike the Big Bang rewrite of Yankee Stadium, Madison Square Garden is an iterative success story. MSG has been iteratively upgraded over the years, resulting in an ever more intimate and popular venue.
Suites have been added, and removed. The original 200 seating level was merged with the 100 level, the 300 and 400 levels were merged. Floors were raised to move fans closer. MSG has tweaked food options and added overlooks without needing to start over from scratch!
Developers still want to build a new arena down the street and knock Madison Square Garden down, but not because they want to build a better Madison Square Garden. Instead, the calls for a new arena are secondary to freeing up the land for new condos, office towers, and an improved Penn Station.
The developers may end up building a new Madison Square Garden. If they do, their pitch will be about freeing the land to build something new; not “The Same, But Better”.
Hat tip to my friend Zach for pointing out that MSG is an example of iteration. Details for this post came from the Madison Square Garden article on Wikipedia
My core development tenant is Never Rewrite. You will be better off iteratively fixing and improving anything that is providing value to your customers. It is common for developers and managers to say that a rewrite will fix all problems and make new development easier. These claims usually come with the added benefit of some exciting new technology.
The odds of a rewrite being better for your business and your customers are vanishingly small. Sometimes though, a team manages to deliver on a big bang rewrite. Are the results better for the business and customers?
The Yankees, yes the baseball team, executed a real life big bang switchover. They built a new stadium on a literal green field, across the street from the original stadium. The team moved over, destroyed the original, and replaced the green field.
I can’t imagine a more perfectly executed big bang cutover.
But, were the results better for the business and customers?
The Yankees Organization completed a remodel of Yankee Stadium in the late 1970s and immediately decided that their baseball stadium was too difficult to upgrade. The team rejected multiple modernization attempts in the 1980s and 1990s, only a new stadium would do! Eventually, in 2006, the Yankees were allowed to build a new stadium on parkland across the street from the original stadium.
Baseball fans, the customers, went without serious upgrades for over 25 years. The local community lost their park for 5 years while the big bang was underway.
Surely though, with 25 years of planning and an unlimited budget, the new stadium was a marvel for the fans! Nope, the seating was further from the field, the bleachers had obstructed views, and the best seats were so expensive that they went empty, even in playoff games.
But it made for great baseball right? No again, the wall slope encouraged home runs, the spread out seating killed the noise of the crowd, and the empty premium seats also blocked the fans from getting autographs.
What did they get right? Better food, more elevators, more bathrooms, and bigger seats. Oh! And everyone has a cupholder now.
Was it worth making the fans wait 25 years for cup holders?
Would iterating on seat design, restaurants, bathrooms and elevators over 25 years have been better for everyone?
Even when a big bang is successful, the results won't be better for the business and customers.
Details of the Yankee Stadium history come from this wikipedia page
My last three posts laying out a brief history of User Defined Field Patterns past, present, and future. This post lays out a framework for migrating. If your CRM is using the Relational or NoSql pattern and you’re ready to move to the more efficient, cheaper, and simpler future, this is the post for you!
Before going into how to migrate, a reminder of my philosophy:
There are many ways to migrate from one pattern to another. This strategy will minimize risk and maximize customer value.
The Relational and NoSql patterns make use of a relational database.
Step 1 is to add a JSON column to your existing contacts column.
Your new schema should look like one of these two models
That’s it - deploy an additive schema update to your database. Since there’s no code to access the new columns, there’s no coordinated deployment. Just the regular, minimal, risk of updating your database.
Now that the new schema is in production, it is time to extend your query code.
Add a new code path that checks to see if any data is present in the new schema. If there is data available, execute the query using the new JSON column. When there’s no data, use the original query method.
You will need to develop this code hand-in-hand with the code for migrating the data from your original system to the new schema. The important piece is that you should always be deploying with the READER code on, WRITER code off.
When you deploy this code, there won’t be any data in the JSON column. The new code will be available, but unused.
Since the new code won’t be used, this step is also extremely low risk.
At this point your system will use the new schema whenever data is present.
This gives you a single switch to flip - to use the new system, start writing to the new column IN ADDITION to the original method.
Mistakes at this step are the most likely to cause customer impact! It is also the most expensive in time and resources because you are writing the data twice.
However, this also gives you a very quick fallback path. The original writing process is untouched!
If there’s a problem, turn off the double write and delete the data in the new column. Thanks to the work in Step 2 you’ll instantly fall back with NO DATA loss.
Migrations are hard! Preventing data loss minimizes the risk.
The final step is to stop writing to the original data store. This ends your ability to fall back so make sure to confiscate copies of the data deleter from Step 3!
Ending the double write should be low risk because you were only doing it as a fallback at this point. You should see an immediate bump in performance and drop in costs. This trend will continue as the data migrates from the old system to the new.
At some point you’ll be ready to shut down the old system.
The last step is to decide what to do with the unmigrated data. Depending on how long you’ve waited you’re looking at customer data that hasn’t been accessed in months. Look at your retention promises; maybe you don’t have to migrate the data at all.
Either way, clean it up and shut down the old system at your leisure.
You can migrate User Defined Field code to the latest pattern with very little risk by using the 5 step strategy laid out in this article.
The Hybrid Solution offers excellent scalability and performance for reasonable costs. If your CRM is using one of the earlier patterns, it is time to start migrating.
Take control of the process with small, low risk, steps and never rewrite!
Part 2 covers how NoSQL emerged as an improvement over the classic relational database solution for User Defined Fields. NoSQL delivers speed and scalability by being expensive and fragile. In part 3 I’m going to cover the emerging Hybrid Database solution for User Defined Fields.
Hybrid Databases allow you to combine the best aspects of the relational and NoSQL models, while avoiding most of the downsides.
A hybrid implementation looks like this:
The hybrid model brings the data back to a single server, but without the Contact->Field relation. Instead the field data is stored as a JSON object in the Contact table itself.
If your SaaS is implementing User Defined Fields from scratch today, go with the Hybrid model. If you already have the classic or NoSQL pattern in place, it’s a good time to start thinking about how to evolve towards a hybrid solution.
I’ll cover how to evolve your existing solution in Part 4.
In part 1 - I covered the classic solution for User Defined Fields; simple but unscalable.
NoSQL emerged as a solution to relational fields in the late 2000s. Instead of having a meta table defining fields in a relational database, the User Defined data would live in NoSQL.
The structure would look like this:
This model eliminates the meta programming and joining the same table against itself. The major new headache that this model creates is difficulty in maintaining the integrity of the field data.
NoSQL systems solve the scaling problems with setting up User Defined Fields in a relational database. The scaling comes with high costs in terms of complexity, fragility and costs.
Reducing the complexity, fragility, and costs leads to the upcoming 3rd shift, covered in part 3.
This series covers a brief history of the 2 historic patterns for implementing User Defined Fields in a CRM, the upcoming hybrid solution that provides the best of both worlds, and how to evolve your existing CRM to the latest pattern. If you care about CRM performance, scaling, or cost, this series is for you!
Every CRM provides a basic fields for defining a customer. Every CRM’s basic field set is different depending on the CRM’s focus. So, every user of a CRM needs to expand the basic definition in some way. Birthdays, purchase history, and interests are three very common additions.
The trick is allowing users to define their own fields in ways that don’t break your CRM.
At a high level, there have been three major architectures for implementing Custom Fields. Most of the design is driven by the strengths and weaknesses of the underlying database architecture.
Pattern 1, generalized columns in a database, spanned the dawn of time until the rise of NoSQL around 2010.
Pattern 2, NoSQL, began around 2010 and continues to today.
Pattern 3, JSON in a relational database, began in the late 2010s and combines the best of the two approaches
Before the rise of NoSql there was pretty much one way to build generic user defined fields.
The setup is simple, just 3 tables. A table of field definitions, a table for contacts, and a relational table with the 2 ids and the value for that contact’s custom field.
The classic relational database pattern is easy to set up, but has terrible scaling. This super simple example would bog down by 1,000 contacts and 50 fields.
There are lots of ways to redesign for scale, but this is a SHORT history. Suffice it to say that it takes extremely complex and finicky systems to scale past 100,000 contacts and 1,000 fields.
The solutions to the classic pattern’s scaling led to the NoSQL revolution, covered in part 2.
This weekend I cheerfully went to lose at a fencing tournament.
That’s not negativity, I’ve been fencing off and on for 30 years and I just started up again after a multi-year hiatus. I fenced terribly, which I already knew from practice. What I didn’t know, what practice couldn’t tell me, is where I would break down when I fenced for real.
I was just competent enough at the things I knew to watch for, that my opponents destroyed me in ways I didn’t even remember to practice.
What does this have to do with software and my mantra of Never Rewrite?
After years of not doing maintenance, I am a legacy system. Practice is literally a testing environment. If I had spent months working on my known problems, when I finally got to production, I would still have been destroyed by my blind spots.
You can choose to spend months rewriting only to have your system fail when you finally make it to production.
Or, you can choose to cheerfully go to production with what you’ve got so that you can see what breaks and iterate.
