SmileAndDial CRM was built around the concept that people are nothing but a number. Phone numbers were a fundamental constraint in the software; it was impossible to have a contact that didn’t have a number. With the introduction of TheeSeeShipping and months of Invisible Shipping SmileAndDial has been able to free themselves of this incremental constraint. Through many iterative releases the company has slowly been transforming itself.
The data model has gone from using Phone as a key in all tables.
To a normalized model with foreign keys that don’t carry extra meaning.
Changes were small. They were low risk and had no customer impact. There was little work in flight, and occasional priority shifts didn’t result in code rot. Their slow accumulation removed a fundamental constraint that was built into the company’s founding principles.
Happy Ending?
SmileAndDial CRM is still the go to CRM for boiler rooms, pump and dumps, extended car warranties, and other outbound call center activities.
Now, as if by magic, they are also ready to help their customers expand and spam prospects with ads for bitcoin, blockchain technologies, and of course, male enhancement pills.
Thankfully this is a fable and we can pretend that this is a happy ending!
Onward with SmileAndDial CRM, and their quest to remove phone numbers as a fundamental constraint! Part 2 introduced the concept of TheeSeeShipping, a low risk, iterative way to make large sweeping changes. At the end of part 2, the database looked like this:
New Id columns have been added, but they are unused and null.
Now SmileAndDial’s developers can TheeSeeShip small, incremental, low risk changes to production. These changes are invisible to users and can be worked in with day-to-day work.
TheeSeeShipping works like a ratchet; once a change is complete, it stays complete. Whether the work is done quickly or slowly, change is inexorable and one way.
The Second Series of Releases - Propagating ContactId
The second series of releases is to insert the new Contact.Id value into other tables’ ContactId value when rows are inserted.
This requires lots of small changes - getting the new Id after inserting into Contact and adding it to the insert statement in other tables. But, because the fields are nullable, there are no foreign keys, and no query logic; these changes can be done incrementally.
Fix one or two inserts every release. These are small, easily testable changes. Grab the Id, add it to the insert, confirm that the row was written with the correct ContractId.
TheeSeeShip it in low risk, two line changes until all new rows have a ContactId.
The next step in this series is to backfill the existing data. The Contact table has an Id for all rows. Run an update query for each table to backfill the data.
Guess what, since the data still isn’t used in the system this is also a low risk query.
Now every row in the database has a value for ContactId, and every new row will get a value on insert. The time has come to remove the nullability and add in the foreign keys.
The Third Series of Releases - Updating Selects
With ContactId propagated to every table in the database, the time has come to update the Select queries. Instead of selecting from every table by phone number, SmileAndDial needs to run off of ContactId, or join Contact to use Phone
Each of these changes can be made incrementally in a separate release. They are small, easy to test, and independent of each other. TheeSeeShip it; the data model is different, but to customers it is still SmileAndDial.
After any bugs or missing ContactIds get fixed, it is time to remove the foreign key constraint on the Phone field. Note that this can be done at any time that is convenient to the DB team. There’s no rush and no impact if a bug with ContactId does turn up.
The Fourth Series of Releases - Removing the Fundamental Constraint
Everything in the first three series of releases was incremental and non-breaking. Each change was minimal and non-breaking.
Even at the end of phase 3, the discovery of an unmodified table would have no impact on the system. The table and code would continue to work as it always had. Any mistakes were easy to roll back.
Everything had changed, and everything was still the same.
In the fourth series of iterations, Email will be removed as a fundamental constraint in the system. This, finally, is a breaking change that is not trivial to undo.
But, it is also a change that is easy to do in an incremental, low risk way.
Incrementally remove the Phone column from all tables except for Contact. At this point there are no foreign relations or Select statements that use Phone.
Finally, when Phone is gone everywhere else, make Contact.Phone nullable. Update the service and UI logic to allow the creation of a contact without a phone number.
Phone Is No Longer Fundamental
There are a lot of incremental changes and releases in the four series. The key is that each change was very small and very low risk. After each release customers had a fully functional, up to date, version of SmileAndDial.
Unreleased work in progress was low and work could have halted for months at a time without any of the completed work suffering from code rot.
TheeSeeShipping changed the fundamental nature of SmileAndDial without ever taking SmileAndDial out of the hands of its customers.
In part 2, I’m going to show you how to TheeSeeShip these changes to release small, low risk, bits of functionality and avoid a high risk, high lag rewrite.
The First Release Series - Adding IDs
The first series of incremental releases is to add new columns to the existing tables. The Contact table needs an auto-incrementing Id column that will eventually become the new, non-phone, primary key throughout the system. All the other tables need an int ContactId column that is nullable for now.
Because we are TheeSeeShipping, we are not making Id a primary key, allowing ContactId to be null, and not setting up any foreign relations. We will do all of those things over time, but doing them all at once makes the changes large and risky.
Instead these changes are entirely additive. The changes are also completely independent of each other, making it possible to update the schema one table at a time. Whether this goes out in one release or 4, they are low risk changes.
Data assumptions are baked into your CRM’s makeup and can seem impossible to change. Email marketing requires contacts to have an email, because you can’t do email marketing without one. Call center software requires prospects to have phone numbers so that agents can do outbound sales.
But what happens when your business needs to change and your fundamental constraints are no longer fundamental? How do you change your core data model assumptions without starting over or freezing development?
SmileAndDial CRM has spent years positioning themselves as the go to CRM for boiler rooms, pump and dumps, extended car warranties, and other outbound call centers on the strength of their dialer integrations. They’ve done well, making a quality product for horrible people. But the FTC is cracking down on junk calls and putting their customers out of business. They need to expand into email spamming and help support their horrid customers. After all, keeping your customers out of jail massively extends Customer Lifetime Value.
This series will follow SmileAndDial on their journey to remove phone numbers as a fundamental constraint in their software.
I cohost a podcast devoted to the idea that starting over and rewriting your system is a mistake that will lead to failure. But I have struggled with explaining the alternative, iterative replacement.
One commenter summed it up as: Don’t rewrite, instead rewrite.
I’m inventing a new term, TheeSeeShip, to highlight the difference. Based off the Ship of Theseus, when you TheeSeeShip, you are iteratively replacing parts of the current system that are broken, don’t scale, or just aren’t useful anymore.
A rewrite creates a second system, with the hope of one day becoming the sole system. Until that day, you have the system you use, and an ever growing mass of untested work in progress.
When you TheeSeeShip, there is only ever one system and everything will be in production the whole time. Over time you’ll replace every line as you add and remove services, features, and scaling patterns. Everything changes, but the system remains.
The opposite of a Rewrite is to TheeSeeShip. TheeSeeShipping is lower risk, provides more value to your customers, and boosts morale. I’ll dig into why in the posts ahead.
Unlike the Big Bang rewrite of Yankee Stadium, Madison Square Garden is an iterative success story. MSG has been iteratively upgraded over the years, resulting in an ever more intimate and popular venue.
Suites have been added, and removed. The original 200 seating level was merged with the 100 level, the 300 and 400 levels were merged. Floors were raised to move fans closer. MSG has tweaked food options and added overlooks without needing to start over from scratch!
Developers still want to build a new arena down the street and knock Madison Square Garden down, but not because they want to build a better Madison Square Garden. Instead, the calls for a new arena are secondary to freeing up the land for new condos, office towers, and an improved Penn Station.
The developers may end up building a new Madison Square Garden. If they do, their pitch will be about freeing the land to build something new; not “The Same, But Better”.
Hat tip to my friend Zach for pointing out that MSG is an example of iteration. Details for this post came from the Madison Square Garden article on Wikipedia
My core development tenant is Never Rewrite. You will be better off iteratively fixing and improving anything that is providing value to your customers. It is common for developers and managers to say that a rewrite will fix all problems and make new development easier. These claims usually come with the added benefit of some exciting new technology.
The odds of a rewrite being better for your business and your customers are vanishingly small. Sometimes though, a team manages to deliver on a big bang rewrite. Are the results better for the business and customers?
The Yankees, yes the baseball team, executed a real life big bang switchover. They built a new stadium on a literal green field, across the street from the original stadium. The team moved over, destroyed the original, and replaced the green field.
I can’t imagine a more perfectly executed big bang cutover.
But, were the results better for the business and customers?
The Yankees Organization completed a remodel of Yankee Stadium in the late 1970s and immediately decided that their baseball stadium was too difficult to upgrade. The team rejected multiple modernization attempts in the 1980s and 1990s, only a new stadium would do! Eventually, in 2006, the Yankees were allowed to build a new stadium on parkland across the street from the original stadium.
Baseball fans, the customers, went without serious upgrades for over 25 years. The local community lost their park for 5 years while the big bang was underway.
Surely though, with 25 years of planning and an unlimited budget, the new stadium was a marvel for the fans! Nope, the seating was further from the field, the bleachers had obstructed views, and the best seats were so expensive that they went empty, even in playoff games.
But it made for great baseball right? No again, the wall slope encouraged home runs, the spread out seating killed the noise of the crowd, and the empty premium seats also blocked the fans from getting autographs.
What did they get right? Better food, more elevators, more bathrooms, and bigger seats. Oh! And everyone has a cupholder now.
Was it worth making the fans wait 25 years for cup holders?
Would iterating on seat design, restaurants, bathrooms and elevators over 25 years have been better for everyone?
Even when a big bang is successful, the results won't be better for the business and customers.
My last three posts laying out a brief history of User Defined Field Patterns past, present, and future. This post lays out a framework for migrating. If your CRM is using the Relational or NoSql pattern and you’re ready to move to the more efficient, cheaper, and simpler future, this is the post for you!
Migration Philosophy
Before going into how to migrate, a reminder of my philosophy:
Minimize risk by taking small incremental steps
Focus on providing value to the customer
There are many ways to migrate from one pattern to another. This strategy will minimize risk and maximize customer value.
Step 1 - Extend Your Relational Database
The Relational and NoSql patterns make use of a relational database.
Step 1 is to add a JSON column to your existing contacts column.
Your new schema should look like one of these two models
That’s it - deploy an additive schema update to your database. Since there’s no code to access the new columns, there’s no coordinated deployment. Just the regular, minimal, risk of updating your database.
Step 2 - Query The New Schema
Now that the new schema is in production, it is time to extend your query code.
Add a new code path that checks to see if any data is present in the new schema. If there is data available, execute the query using the new JSON column. When there’s no data, use the original query method.
You will need to develop this code hand-in-hand with the code for migrating the data from your original system to the new schema. The important piece is that you should always be deploying with the READER code on, WRITER code off.
When you deploy this code, there won’t be any data in the JSON column. The new code will be available, but unused.
Since the new code won’t be used, this step is also extremely low risk.
Step 3 - Double Write
At this point your system will use the new schema whenever data is present.
This gives you a single switch to flip - to use the new system, start writing to the new column IN ADDITION to the original method.
Mistakes at this step are the most likely to cause customer impact! It is also the most expensive in time and resources because you are writing the data twice.
However, this also gives you a very quick fallback path. The original writing process is untouched!
If there’s a problem, turn off the double write and delete the data in the new column. Thanks to the work in Step 2 you’ll instantly fall back with NO DATA loss.
Migrations are hard! Preventing data loss minimizes the risk.
Step 4 - Only Hybrid Write
The final step is to stop writing to the original data store. This ends your ability to fall back so make sure to confiscate copies of the data deleter from Step 3!
Ending the double write should be low risk because you were only doing it as a fallback at this point. You should see an immediate bump in performance and drop in costs. This trend will continue as the data migrates from the old system to the new.
Step 5 - Clean Up
At some point you’ll be ready to shut down the old system.
The last step is to decide what to do with the unmigrated data. Depending on how long you’ve waited you’re looking at customer data that hasn’t been accessed in months. Look at your retention promises; maybe you don’t have to migrate the data at all.
Either way, clean it up and shut down the old system at your leisure.
Conclusion
You can migrate User Defined Field code to the latest pattern with very little risk by using the 5 step strategy laid out in this article.
The Hybrid Solution offers excellent scalability and performance for reasonable costs. If your CRM is using one of the earlier patterns, it is time to start migrating.
Take control of the process with small, low risk, steps and never rewrite!
