After a few weeks, a customer’s first page of contacts will become static. Same with tags, lists, and every other object that your CRM supports. When the data on the first page becomes static, users stop seeing it entirely.
Instead the first page becomes muscle memory on the way to your user’s real actions.
How long do you make your customers wait to load a page of data objects that won’t even register in their minds? How many extra hoops do they have to jump through to get to the actions they want to take? How much slower is the process for your biggest customers?
Customers log in to take actions, not objects. Don’t waste their time showing data objects until you know enough context to show meaningful data.
Data scales, actions and attention don’t. You can wage a constant fight to scale your UI, or you can choose Actions over Objects, and avoid the issue entirely.
I cohost a podcast devoted to the idea that starting over and rewriting your system is a mistake that will lead to failure. But I have struggled with explaining the alternative, iterative replacement.
One commenter summed it up as: Don’t rewrite, instead rewrite.
I’m inventing a new term, TheeSeeShip, to highlight the difference. Based off the Ship of Theseus, when you TheeSeeShip, you are iteratively replacing parts of the current system that are broken, don’t scale, or just aren’t useful anymore.
A rewrite creates a second system, with the hope of one day becoming the sole system. Until that day, you have the system you use, and an ever growing mass of untested work in progress.
When you TheeSeeShip, there is only ever one system and everything will be in production the whole time. Over time you’ll replace every line as you add and remove services, features, and scaling patterns. Everything changes, but the system remains.
The opposite of a Rewrite is to TheeSeeShip. TheeSeeShipping is lower risk, provides more value to your customers, and boosts morale. I’ll dig into why in the posts ahead.
In part 1 - I covered the classic solution for User Defined Fields; simple but unscalable.
NoSQL emerged as a solution to relational fields in the late 2000s. Instead of having a meta table defining fields in a relational database, the User Defined data would live in NoSQL.
The structure would look like this:
This model eliminates the meta programming and joining the same table against itself. The major new headache that this model creates is difficulty in maintaining the integrity of the field data.
No complicated meta programming. Instead you write a filter/match function to run against the data in the Collection Of Fields.
No more repeated matching against the same table. Adding additional search criteria has minimal cost.
Open ended/internet level scaling. For a CRM or SaaS, the limiting factor will be the cost of storing data, not a hard limit of the technology.
Much more complicated to set up and maintain. Even with managed services supporting two database technologies doubles the difficulty of CRUD. Multiple inserts, multiple deletes, tons of ways for things to go wrong.
Without a relational database enforcing the data structure, poisoned or unreadable data is common. Being able to store arbitrary data collections means you’ll invariably store buggy data. You’ll miss some records during upgrades and have to support multiple deserializers. You will lose customer data in the name of expediency and cost control.
It’s more expensive. You’ll pay for your relational database, NoSQL database, and software to map between the two.
NoSQL systems solve the scaling problems with setting up User Defined Fields in a relational database. The scaling comes with high costs in terms of complexity, fragility and costs.
Reducing the complexity, fragility, and costs leads to the upcoming 3rd shift, covered in part 3.
This series covers a brief history of the 2 historic patterns for implementing User Defined Fields in a CRM, the upcoming hybrid solution that provides the best of both worlds, and how to evolve your existing CRM to the latest pattern. If you care about CRM performance, scaling, or cost, this series is for you!
What are User Defined Field Patterns?
Every CRM provides a basic fields for defining a customer. Every CRM’s basic field set is different depending on the CRM’s focus. So, every user of a CRM needs to expand the basic definition in some way. Birthdays, purchase history, and interests are three very common additions.
The trick is allowing users to define their own fields in ways that don’t break your CRM.
The Three Patterns
At a high level, there have been three major architectures for implementing Custom Fields. Most of the design is driven by the strengths and weaknesses of the underlying database architecture.
Pattern 1, generalized columns in a database, spanned the dawn of time until the rise of NoSQL around 2010.
Pattern 2, NoSQL, began around 2010 and continues to today.
Pattern 3, JSON in a relational database, began in the late 2010s and combines the best of the two approaches
Pattern 1 - All in a Relational Database
Before the rise of NoSql there was pretty much one way to build generic user defined fields.
The setup is simple, just 3 tables. A table of field definitions, a table for contacts, and a relational table with the 2 ids and the value for that contact’s custom field.
This design is extremely simple and can be implemented by a single developer very quickly.
Basic CRUD operations are easy and efficient.
Building search queries requires complicated techniques like metaprogramming.
Every search criteria results in a join against the ContactFields table. This results in an exponential explosion in query times.
The lack of defined table columns handicaps the database’s query optimization strategies.
The classic relational database pattern is easy to set up, but has terrible scaling. This super simple example would bog down by 1,000 contacts and 50 fields.
There are lots of ways to redesign for scale, but this is a SHORT history. Suffice it to say that it takes extremely complex and finicky systems to scale past 100,000 contacts and 1,000 fields.
The solutions to the classic pattern’s scaling led to the NoSQL revolution, covered in part 2.
I recently stepped into an elevator and saw this panel:
The panel was clean, full of high quality materials, and everything worked.
Quality is about more than the functionality, does this look like a quality panel?
Push a button and it lights up!
Sure, it might light up red, green or blue. And the light might be around the edge or in the center. And some of the buttons are flush with the mount, while others extend out; but that doesn't impact the light turning on.
There are 12 possible button implementations, and 5 of them appear randomly.
But when you push the button, the light turns on!
What does that have to do with SaaS Scaling?
No matter how excellent any individual endpoint implementation is, having an API with endpoints that work differently decreases the overall quality of your product.
Having a UI with mismatched widgets and styles increases the user’s cognitive load and decreases quality, even when the differences don’t change any functionality.
Consistency during the scaleup period can be difficult as multiple new teams spin up, but it’s critically important if you want a quality product.
But how do you get started? How do you shorten your stride from shooting the moon, to one small step?
The next series of posts is going to lay out my scaling iterative delivery framework. This site is about scaling SaaS software, and this framework works best if you want an order of magnitude more of what you already offer your clients. This isn’t a general framework, and it certainly isn’t the only way to get started with iterative delivery.
Work your way through these steps:
Pick a goal - 1 sentence, highly aspirational and self explanatory.
You become a Scaleup when your SaaS’s service offering becomes compelling and you start attracting exponentially more clients.
All at once you have a lot more clients, clients with a lot more data.
Solutions that support 1,000 clients buckle as you pass 5,000. Suddenly, 25,000 clients is only months away.
Services that support hundreds of thousands of transactions a day fall hopelessly behind as you onboard clients with millions of transactions.
You finally know what customers want. You quickly find the edges of your system. Money is rolling in from customers and VCs. You can throw money at the problems to literally buy time to find a solution.
But you’re faced with a looming question - moonshots or baby steps.
Moonshots Are About You, Baby Steps Are About Your Clients
It’s not about you or your SaaS, it’s about your client’s outcomes.
Moonshots are appealing because they take you directly to where you need to be. Your system needs to scale 10x today and 100x next year; why not go straight for 100x?
Baby steps feel like aiming low because the impact on you is small. But it’s not about you! Think about the impact on your clients.
From a technology perspective, sending emails 1% faster is ::yawn::
But for your clients, faster emails means more engagement, which means more sales.
Would your clients rather have more sales this week, compounding every week for the next year, or flat sales for a year while you build a moonshot?
Clients who churn, or go out of business, won’t get value from the moonshot. Even if you deliver greater value eventually, your clients are better off getting some value now.
Are you delivering value to your SaaS or your clients?
The Chestburster is an antipattern that occurs when transitioning from a monolith to services.
The team sees an opportunity to exact a small piece of functionality from the monolith into a new service, but the monolith is the only place that handles security, permissions and composition.
Because the new service can’t face clients directly, the Chestburster hides behind the monolith, hoping to burst through at some later point.
The Chestburster begins as the inverse of the Strangler pattern, with the monolith delegating to the service instead of the new service delegates to the monolith.
Why it’s appealing
The Chestburster’s appeal is that it gets the New Service up and running quickly. This looks like progress! The legacy code is extracted, possibly rewritten, and maybe better.
Why it fails
There is no business case for building the functionality the new service needs to burst through the monolith. The functionality has been rewritten. It's been rewritten into a new service. How do you go back now and ask for time to address security and the other missing pieces? Worse, the missing pieces are usually outside of the team’s control; security is one area you want to leave to the experts.
Even if you get past all the problems on your side, you’ve created new composition complexities for the client. Now the client has to create a new connection to the Chestburster and handle routing themselves. Can you make your clients update? Should you?
Remember The Strangler
If you want to break apart a monolith, it’s always a good idea to start with a Strangler. If you can’t set up a strangle on your existing monolith, you aren’t ready to start breaking it apart.
That doesn’t mean you’re stuck with the current functionality!
If you have the time and resources to extract the code into a new service, you have the time and resources to decouple the code inside of the monolith. When the time comes to decompose into services, you’ll be ready.
The chestburster gives the illusion of quick progress; but quickly stalls as the team runs into problems they can’t control. Overcoming the technical hurdles doesn’t guarantee that clients will ever update their integration.
Success in legacy system replacement comes by integrating first, and moving functionality second. With the chestburster you move functionality first and probably never burst through.