When your SaaS Scaleup can't scale, I can help
The only client experience you can offer clients is the one you have today. Yesterday’s experience is a memory, and tomorrow is a promise.
Delivering iteratively, baby step improvements to your client’s experience, makes the experience better today, tomorrow’s promise more believable, and memories of things improving.
Promising a moonshot keeps the only experience your client’s have exactly the same; and if it was a good one, you wouldn’t need a moonshot.
When your customers struggle with a non-intuitive interface, would they rather:
Wait 6 months while you work on an amazing redesign and brand refresh, or get weekly tweaks that make using your product easier and more intuitive?
Branding and color schemes are about you, improving the UI is about your customers.
Don’t put off customer steps while working on a moonshot like a branding refresh.
You become a Scaleup when your SaaS’s service offering becomes compelling and you start attracting exponentially more clients.
All at once you have a lot more clients, clients with a lot more data.
Solutions that support 1,000 clients buckle as you pass 5,000. Suddenly, 25,000 clients is only months away.
Services that support hundreds of thousands of transactions a day fall hopelessly behind as you onboard clients with millions of transactions.
You finally know what customers want. You quickly find the edges of your system. Money is rolling in from customers and VCs. You can throw money at the problems to literally buy time to find a solution.
But you’re faced with a looming question - moonshots or baby steps.
It’s not about you or your SaaS, it’s about your client’s outcomes.
Moonshots are appealing because they take you directly to where you need to be. Your system needs to scale 10x today and 100x next year; why not go straight for 100x?
Baby steps feel like aiming low because the impact on you is small. But it’s not about you! Think about the impact on your clients.
From a technology perspective, sending emails 1% faster is ::yawn::
But for your clients, faster emails means more engagement, which means more sales.
Would your clients rather have more sales this week, compounding every week for the next year, or flat sales for a year while you build a moonshot?
Clients who churn, or go out of business, won’t get value from the moonshot. Even if you deliver greater value eventually, your clients are better off getting some value now.
Are you delivering value to your SaaS or your clients?
Iterative Delivery is a uniquely powerful method for adding value to a SaaS. Other than iterative, there are no respectably named ways to deliver features, reskins, updates and bug fixes. Big bangs, waterfalls, and Quarterly releases fill your customer’s hearts with dread.
Look at 5 antonyms for Iterative Delivery:
If your customers used these terms to describe updates to your SaaS, would they be wrong?
Iterative Delivery is about delivering small pieces of value to your customers so often that they know you’re improving the Service, but so small that they barely notice the changes.
Don’t be overwhelming, erratic or infrequent - be iterative and delight your customers.
3 Signs Your Resource Allocation Model Is Working Against You
After 6 posts on SaaS Tenancy Models, I want to bring it back to some concrete examples. When your SaaS has a Single Tenant model, clients expect to allocate all the resources they need, whenever they want. When every client is entitled to the entire resource pool, no client gets a great customer experience.
Here are 3 signs your Resource Allocation Model is working against you:
This is a classic “noisy neighbor” problem. Each client tries to claim all the shared resources needed to do their work. This isn’t much of a problem when none of the clients need a significant percentage of the pool. When a large client comes along, it drains the pool, and leaves your small clients flopping like fish out of water.
When having multiple large clients in a cell affects stability, the short term solution is to migrate some clients to another cell. Large clients can impact performance, but they should not be able to impact stability. Moving clients around buys you time, but it also forces you to focus on smaller, less profitable clients.
This is advice that often pops up on SaaS message boards. Don’t try to run your job during the day, schedule it to run in the evening so it is ready for the morning. When clients start posting workarounds to your problems, it’s a clear sign of frustration. Your clients are noticing that performance varies by the time of day. They are building mental models of your platform and deciding you have load and scale issues. By being helpful to each other, your clients are advertising your problems.
Conclusion
These 3 issues have the same root cause; your SaaS’s operational data is mixed in with client data. If you have any of these three problems, the time has come to separate your data from the clients’.
Fixing these problems won’t be easy or quick!
The good news is that you can separate the data and change your resource allocation model in an iterative fashion. Start by pushing your job service across the tenancy line.
Get value and regain control one incremental step at a time, and never do a rewrite!
This is part 4 in a series on SaaS Tenancy Models. Parts 1 , 2 , and 3.
SaaS companies are often approached by potential clients who want their instance to be completely separate from any other client. Sometimes the request is driven by legal requirements (primarily healthcare and defense), sometimes it is a desire for enhanced security.
Often, running a Multi-Tenant service with a single client will satisfy the client’s needs. Clients are often willing to pay for the privilege of their account run Single Tenant, making it a potentially lucrative option for a SaaS.
A Cell is an independent instance of a SaaS’ software setup. This is different from having software running in multiple datacenters or even multiple continents. If the services talk to each other, they are in the same cell regardless of physical location.
Cells can differ with the number and power of servers and databases. Cells can even have entirely different caching options depending on need.
The 3 most common Cell setups are Production, Staging (or Test), and Local.
Cell architecture comes with a few distinct properties:
In part 3 I covered the difficulties in operating in a true Single Tenant model at scale. A Cell with a single client effectively recreates the Single Tenancy experience.
Few clients want this level of isolation, but those that need it are prepared to pay for the extra infrastructure costs of an additional Cell.
For SaaS without global services, a Cell model enables a mix of clients on logically separated Multi-Tenant infrastructure and clients with effectively Single Tenant infrastructure. This allows the company to pursue clients with Single Tenant needs, and the higher price point they offer.
The catch is that Single Tenant Cells can’t exist in an architecture with global services. If there is a single service that must have access to all client data, Single Tenant Cells are out.
If you are enjoying this series, consider subscribing to my mailing list (https://shermanonsoftware.com/subscribe/) so that you don’t miss an installment!
For SaaS with a pure Single Tenant model, infrastructure consolidation usually drives the first two, nearly simultaneous, steps towards a Multi-Tenant model. The two steps convert the front end servers to be Multi-Tenant and switch the client databases from physical to logical isolation. These two steps are usually done nearly simultaneously as a SaaS grows beyond a handful of clients, infrastructure costs skyrocket and things become unmanageable.
Considering the 5 factors laid out in the introduction and addendum - complexity, security, scalability, consistent performance, and synergy this move greatly increases scalability, at the cost of increased complexity, decreased security, and opening the door to consistent performance problems. Synergy is not immediately impacted, but these changes make adding Synergy at a later date much easier.
Why is this such an early move when it has 3 negative factors and only 1 positive? Because pure Single Tenant designs have nearly insurmountable scalability problems, and these two changes are the fastest, most obvious and most cost effective solution.
Shifting from Single Tenant servers and databases to Multi-Tenant slightly increases software complexity in exchange for massively decreasing platform complexity.
The web servers need to be able to understand which client a request is for, usually through sub domains like client.mySaaS.com, and use that knowledge to validate the user and connect to the correct database to retrieve data.
The difficult and risky part here is making sure that valid sessions stay associated with the correct account.
Database server consolidation tends to be less tricky. Most database servers support multiple schemas with their own credentials and logical isolation. Logical separation provides unique connection settings for the web servers. Individual client logins are restricted to the client’s schema and the SaaS developers do not need to treat logical and physical separation any differently.
The biggest database problems with a many-to-many design crop up during migrations. Inevitably, web and database changes will be incomparable between versions. Some SaaS models require all clients on the same version, which limits comparability issues to the release window (which itself can take days), while other models allow clients to be on different versions for years.
The general solution to the problem of long lived versions is to stand up a pool of web and database servers on the new version, migrate clients to the new pool, and update request routing.
The biggest risk around these changes is database secret handling; every server can now connect to every database. Compromising a single server becomes a vector for exposing data from multiple clients. This risk can be limited by proxy layers that keep database connections away from public facing web servers. Still a compromised server is now a risk to multiple clients.
Changing from physical to logical database separation is less risky. Each client will still be logically separated with their own schema, and permissioning should make it impossible to do queries across multiple clients.
Scalability is the goal of Multi-Tenant Infrastructure Consolidation.
In addition to helping the SaaS, the consolidation will also help clients. Shared server pools will increase stability and uptime by providing access to a much larger group of active servers. The client also benefits from having more servers and more slack, making it much easier for the SaaS to absorb bursts in client activity.
Likewise, running multiple clients on larger database clusters generally increases uptime and provides slack for bursts and spikes.
These changes only impact response times when the single tenant setup would have been overwhelmed. The minimum response times don’t change, but the maximum response times get lower and occur less frequently.
The flip side to the tenancy change is the introduction of the Noisy Neighbor problem. This mostly impacts the database layer and occurs when large clients overwhelm the database servers and drown out resources for smaller clients.
This can be especially frustrating to clients because it can happen at any time, last for an unknown period, and there’s no warning or notification. Things “get slow” and there are no guarantees about how often clients are impacted, notice, or complain.
There is no direct Synergy impact from changing the web and database servers.
A SaaS starting from a pure Single Tenant model is not pursuing Synergy, otherwise the initial model would have been Multi-Tenant.
Placing distinct client schemas onto a single server does open the door to future Synergy work. Working with data in SQL across different schemas on the same server is much easier than working across physical servers. The work would still require changing the security model and writing quite a bit of code. There is now a doorway if the SaaS has a reason to walk through.
As discussed in the introduction, a SaaS may begin with a purely Single Tenant model for several reasons. High infrastructure bills and poor resource utilization will quickly drive an Infrastructure Consolidation to Multi-Tenant servers and logically separated databases.
The exceptions to this rule are SaaS that have few very large clients or clients with high security requirements. These SaaS will have to price and market themselves accordingly.
Infrastructure Consolidation is an early driver away from a pure Single Tenant model to Multi-Tenancy. The change is mostly positive for clients, but does add additional security and client satisfaction risks.
If you are enjoying this series, please subscribe to my mailing list so that you don’t miss an installment!
In the first post on Saas Tenancy Models, I introduced the two idealized models - Single and Multi-Tenant. Many SaaS companies start off as Single Tenant by default, rather than strategy, and migrate towards increasingly multi-tenant models under the influence of 4 main factors - complexity, security, scalability, and consistent performance.
After publishing, I realized that I left out an important fifth factor, synergy.
In the context of this series, synergy is the increased value to the client as a result of mixing the client’s data with other clients. A SaaS may even become a platform if the synergies become more valuable to the clients than the original service.
Another aspect of synergy is that the clients only gain the extra value so long as they remain customers of the SaaS. When clients churn, the SaaS usually retains the extra value, even after deleting the client’s data. This organically strengthens client lock in and increases the SaaS value over time. The existing data set becomes ever more valuable, making it increasingly difficult for clients to leave.
Some types of businesses, like retargeting ad buyers, create a lot of value for their clients by mixing client data. Ad buyers increase effectiveness of their ad purchases by building larger consumer profiles. This makes the ad purchases more effective for all clients.
On the other hand, a traditional CRM, or a codeless service like Zapier, would be very hard pressed to increase client value by mixing client data. Having the same physical person in multiple client instances in a CRM doesn’t open a lot of avenues; what could you offer - track which clients a contact responds to? No code services may mix client data as part of bulk operations, but that doesn’t add value to the clients.
Sometimes there might be potential synergy, like in Healthcare and Education, but it would be unethical and illegal to mix the data.
Two of the factors, complexity and scalability, are generally invisible to clients. When complexity and scalability are noticed, it is negative:
A SaaS never wants a client asking these questions.
Security, Consistent Performance and Synergy are discussion points with clients.
Many SaaS companies can adjust Security concerns and Consistent Performance through configuration isolation.
Synergy is a highly marketable service differentiator and generally not negotiable.
As much as possible I’m going to treat and draw things as 2-tier systems rather than N-tier. As long as the principles are similar, I’ll default to simplified 2-tier diagrams over N-tier or microservice diagrams.
Coming up I’ll be breaking down single to multi-tenant transformations.
Why a SaaS would want the transformation, what are the tradeoffs, and what are the potential pitfalls.
Please subscribe to my mailing list to make sure you don’t miss out!
Recently, I’ve spent a lot of time discussing the evolution of SaaS company Tenancy Models with my colleague Benjamin. These conversations have revealed that my thinking on the subject is vague and needs focus and sharpening through writing.
This is the first in a series of posts where I will dive deep on the technical aspect of tenancy models, the tradeoffs, which factors go into deciding on appropriate models, and how implementations evolve over time.
There are 2 ideal models, single-tenant and multi-tenant, but most actual implementations are a hybrid mix.
In the computer realm, single-tenant systems are ones where the client is the only user of the servers, databases and other system tiers. Software is installed on the system and it runs for one client. Multi-tenant means that there are multiple clients on the servers and client data is mingled in the databases.
Pre-web software tended to be single-tenant because it ran on the client’s hardware. As software migrated online and the SaaS model took off more complicated models became possible. Moving from Offline to Online to the Cloud was mostly an exercise in who owned the hardware, and how difficult it was to get more.
When the software ran on the client’s hardware, at the client’s site, the hardware was basically unchangeable. As things moved online, software became much easier to update, but hardware considerations were often made years in advance. With cloud services, more hardware is just a click away allowing continuous evolution.
The main factors driving tenancy decisions are complexity, security, scalability, and consistent performance.
Keeping client data mingled on the servers without exposing anything to the wrong client tends to make multi-tenant software more complex than single-tenant. The extra complexity translates to longer development cycles and higher developer costs.
Most SaaS software starts off with a single-tenant design by accident. It isn’t a case of tech debt or cutting corners, Version 1 of the software needs to support a single client. Supporting 10 clients with 10 instances is usually easier than developing 1 instance that supports 10 clients. Being overwhelmed by interested clients is a good problem to have!
Eventually the complexity cost of running massive numbers of single instances outweighs development savings, and the model begins evolving towards a multi-tenant model.
The biggest driver of complexity is the second most pressing factor - security. Ensuring that data doesn’t leak between clients is difficult.
A setup like this looks simple, but is extremely dangerous:
Forgetting to include client_id in any SQL Where clause will result in a data leak.
On the server side, it is also very easy to have a user log in, but lose track of which client an active session belongs to, and which data it can access. This creates a whole collection of bugs around guessing and iterating contact ids.
Single-tenant systems don’t have these types of security problems. No matter how badly a system is secured, each instance can only leak data for a single client. Systems in industries with heavy penalties for leaking data, like Healthcare and Education tend to be more single-tenant. Single tenant models make audits easier and reduce overall company risk.
Scalability concerns come in after complexity and security because they fall into the “good problems to have” category. Scaling problems are a sign of product market fit and paying customers. Being able to go internet scale and process 1 million events a second is nice, but it is meaningless without customers.
Single-tenant systems scale poorly. Each client needs separate servers, databases, caches, and other resources. There are no economies or efficiencies of scale. The smallest, least powered machines are generally way more powerful than any single client. Worse, usage patterns mean that these resources will mostly eat money and sit idle.
Finally, all of those machines have to be maintained. That’s not a big deal with 10 clients, or even 100. With 100,000 clients, completely separate stacks would require teams of people to maintain.
Multi-tenant models scale much better because the clients share resources. Cloud services make it easy to add another server to a pool, and large pools make the impact of adding clients negligible. Adding database nodes is more difficult, but the principle holds - serving dozens to hundreds of clients on a single database allows the SaaS to minimize wasted resources and keeps teams smaller.
Consistent Performance, also known as the Noisy Neighbor Problem, comes up as a negative side effect of multi-tenant systems.
Perfectly even load distribution is impossible. At any given moment, some clients will have greater needs than others. Whichever servers and databases those clients are on will run hotter than others. Other clients will experience worse performance than normal because there are fewer resources available on the server.
Bursty and compute intensive SaaS will feel these problems more than SaaS with a regular cadence. For example a URL shortening service will have a long tail of links that rarely, if ever, get hit. Some links will suddenly go viral and suck up massive amounts of resources. On the other extreme - a company that does End Of Day processing for retail stores knows when the data processing starts, and the amount of sales in any one store is limited by the number of registers.
Single tenant systems don’t have these problems because there are no neighbors sucking up resources. But, due to their higher operating costs, they also don’t have as much extra resources available to handle bursts.
Consistent performance is rarely a driver in initial single vs multi-tenant design because the problems appear as a side effect of scale. By the time the issue comes up, the design has been in production for years. Instead, consistent performance becomes a major factor as designs evolve.
Initial forays into multi-tenant design are especially vulnerable to these problems. Multi-tenant worker pools fed from single-tenant client repositories are ripe for bursty and long running process problems.
Fully multi-tenant systems, with large resource pools, have more resilience. Additionally, processing layers have access to all of the data needed to orchestrate and balance between clients.
In this post I covered the two tenancy models, touched on why most SaaS companies start off with single-tenant models, and the major factors impacting and influencing tenancy design.
Single tenant systems tend to be simpler to develop and more secure, but are more expensive to run on a per client basis and don’t scale well. Multi tenant systems are harder to develop and secure, but have economic and performance advantages as they scale. As a result, SaaS companies usually start with single tenant designs and iterate towards multi-tenancy.
Next up, I will cover the gray dividing line between single and multi-tenant data within a SaaS, The Tenancy Line.
Pixel Tracking is a common Marketing SaaS activity used to track page loads. Today I am going to try and tie several earlier posts together and show how to evolve a frustrating Pixel Tracking architecture into one that can survive database outages.
Pixel Tracking events are synchronously written to the database. A job processor uses the database as a queue to find updates, and farms out processing tasks.
This design is governed by database performance. As the load ramps up, users are going to notice lagging page loads. Worse, each event recorded will have to be processed, tripling the database load.
You can relieve the pressure on the user by making your Pixel Tracking asynchronous. Moving away from using your database as a queue is more complicated, but critical for scaling. Finally, using Topics makes it easy to expand the types of processing tasks your platform supports.
Users are now completely insulated from scale and processing issues.
There is no database in the final design because it is no longer relevant to the users’ interactions with your services. The performance is the same whether your database is at 0% or 100% load.
The performance is the same if your database falls over and you have to switch to a hot standby or even restore from a backup.
With a bit of effort your SaaS could have a database fall over on the run up to Black Friday and recover without data loss or clients noticing. If you are using SNS/SQS on AWS the queue defaults are over 100,000 events! It may take a while to chew through the queues, but the data won't disappear.
When your Pixel Tracking is causing your users headaches, going asynchronous is your Best Alternative to a Total Rewrite.
