SaaS Metrics Maturity Model - Part 2

This episode dives deep into the SaaS Metrics Maturity Model, exploring how companies can evaluate and improve their use of SaaS metrics.

1. Foundational Metrics: Emphasize the importance of having clear, well-documented definitions and calculations for metrics.
2. Trust in Metrics: Trust is built through consistency and continuous improvement in metric reporting and interpretation.
3. Strategic Alignment: Metrics should be directly linked to the company’s strategic goals to ensure relevancy and impact.
4. Data-Driven Culture: Cultivating a culture that prioritizes data-driven decision-making is crucial for operational success.
5. Future Trajectories: Use metrics to project future business trajectories and make informed strategic decisions.

1. Introduction

The hosts introduce the topic and set the stage for the discussion. They recap the previous episode’s insights on the problems and root causes in SaaS metrics usage. Ray Reich: "Last episode we talked about the 15 problems with the use of SaaS metrics." Dave Kellogg: "Let's dive into the five levels of the SaaS Metrics Maturity Model."

2. Laying the Foundation

The necessity of clear definitions and calculations for metrics is emphasized, including how these foundational aspects support accurate metric-driven management. Dave Kellogg: "You can't do metrics-driven management well without clear definitions and calculations." Ray Reich: "Most companies don't document their metrics adequately, which is puzzling."

3. Building Trust

Discussion on building trust through consistent metric templates and the continuous improvement of metrics reporting. Dave Kellogg: "Templates build trust, and continuous improvement shows commitment to accuracy." Ray Reich: "Trust is essential for metrics to be effective and believed by all stakeholders."

4. Strategic Linkage

Explains how to tie metrics directly to company strategy, enhancing the relevance and impact of the metrics used. Dave Kellogg: "Tying metrics to company strategy is crucial for operational alignment." Ray Reich: "Strategic linkage helps ensure that metrics contribute to larger business goals."

5. Cultivating a Data Culture

Focuses on creating a culture that values and utilizes data for decision-making. Dave Kellogg: "A culture of data-driven decision making is pivotal for long-term success." Ray Reich: "It's about making everyone in the organization numerate and metrics-aware."

6. Looking Towards the Future

Discusses using metrics to project and plan for future business trajectories. Dave Kellogg: "Use metrics to define long-term goals and the steps needed to reach them." Ray Reich: "It's about where we are heading, not just where we are."

1. Document Your Metrics: Ensure all metrics are clearly defined and calculations are documented.
2. Regularly Review Metrics: Implement regular reviews to maintain accuracy and relevance.
3. Align Metrics with Strategy: Continuously ensure that all metrics are aligned with the strategic goals of the company.
4. Promote Transparency: Be transparent about how metrics are calculated and used, building trust across the organization.
5. Educate Your Team: Regularly educate and train your team on the importance and use of metrics.
6. Encourage Feedback: Foster an environment where feedback on metrics is encouraged and valued.
7. Plan for the Long Term: Use metrics to plan strategic trajectories and prepare for future challenges.

Dave "CAC" Kellogg and Ray "Growth" Rike continue their discussion on the SaaS Metrics Maturity Model which includes the below five levels:

People

Ray Rike, Dave Kellogg

Companies

Benchmarket

Books

"The Crux" by Richard Rumelt, "Good Strategy Bad Strategy" by Richard Rumelt

Guest Name(s):

None

Content Warnings:

None

Dave Kellogg
Live from schedule, New York, it's SaaS talk with the Mettrix brothers, growth and CAC. And I'm growth, better known as Ray Reich, the founder and CEO of Benchmarket. And I'm Kak, better known as Dave Kellogg, independent consultant, eir at Baldrton Capital and the author of Kelblog. And together we are the Mettrix brothers. And we go together like a wink and a smile.

Ray Reich
Well, I hope that I'm actually this smile because I'm such a pleasant person. Dave. I'm the wink then. Okay. Okay.

So you, so now we could go wink and CAC. I like it. You're winking CAC. There we go. And no alcohol for once.

Dave Kellogg
In my analogy of the week, though. I must admit I do wink more at people after I've had a few alcoholic beverages. Dave. There you go, Ray. There you go.

Ray Reich
Okay, so, hey, last episode we talked about really the 15 problems with the use of SAS metrics and the five root causes of that, but it all led up to getting into more detail of the SAS metrics maturity model that you developed. So would you mind just right after we hear a word from our presenting sponsor, provide us the five levels and then we're going to level one. Okay? Yes, sir. SaaS talk is presented by Gainsight, the first digital customer platform, including customer success, management, product experience, customer communities and customer education.

Find out why more than 1500 companies, including SaaS leaders like Zoom, Atlassian and Okta, and hundreds of early stage startups, rely on gainsight to efficiently retain and expand existing clients through an integrated digital first post sales customer journey. Gainsight has affordable packages for younger companies and goes live in two to four weeks or less. Visit ww dot gainsight.com. Now back to the show. Okay, Dave, so let's get into the maturity model and why don't you start with those five levels again and then we'll dive right into level one.

Ray Reich
Sure. So as a reminder for folks listening in, this is effectively a second episode on the subject of the SAS metrics maturity model. This was the subject of my 2023 talk at SASTR, called basically the strategic use and abuse of SAS metrics, where I started out by talking about 15 SAS metrics problems. Then we looked at the five underlying causes of those 15 problems and then we introduced very quickly this notion of a SAS metrics maturity model, kind of a multilevel maturity model that kind of lets you do two things. One, say where am I?

Dave Kellogg
So you can do kind of where you are here analysis. And then two, help based upon where you are. Figure out an action plan for what you want to do next. So that's where we are. We'll talk about the five layers of the maturity model, and basically, layer one is the foundation.

So you can't do SAS metrics well, or metrics driven management well. If you don't have clear definitions for metrics, clear calculations, how we calculate them. Simple example, do we do LTv to CaC, smooth or unsmooth? Do we have a definition in LTV to CAC, how we're getting lifetime or using GRR, NRR or some other churn rate? Clear semantics.

So, so what do things mean when we say them? So, so that's the foundation layer. Layer two is trust, which is, do we trust these things? I have a couple of sayings. One is that templates build trust, so do we have good templates that we stick with?

The other key part of trust to me is continuous improvement. Are we always making it better? Because we're always going to find problems. But this finding a problem kind of go into the bit bucket? Or does finding a problem mean, oh, we've got something to go fix, and we fix it.

And people see that fixed in the next release of the dashboard. Level three, strategic linkage tying metrics to the company strategy. Level four, data culture. So, a culture of data driven decision making. And layer five, which I call trajectory, which is just once we get out of kind of day to day operations, or we're using metrics to define, over the long term what we're trying to build.

Ray Reich
Perfect. So let's go to level one. And just to make sure, you know, I read the episode briefings, when you put them together about laying the foundation. The first part of that, Dave, it really spoke to me, being on the SAS Metric Standards board, as you know, well, as one of our advisors. But it's the need to have those definitions and calculations you talked about well understood, documented and used consistently.

And one of the things I find all the time is most companies don't go to the length of documenting what they decide. Why do you think that is? I'm not sure. I don't actually know why. Look, my instinct, Ray, is that there's two ways to solve this problem rather than talk about why it exists.

Dave Kellogg
One, footnote, it. So every time in a deck, if you're building the deck of the template, every time a new metric is introduced, you just put a super tight definition in the footnote. It's actually good for two reasons. One, it forces you to come up with really short definitions, because when you make a document, the definitions tend to be like endlessly long, unnecessarily long, whereas if you're forcing yourself into a footnote, you end up making like Maritech does this. The definitions of maritech metrics on maritech public comps, they're like this long, they're teeny, and it's because they, you know, necessity is the mother invention.

They just didn't have much space, so they found a really compact way to define things. The other problem I think is, and this is going to kind of contradict the first one, when you use footnotes, the problem is you can get inconsistency across decks. So some people create a centralized document, but they didn't ever reference it. So that's bad. So for me, I'd rather have a footnote with a link to the centralized deck.

And by the way, I think the people who do this, right, this is actually almost a trust layer thing. When there is a problem, we go to the source and we fix it. So maybe there is a central document, maybe in that central document we do the definition, and maybe part of the definition is the short definition of the metric that we copy and paste onto a slide. So somebody doesn't need to see the central document. But if we're caught in a.

Wait a minute, are we accelerating sales commissions in the CAC or not? Right. First you might get that into the compact definition, but if you can't, if you have a link to the central definition, you can see it there. I tell you, when I was entering as a new executive often, and I would look at some of the dashboards and reports, it was so helpful for those companies that said, oh, here's a way, we're calculating CAC ratio. So at least I knew what the reports I was reading to get onboarded.

I like it just to be anal here, Ray. I like it better as a footnote than a PowerPoint slide note, because very few people see the slide notes and they just get lost. Totally get it. Now, you brought up a second part about laying the foundation, and that's to have clear understanding of semantic references, things like forecast, because I still remember, people say, well, my forecast is 3.2 million, but I didn't know if that was their best case or their probable case or their commit number. So you're saying it's really important to define those semantics right up front.

Yeah, absolutely. You can go to the forecast meeting to pick that example, and you're going to hear people use words like best case, forecast, commit, downside, upside. You're going to hear a lot of words. And the question is, do we have any common understanding of what those words mean? And at many, many companies, the answer is no.

It's just all kind of intuitive. And if you ask, you know, hey, what's, what do you think the probability is that we hit our commitment? You know, some sales guys will be 100%. That's what we do. We know.

And it's like, well, as a mathematician, this is not going to be 100%. I know that. So am I going to factor that at 90% or if it's not committed, but only forecast, is that a 0.7? And whose forecast is it and how is it calculated? So to me, just making sure we understand what the words mean, because most people know they need to define SAS metrics.

That's semi obvious. But the thing they often miss is, is all the words that go with them. Totally agree. The other thing I love that you highlighted in laying the foundation was denied. I used the words like instrument and automate, but you talked about the need to sometimes fundamentally rework the underlying systems.

Ray Reich
As an example, you mentioned p and L in the last episode. A lot of companies don't have subscription revenue separated out from services revenue. So it's hard to do things like CAC ratio on an ARR basis. So, yeah, can't do CAC, period. Right.

Dave Kellogg
If you don't have a subscription gross margin, and you can't really do CAC, payback, period. Totally agree. The other thing I'd say right here on the foundation. So, so look, it includes a definition. In calcs, most people get that.

It includes semantics, which can be subtle, like what does best case mean? It often includes reworking underlying systems. As we talked about, if your p and l is not the standard SAS structure, you can't even, you kind of blow up on the launch pad in doing SAS metrics, I'd say more subtly, it's about benchmarks. Do we agree what sources are going to use as benchmarks? What's applicable to our size, right, in terms of data sources, let alone the targets we're shooting for, which would be more of a trajectory issue.

And the thing I would suggest, Ray, I do have a concrete suggestion there, because I've done this a couple of times, is get a coalition of the willing to go do this. So typically it looks like me, maybe one other board member, the CFO, the Ops person, and that works as a committee. You do not want to define all this stuff in a board meeting, right. You want to take it offline and you want the coalition of the willing, the metrics committee, as I call it, to come back and say, hey, we've got people on the management team, we have people on the board. We've dove in pretty hard here, and we are now all comfortable that we agree on definition, semantics, et cetera.

Ray Reich
Dave, I know how much you love actually being involved in those metrics committees, because I do have you involved with the SAS metrics board. People love being on metrics committees with me, Ray, as you could attest. Yes, I can. Well, let's go to the next one, because you talk a lot about this, and it's the level two, and that's all about building trust. Yeah.

Dave Kellogg
So we need to have trust in this so we can make all these definitions and we write down the semantics, but people trust the numbers, and there's a lot of ways to erode trust. You know, first, subtle sleight of hand on definitions. Like, you know, look, ASC 606 amortizes sales commissions. If you're. I believe the spirit of the CAC ratio is not to use amortized commissions.

So you should, you know, how you define them will affect your trust. Right? Or if you. If you have csms who actually do some upsell and you don't include their cost in a CAC ratio, that. That erodes trust.

Right. It's not common sense. So. So having these definitions, making sure the definitions kind of make sense, that's trust building. Another form or another way to erode trust is through cherry picking.

Like, you just show the metrics that were good last quarter. So if you had good logo churn, you show logo churn in q one. If you had good ARR churn, you show ARR churn q two, and you're like, hey, the metrics are always changing. I wonder why. We used to care about logo churn and now we care about dollar churn.

And if you excavate that and find out the reason is we're just cherry picking. Like somebody in finance is putting 37 candidates up and we pick the five best ones that we most want to talk about that erodes trust. So that's why I say templates build trust. If you always show the same template with history, right, with maybe the forecast for next quarter, and it's always the same metrics, then I know you're not playing any games. Hey, Dave, I gotta highlight one example, and you're the one who provided this.

Ray Reich
I think it was a blog or one of the presentations, the CRO who shows up to the quarterly board meeting and says, oh, our total arrangement goal was 38.8 million, and we're at 38.9, so we beat it. But that same CRO forgot to mention that new ARR, misplanned by 25%. That's cherry picking. Yeah, ending. You know how they say, well, I can't remember who said it, but patriotism was the last refuge of a scoundrel.

Dave Kellogg
Like, to me, ending ARR is the last refuge of a scoundrel because it's inherently damped. Right. If you start out with 20 million of ARR in the bucket and you're supposed to sell three and you come up at two against the three, you've done a 66% performance on New ARR, but you'll have done 22, 20 thirds, which has got to be coming up on 96% of ending ARR. So, yeah, that's another great. Another one.

Beware of Greeks bearing gifts, the trojan horse. Beware of sales vps talking about ending ARR. To me, it's a giant red flag when I hear people talking about it. But it does happen, and it is done in this context of cherry picking. And I think cherry picking erodes trust.

So the other two things are. Trust are one footnotes. Just never hide exceptions. Always address them. Don't leave out, oh, churn was 200k, except for the 500k from the one big account.

Right? Like, thanks for playing. Churn was seven hundred k. And put a footnote, you know, 500 if it came from one big account. So we think it's not repeatable, but don't play those games.

And then cadence is part of templates as well, which is what gets presented when. Right. Because you do, you should look at different things on a weekly basis than a quarterly basis, then at a board meeting. But. But they should be consistent across the meeting type.

Ray Reich
Now, the other thing to build trust is that it's not just a one time event, that there's continuous focus and improvement quarter over quarter, and that you're fixing data to source. Is that part of level two, Dave? Yeah, it is. I think continuous improvement is a really key part of level two, which is there's always going to be problems. Right.

Dave Kellogg
We're always going to have screwed up a definition, screwed up a calculation. It happens. But what I want to see when that happens, I want to see somebody go to the root sauce document and fix the definition. This is the official document. We are now updating it to say we accelerate commissions to CAC ratio.

And I want to see it in the template when I come back to the next board meeting. Specifically, I want to see the template improved to fix whatever the problem was. I like concrete examples. If we were smoothing LTV to CAC to make it look better, which is a very common trick, which I can't stand, then I want to see the next version of the template, come back with an unsmoothed one. My answer is always, I can smooth it myself, thanks.

It's very hard to unsmooth if you show me a smooth metric. Hey, Dave, would you do me a favor? Would you just define what you mean by smoothing? Because when I first heard it, I'm like, I wonder if I'm defining it the same way. So here's the best practice of defining a term.

So look, smoothing is, is going to be taking a trailing twelve month period. For example, if you wanted to do LTV to CAC, you can do it either in the current quarter, right? So what do we spend sales, marketing last quarter to get the CAC part the bottom, and then LTV, you'd say, well, what was the average ARR for new accounts last quarter? I know you'd use all accounts, but you say, what was the average r for new accounts last quarter? Then multiply it by one over the churn rate of last quarter.

Right. And that would get you an LTV decac, maybe this quarter. No, it'll be last quarter. Well, it depends. But you get a churn rate and multiply recent churn rate.

Multiply it. Some people say, well, wait a minute, that's inherently volatile. In Q four, we sell a ton of ARR, and the sales and marketing costs are pretty high in Q three. And we really should look at this on a trailing basis. And that's fine, it's a reasonable argument.

But smoothing it is. When you just put LTV to CAC in the field, you don't say anything and you're actually taking trailing twelve months, sales and marketing costs trailing twelve months new ARR. Inverting that with a annual churn rate and all you're doing is damping out volatility. And look, there's an argument. To say it analytically it's more pure, and it's actually not a bad argument.

But my argument is, if you're going to smooth it, you got to tell me by default, I assume it's quarterly metric, and you're not smoothing. So that's why it's a trust issue. There's two different debates there. One is a trust debate where if you haven't indicated it, I think you're eroding trust. The other is, what should we be looking at?

I still think we should be looking at the quarterly variance because as a board member the variances are what's interesting and I'm old enough to go, oh wait a minute. One queue is typically light on new ARR because it's the first quarter. We cleared the table in Q four and Q four is typically high on sales and marketing expense. So our CAC ratio is going to typically look pretty bad in Q one. And I promise to be thoughtful and not have a tantrum about that.

Right. So you don't have to be afraid to show it to me. Got you. So what you're saying is that Dave Cat Kellogg is not a fan of smooth operators. Yeah, that's right.

I'm not a smooth operator and I don't like smooth metrics even more. I'm okay with smooth operators. Okay, well, let's move to level three of your maturity model. And that is about the linkage to strategy. So talk a little bit about the details of that.

Yeah. So look, this is going to include a quick primer on how I think companies should do strategy because that itself is hard. But look, if you want to do strategy, in my opinion, if you don't know much about it, go read a book called the Crux. If you have time for two books, go read the Crux. And then another book called good strategy, bad strategy.

They're both by the same author, Richard Rumouth, who's a UCLA, retired UCLA professor of business. And those are the two best books on strategy that I know. And I know a lot of good books on strategy. So in the point of the crux is basically to say every company at every moment in time has one big challenge they need to overcome and you need to focus on identifying that big challenge. And that is like half the battle, maybe three quarters of the battle.

What is the most important thing we need to do right now? Because I think one of the occupational hazards of running a company is you got 50 people around you and everyone's got things that are broken. Customer support isn't good enough. The product has bugs, sales is inefficient. We hired somebody stupid, whatever.

You have all these problems. And sorting out the really big levers from the small levers is a very important part of the CEO and the exec team's job. And Rommel does a great job in both those books of saying, look, try to find your crux. What's the crux? It's a reference to rock climbing, apparently.

What's the crux of the route? What's the hardest part of the route. That's step one. Figure out your strategy. And that, in Rumel terms, is nothing lofty or fancy.

It's what's our biggest problem, and then what are we going to do to solve it? So then we make a bunch of goals to solve it. Right? For example, one of the examples in the book is Netflix. You can imagine all the problems Netflix had back in the days when they're shipping CDs around.

Operational problems. Disks are going to the wrong place. People are getting and not returning disks. Discs are coming back damaged. You can imagine all these operational problems when you're in the CD shipping business.

But their biggest problem was first, streaming. Like, what are we going to do about streaming? And then once we made that transition, the biggest problem was, holy cow, where are we going to get content from? Because the content industry is changing around us and we run at real risk of being a distributor without content. So we need to get in the content business whether we want to or not.

Personally, I thought it was crazy, just so you know. But if you'd never identified that as your biggest challenge and put commensurate resources behind it, we're going to go hire world class people to go build amazing content that we own. That was their crux. Just as an example. So basically, in the two minute strategy course, one, get the crux, two, get like four to six strategic goals that associate with it, and then hook those goals to okrs.

As we talked about earlier, the example of ASP is a key result. OKR means objective and key result. So key result for an upmarket push should be increased ASP. And if we're not increasing the AsP, well, that was the whole point of the upmarket push, so why isn't it going up? And what are we going to do about that?

And then in a perfect world, you find the KPI's the key performance indicators, sorry for all the acronyms that lead to KRS specifically. Wait a minute. If we want the win rate, average sales price to go from 25 to 50, shouldn't we first see that in the pipeline? Average sales price, the opportunity value. And the answer is, you should, so go look for that, too.

Ray Reich
I can't believe it. Here we are in the second episode talking about the SAS metrics maturity model. We're almost out of time, but I wanted to just talk for a minute or two on your level four, which is having a culture that's built around metrics. I mean, that's why I actually created the company and I went into benchmarking, because I truly believe that if you're a metrics informed, benchmark, validated culture, you're going to have superior operational performance. So I really love this one, Dave.

It's like I said, it is who I am. And the first thing you talked about is you got to demand that numeracy that everyone in the organization thinks about how their objectives can be measured and how those measurements impact the overall company's strategic goals. So I'm going to let you take it from there. But I love this one. Yeah, so.

Dave Kellogg
So do I. And basically, Ray, preaching to the choir here, I'm pretty sure. But you want to build a metrics culture where numeracy is demanded, where we have cadence and discipline about reporting and discussing numbers. As I touched on earlier, I think the biggest miscomprehension here is people think, oh, we're going to spend all our time talking about the numbers. And that's actually the symptom that you've not solved the problem if you spend all your time.

Like, I remember one time, I think I talked about this in the talk. I was at a meeting where we got caught into a definition of an MQL because the marketing team counted every user conference attendees. And MQL, all hell necessarily broke loose thereafter because sales certainly didn't think that all the existing customers were MQL. It's just because they came to the annual conference and we ended up in like a half hour scrum about what an MQL was. That is not a good example of discussing the business using metrics.

That was discussing metrics. That's a foundation layer problem. What is an MQL? So to me, that discussion about what an MQL is in a quarterly business review was a symptom of a problem. That is, we didn't have a foundation tight.

It's not the goal. When I say let's have conversations about the business using metrics, it's not conversations about the definitions of metrics. It's not what is an MQL? We all know that we agree to it. It's written down foundation layer stuff.

Now let's talk about how mqls are changing. What does it mean? Where are they coming from? How do we get more? How are they converting?

That's a conversation about the business using numbers. Not, not, not a conversation about the numbers. That's why I love my little phrase. I use metrics, informed decision making. We're talking about the business decisions, not about the metric.

Ray Reich
But we got to wrap up here. So take us home with level five maturity. That penultimate here's where we want to get to. Yeah. So we've talked on this, Ray.

Dave Kellogg
I mean, to me, once you've got, I mean, look, if you can get to level four, you're in really good shape because you've got a metrics driven culture, you know, with metrics linked to strategy that you trust, that are all, you know, well founded that that's awesome. Most people aren't there. Once you get there, then to me, you start looking further out and you work on your strategic trajectory. You build a long range model. You show the values of those drivers in the model.

You have the difficult conversations with the board about what your long term targets are. We don't just take your benchmark, much as I love your benchmarks to go. Ray said it should be 1.4 Chevy in 1.4. We talk about for our business, our competitors, our sales model, what should the CAC ratio be? Finally, we talk about timeframe, right?

When are we going to do what? And then also sequencing, which doesn't get much term, but like for example, say we think we need to open an Asia Pacific and we need to have a second product. Well, which are we going to do next year? And you can use a model that says, show me the default plan for next year. Now overlay a product launch.

Now take that off and overlay an Asia Pacific launch. And now show me both. Right. And do this kind of layered scenario analysis to determine what you think you could afford to do. When is a fantastic use at this level five.

Ray Reich
Perfect. So, Dave, we got a wrap up and I just want to summarize the five levels. And then maybe you can just let people know where they could go see a little bit more about this because I think it might be in your sasser presentation. But before we do that, level one of the returning model is getting the right foundation in place. Level two, making sure you build trust.

Number three, ensuring there's strategic linkage of the metrics to corporate strategy. Level four is to really have a culture centered on metrics driven. And then fifth is that trajectory you just talked about. Did I get it? You did, Ray.

Dave Kellogg
And in terms of if you want to see this stuff, the best place to go is the Kellogg. You could just kind of, I would Google Kelblog Saster 2023 video and it'll take you to a video of the entire presentation along. And that post itself includes a link to the slides. Perfect. Dave, thank you so much.

Ray Reich
I really enjoyed these last two episodes and I really look forward to our next one. Thanks, Ray. It's been a lot of fun. Okay, bye bye now. Bye bye.

Dave Kellogg
SAS Talk is a production of the Metrics Brothers Growth and CAC and a member of the Benchmarket podcast Network. By accessing this podcast, you acknowledge that the metrics brothers make no warranty, guarantee or representation as the accuracy or sufficiency of the information presented or the humor content of the jokes provided. The information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for general information only. This podcast should not be considered professional or for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, undirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising out of any use of or God help you, reliance upon the information presented here.

Ray Growth Reich is based in New York City and available on Twitter x rayreich. Dave Kak Kellogg is based in Silicon Valley and available at Kelblog. Schedul, which is French for unspellable, is not our actual production location. You can reach us@sastalkpodcastmail.com thanks for listening.