Office Hours: Taming AI Margins

How about now? Can you hear me now? Awesome man. still waiting for Ulrich and Emil to get in here. So just a minute. We'll probably kick things off at about five after.

Okay, no problem. Exciting. I didn't realize there would be back and forth possibilities. Yeah.

yeah. Yeah. It's we're we're just getting used to to Riverside. We used to run these on Google Meet, but Riverside makes it a little bit more challenging. You can either do it this way or you can do like a webinar, and with the webinar the attendees need to like I think it's called like a live call in or something like that. So that they're it gets a little bit more complex, but but yeah, it it should be pretty easy and seamless today, hopefully.

exciting. Exciting. How's it going with getting these off the ground and yeah.

Pretty good. Yeah, I hope I hope everything is working out well today. I'm surprised there's nobody we had a bunch of registrants, but nobody in here yet. And sometimes that happens, but

It's a little bit complicated to figure out. I had I I spent like five minutes figuring out how to get I got recycled back to ask for permission. yeah, like and eventually I found on my calendar two sets of invites. One was basically just save the date but no access. And then the second one, if there's a screen share, I can show you. so you know.

Okay. shoot. one sec. No what? Emile has the okay, whoops, that is the wrong one. yeah, sorry man, one second. I've just got to I've gotta shoot a link over to Emil and then I'll take a look at what you're what you're sharing.

Yeah. No worries, no worries, no worries.

Is this here we go? Invite What's guys? Hey Shiv, hey Jeff.

There we go. I always send a like a placeholder for the hosts and I think my placeholder accidentally created a Google meet and so that's where that's where Emil is right now. So he should be over here shortly. and I'll make sure Ulrich gets in here in just a second.

Go. And I mentioned this earlier to Ali. Just just a heads up guys, we'll probably kick off about five minutes after, assuming assuming we can get Ulrich in here. what's up, Ulrich?

I'm here. yeah, I so sorry I my my my the the thing in my calendar didn't have the link to Riverside, so I I became helpless for a few seconds. So I had to sort of figure it out.

Yeah. I was just explaining I was explaining my my like placeholder strategy. So like I'll s which is a very flawed strategy, but I'll I'll usually create a placeholder like a month in advance and then sometimes Google Meet will create like an automatic because I I use Google, it creates an automatic Google Meet. So I think Emile was in there. so he should be in here shortly and and we can get things going. But I mentioned to these guys Ulrich that we'll probably kick off about five minutes after. And at that point I'll hand it over to you to kinda set the stage. gonna make sure a meal is good to go.

All right. What do you think, Ulrich? Do you wanna kick things off and we can let Emil trick I think Emil will probably be in here shortly.

Yeah, let's Let's do that. Okay, so

I can set the stage too. So here's Emil. There you go. so these are these are kind of like my favorite office hour sessions because we get hey Emil, sorry about that. you're muted too. Here, let me get you there you go. I I explained to everybody the the placeholder problem. so sorry about that. We're we're just about to kick off and Oric was gonna jump into just like a quick five minute download to set the stage, but

These are my favorite office hour sessions because we kind of get both sides of the equation. We get pricing strategy, we get pricing technology. And so the way that these typically work is I'm kind of moderating. I'm gonna go through the questions that people submitted as they registered and pick out some of the questions from the people that are here and turn it over to Ulrich and Emil to answer. And I will say, like if you have like a short follow up, that's great. Like feel free. But want to try to make sure that that as many people who are in here can ask their questions as possible. So with that I will hand it over to Ulrich and we can kick things off.

Okay, so great to have you here, everyone. So we I I kind of looked through the the questions that that that Rob sent and and said, okay, so we can answer each of them and then there's gonna be like a like an outline of a framework. and then I can leave you all to sort of figure it out, or I could just like try to give you the framework like in the first five minutes, and then we can answer the questions like with reference to the framework. So So and then it actually it so happens that that this exact framework was something that I shared at a digital route event I think last week or the week before the last week where so where Emile's the CPO. So I thought, okay, this is this sort of this just makes sense. So here's the framework. So the topic is AI margins, how does it work? Like how do how do we how do we tame them? And and so so I was thinking about this for a long time. And obviously with AI being everywhere, it's been it's been obvious to sort of write a book on AI pricing for a while. And I've just I've just held off on writing a book about it because I didn't sort of feel that I had like the insight that would allow me to sort of write the right book about AI pricing. And then it just so happened in April of this year that I've that I I think I got that insight. And so I just want to share it with you. So, and the inside is essentially what I call the double hockey stick problem. So, and it goes something like this: so it has been for all products, all time, so that value is unevenly distributed. So, whatever you sell, whether it's a bottle of water or it's like an enterprise software solution, some customers are gonna get more value out of it than other customers, and That is going to be very unevenly distributed. So you're gonna have this sort of power law 80-20, where like 20% of customers get 80% of the value, that kind of thing. Traditionally, for software, you would see that the way that it was then priced and monetized would be like, okay, so we have some customers that pay whatever 1x of the price, and then some customers pay 2x, and then some customers pay pay 10x. So you had some sort of like distribution of like revenue across customers. And then

Often that was like relatively flat. So often we would see, and this is sort of the classic pricing work we did with with B2B SaaS, is that we would go in and we would find customers that had like 200x the pricing potential compared to others. And then we sort of identify that good pricing, quote unquote, is a pricing model that is able to sort of tease this out and create this these kinds of price points in a in a in a structured manner. That's all good. And this worked. For traditional SaaS companies, because they had flat SaaS costs. So whether you do whether you're you have the good pricing that is able to capture the tip of your value hockey stick or not doesn't matter because everything is pure margin. So you can actually have this opportunity cost just floating around where you don't monetize your customers because you don't have any costs to punish you. And this is what changes with AI. So AI comes in and suddenly you have another, like a second hockey stick, because some customers are exponentially more costly than others. So we very routinely see in the projects we do distributions like 1% of users generating 50% of the cost. And this just was never so in traditional B2B SaaS. So suddenly organizations have to deal with we have power law distributed something which is it shows up as cost first, but actually we figure out that it's actually also value. So often we have these, we have issues where we are we the margins are low, and that actually happens because we have all these sort of high costs and we haven't had a pricing model yet that can sort of figure out how to monetize this. And of course, we should bring some sort of balance to the picture. So this is sort of fundamentally the double hockey stick problem. It has some side issues here. So I just want to sort of point out a few things. So one is we have two hockey sticks, that's sort of problem number one. The second is that the ho the cost of AI will show up on its own. It just shows up when you deploy the product. The money, the the revenue will only show up if you have a pricing model that actually captures the value. So if you have a flat proceed-based pricing.

You can have a ton of value in that model, but if you don't price it, you don't get the revenue. But the AI consumption cost will show up. So one is like omission, the other is commission. Then the second issue is that symmetry is not guaranteed. We had a customer, they had a very, very spiky, they had 0.1% of their customers responsible for 75% of their costs. The most unprofitable customer was 380 times the cost of the revenue they brought in. For reasons due to that vertical and that customer base, the value wasn't there. We couldn't thousandx the pricing of that customer. They would just immediately churn. Like it was more like a pro sumer, like B2C type context. So there's just no way that that customer would pay like whatever, fifty thousand dollars a month or whatever the number was. So you can actually sometimes we AR products see, we have found a great way to take a million dollars of compute and only generate like fifty thousand dollars of product value. So it's not guaranteed that just because you have cost, you also have value. And vice versa, sometimes you have found a great way to use a little bit of AI to actually create a tremendously value pro valuable proposition where you just have like a very, very spiky value stick and the and the cost is super manageable and you don't actually need to worry about it. The problem though is, number four, that the sticks will move. Model costs will change, customers will learn to use the product, different ways of behavior will change. So the hockey sticks, I call them dance. Whatever relationship they had yesterday is going to be different tomorrow. And then five, we don't know as organizations how to deal with this. We are used to the world being stable and there not being any cost to deal with, and now we have unstable worlds with costs. So we need to learn that. So fundamentally, you get these sort of four cost value scenarios. You have one where you have super spiky value and super spiky cost. That's what I call double spiky. Then you have ones where you have no value but you have lots of costs. You deal with that with fair usage. You just cut away the cost. You're just left with something that is relatively reasonable. Then you have the reverse.

Then you have like super spiky value, no cost. You just ignore the cost. You just should have a really good pricing model that can actually capture this. So this is like we need some something s sophisticated on the pricing. And then you have scenarios like it's actually not that spiky. Your world is pretty flat, both on the cost and the and the value side. So spiky value, you need a good pricing model. Spiking cost, you need a like a great fair usage model. Spiky both, you need both things. That's fundamentally the framework. Okay.

Love it. Thanks, Ulrich. with that, let's jump right in. Jeff from Miro, I'd love to kick it off with you. and I'll throw kind of like on deck in the in the chat here in a second. But Jeff's question, and feel free to jump in and add any additional context. As we move to agentic capabilities, the market standard seems to be moving towards some version of selling value on top of either costs or tokens. From an internal perspective, are we starting to see a standard model around how variation of cost and tokens are captured and how value is added on top, either as a margin dollar number or as a dollar value? Are you seeing any pushback on companies being frustrated with only knowing the cost of an action after the prompt is run? and feel free to chime in there, Jeff.

So so give give give me some colour here, Jeff. So you're saying from an internal perspective, are we starting to see a standard model around how variation of cost tokens are captured?

Yeah, so like yeah, maybe so th there's two questions in there, which isn't lost on me. so like question number one is is like I think like we're we're seeing more again, like with agenda capabilities, like we the there's more uncertainty around like what actions are being taken by the AI and like where it's going and the costs associated with that. Like that's leading to

two things from my in my mind like number one it's leading to almost like at least again like you almost have to pass through some cost because it is so variable and so like question number one is like when people go to like capture the value that their product is creating on top of an LLM is it usually done as like I think

Gumloop just released one that's like it's eight percent on top of whatever your cost is. Or are you seeing, like, you know, we're gonna charge one dollar per million tokens or whatever. And so, like, is it value oriented or is it margin oriented, like percentage oriented? And then number two is like everything is becoming more obfuscated so that you can and so like is how do we work through? Essentially of like, you know, we're seeing most most actions like you're not told a price until after you take the action. And like it feels odd at best.

Okay, I I'll give you some concepts that I think will be valuable to you to sort of think through this. Okay. So The first is the distinction between what are called defined and undefined work. So undefined work is what Claude does for you today. You just open it up and you say, hey, what do you want to do today? And then you go, like in a million different directions. So that is undefined work. So undefined work is impossible to value price because it can be anything. You can literally use the AI for any kind of outcome. So it fits into any value chain, any value creation process. And you can sort of outcome-based price that because it's like it the outcome is not known until it's done, and it's super individualized from one customer and use case to the other one. So it's not productized. Defined work on the other hand is you have something like I'll see if I can find something like relatively in Miro's case. So so we if you have design, for example, they say, hey, I'm gonna edit this one picture.

That's a very specific like outcome or I'm gonna create like one minute of video or whatever. So you have to have defined work. So the second concept is to think about these things on a timeline basis. So where are you as an organization and are you experimenting or are you monetizing what are you doing? So what we see often is that cust like like a way to do this, you can argue that figmented some of the same is like you deploy AI in a very undefined way, you make it accessible in the product, and you just like and then on undefined work, you just have to like Cost plus price in some way. You say, okay, so essentially what we have is we have the model, it's a horizontal input, it's just like electricity, and we just have to put a margin on top. Whether you do that on a net dollar basis or percentage basis, or bring your own and you have a like a like like a fee on that, doesn't really matter. Like doesn't really matter to be honest. Like just find a way to do it, maybe multiple ways.

The fundamental thing is that you have a margin. You can also subsidize subsidize this. Like some customers do, like, hey, whatever, we'll just do this for free and then like have at it, and then we'll see what kind of adoption pattern that emerges. And then out of that adoption pattern, over time, we'll then find defined work. So we'll say, okay, maybe we still have the undefined like ability in the product, but if you do any of these like eight things or 80 things or whatever. Those are now defined outcomes and those we price on an outcome basis per outcome, et cetera, et cetera. Another aspect, margin relevant, is that usually you need the heavier frontier models for the undefined work, but you can have the cheaper, like lower price models for the defined work because well, they're defined.

So the the outcome is super repeatable and you know exactly what customers are. So at the beginning of the timeline, you're like, hey, here's the AI, and we're gonna do it on a cost plus basis, and the value is like hard to articulate, so the margin is not great, and the revenue is not great. And at the end, the revenue is great because we're pricing the outcome, but in the margin is also great because we're actually delivering that outcome in a super cheap way. So that's sort of the the the timeline that you want to travel. Make sense?

Okay, got it. Okay. Yep. Cool. Thank you.

Anything to add, Emil? Yep. Yep.

But I th I think yeah, I'm I was actually reflecting on the the the pushback part. I mean I I I sell AI monetization software. I hear pushback from from my customers, my customers and customers basically every day. And I I you know, Uluk has touched upon quite a lot of good stuff here, but I think one one thing that I work a lot with i is is transparency. I mean it doesn't really resolve it fully.

But at least being transparent throughout a month, you know, if if your billing period is a month, being transparent throughout that month so so customers and users can actually see as close to real time as possible, actually see what are they consuming. I think that that is also a great way to address that that potential pushback.

Yeah, we we talked about that in the past I mean as well. Like that is sort of also the prerequisite for the second level of that, which is control mechanisms. Like the ability to actually control and like give certain people access to this much consumption and so forth. Yeah.

Okay. Do you feel like the market's moving against transparency right now? Or like is moving away from transparency, at least with like undefined work? That's how it feels to me.

So yeah, I I'd actually be curious to hear hear your take on it, I mean. So so I would actually answer it with on the pricing part, people are trying to be as transparent as possible. It's really hard. So a a good substitute for transparency is measurability and control. So transparency almost assumes like I'll know it before the fact, but you can't, like often it's just impossible. So but if you can measure and control, then that's fine.

On the cost hockey stick side, I would argue that we're going we're going for less transparency. So that means that a lot of these sort of fair usage models are just total black boxes. And I happen to be of the opinion that they should be. It's the only way to sort of commercially operate it. So it's on the yeah, the one hockey stick is super transparent and people are trying trying to do everything to make it more transparent and controllable and measurable. And the cost hockey stick is just like shrouded in in fog and locked away.

Yeah. If the cost hockey stick is is not transparent. I guess like I I I'm like looking at some of my stuff of like, you know, Figma describes it as like the cost is like some combination of like model, complexity of task and context. And like they all kind of have these like ways that like Microsoft is similar. Like I do know I I mean I it's because they don't know, I think.

No, well well well, even if they knew, they would be stupid telling you because any definition is gonna get challenged immediately by the people that you don't wanna have the argument with.

Yeah. I guess my my question is like like I I totally understand it from a like that's how you choose to go to market, but like everybody on this call has to make the model. And so like like we all have to know the math. And so like

Yeah. So what do you do so the the the right control metric for cost is cost. So if so so so we we had a we had a customer, they were doing about twenty million a month AR, they were losing money on the AI, not not on the they were losing money overall, but because of the AI. And they had I think they had They ha they had around twelve million of AI cost that was just like killing them. And it had grown very explosively. We put in place fair uses mechanism. It was one of these where like point zero, like zero a tenth of a percent was producing seventy five percent of the cost. We put in place fair uses mechanism and we said, okay, and we had very very very good real-time cost measurements, like the kind that you can buy from Emil. And then say, okay, so we knew down to an action level. Yeah, we I'm I'm plugging him every time. So so we knew down to an action and user level what the dollar cost of an AI action was. And then we just like the way that we like like physically like operate fair users is number of dollars consumed. So as soon as a customer hit like $80 of consumption cost, we would start to throttle him, whatever the number was, like just making it up, right? Because that is much more robust than saying 100 edits or like 500 of these things or two seconds of that model or whatever, because tomorrow all of that is going to change. They have like they can do different things, the costs are different, and so forth. But but just like doing it on the actual dollar level, that is extremely stable over time. And and non ambiguous and and operable from from the team perspective. So yeah.

But but sorry, like but but I mean it I'm just curious now i isn't that a tricky one though, you know, enforcing and and and throttling or maybe even ultimately cutting people off. I was actually having conversation a couple of weeks ago with someone working in in IT secur security where where they're they're launching AI feature and and I suggested the same to him and he's like, Yeah, but i if I if I actually cut my customers off when they he hit a threshold, I no longer have any customers. Like how how how how can you work around that?

Yeah, so so so any any system has to have a kill point. Like just just like any physical system, like you can't do whatever a trillion, trillion volume. Like so so so whether either it breaks down on its own or you decide when that is before it happens. So ev even with mission critical like we've done it for mission critical systems for defense and like things that they say, I we w they will come and kill you in your sleep if it breaks down. So you but you still have to have a point where the system just doesn't operate anymore. So you might as well define it like up front. So then you do that. And then we we call the process boiling the frog. So you'll know between whatever zero consumption and the kill point, how many customers are along, like at what points of that route. And then you can say, okay, the first break is just that we politely tell customers to back off. Like whatever. Like you so every every every time you create a break, it has a notification point. Like you communicate something with the customer, and then it has some sort of consequence, like it slows down the solution. And then you can go back to the kill point and say, okay, nobody has reached the kill point. And then you can say, okay, let's put it at whatever, this like a million of consumption. And then because you know your consumption rates, you can say this is gonna affect 20 accounts out of a million. And then you can do that. And then you can test and then you can put it in and you can see how those 20 accounts react. And then you can move the point back and then you can affect 500 counts and then five thousand and then twenty thousand. And then that means that you can actually slowly like move the the point to where you get to measure how it affects customers. And it can only affect them in one of four ways. Either they keep consuming, they just don't care, they just they they change behavior to keep their their level of consumption up, or they slow down. Or they churn, or if you've given them the option, they upgrade and buy more so they they can keep going. Those are the four different outcomes. You can you can you can you can just like measure the you can just measure it and say, okay, so this particular throttle or break created this dispersion of outcomes when we hit it with 10,000 customers. And then then you get that. And then you can say, okay, we want it, we we want to place it, so it affects five percent of customers.

And then you chase it with another break, like break two, which is more aggressive. So this slows them down even more and tells them to really knock it off or whatever, right? And then you push that in behind. But instead of doing it all at once, just like throwing in fair usage mechanism and like disturbing your customer base, you start at the kill point with the weakest possible break, and then you move it like through your customer base, like figuring out okay how do they react. And then you you you measure it and then you can chase it with another break, another, and then you stack the breaks. And literally in this way, you get to, and I'm not kidding, you get to engineer your margins. You get to just decide this is the gross margin I want of this product. And and then the trade-off is how many customers am I affecting in what way? So we we moved gross margins twenty-two percent with this account that I told you about, like they did twenty million a month. and we did it in a matter of six weeks. And we we had a little bit of customer like Reddit noise, this kind of thing, but not a lot because it didn't happen all at once. And we didn't tell people that we had these particular limits and they were like the system just like politely started to sort of introduce it over time. And there wasn't this sort of big like one moment where suddenly the the like the product experience was different for everyone. It was like for different users in different ways over the course of several weeks.

Awesome. you good, Jeff? That helpedful? Cool. awesome. Shiv, I wanna move to your question next. a bit more kind of technology focused. So I'll read it. If you have any additional nuance that you wanna share there, feel free to chime in. But Shiv's question, how do you spec out what all the tele telemetry requirements are for a new pricing model? And you're muted shift, but feel free to there you go.

Sorry, I didn't really add too much context there. I mean quite a broad question. telemetry feels like the the biggest barrier we're facing when it comes to launching a lot these new pricing models. So yeah, anything that you can kind of help to explain how we should think about that process of defining what the right telemetry is and

You think about what to measure or how to measure it and like how fast or like give me what's the

I think both. I think obviously part of it is linked to your pricing model, right? If you're doing a token, you know, cost plus model, then naturally it's easier and more obvious. But I guess even in that side of things, it's probably even more important to think about what else you'd want to measure because to your point, Auric, if you want to be moving to something that is closer towards outcomes, you need to be measuring that from the start. so yeah, I think both is

I'll I'll let you handle a lot of the technical parts of meat here, but maybe start there. Yeah.

Yeah. Yeah, so so I mean, look when I talk about usage-based b business models day in, day out, the two main barriers that we come across all the time i is connected to telemetry or or or usage data as we we typically refer to it. And and the the first point is how do I actually get that usage data? So what we'll typically hear is that a a lot of companies out there think, shit, I don't have any of this. I'm gonna have to spend you six, twelve months massive engineering project just to build, just to get this usage data. That that is that is as far from the truth as as you can get. Every product, almost every product out there today, does emit some sort of product telemetry that can be the foundation for usage-based business models. So that that that's like the starting point. What I would then do is you know work backwards from from the invoice line. So for each billable metric, you know, define the event, the unit, the dimensions. So who, what, and where, and the time to stamp that counts. You know, you need to look for deduplication keys. You need to get that source of truth. So and I think that there are two main architectural decisions if we talk from a technical perspective. You can either go with with an event or API driven approach. That's where Shiv a lot of the heavy heavy lifting will be on you and your your engineering side. Or you can go with a a a capture first approach, which is where I don't want to promote D2Tru software too much, but essentially that that is what we do. We will connect to your existing product stack and collect that usage data and then transform it so that it is trusted to go on a customer invoice.

I'll I'll I'll add I'll add a few things here. So so just just a sort of a framework for for you to sort of think about. So one thing is that the the way that we think about it from a sort of like a like all the way back from a design perspective, like when we start out, is to set, okay, so you have essentially value metrics and cost metrics. And so cost metrics are you you can either have direct costs or you can find proxies, there are different sort of ways to do this. So maybe if you're a if you're a bank, maybe you have a number of accounts or customers or trades or they can do different things. And then value value metrics can be whatever you think is valuable. One of the things that we often find is that you can have value metrics that are not directly product related. So for example, I'll just give like a very basic example. Number of employees sometimes is a great way to measure value, but it isn't something that is sort of inside the product anywhere. So sometimes you need to just like take a step back and say, okay, what actually makes what is a good measurement of value for this solution overall? And just decide that, hey, maybe we need to go find that number somehow. Then value matrix and cost metrics, they can they, that's using my terminology, they get deployed. Into either pricing or control metrics. So they those are the two, like those are the two hockey sticks, and you sort of are you control the cost and you and you price the value. That's sort of that idea. Where we often see is that from a from a from a process point of view, when you are still sort of figuring out what you want, you probably want to measure and track more things. From value and cost, then you're actually gonna deploy. So while Emil is right in that of course the things that you put on the invoice, you should absolutely be able to measure and put on the invoice.

There might be other things that you want to also measure as you go along, as this might be something you want to price in the future. And then you'd want the history of that. So for example, the example I gave before with defined and undefined work. Hey, if we're just doing undefined work, at the beginning we just measure tokens and we do sort of the cost plus thing, and that's it. But then we might also might want to set up some telemetry to measure different types of actions. So we're we're generating these value metrics, and then at some point we deploy them into pricing metrics. Now we we define them closely, this is how they are, and then we get to price them. So you should think about having many value and cost metrics and a few pricing and control metrics. And then so the value and cost metrics become this like this menu that you can that you can choose to deploy from at any point in time, if that makes sense. Yeah.

Yeah. It does make sense. Maybe one kind of slightly separate question related to your, you know, first discussion point when it comes to the undefined work. Is there a risk if you're starting there with the intention to move towards outcomes that if you're gonna start with a cost plus approach, you anchor customers to think in that way, you might kind of confuse customers from the beginning because you're talking about tokens or something that's kind of quite unpredictable. I'm just trying to think through that tension of I think there's some cases where it's always going to be undefined and you're going to have to have some sort of cost plus model. But if you think there's a route to being, you know, a little bit more kind of fixed credits, et cetera, then is there a risk by starting on that kind of cost plus model that you you confuse customers and you anchor yourself there?

So yeah, so so I think I think there are some there are some there are there are there are two I call them actually I call them guardrails. Like they're there are two mechanisms that you need to have set up that allow you to make this kind of future change. And one is that the contract framework that you sign with customers should allow you to price in a different way in the future without suddenly being like locked into heavy renegotiations. So that's so like from the contract level, you sort of at the beginning, even when you're doing undefined work, you should sort of prepare that you can introduce items and you can reprice and you can do things later. So just from the contractual point of view, that's set that's said and done. Then the more sort of fluffy version is that you should also prepare the customer narrative and the expectations around it. So public example, Figma goes out, they launches AI across the product. They do it in a very undefined way, doesn't even have credits to begin with, just like, hey, we now AI capabilities in the product. They have another guardrail, which is that they have really good fair users mechanisms in place when they do it. so they get to control margins. They actually drop from like 92% to 82, 84%. So they drop around 10 points like during that experimentation process. Then they start to define the work more, they add credit systems, and then they start to monetize the credit system at a later point. So so in short, the when the contracts are there that allows you to actually do it, your the the one thing that you should that you should also ensure to do is to Is to tell people that you are going to make changes in the future and that nobody knows where AI is going, and that you might say, hey, this is how we do it today, but I have no idea what kind of product I'm gonna or solution I'm gonna deliver to you like nine months from now because, well, things are moving so fast. So you should expect it to change. And then the stable thing, the thing that will make customers not

revolt and kill you, is that you should focus very clearly, in my opinion, on your on the mission and outcome you have for your customers. And saying, hey, we are we are and we always will be like the number one like you know creative platform for movie makers, whatever it is that your product does, right? And then that's our mission and our goal. And what whatever way we happen to solve that from an actual technical perspective tomorrow, we'll see with AI, right? But but this is our this is the job to be done that we put in the market to do. And then whenever you do a new pricing change, say, hey, you know, we're not pricing tokens anymore, now we're pricing these seven actions inside of a credit system. Because as you remember, we are still committed to being the number one creative platform for movie makers or whatever it is that your mission is. And this is a much better way to do that. So this gives you this gives you this idea where customers understand why you're making the change at all times. And that that enables you, like then they can live with almost any how. But if you're just like making a change and they have and you're they are left with no reasoning. for why these changes are happening, that's when they start to get mad at you. So this this the the narrative is extremely important to to to invest in upfront if you're planning to have a lot of downstream change.

Awesome question. Ali, you're up next. I'll I'll read it. If you want to add any nuance or any context, feel free. So Ali's question how to think about subscription seat and AI combo pricing. Anything that you want to add there, Ali?

Well, that's a little bit maybe Ulish presented in the with the hockey sticks where we just launched our AI product this week and so it's I'm just new, that's why I'm trying to soak in a little bit of understanding of how I should go about pricing it. But yeah, we have a combination of subscription in different tiers and then on top of it now we are adding metered AI by dollar.

So if that's the right way to go about it or not.

We'll find out. So I I think it I think it it's I have so very often and so okay, I'll I'll go off on a philos philosophical tangent for like a minute and then you'll just have to forget me. So very often we get this we get paralyzed because we have this idea that there's one perfect model out there and it's we could just like get enough of the data and get enough of like the smart minds in a room, we can like Find that perfect thing. And then we can just do that and then we're done. Right. And I think that's just impossible, like an impossible demand to have of yourself. And also, things change all the time. Markets change, your competitors change, the models changed, you're whatever. You the somebody gets a great idea and product, and now that changed, customers change, like they learn to use your. So whatever AI consumption pattern you have right now with your product is gonna be different a month from now. Because customers learn to use it. So things change. So whenever you you kind of need to change with it instead of like, okay, so you you almost like you have this like instrument with a lot of dials on which, okay, maybe we position this a bit different, maybe we take this discount off, maybe we add that new credit item to the box, whatever it is. Like you you have a ton of ideas. They should roughly get you in the right direction, and that's a conversation you need to have as a team. And then you should make sure that they don't carry like risk of blowing you up. So that's why you have the guardrails and so forth. But inside of like those like strong requirements, you're gonna have a set of what I call viable actions, like the viable set of alternatives that you can move forward with. And in my experience, the most successful teams are just the ones that do the most decisions on on executing on viable actions. As opposed to finding the best one.

Because there is an exploration in just like executing towards the goals like, okay, we'll do this. Okay, it didn't work, like let's do the other thing. But as long as you don't take major risk in blowing things up and then the experimentation itself is going to incrementally and sometimes in in like big steps, improve the business. And then you're gonna end up with something like over the course of a year that is just like fantastically different compared to if you like sat down and deliberated really, really closely.

I had a customer very recently, they he they they went from from zero to two hundred million ARR by the founder coming up with the perfect pricing model ten years ago. And they are now being disrupted very heavily by AI. And they have no idea what to do. Because they have never thought about changing pricing and packaging. Because we just did this one thing. We had one package and we charged whatever this amount of money for it and like the some whatever the terms are. And it just took us to 200 million. And so now the the capability in the organization to even have discussions around how pricing works and what is value for customers, all these things, is something they have to relearn as if they were like a like a very young startup, right? So I think it's just one of these things where they could probably have gotten to whatever. I'm just imagining something like a billion revenue if they had actually like kept working on the model. But they just didn't have to because like, hey, it just like it worked out of the gate, and then they just went building great product and doing other things, right? so I think okay, that was the tangent.

But but but Ullick, I mean and look looking here, so essentially we're talking about a a a traditional, you know, hybrid model, which I personally feel is a a fantastic way to to dip your toes into the the world of usage based business models. I mean th there's there's also looking at the positive side of a hybrid model, you you're able to maintain a certain level of of predictability whilst protecting you you on the the cost side by having the AI consumption on top of your already

existing subscription. And you're also giving that to your your end customers. So allowing them also have t predictability, what is my my my monthly bill going to look like to some extent, you know, without the without the usage on top at least. And I think one thing to be be cautious of here though is the the seat compression. You know, as AI features become more and more mature, we're going to see more and more more agents do do the work that your tip typically your users would do.

you know, that seat count is gonna start to fall. And then you have to make sure that you can capture that more upside from the consumption part on top.

Hmm. Yeah, we actually shifted then from seat to d data silos for partly for that for that reason. yeah. the the data the department or however they want to structure who has access to what data defines then the the volume pricing of the of the subscriptions. So rather than how many people within that department or how many agents within that department is using the AI using the software. So so it yeah we we managed to go around that problem. Yeah. But it's still this dual model. It's still subscription plus consumption at the same time. And I was just a little bit worried that it becomes just messy rather than clarity. But the way I'm understanding it right now, at least as a trial version, is worth if it's worth pursuing that idea as a concept.

I de I definitely I definitely believe so. I think you know, things could could get messy on the the financial side with with your recognition and and all of those parts. It depends how you treat it. Are are you are you billing and metering all of that AI consumption completely separate from the rest of your subscription business? Then then then it could turn into a bit of a headache, at least for the the office of the CFO at some point in time.

Mm. Mm. I need to investigate.

Awesome. Thanks, Ollie. And thanks, Ulrich and Emile. All right, Carl. I like your question. I think this is an interesting one. It kind of touches on technology and on strategy. How to track token utilization by client and use case. Do you want to expand on that at all, Carl? Any any examples or any? nice. Love it.

Yeah, I I switched off my video, I got interfered a bit. So sorry about that, but I'm listening. Yeah, exactly. I mean a couple of questions were wanted for me already right now. But for me especially like I work a lot with financial modelling. So if we make a new pricing model, I like to come to a pricing model by by scenarios and financial models. But the key part is for me is like how do you know how this

Yeah. No problem. Got another attendee, I love it.

So so sorry, Carl, it's just super low. it's just me.

It is a little bit it's a little bit quiet, yeah.

Okay, now what did these one moment please? one moment.

It it it's it's low but it's it's works.

Wait, they can do it better.

Kinda kinda the same. But yeah, you get let it rip. You get you w we'll we'll listen closely. Yeah.

Okay, I'll try. Anyways, yeah, for me it's just about I I like to work with financial models to develop the pricing and for that the big challenge is like how do we get transparency? So exactly What is the usage behavior? How can you connect usage behavior of clients to the costs? And how can you connect at the same time the revenues of those clients to those AI costs? So I'm trying to get this transparency just from client usage to cost incurred to revenue achieved.

Okay. So the way I'm hearing it's like how do you create transparency, especially sort of sort of the client side? And and so as I'm hearing it, it's both in the sales process but also once they're up and running, like what how the consumption works.

No, I think if you have a new product of course you'll just make assumptions. But then later on how can you track it? How can you track it? the connect the cost bimodal per use case and align that with the revenues per use case. Make sense?

So so so how do you so is it is it the idea that is it essentially what you're describing as like the estimation problem where customers are we don't know how much we're going to need and we have we know roughly what kind of work we do, but how it exactly sort of translates into your consumption model, that's the difference that's that's the difficulty.

More sorry, so you said before like if you make something like twenty times it will come to a defined outcome, right? If you don't know at the beginning, you just track it, like for twenty time and over time you will know which use case uses how many credits, how many tokens in the beginning. Does that make sense? And now my question is, how can you track the usage of that? Sorry, I'm a bit because I'm now with my daughter, so it's a bit difficult. okay, let me think again. So Rob, can you quickly on give me the question again? What I wrote back then.

Yep. The question was how to track token utilization by client and use case.

Yeah, exactly. So do you know like how can you get the information about tokens used? so that you can afterwards get to those defined outcomes and at the same time can you somehow connect those tokens to revenues? Is there any a line of process when you can look, okay, last month they did that, and can you connect the revenue side with the cost side?

So okay, so l so it's the it's the actual technical problem of a customer doing something the product and then measuring how many tokens that actually consumed, and then the next part of it which obviously like connects you to cost, and then the next thing is like how does that actually, depending on your model, then connects to the revenue. So and implicitly in that question, you also have there's a margin question, like how to actually do that, right?

Yeah. Yeah, exactly what I'm trying to is align the the two hockey sticks you sta showed in the beginning. I'm trying to align the revenue hockey stick with the cost hockey stick.

Yeah. So Emil, you're much more equipped to to answer the actual token utilization question.

Yeah, but maybe maybe starting there then. So so if if we assume you know you're you're shipping you you're using AI either towards your clients or you're shipping an AI AI product or feature and you're you're dependent on you know some LLM in the background, whether it's open AI or something from from Anthropic. Please correct me if I'm wrong, but but essentially you could be getting an invoice from them. You've consumed, I don't know. A billion tokens, but you then have no way to understand, you know, has customer A consumed five percent of that? Customer B consumed ten percent of that, etc. Because because essentially what the only thing that OpenAI or Anthropic has is your your company's API key. So you can see your consumption, but you can't spread it across customers or or usage. So what I what I would do then is you actually need to tag at the source. So label each call when it happens. You need to include, you know, tenants. So which customer is behind that? Who is the actual individual at that customer site using that AI? What feature or what use case were they doing? And if you also have a a a task ID. And and essentially, you know, start from the logs that you already have. Because I'm I'm fairly certain that that you maybe Carl have more logs than than you actually realize that you have. You know, you've got gateways, you've got API servers, cloud services. All of that can provide real important labels that you can tag that raw usage coming from the L L

Yeah. and my question is, are clients doing that already? Are they really tracking the usage, aligning the cost and also tracking the revenue that comes with this usage? You are answering my questions. My question is is it being done at the moment?

I I I I have a concrete example. this is from from a couple of weeks ago talking to one of the largest you know consultancy management consultant firms. They they rely heavily on on AI from Anthropic when delivering their their customer projects. They don't actually they don't actually, you know, charge their customers based on on AI usage, but they need to understand the cost. It's like which projects are consuming a lot of AI, which which which employees are consuming a lot of AI and w and we actually do that for them. So track all of that raw raw tokens coming from the LLM and then attributing them to projects and users so that they at least have an understanding, okay, this project drove this much cost. So that they can go like that in the end and and put a put a price on it. So yes.

And next question would be because first is the tracking of the cost, but the second part would be looking at the most cost efficient LLM. So are do you also have like feedback groups on how you could optimize on this level? Or are they also doing that or is that not the topic yet? Would that be the next level?

So so th th this example hasn't gotten there yet. Right now it's just keeping track track of that cost. And I think o over time that is something that they they will start looking into, you know, some mundane tasks. Can they they optimize which models they run in the background to bring the to bring that cost down?

Yeah, I can I can quickly also say like I've I've been working on building out those exact models, Carl, and we're kind of using that with our pricing and packaging teams and customer success teams to like protect the margins but also grill the devs on why some features are way more expensive than others, but they're tracking it internally as if they're the same.

Awesome. thank you guys for for that answer and for Roheen. Dan has one more question before we get out of here. I wanted to make sure we we answered this one. So Dan, I'll I'll read it real quick, but feel free to chime in. So Dan's got a new product, it provides AI governance, containment, et cetera, and they broken it into two meters, user seats and agent seats. Would love to get feedback on ideas on how to price. So Dan, feel free to add any additional color in there.

Yeah. So been on a couple of these before. So we we're a cybersecurity company in California doing primarily bot management and API security. And one of our real strengths is understanding the the intent of these calls. And so we've recently put out a product that helps to prevent the hugging faces scenarios, right? It does containment governance about AI. And so we've kind of taken the approach of two different meters. So there's a user meter, which is, you know, like Ulrich says, I put Prompt clot, it comes back. And but then you know, a lot of these companies are looking to enable autonomous agents, right? Non-human identities. And you know, where I would naturally think, great, so an autonomous agent, if I assume that the user agent is only using AI for four hours a day. And you want to set up these autonomous agents that are running 24 hours a day, that's basically six times the user agent price type thing. it's it's the initial feedback from customers is while they acknowledge the value of the agent, they don't think they should be paying almost anything for it. So I just wanted to get some feedback if yeah.

Mm. Okay, that's interesting. So so we have The good thing is you can price anything that you can measure, right? So so so then w whether you should, different question. So and whether they will let you, different question. So so so often sort of seat-based models for agents does not make sense for several reasons. one is that I think actually sort of So so right now we have we have seed contraction. So we used to have ten humans and now we have only five humans and then we have some agents. Okay, so that's so now now the idea is to just like charge for the agents. I think actually you'll see that agents also have seat contraction. Like we'll used to we we start everybody's building agents and now we have like twenty agents doing things. I think like we're gonna see people just consolidating to just have like three agents doing things because why have twenty agents when you could just have like three bigger ones that do so so I think I think you're gonna have the seat contraction on the agent level as well. and then The other thing is that there is only like a human can only push a button that many times per day. An agent can do it like disproportionately so, right? So it can do it like trillion times a day. So so where you like the power law distribution for a human user is just like not that.

Crazy, but for an agent user, it can be it can be wildly different. So you can have two, so the the technical way I meant to measure this is through a concept which I call metric density, which is like what is the uniformity of value of the thing that you're measuring. So for example, Stripe, they measure dollars of payment. And that has perfect density because one dollar is exactly the value of another dollar. A user for, let's say, a word processor like Microsoft's Word is like. Is all like is like low density, but they don't care. Like I'm not getting as much value out of Microsoft Word as like JK Rowling is. He writes Harry Potter, I whatever I write is not that, right? And then and so forth. So you can have different sort of density. API calls famously has really low density. Some API calls are just like super valuable, others are not. Some API calls are done in the billions with low values, others are like very high value trades, they're only done in hundreds, but they are like super valuable. And so on. So I think I think you kind of wanna drill down to like what are the agents actually doing? and then some of those things are gonna be relevant, some are not. And then you're gonna figure out okay, what can we actually operationally measure? What can we articulate and agree on with the customer? That becomes like your viable set. Then some of those things are valuable to customers, they actually want it. Some of these things are acceptable. Customers will think they're fair. Some of them are gonna have relatively good or not good density. So that means that actually they'll monetize pretty well. And then some of them will also grow in the future. So you have different sort of evaluation criteria for like choosing good metrics. So you can run the numbers on some of these, like if customers were to accept this, then we'll make this much money on metric A, B, C, D, whatever. And then you can actually go, you can ask customers. You can say, hey, what if I price per whatever this incident or that call type or this particular action? Would that make sense to you? And they will tell you.

Okay, that's good. yeah, we we actually measure all the tool calls behind the scenes. So that's that's one of the ways we're gonna do it. I forgot to mention that w we've we're starting down a path also of offering a hybrid deployment where the data plane is on their side. So we can offer and on that once we get there, then you know the cost of the control plane isn't super expensive to us so we can we can get like seats seem to resonate a lot more. It's more when we're talking about the SAS deployment where we're hosting both that

Be careful you don't give away the value value. Right? So so this is this is a problem with like old school on-prem solutions, is that hey, we're like they're doing the same software to whatever two companies, and one of them is just getting like a ton more value compared to the other one, right? So if so so just because just because the customer is carrying the cost of doing whatever the computer, the inference, doesn't mean that they also should get all the value for free.

Right. So so you you still need to have something some some some telemetry and measurability of whatever that is and just say, Hey, we're gonna we're gonna charge you based on this and the customer's gonna say, Well, but that's a hundred percent margin for you. And then you can say, Well, don't buy it then. Like it like fundamentally that's what it comes down to, right?

Awesome guys. thank you to everybody who s who stuck through through the whole session. This was awesome. And thank you so much to Ulrich and Emil for sharing your time and expertise. I'll shoot out a recap after this and make sure to make introductions as as it makes sense. But hope everybody has a great rest of the day and great weekend.

Thanks Rob, thanks Ulric, thanks Emil.