Agents and Outages
This week I attended a Salesforce webinar on “Proving AI ROI: How to Measure Success with Agentforce”. I’m always interested to hear how Salesforce uses Salesforce as I feel this is probably the closest thing to the “there is an unlimited budget” assumption that myself and many others started out with at the CTA review board. Yes I’m sure there are wooden dollars involved, and there will be pressure to show a return on the investment, but right now I’ll wager Salesforce is the easiest place on earth to get an Agentforce pilot approved. I’ll also wager they get mate’s rates rather than the rack rate $2 per conversation.
Some of the stats were truly impressive - 50,000 conversations a week with an 80% resolution rate and a 1% escalation rate. This does suggest there’s a group of customers out there, making 19% of calls, that are happy not to have a resolution or an escalation - maybe they just want to talk to someone and even an AI chatbot will do.
As the majority of these conversations are unauthenticated there weren’t any stats about repeat callers, which is an area I’ve been thinking a lot about recently. What happens when a tool like Agentforce makes it so easy to call for help that customers don’t try to help themselves? If we place documentation online you give customers a way to help themselves and hopefully not call the support line, but if it’s easier to hit an AI helper than read a document, are we in danger of infantilising our customers? Worse, will we end up spending a fortune re-solving the same problems ad-infinitum? Maybe not yet, but as the agents gain capabilities and humans lose them, who knows?
Another interesting aspect of the stats was the escalation rate increasing as we approach a release window - customers are asking questions that the Agent can’t handle because the work is in progress. Not that interesting you might feel, but the overall volume of calls remained broadly the same, which implies that the people who were asking easy questions the Agent could handle suddenly pivoted to asking about the new release, or maybe those that were happy with no resolution become more picky when asking about new features. Definitely an area that would benefit from additional research!
The reason escalations increase is because the new release information is bang up to date and subject to change, so the Agent likely won’t be grounded with it. This got me thinking - what happens if there’s something going on right now that the Agent doesn’t know about, and how does that affect performance and cost? Consider a service outage, where every customer you have is hitting up an Agent trying to find out more about it. In this case the Agent will probably not be able to help, but it’s also likely to annoy customers as it tries to diagnose a problem that is caused by the service being unavailable. Not only will you be trying to handle the outage and restore the service, you’ll be spending a fortune routing all of these requests through an Agent that will likely have to hand the request to a human - a real 1-2 punch. I’m not sure what the solution is here to be honest - do you take down the Agent and thus look like you are having multiple outages, or try to ground the Agent with information about ongoing service issues so it can advise users to check a status site to find out more.
The Salesforce ethos is Agent conversations can always be escalated to a human - there are no scenarios that would result in the Agent becoming a dead end. This does mean that you need enough humans to handle all escalations, which becomes quite the bottleneck in the event of an outage. Across the industry we hear a lot about the millions of autonomous Agents that will inevitably handle most of our work, but very little about the need for us to act as backups when the Agent taps out.
This is another area where it feels like the Agentic approach will struggle when there’s an outage - as Martina Navratilova famously said “What matters isn't how well you play when you're playing well. What matters is how well you play when you're playing badly.". In an outage your customers are already likely to be losing faith and feeling more than a little tetchy, and these emotions will intensify if the Agent isn’t aware there’s a problem and takes them on a wild-goose chase before adding them to a long queue to wait for a human to become available.
Changing tack slightly, what also jumped out at me from the webinar was how the language around Agents is very different to that which we usually employ when talking about technology. No longer is there talk of “configuring” or “updating settings” when changing the behaviour of an Agent, instead we are “coaching” or “retraining” it, even though what we are actually doing is changing static instructions that are passed to the reasoning engine along with a request. We’ve anthropomorphised Agents in very short order and see them as willing students with ourselves as the wise teachers, but much like the infantilisation of users, over time we are likely to get worse while the Agents get better - who will be teaching who when that comes to pass?
If you’d like to hear more of my thoughts on AI, I’ll be appearing as a panelist at the Salesforce ISV community group’s “Become AI Ready In One Week And Transform Your Business“ on 16th April at 6pm BST. Hopefully I’ll see a few of you there!