In 1997, Microsoft introduced their infamous “Clippy” assistant for Office: a cartoon paper clip with googly eyes that popped up uninvited to give unhelpful advice.  The reaction from users was overwhelmingly negative, with at least half of participants in a 2003 Stanford University survey finding Clippy incredibly annoying.  

Although the technology behind software support chatbots has advanced in recent years, user perceptions haven’t improved much. In fact, between 50% to 80% of users still find chatbot interactions frustrating and unhelpful—primarily due to the bots’ inability to understand context.

Yet, despite the lackluster customer feedback, companies continue to invest in automated support, in part because customers themselves seem to want it.  According to Adweek, 65% of software users would rather access self service resources than talk to a human, and 63% would happily talk to a chatbot if they knew it could give a useful answer.

So, can today’s generative AI powered customer support agents succeed where Clippy and his various permutations failed?  Based on my own company’s experience developing generative AI solutions for software publishers, there’s reason to be optimistic, though it requires an understanding of how generative AI works and what it takes to create an agent capable of providing five-star support.

More Than Words: How LLMs Offer More Robust Assistance

Traditional chatbots behave more or less like a Google keyword search.  If you ask- “How can I update a user’s billing address?” it might return three or four articles in order of relevance, starting with an article on editing billing information followed by something more tangential, like how to set up monthly billing (because of the keyword ”billing”) or how to update your email address (because “address”.) 

However, the power of modern AI models is their ability to generate original responses beyond existing data. So if you asked an AI based support agent “How can I  make a dashboard showing monthly safety incidents at our factory?” the agent might give an answer then follow it with, ‘Would you also like to set up automatic notifications for your managers when the dashboard is updated?’

In other words, a modern AI support agent can go beyond “What button should I click on this screen?” to “How can I accomplish my ultimate goal?’.

Software / User: Helping AI Agents Understand Your App

One reason why modern AI models can give more contextualized software support is because they are “trained” on vast amounts of information about software applications, in general.  So an AI agent will recognize that clicking “X” on a pop up window usually closes it, or that a menu item labeled “My Account” is probably where you go to change your user name. 

This general knowledge can be a double-edged sword when an AI model encounters a less familiar app.  Without the benefit of countless tutorials, manuals, and message board discussions to reference, an AI model might ‘hallucinate’ features based on similar software or make incorrect assumptions about how the app ought to work.

For example, when we developed our first prototype AI support agents, we used well known apps like SAP, Asana, and Google Workspace as test cases.  And the popularity of these apps worked in our favor, as the AI models already had detailed information about these apps in their training data (e.g., if you want to know how to create a process template in SAP, there are plenty of tutorials on the Internet.)

But when we started building support agents for niche software products from small to midsize publishers, we had to implement guard rails to ensure the AI agent didn’t make incorrect assumptions based on its knowledge of other apps.  In one case we had to explain there was no “save” button for projects and, instead, changes would be saved when the user “submitted” a project for review.  And we had to address dozens of similar cases without overburdening the AI with ad-hoc rules.

But, while testing the AI’s understanding of an app could be tedious, it offered some surprising benefits.  Because the AI’s incorrect assumptions were based on “all other software documentation on the Internet”, anything the AI struggled with would likely be counterintuitive for human users as well. In some cases, our software publisher clients opted to tweak the interface, rather than trying to explain a wonky user experience to the AI.

Supporting AI: RAG, Screen Readers, and Beyond

Even under ideal circumstances, where a software application is highly intuitive and exhaustively documented, no AI will automatically know everything out of the box.  For example, I’ve successfully used Google Gemini to write advanced automation scripts for Google Sheets, but sometimes it gives bogus answers for obscure questions like “Can I use ARRAYFORMULA() with VLOOKUPs?”

The good news here is that we don’t have to make a strict either / or decision between modern AI and old school keyword search. Using retrieval augmented generation (RAG) we can have the AI agent run keyword based searches of software documentation in the background, and incorporate the results into its responses as appropriate.

We also gave our AI support agents the ability to read the user’s screen (via a Chrome browser extension), which provides instant context for what options are available in the software interface and what the user is doing.  And because the agent “sees” all of the underlying source code for a page, we can embed helper text in the page’s metadata (e.g. “the navigation menu for this page will appear on the left on desktop browsers and as a collapsible drawer menu accessible by tapping the three lines icon on mobile.”)

For extremely large organizations, training a specialized LLM to support a particular software suite is an option, though the hundreds of thousands if not millions of dollars required to train an LLM might be too much for smaller companies, especially startups who are implementing major new features every quarter.

Keeping Current: Managing Product Documentation for AI Agents

Updating software documentation is a challenge for every publisher, and AI support agents add one more audience that needs to be kept informed of the latest updates and changes to an application. That said, there’s a huge upside as, once the updates are made, the agent can support unlimited customers for an extremely low cost.

This is where a component content management system like Paligo or K15T’s Scroll add-on for Confluence can be extremely valuable, as they allow teams to maintain multiple versions of the same knowledge base article – in this case one version for human audiences and another for AI.

Conclusion

There’s a reason why software companies continue to pursue the dream of fully automated software support.  For publishers, automated support agents promise significant savings.  Meanwhile, for users, on demand support is preferable as long as AI support agents can deliver the guidance they need.  

The days of static FAQs, user forums, and keyword searches aren’t completely behind us: those resources will continue to  play a role in support ecosystems for the foreseeable future. But as software companies integrate AI-powered support agents into their applications, the line between “support” and “user experience” will likely vanish as users expect software to simply explain itself (but without the paperclips and googly eyes.) 

Hopefully you found this article useful: if you’re interested in exploring generative AI support agents for your company’s software, please reach out to Sonata Intelligence for a consultation.