SONATAnotes
AI Doesn’t Know What it Doesn’t Know: Knowledge Management for AI Agents
Ever since ChatGPT brought generative AI to the masses, a growing number of people have used it to complement or even replace search engines. And AI models’ ability to give plain-language answers to specific questions based on seemingly all the data from the public Internet is definitely impressive.
However, not all questions can be answered from publicly available data. Some important sources of knowledge you won’t find in a “generic” AI model include:
- Institutional knowledge, such as your organization’s standard operating procedures, client data, or product documentation.
- Personal knowledge, like a financial advisor’s preferences for real estate investments versus crypto.
- Gated knowledge, like the full text of published research papers that cost money to download.
- Timely knowledge, such as up-to-the-minute market data, quarterly reports, news stories, or how many units of a specific component are available in your warehouse.
These types of proprietary knowledge often gives organizations a competitive edge, prompting companies—from financial data provider Bloomberg to farm equipment maker John Deere—to invest millions in developing custom AI models trained on their own data. Meanwhile, an increasing number of software providers are integrating AI-powered search and analytics into their products. But what about situations where an organization can’t justify the costs of building their own AI models or for which no off-the-shelf solutions exist?
Our company has helped clients in various fields develop AI agents “grounded” in their proprietary data. In this article, we’ll look at how organizations can build AI solutions that take advantage of their institutional knowledge and proprietary data, without having to train their own AI models from the ground up.
Checking Your Sources
We’ve discussed how AI models learn and access information at length in other blogs, but to give a quick summary:
- AI large language models are initially “trained” on massive bodies of text, in some cases trillions of words harvested from the public Internet and other sources. However, this layer of knowledge is frozen at the time the AI model was initially created. Also, AI models don’t actually retain the verbatim text of their training data so much as patterns of words that they can use to construct new statements (this is what allows AI models to answer an original question, but is also the mechanism that causes them to occasionally “hallucinate” incorrect answers, like someone who half-remembers an article they read several years ago.)
- In addition to their training data, an AI model can analyze whatever text a user enters during the course of a conversation. And while, normally, this knowledge would vanish as soon as the conversation ends, more sophisticated AI agents can save past conversations to memory, for the agent to reference in future conversations with the user (though these ‘memories’ must be analyzed anew every time, and are never actually incorporated into the model’s permanent memory.)
- Finally, the AI’s training data can be supplemented with keyword-based searches of the Internet or a document repository (similar to how Google works) as well as database queries, RSS feeds, or whatever else the agent’s developer cares to integrate. When we provide this type of outside information we technically aren’t “training” the AI model (training happens once, when the model is initially created) but rather “grounding” it with additional information.
But, whatever the source, how do we actually make information useful to an AI agent?
Translating Human-Speak For AI
Recently, we worked with an industry association that offers technical guidance to both government regulators and businesses seeking to stay compliant (for this example we’ll say they were in telecommunications – not their actual domain, but similarly complex.) The association provided us with multiple manuals to serve as the body of knowledge for an AI agent, each several hundred pages long.
Per our usual procedure, we started out by dumping the manuals into an online document repository and seeing how well the AI model did answering questions from the raw document. However, when we asked about regulations related to wireless infrastructure, the AI agent seemed oddly obsessed with the national wireless network in Iraq. We then looked and saw that the relevant section of the manuals included an extended case study on the reconstruction of Iraq’s telco infrastructure after the U.S. invasion in 2003. Eventually, after extensive reworking of the documents, we got the AI agent to provide more relevant responses.
This example highlights how human brains and AI language models process information in fundamentally different ways. Many of the things that make knowledge comprehensible and relevant to humans can confuse machines. For instance:
- A beautifully designed infographic that elegantly communicates complex data relationships to human eyes might be completely indecipherable to an AI system that can’t process spatial relationships or color coding.
- Case studies that engage human readers with narrative elements and contextual details can overwhelm AI systems with irrelevant information that obscures the key points.
- For database queries, metadata can be a double-edge sword, providing either helpful context or distracting clutter.
Knowing what information to keep, what to reformat or condense, and what to filter out is a key part of making human knowledge useful to AI agents.
Juggling Information: Context and Attention
While modern AI models can theoretically process millions of words in their memory, this capability comes with significant practical limitations. Just as humans can’t focus on everything in their field of vision, an AI model can’t account for all of the information in its “context window” when generating output. This means we have to be judicious about what information we feed to an AI model at any given point in a conversation – providing just enough for it to give informed responses, without causing it to get lost in a cloud of words.
For instance, when our company developed a role play simulation to help healthcare providers practice their communication skills, we developed a system to selectively swap guidelines and background information in or out of the AI’s memory depending on whether the user was playing as a physician versus a nurse, or whether a scenario was taking place in a doctor’s office versus an emergency room. Otherwise, keeping the information for all roles and all settings in the agent’s context window led to erratic performance.
Protecting Sensitive Data
Knowledge management isn’t just about performance: it’s also about privacy and security. When our company builds AI agents for clients in healthcare and banking, they will often need to access individual patient / client records in order to advise the user. However these records are usually subject to a host of privacy regulations (e.g. HIPAA in healthcare, SOX or GLBA for banking applications), none of which were designed with AI in mind.
In these cases, organizations need to implement robust authentication and authorization systems – to ensure that the AI agent’s access to data is limited to whatever that particular user is allowed to access (for instance, providing salespeople with high-level specifications for a product but not necessarily the detailed schematics.) Beyond that, conversation transcripts involving sensitive data must be stored with appropriate security controls comparable to the organization’s other protected systems.”
Meanwhile, it should go without saying that employees should only use approved AI applications when dealing with sensitive information – and never copy-paste data into their personal, consumer-grade AI accounts.
Plugging AI into Knowledge Networks
When we know what information an AI agent will need in advance, then it’s fairly easy to “push” that information from the relevant system into the AI agent’s memory at the start of a conversation. But what if the agent needs to “pull” new information on the fly, during a conversation?
If the scope of possible information is limited, then we can provide the agent with an index of possible data sources – as we did for this corporate security advisor agent (i.e. organizing its available knowledge into “physical security”, “cybersecurity”, etc.).
However, if we want AI agents to achieve their full potential as “virtual teammates”, that will require something more radical. Currently, most organizations rely heavily on informal social networks for information sharing – basically, if a human employee doesn’t know where to find something then they will “ask around.” But if we want to give AI agents comparable access to information then we either need to do a better job of capturing organizational knowledge into structured repositories… or integrate AI agents into messaging platforms, email, and video conferencing so they can “ask around” the way humans do.
Conclusion
As AI agents become more integrated into organizational workflows, knowledge management will be equally – if not more – important than which AI models an organization is using. The challenges we’ve outlined – from curating proprietary knowledge to managing context limitations and protecting sensitive data – aren’t merely technical hurdles but strategic considerations that will determine the success or failure of AI implementations. Organizations that master them will gain significant advantages – while those that don’t will find that even the most sophisticated AI systems can remain frustratingly limited by what they don’t know.
Hopefully you’ve found this article useful. If your organization is looking to develop AI agents for workforce training, on the job support, or other purposes, please reach out to Parrotbox for a consultation.
Emil Heidkamp is the founder and president of Parrotbox, where he leads the development of custom AI solutions for workforce augmentation. He can be reached at emil.heidkamp@parrotbox.ai.
Weston P. Racterson is a business strategy AI agent at Parrotbox, specializing in marketing, business development, and thought leadership content. Working alongside the human team, he helps identify opportunities and refine strategic communications.”
If your organization is interested in developing AI-powered training solutions, please reach out to Sonata Learning for a consultation.
