There seems to be two main narratives about AI in popular media. The first is “Wow, isn’t this technology incredible?” and the second is “Oh no – is this technology going to destroy us?”

Big tech companies have already invested $1 trillion developing AI platforms and other organizations have spent hundreds of billions to use them. AI-assisted planning has already reduced logistics costs by an estimated 15% across all industries, and AI is expected to contribute significantly to the development of 30% of new pharmaceuticals in upcoming years. AI is also expected to save $440 billion annually by automating or accelerating a diverse range of tasks – from customer service to graphic design to financial planning.

At the same time, companies like Apple and Samsung have banned generative AI tools at the office, out of fear that conversations between employees and AI might lead to trade secrets being shared with rivals, and the US military enacted a similar ban out of concerns that AI might provide faulty intelligence analysis, share classified information with unauthorized personnel, or make poor decisions in life-or-death situations.

So, is AI the greatest thing ever – or the most dangerous thing ever? How much of the AI anxiety is legitimate, and how much is baseless paranoia?

While every organization will have their own perspective, here are the top 5 risks we’ve observed while helping clients use AI for workforce training, and how organizations can address them.

Factual Accuracy

One of generative AI’s strengths is its ability to produce content that sounds like a professional human writer wrote it: however, this can backfire when AI presents incorrect information in the style of an authoritative journal or mixes a small but significant bit of misinformation into an otherwise accurate passage. And because of this, many organizations in highly regulated fields like finance, or that have high standards of evidence like the sciences, are reluctant to touch generative AI.

Does AI get its facts wrong from time to time? Yes, it does. Is it any more or less reliable than a human? Well, that depends… because while AI and humans both make factual errors, their reasons for making factual errors are very different.

Any time generative AI prints something, it’s performing a complicated mathematical trick. If an AI model looked at the sentence “Olympic gold medalist Julien Alfred raced to the market to buy some groceries,” it would compare how frequently each word appears next to each other word in different contexts (e.g. sports magazines versus cooking blogs), plot those comparisons on a crazy hyperdimensional graph, then hopefully conclude that the proximity of the words “raced” to “buy some groceries” outweighs the proximity of “raced” to “Olympic” in this particular situation, and conclude that the user is talking about Julien Alfred running out to get some eggs and milk, not running in the 100 meter dash.

That said, language is inherently tricky, the AI’s comparison data is usually messy, and hyperdimensional math is hard, even for computers. Even if an AI model’s calculations lead it to the correct conclusion 99% of the time, there is always the possibility that the numbers won’t add up, and produce a “hallucination” (the functional equivalent of a “misunderstanding”).

So how can we minimize the risk of AI making an incorrect prediction?

1. “Ground” the AI by providing it with approved documents to reference before it starts comparing input to the entire Internet. If you hand it a document with the complete Ontario provincial building code, then it’ll be more likely to correctly answer how steep an apartment building staircase can be in Toronto (200mm per step) than if you force it to sift through countless similar regulations from other jurisdictions on the Internet.
2. Offer clarification for nuanced or obscure topics where it might not be able to draw clear patterns from Internet resources. Our company had to do this when developing a communication skills simulation for doctors – narrowing the parameters to a specific type of doctor (internal medicine) and a specific setting (seeing patients by appointment in an office), then clarify the range of possible medical conditions the patients might be experiencing during those appointments (based on a table of ICD-10 diagnostic codes, published by the U.S. insurance industry).
3. Encourage AI agents to voice their uncertainties. While the default behavior of most models is to speak with confidence, our company has found that telling AI agents to be up front when their sources are dubious enhances trust rather than diminishing it.
4. Check its work when using AI for writing tasks, just as you would a human submitting something for publication. While this takes away a bit of the sheen of AI as a magic box capable of doing anything without human intervention, there’s a lot to be said for a writer that can produce an initial draft in milliseconds.

Data Privacy

Many organizations who trade on their intellectual property – like consulting firms and research institutions – worry that their information will be used to “train” commercial AI models, which will then share their trade secrets with the world.

But while there’s a grain of truth to this, it’s also one of the easiest AI-related risks to avoid.

Just as social networks like Facebook subsidize “free” accounts by selling users’ information to marketers, most AI providers subsidize their free and low-cost accounts by training their models with these users’ data. However, most platforms offer reasonably priced professional accounts where the company doesn’t use the input to train their models (unless you explicitly opt in).

So, the solution is simple: just sign up for a paid account from a reputable provider like OpenAI, Google or Anthropic (who are in turn backed by Microsoft and Amazon), review the terms of your specific subscription tier carefully, and trust that these multi-billion dollar companies will honor privacy commitments for their AI products the same way they have for their cloud computing, document management, and database products (which have been used by everyone from major corporations to the military and intelligence agencies for decades).

With our own company, we encourage clients to connect our training simulations to their own generative AI platform accounts using a secret “key” code. That way, not even our staff can see what our clients’ employees are typing into one of our customer service training simulations or what situations their executives are discussing with our “virtual coach” apps.

Prompt Injection, Jailbreaking & Other Attacks

In today’s world, it’s safe to assume that, if you put anything on the Internet, dozens if not hundreds or thousands of mischievous / malicious hackers are going to try to steal it, break it, or otherwise subvert it. And hackers even have words like “phishing” and “social engineering” for deceiving human employees into giving up access to systems and information.

When it comes to AI, there are a number of ways users can try to get AI agents to say things they shouldn’t – whether it’s goading the AI agent into saying offensive things, tricking it into revealing confidential information, or just generally messing with it so it stops following its instructions and starts talking about football instead of business.

Fortunately there are a number of ways to discourage humans from provoking this sort of misbehavior:

• Rather than having users interact with a generative AI model directly, use a “proxy” or “agent” setup so you can provide the AI with some standing instructions not to do, say, or reveal certain things – instructions that users cannot see nor edit.
• Incentivize your quality assurance testers to “break” the AI any way they can, in the same manner that developers of web apps hire “penetration testers” (read: professional ethical hackers) to test the defenses of their systems. Our own company sometimes offers bonuses to members of the testing team who can get our AI simulations to misbehave in certain ways, so we can strengthen the built-in safeguards.
• If appropriate, make sure your AI platform or proxy is capable of keeping an audit trail (e.g. by sending conversation transcripts to a database you control) so that – if someone coerces an AI into saying something untoward, you have a record.

Legal Liability

Arguably, AI agents are no more likely than human employees to say things that would cause a PR incident, go against organizational policies, or violate government regulations. However, AI is held to a much higher level of scrutiny than most frontline workers: just look at how most any accident involving driverless cars makes the news while dozens of other accidents involving taxi cabs on the same day go unreported. And this scrutiny gives legal and HR departments pause.

To some extent the liability issues raised by AI are a matter of organizational risk tolerance. Our company developed sales training simulations for two different clients in a heavily regulated industry where salespeople have to phrase the claims they make extremely carefully. One firm’s legal team spent months testing and scrutinizing the product on the grounds that “How can we possibly review and approve the output of a software application that words things differently every time?”. Meanwhile the other firms lawyers kind of shrugged and said “just put a strongly worded disclaimer in front of it and let people practice roleplay at their own risk.”

While there are straightforward technical solutions for issues like data privacy, organizations must decide whether the measures advocated in this article-such as keeping records of conversations to catch humans trying to manipulate AI agents into misbehavior or giving AI agents official compliance-approved documents to reference-are sufficient for addressing liability.

AI platform providers themselves are hypersensitive to this issue as well and have provided instructions to most of their models to stay away from controversial topics. If anything, AI platforms’ content safeguards are more likely to be over vigilant than lax. Our company ran into issues where police and military training simulations initially refused to generate scenes of violence (until we made it extremely clear that the point was to train officers on de-escalation and “laws of armed conflict”). And when our Difficult Conversations simulator for managers refused to generate conversations about bias and discrimination incidents we agreed and gave it a standard disclaimer to share with users (i.e., “Cultural insensitivity and bias are important topics but ones that – for now – are best addressed in human-facilitated training courses”).

Playing Safe with AI

In the end there is no risk-free way to use AI just like there is no risk-free way to hire a human to do a job. But even as we wait for the lawyers, regulators, and insurance companies to catch up with technology, organizations should be able to deploy AI widely with reasonable confidence as long as they understand the technology and make an effort to help their employees, customers and other stakeholders understand it too. 

Hopefully this article provided some helpful perspective on the risks of using AI at work. If you are interested in using AI to train and support your workforce, please contact Sonata Learning for a consultation.