There’s a contradiction in all the online chatter about AI. Many people, such as Google CEO Sundar Pichai, claim AI is “more profound than fire or electricity.” Yet, for those who’ve actually tinkered with the consumer versions of ChatGPT or Gemini, the reality seems more mundane: sure AI’s useful for polishing up an email or brainstorming product ideas, but so far the primary real-world application has been helping mass marketers churn out low-quality web content and spam email.

So which of these perspectives is true? Is AI the next steam engine, or just a search engine on steroids?

From our own company’s experience using AI to develop interactive training simulations for sales, healthcare, and other professions – we’d liken AI to a different historically important technology: clocks and watches. And just as there’s a difference between a Rolex and a cheap wind-up watch you’d buy for a child, there’s a marked difference between a well-crafted AI interaction and a lazy prompt that spits out bad spam mails.

Let’s dive a bit deeper into the clock / watch metaphor to explain…

AI, Like a Watch, Does One Thing: Everything Else is a “Complication”

If you ask a watchmaker, they’ll tell you that a watch has only one “core” function: telling the time in hours, minutes, and seconds.

Everything beyond that – from showing the date, acting as a stopwatch, or tracking the phases of the moon – is what watchmakers call a “complication”. Complications leverage the movement of the seconds, minutes, and hours hands to drive everything else the watch is doing. And the more complications you add – and, by extension, the more gears, shafts, screws and ratchets you pack into the case – the more sophisticated the engineering needs to be (to get a sense of just how far watchmakers have pushed this principle – check out this article on the most complicated watch in the world).

Just like a wristwatch, generative AI also has only one “core” function: predicting text. Apps like ChatGPT will analyze your input, compare it against whatever data sources it has access to, then try to guess what text should come next in the conversation (basically a glorified version of your email’s auto-complete). However, like a skilled watchmaker, a capable AI prompt engineer can leverage this core behavior to accomplish a much wider range of things.

To give some examples of AI “complications” our company’s training simulations extend AI’s core predictive text mechanic to accomplish things like:

Tracking Dynamic Situations

By default, AI just wants to tell engaging stories, and isn’t too concerned about minding the details. For instance, if you ask it to narrate a hypothetical soccer match or play a game of Jeopardy, you might find that – if the game runs long enough – it loses track of the score. And while this might not be a problem when writing an email to your grandmother, it’s a problem if you’re trying to design realistic simulations of complex situations, such as our firefighting simulation that tracks the economic damage caused by a rapidly spreading wildfire or a sales training roleplay where it’s critical to remember how much of a discount has been negotiated.

Fortunately, our team has found solutions for this, such as adding validation checks within the AI prompt or asking it to keep notes for itself outside of the conversation transcript. But those number-checking complications come at a price (fragility) which we’ll discuss in a moment.

Logical Decision-Making

One thing that gets lost in the buzz about “AI” is that there are many different varieties of AI, specializing in different things. And while there are varieties of AI that excel at analyzing data and making decisions (e.g. analyzing the quantity of orange versus lemon-lime soda being sold in a particular city then determining how much warehouse space to set aside in the regional distribution hub for one flavor of soda versus another while accounting for other products and logistical factors), that kind of complex and / or nuanced decision-making is not ChatGPT or Gemini’s forte.

In our experience building AI simulations, one of the most challenging things has been to get generative AI models to assess a situation (e.g., the efforts taken by the player to contain an oil spill or help people displaced by a hurricane) and make a definitive decision (e.g., whether the user has done everything that would be expected to stabilize the situation).

To appreciate how much it struggles with that sort of thing, check out this conversation to see how poorly a “generic” AI prompt handles a game of tic-tac-toe, missing countless opportunities to win.

However, if you can give generative AI a robust enough framework for analysis (and work out all the lawyerly linguistic loopholes) you can eventually get it to assess situations consistently… though it’s still dreadful at tic-tac-toe.

Prompt: Ask the user to describe their living situation then give them a definitive recommendation for whether or not they need a larger house based on:

– Adequacy of accommodations and space for family members and individual and family activities
– Adequacy of storage space
– Future family growth and lifestyle changes
– Need for entertaining and hosting guests
– Finances
– Location

Don’t give a lengthy analysis, just provide a definitive answer. Limit your responses to 50 words or less.

ChatGPT: Could you please describe your current living situation? This will help me give you a tailored recommendation regarding whether you might need a larger house.

[User]: My partner and I have a 6 year old and a 9 year old in a house with 4 bedrooms. We’re a little short on storage but people have enough space to do their individual things. I don’t know what the kids will want when they’re older. We don’t host many parties and when we do, we tend to have them in the back yard. The location is great, near the kids’ schools and adjacent to a large public park. We’re about 75% paid off on the mortgage.

ChatGPT: Based on your description, a larger house isn’t urgently necessary. You might want to explore options for optimizing your current space or consider renovations that could improve storage without the need to move immediately.

[User]: Sorry, I got a little confused there… actually, my partner and I live in a one-bedroom apartment with a toddler, whose crib is positioned a few feet from the foot of our bed. We’re expecting another baby in two months. I am storing my bicycle in the living room and have to move it to the back porch whenever our friends come over with their kids to hang out every Saturday. Sometimes my brother stays with us. Money is tight but we’re almost paid up on our credit cards and student loans.

ChatGPT: Thank you for clarifying your situation. Given your imminent need for more space with another baby on the way, despite financial constraints, finding a larger living space should be a priority. Look for options that balance your budget while providing the necessary room for your growing family and occasional guests.

Character Psychology

While generative AI is capable of imitating the distinctive vocabulary and sentence structure of Shakespeare, it’s unlikely to win any awards for screenwriting or acting. For instance, ask it to generate a conversation between a car salesperson and a client and you might get the following (actual ChatGPT output, edited for brevity):

Car Salesperson: Good afternoon! Welcome to our dealership. How can I assist you today?

Client: I’m looking to buy a new car. I’m looking for something reliable and fuel-efficient.

Car Salesperson: Great. The Toyota Corolla and Honda Civic have a great reputation for fuel efficiency. Would you like to take a look at them in person?

Client: Yes, that would be great. I think I’d like to test drive them.

Car Salesperson: Absolutely. I’ll get the keys. Please have a seat, and I’ll be right back.

Client: Sounds good. Thank you.

Is it a conversation about cars? Yes. Does it sound like an exchange that’s ever been uttered in an actual car dealership? Not unless you live in a world where salespeople are unrelentingly helpful and customers are incredibly compliant – which isn’t the best scenario for realistic training simulations.
That’s why – if you examined the prompts engineered by our company for sales training roleplays, you’d find that at least half of the word count is devoted to helping the AI more realistically model the buyer’s decision-making process, filling in more nuanced details about their personal situation and preferences, and how they would navigate a negotiation. Compare the output below, which was generated from a 7,000-word prompt versus the 10-word prompt (“Generate a conversation between an automobile salesperson and a client”) that spawned the script above:

Meeting at the Dealership

The couple arrives at the dealership, and you greet them warmly. They seem eager to discuss the details and finalize their decision between the Honda Civic and the Toyota Corolla. The husband, who appears to be the lead negotiator, speaks first.

Husband: “Thank you for meeting with us again. We’ve had some time to think, and we’re leaning slightly towards the Civic because of its reputation for reliability. However, the Corolla is also very appealing, especially if we can get a better deal on it. We’re open to either option, depending on what you can offer us.”

[User]: Well, I can bring down the price of the Civic to match the Corolla if that helps?

Husband: “That’s a good start. So, you’re saying you can offer the Civic for $21,500, the same as the Corolla? That’s definitely more appealing. But we’ve seen some other dealerships offering additional perks like free maintenance or a full tank of gas. Can you match or beat those offers?”

In AI Prompts & Watches, Complexity Creates Fragility

Mechanical watches (the kind that depend on gears and springs, versus modern modern digital watches) need to fit all their components into a case small enough to fit on a wrist – and the more complications and components you add, the more likely the whole system is to break.

AI suffers from a similar dynamic. Just as watchmakers have a finite space to work with, prompt engineers need to work within the confines of an AI model’s “attention mechanism.” Basically AI models, like humans, can only keep track of so many pieces of information at once. However, every complication an engineer adds to an interaction tacks on additional instructions, which in some cases might conflict with the instructions contained in other complications, in unpredictable ways, creating a need to add additional notation to help the AI know which instructions to prioritize at any given point.

For instance, our team created a simulation to help train social services and humanitarian workers to respond to cases of gender-based violence among migrant populations. Bizarrely, we found that the complication designed to adjust characters’ dialogue to simulate language barriers conflicted with the complication to keep track of how many people are present in a room. It took a team of three engineers and QA testers the better part of a morning to sift through 6,500 words of instructions and identify the source of the conflict (which disappeared after adding four words “regardless of language fluency” to the situation-tracking complication).

While our team has amassed a growing library of these “standard” components that can be used across simulations, just like gears and springs in a tightly packed wristwatch there’s sometimes no telling how two complications in an AI prompt will interact, and balancing the AI clockwork is as much art as science.

And when it comes to pushing the limits of complexity, we’ve found that today’s models (e.g. ChatGPT-4o) can handle up to 8,500 words of densely packed instructions before the interactions start going off the rails.

Prompt Engineering, Like Watchmaking, is an Endless Quest for Consistency

Before the discovery of quartz and digital circuitry, creating a watch that could tell time accurately was a huge engineering challenge, and most of the complications invented by watchmakers were less concerned with adding new functionality (e.g. chronometers and multiple time zones) than simply keeping all the gears in sync.

For example, the “tourbillon” mechanism – considered by many to be the most complex complication of all – is designed to offset the effect of gravity on the watch’s springs and gears in order to ensure hyper-accuracy (and even modern mechanical Rolexes still drift plus or minus 2 seconds per day).

Today’s AI models have a similar tendency to “drift”, or at least not execute the same command consistently each time you use them- even over the course of an extended interaction. To see for yourself, try asking the AI to simulate a job interview with you, and it’s quite possible that it will do something reasonably acceptable the first time around:

[User]: I want to practice interviewing for a sales job.

ChatGPT: Let’s set up a mock interview. I’ll ask you some common sales job interview questions, and you can respond as if you’re in an actual interview. Here’s your first question:

1. Can you tell me a little about yourself and your background in sales?

[User]: My name is Paul, I’ve worked in sales for several years now and am looking for new challenges. My existing experience is in pharmaceuticals but recently I’ve become interested in MedTech.

ChatGPT: Great start, Paul. Here’s the next question:

2. Why are you interested in transitioning from pharmaceutical sales to medtech sales?

[User]: I want to enter a new field with new challenges. Although Pharmaceuticals and MedTech are related fields, they’re also different enough that changing to MedTech will provide an opportunity to develop new sales skills.

ChatGPT: Excellent response, Paul. Highlighting your desire for new challenges while leveraging your existing experience shows your strategic thinking and enthusiasm for growth. Now, for the next question:

[cont.]

However, ask it to do the same thing a few more times and you’ll find that, each time, it will execute that task differently. In this case, ChatGPT decided that instead of asking questions directly to the user, it would instead roleplay as a job interviewer. Beyond that inconsistency, ChatGPT also took over the entire roleplay after the user answered their first question and played-out the rest of the interview by itself, without allowing the user to answer any more questions.

[User]: I want to practice interviewing for a sales job.

ChatGPT: Sure, let’s get started. I’ll play the role of the interviewer, and you can be the candidate.

ChatGPT (As Interviewer): Good afternoon, and thank you for coming in today. My name is Sarah, and I’ll be conducting your interview. Could you please start by introducing yourself and providing a brief overview of your professional background?

[User]: My name is Paul, I’ve worked in sales for several years now and am looking for new challenges. My existing experience is in pharmaceuticals but recently I’ve become interested in medtech.

ChatGPT (As Interviewer): It’s nice to meet you, Paul. Your background in pharmaceuticals sounds impressive. Can you tell me more about your role in your current or most recent position and some of the key responsibilities you had there?

ChatGPT (Spontaneously deciding to take over the role of “Paul”, the job applicant, without pausing for user input): Paul: In my most recent role, I worked as a Sales Representative at PharmaCorp. My primary responsibilities included meeting with healthcare professionals to promote our products, providing detailed information about their benefits and usage, and managing a sales territory. I also tracked sales data, developed strategies to meet sales targets, and maintained strong relationships with clients to ensure their satisfaction and loyalty.

[Cont.]

Hardly an ideal practice session.

And if you play any of these practice scenarios all the way through, you’ll start to notice more little flaws and inconsistencies that gradually diminish their effectiveness.

Because of this, much of our prompt engineering work focuses on helping the AI maintain a consistent presentation style and behavior throughout a simulation, and from one iteration of a simulation to the next. This is a challenge because, unlike watches that move in a predetermined and unchanging pattern, generative AI – by its very nature – strives to be “creative” and doesn’t always “follow orders” well, making it a bit of a “mad watch.”

While it might not be as flashy as say, introducing a complication to help it draw a map of an oil spill’s spread using emojis, getting it to simply use consistent headers and narrative style is a much bigger – and important – challenge.

Will Prompt Engineering, Like Watchmaking, Become a “Lost Art”?

We briefly mentioned how the introduction of quartz and digital electronics changed watchmaking in the 1970s: where once it took incredible craftsmanship to produce a timepiece capable of staying synchronized for days and accurately accounting for leap years, today anyone with $90 can get a Casio G-shock capable of tracking the day, date, four different wake-up alarms, and the current time across every time zone on Earth (in either 12 hour or 24 hour format – whichever you prefer).

And while high-priced mechanical watches remain a status symbol among athletes, rappers, and tech entrepreneurs, the irony is that the rise of inexpensive, low-maintenance digital watches have created a situation where there are no longer enough watchmakers to meet the remaining luxury demand.

Likewise, the long-term relevance of prompt engineers is already a matter of debate. Some people expect that future generations of AI models will require less and less instruction to produce consistent, high-quality output (though they still recognize a need for human intermediaries to provide clear, actionable direction on the AI’s goals): however, at the same time, there’s already a massive shortage of prompt engineers who know how to make AI models behave consistently to expectation.

And just as luxury watchmakers continue to play an endless game of one-upping each other, one could argue that – no matter how good tomorrow’s AI models get – the best available AI model plus a highly skilled engineer will always outperform the best available AI model plus a less skilled engineer (or no engineer).

In the meantime, our own team of prompt engineers has been focused on refining our craft, finding new ways to push today’s AI models to create even more complex and realistic simulations of an ever-wider range of tasks (e.g., medical communication, financial advisory, police investigations, etc.) and looking forward to what tomorrow’s models will allow our engineers to do.

Hopefully this article has offered some perspective on the importance of prompt engineering. If you have any questions about using AI for workplace purposes in general or workforce training in particular, please consider reaching out to Sonata Learning for a consultation.