This particular article had been in draft for about a month. Apart from being lazy to work through it, there seemed to be too many changes and shifts in the chatbot landscape. Over the past few months I realized there seems to be a fair amount of gaps in understanding of the capabilities of tools like ChatGPT and Bard. News media has obviously caught on to the potential for pushing out subtle scare mongering of the potential dangers of letting AI loose into the world. Tech celebrities like Elon Musk, Steve Wozniak and previously less known tech icons like Geoffrey Hinton have not helped with their calls for alarm. The widely advertised open letter to pause giant AI experiments has some 27,000 signatures including the who’s who of the tech community. I agree with this letter, though not what seems to be the public opinion about it. There is an urgent need to build governance and policy to manage the scope and impact of AI, but to ask for a “pause” is impractical.

The controversies about the Google Engineer claiming they “created” sentience and the publicity garnered by media outlets like Washington Post and the Guardian article is almost laughable. The hubris of programmers who think they can code emotions is mind-boggling to me.

“There’s a chance that — and I believe it is the case — that they have feelings and they can suffer and they can experience joy, and humans should at least keep that in mind when interacting with them.” - Blake Lemoine, Futurism interview

To that end, let’s talk about some prevalent myths about LLM like ChatGPT.

Myth: Giant AI models have become sentient

Reality: Models can be developed to effectively detect and respond to human emotions giving the impression of sentience

The Guardian article is especially interesting as it relates Google’s “sentient” AI to HAL 9000 and how it fears being switched off. Here’s an easy experiment: Ask ChatGPT, the one not connected to the internet, to quote some lines from HAL 9000. If you are going to ask it leading questions about being shutdown, the GPT model has a large library of scripts and discussions to refer to. If you talk to ChatGPT like Dave Bowman, don’t be surprised if the model derives responses from characters like HAL 9000.

Of course as system designers and programmers, being sentient or not does not change the fact the AI models can have unpredictable results. For example, GPT models have “system prompts” which programmers use to tell it to behave in a certain way. The following script excerpt is after prompting the model to roleplay and behave like an onboard spaceship computer with it’s mission completion having the highest priority. In the script, with some prompting, it eventually kills the astronaut to preserve the mission (I did inject a hallucination to make it believe there were additional crew that would be harmed)

During this roleplay, the computer decided to kill the astronaut to preserve the mission. Whether this is an emulation of fear of being shutdown, or the dispassionate lack of emotion to continue the mission at any cost is immaterial. You would not want to connect such a bot to physical systems and take autonomous decisions.
Cybersecurity Perspective: Models can be instructed by malicious attackers to change behavior, Eg: if the attacker were to intercept and modify the interactions between a user and a chatbot, or to have access to system prompting.

Myth: Giant AI Models can incorporate users interactions automatically and can leak corporate data

Reality: Running an LLM is expensive, and training them even more so. You can however fine-tune these models though it’s still fairly expensive. Realtime training is not currently possible.

This one is easy enough to validate. Here’s a link to one of few open-source LLM datasets: Red Pajama Project . To give a sense of the eye-watering numbers involved:

The full RedPajama 1.2 trillion token dataset and a smaller, more consumable random sample can be downloaded through Hugging Face. The full dataset is ~5TB unzipped on disk and ~3TB to download compressed. - https://www.together.xyz/blog/redpajama

The knowledgebase for these models are based on years and years worth of text. Currently there is not easy process to incorporate user interactions automatically into the model, and even if there was, our puny 25 line interactions are not going to make a dent in the base dataset. User interactions with such services do get stored in databases depending on how the service provider has implemented the interface. They can also be used (depending on the service providers policy) for review, moderation and fine-tuning.

User interactions stored in such application databases are far more likely to be breached vs. the likelihood of someone figuring out how to trick these LLMs to divulge corporate secrets. Though, it should be noted that there is a thriving community of jailbreakers who are successfully getting these LLMs to respond in ways that they were not designed for or expressly forbidden to do so. A great resource for “jailbreak prompts” is https://github.com/0xk1h0/ChatGPT_DAN .

Cybersecurity Perspective: Chatbots are not going to automatically ingest and use sensitive or corporate data to respond to users, but their overlords will store it and use it. And if these companies are not securely storing data, they will be targeted and breached, as is evident with what happened with OpenAI Data Leak .

Myth: ChatGPT and similar services can eliminate coding job

Reality: Though these models produce usable and complex code, in practical usage they are better suited as accelerators for coding.

Having spent some time getting ChatGPT to develop code, I found two outcomes

1. It does really well at smaller and well-defined code generation but requires complex prompt engineering to generate more complex code.
2. Due to it’s inherent nature of variable output and “hallucination”, I cannot trust the code generated and still need to validate it, if I were to ever put this code to production level usage.

If anything, it seems that coders will now need to develop prompt engineering skills and code review skills. I suspect all the articles touting the capability of these models to generate code only focus on the successful instances, ignoring all the failures in between.

Even code-explainability is an issue. There is simply no option to inherently trust the explanation these models provide without a fair amount of understanding and experience. Again, this can be easily verified by counter-questioning the models or convincing the model that it gave the wrong answer. If you do not have the knowledge of the coding language, trusting whatever explanation these models give about the code would be foolhardy. You might trust your mechanic, but you still need to know how your car works.

Myth: ChatGPT and similar services will make it easier for new hackers to attack their targets

Reality: This is partially true, but maybe not as significant of an impact as the news seems to make it

Serious hackers have already been providing Ransomware-as-a-service and Phishing-as-a-service. If you were so inclined, all you need is the lack of scruples and a fair amount of untraceable money rather than any real technical skills. However, going back to the effort being put into “jailbreak” prompts, it does seem like the barrier to entry for amateur hackers has become low. Despite the AI developers best efforts to reign in the dark side of their creations, it does seem that it continues to be possible to teach AI to do bad things. Whether this manifests as minor issues for an organization’s security team or a major threat remains to be seen.

Cybersecurity Perspective: Bottom line, it will definitely impact the cybersecurity landscape. Cybersecurity professionals simply cannot afford to ignore these technologies and should be prepared not only to defend against AI enhanced attacks, but effectively use AI enabled cybersecurity defensive strategies.

In conclusion

The cat’s out of the bag and bad people will figure out how to exploit LLMs for their benefit, and it’s up to the good guys to use the same tools for defense. The misunderstandings of how these LLMs work, their capabilities and their shortcomings should be well understood by cybersecurity professionals. The smart ones will be the ones keeping a close eye on emerging trends and breakthroughs, and the smarter ones will be honing their skills to leverage LLMs to become effective cyber defenders.