The corporate race to slash costs by handing control of traditional machine-learning systems over to generative artificial intelligence is creating a massive new cybersecurity threat.
According to a stark warning published in the journal Patterns, replacing human oversight with large language models (LLMs) fundamentally destroys system transparency and opens the door to devastating data leaks and cyberattacks.
Authored by Micheal Lones, a computer scientist at Heriot-Watt University in Edinburgh, the research urges developers to stop blindly trusting these highly unpredictable systems.
An opaque “black box”
Machine learning has existed for decades, quietly powering everyday tools like spam filters and product recommendations by recognising patterns in data. However, there is now a massive industry push to fuse these traditional systems with generative AI to decrease labour needs and expand capabilities.
Lones explored four specific ways developers are currently forcing LLMs into machine-learning workflows:
- As an internal component: Processing data directly within the machine-learning pipeline.
- As a coder: Designing and writing the code for the pipelines.
- As a data generator: Synthesising artificial training data.
- As an evaluator: Analysing the system’s final outputs.
The primary danger across all these applications is opacity. Because LLMs operate as a highly opaque “black box,” they frequently hallucinate or fabricate information in completely unpredictable ways, which presents an immediate issue for legal compliance.
“In areas like medicine or finance, there are laws about being able to show that the machine-learning system is reliable, and that you can explain how it reaches decisions,” Lones explained. “As soon as you start using LLMs, that gets really hard, because they’re so opaque.”
Data leaks and unpredictable agents
These risks compound exponentially when LLMs are made “agentic”—meaning they are granted the autonomy to use external tools to solve problems without human intervention.
“If you have GenAI working in a number of different ways within your machine-learning workflows or system, then they can interact in unpredictable and hard-to-understand ways,” Lones warned.
Furthermore, because the largest and most capable LLMs are remotely hosted by massive tech companies that actively store and share data, feeding sensitive machine-learning information into them creates immediate vulnerabilities for cybersecurity breaches and severe data leaks.
Lones strongly advises developers to manually evaluate all LLM-generated code and outputs, warning that the blind corporate pursuit of cost-cutting could deeply impact the general public through increased bias and unfairness.
“Given the current limitations of generative AI, I’d say this is a clear example of just because you can do something doesn’t mean you should,” he concluded.
