I wish I had the time to write down my thoughts on all the important news and misleading information related to 'AI and cybersecurity'. I don't expect people to be interested in reading my thoughts on this topic, but I feel the need to write something down anyway. Even though I don't have enough time to write a clear and concise summary, recent news has urged me to write something, if only to get my thoughts down on paper. While reading the following, please bear in mind that it is an incomplete summary of my views.
Why should I study Cybersecurity? I can defend my systems by asking the AI. AI is going to harden my defenses and solve all my defensive problems.
Microsoft is one of the companies trying to convince us that AI, including in the field of cybersecurity, will solve all our problems. They are heavily pushing us to use Copilot — one of many incarnations of the LLM models that support ChatGPT — in almost all Microsoft tools.
One of the most heavily promoted applications of Copilot is to help us manage our emails by summarising received messages and drafting responses. To do this, a software module must be able to interact with an LLM and read our emails. What could possibly go wrong?
Microsoft Copilot read confidential emails without permission | Mashable
A bug in Microsoft 365 and Copilot has been causing the AI assistant to summarize emails that were explicitly labeled as confidential...The Copilot security bug reportedly bypassed organizations' data loss prevention (DLP) policies, which are used to protect sensitive information.
Let's put it differently. Not even Microsoft is able to use Copilot to harden its software effectively.
No doubt that if you were to ask a Microsoft evangelist or an AI enthusiast, 'Can I use Copilot to write secure software?', they would undoubtedly answer, 'Of course!', plainly disregarding facts such as this one.
The recent vulnerability in Notepad is even more illuminating. Notepad has been around for so long that it's hard to remember when it first appeared. It is very small software with a tiny, specific use case: handling text files. A few weeks ago, Microsoft introduced Markdown support for Notepad, but failed to write the code for handling hyperlinks correctly. Consequently, a carefully structured Markdown file could trigger code execution (!!!). While the risk posed by this vulnerability is very small for reasons that are too complex to analyse here, the question remains: why did AI not prevent the introduction of this vulnerability? After all, it is trivial software designed to provide very specific functionality.
Zero Day Initiative — CVE-2026-20841: Arbitrary Code Execution in the Windows Notepad
Let's switch to Anthropic, the company that has developed Claude, arguably the most powerful LLM for writing code. One of the tools by Anthropic supports collaboration between members of the same team, so that changes in a certain project made by a team member become easily and quickly visible to the entire team. Such changes may or may not be generated by Claude itself.
Claude's collaboration tools allowed remote code execution • The Register
Security vulnerabilities in Claude Code could have allowed attackers to remotely execute code on users' machines and steal API keys by injecting malicious configurations into repositories, and then waiting for a developer to clone and open an untrustworthy project.
Let's put it differently. Not even Anthropic is able to use Claude to harden its software effectively.
Then a quick look to the offensive side. Every now and then news about "AI-powered attacks" emerge. This fact would deserve a careful analysis (topic for another post), but let's just consider this one:
AI-augmented threat actor accesses FortiGate devices at scale | AWS Security Blog
This is a very interesting report by Amazon Web Services that analyzes a certain attack campaign that the AWS team detected and dismantled. The campaign was powered by AI-generated scripts and code.
This very simple observation contains a very important and very deep fact. One should always keep it in mind:
Notably, when this actor encountered hardened environments or more sophisticated defensive measures, they simply moved on to softer targets rather than persisting, underscoring that their advantage lies in AI-augmented efficiency and scale, not in deeper technical skill.
Another key fact:
This campaign succeeded through a combination of exposed management interfaces, weak credentials, and single-factor authentication—all fundamental security gaps that AI helped an unsophisticated actor exploit at scale. This underscores that strong security fundamentals are powerful defenses against AI-augmented threats.
And then the most important point:
Through routine threat intelligence operations, Amazon Threat Intelligence identified infrastructure hosting malicious tooling associated with this campaign. The threat actor had staged additional operational files on the same publicly accessible infrastructure, including AI-generated attack plans, victim configurations, and source code for custom tooling. This inadequate operational security provided comprehensive visibility into the threat actor’s methodologies and the specific ways they leverage AI throughout their operations.
Translation: "We discovered this attack campaign during our routine operations. We found the hosts used by the attackers, who had not managed to secure their own hosts. We extracted everything from those hosts."
And by the way, why those attackers did not harden their own hosts with the AI?
Commenti