December 31, 2024
by Kerem Proulx, Kyle Bhiro, and Jessica Huang
Fear-mongering of the capabilities and risks of AI has unfortunately led many to ignore the very real threat the use of advanced reasoning systems poses in the cyber domain. Threat groups and security teams have entered into a virtual arms race - wherein novel use cases, models, and techniques are being developed to both defend and infiltrate enterprise systems, consumer devices, government systems, etc. These capabilities are also being applied in the context of social engineering and political warfare (e.g. mass disinformation campaigns).
We need to rapidly develop and distribute advanced defensive reasoning systems to close the capability gap between “us” and our adversaries.
It is evident that Nation-state threat groups have been leveraging publicly accessible AI chat products to aid their offensive operations. It is very likely these groups have pivoted to deploying open source models or even developing their own internal models and AI systems specially trained with offensive capabilities. Although there is little public data on this, with the rise of “defensive” tools such as AI-powered pentesting, not to mention AI-native static and dynamic analysis products, the likelihood of threat actors weaponizing similar technologies is increasingly high. The same way security teams look to deploy modern security products to make their jobs easier, threat actors are modernizing their tech stack to make their “jobs” easier - the issue is that the pace of said adoption may be asymmetric, as criminal groups and threat actors may be quicker to deploy frontier technologies than most enterprises.
Russia: Russia has been accused of deploying AI-driven bots and deepfake technology to interfere in elections, spread misinformation, and conduct cyber-espionage operations against adversaries.
China: China's government reportedly uses AI to enhance cyberattacks and surveillance systems, including advanced facial recognition and AI-driven predictive policing, to maintain control over dissent.
North Korea: North Korean hacking groups, such as Lazarus Group, have utilized AI-driven malware and ransomware to conduct financially motivated cyberattacks, including the theft of cryptocurrencies and financial data.
Security debt, the accumulation of unresolved vulnerabilities within a codebase/application, at medium to large enterprises continues to rise. This is endemic of the prevailing tech industry culture of ship it now, fix it later. Which is not a bad policy in itself - the competitive edge of many organizations comes down to product quality and product velocity. However, product velocity tends to come at the cost of security - and security velocity is not always easy to generate without significant and intentional investment in security infrastructure (both in personnel and technology).
The long tail of vulnerabilities requires defenders to rapidly understand their own attack surfaces, including infrastructure and applications that are hosted on said infrastructure - however as infra gets more complex, visibility suffers. Most companies do not have the resources to invest in many of the “enterprise” cloud and application security products that are meant to solve this visibility problem nor do they have the budget to hire pentesting firms to battle test their software before release.
There are also many startups and SMBs that have become critical nodes in large enterprise tech and product stacks. These SMBs are, however, also the most vulnerable due to their lack of investment in security (due to high capital and implementation costs). Making it easy for these companies to clear their security debt and prevent it from redeveloping is one of the most effective and impactful ways to secure the modern enterprise supply chain. We need more security products that service this level of customer, especially non-security natives. Sure, the CISO and their team need better tools to do their jobs but do we need to sell single panes of glass to security folk? Do we instead need to develop plug-and-play products that actually do the work of securing critical infrastructure and systems. What of the companies that cannot afford large enterprise plans or do not even know where to start with respect to security?
When our adversaries are arming themselves with frontier technologies, it is our duty to arm ourselves and our allies with the same - and this starts with making advanced defensive capabilities accessible to all.