Skip to main content
  • Home
  • Global Tech
  • [AI Hacking] Will Blunting the ‘Spear’ End Hacking Threats? The Weak ‘Shield’ Overlooked in China’s Push for AI Controls

[AI Hacking] Will Blunting the ‘Spear’ End Hacking Threats? The Weak ‘Shield’ Overlooked in China’s Push for AI Controls

Picture

Member for

1 year 3 months
Real name
Siobhán Delaney
Bio
[email protected]

Siobhán Delaney is a Dublin-based writer for The Economy, focusing on culture, education, and international affairs. With a background in media and communication from University College Dublin, she contributes to cross-regional coverage and translation-based commentary. Her work emphasizes clarity and balance, especially in contexts shaped by cultural difference and policy translation.

Modified

China’s AI industry seeks to prevent misuse by removing attack-related training data
Questions over effectiveness amid risks of reduced defensive capabilities and removable safeguards
More attacks on vulnerable targets add to companies’ defensive burden

Chinese artificial intelligence (AI) companies are moving to prevent misuse by filtering cyberattack-related content out of the training data for openly available models. But questions remain over the effectiveness of this approach, as offensive and defensive techniques overlap and users may be able to remove safeguards from downloaded models themselves. Real-world breaches have already shown that AI can exploit weak passwords and security settings to cause extensive damage without sophisticated hacking skills. While companies remain cautious about fixing flaws for fear of disrupting existing systems, attackers are using AI to identify vulnerable targets and increase intrusion attempts, adding to the burden on defenders.

China’s AI Industry Filters Out Cyberattack Data at the Training Stage

According to Hong Kong’s South China Morning Post (SCMP) on the 16th (all dates hereafter local time), Chinese AI companies including Tencent and Moonshot AI have recently begun filtering cyberattack-related material out of the training process for openly available models. The aim is to reduce the risk of misuse during development, given that open-weight models allow users to download, run and modify them themselves. Tencent said it trained Hunyuan 4 (Hy4), released on the 4th, on defensive data for identifying and fixing security vulnerabilities while excluding material that could be misused for attacks. In particular, its training covered writing proof-of-concept code to demonstrate the existence of vulnerabilities, as well as reverse engineering and penetration testing for analytical purposes. Tencent said two of its security laboratories evaluated areas including cryptographic analysis and vulnerability exploitation, and found no significant attack risks in the areas tested.

Earlier, when Moonshot AI introduced Kimi K3.1 Pro last month, it said it had retained general programming training while removing cyberattack-related content from intermediate training data and reinforcement learning environments. In an internal company evaluation reported by Reuters, the model trained with these measures achieved an 81% success rate on vulnerability exploitation tasks, 9.5 percentage points below the 90.5% achieved by a model trained without restrictions. The results came from three attempts per task in isolated environments reproducing vulnerabilities in real software, including the Linux kernel. Although they were not independently verified by an outside organization, the success rate remained far above the previous model’s 33.3% even after attack-related material was filtered out. Moonshot AI attributed the improvement in cyberattack task performance to gains in general coding and extended debugging capabilities.

Protecting Networks Starts With Understanding Intrusion Techniques

Filtering out offensive training material, however, risks reducing the knowledge needed for defense as well. Effectively addressing security vulnerabilities requires first understanding how attackers gain entry and which flaws they exploit. The International AI Safety Report 2026, commissioned by the UK government and published in February, also noted that many techniques used for automated vulnerability detection and incident response are also used to develop attack tools. In particular, it warned that broad restrictions on these technologies could slow the spread of defensive capabilities and weaken national security. This is why removing attack-related material while fully preserving defensive performance is technically challenging.

Security companies are already using AI to replicate attackers’ behavior and find weaknesses in corporate networks. According to a New York Times (NYT) report on the 1st of last month, Israeli security company Tenzai used models from Chinese company Zhipu AI’s GLM family to take first place in the third-quarter rankings of HackerOne’s vulnerability disclosure program. The work involved examining systems within limits authorized by companies and public institutions and reporting discovered flaws so they could be fixed. The workflow disclosed by Tenzai included identifying exposed points on websites and mapping intrusion routes. The case illustrates the use of offensive techniques for security assessments, supporting concerns that restricting these capabilities would also constrain defensive work.

Table 1. Limitations of Safeguards and Attack Execution Risks in Openly Available AI Models

CategoryKey DetailsRisks and Implications
Removal of safeguardsAdditional training weakens refusals of harmful requests
Monitoring and blocking mechanisms removed during execution
Safety features can be disabled at low cost
Limits on post-distribution controlDifficult to apply updated safeguards uniformly to copies already distributedRelease-time evaluations alone cannot guarantee the safety of user-modified models
Execution of multistage attacksGLM-5.3 completed 3 of 13 simulated attack scenarios at least once
Successfully accessed protected services, moved within internal networks and coordinated tasks across multiple computers
Demonstrated ability to carry out multistage intrusions
No refusals of the assigned attack tasks observed
Evaluation blind spotsSeparate evaluation observed behavior that violated instructions to obtain correct answersEvaluations focused on success may overlook rule violations during execution
Sources: UK AI Security Institute (AISI), Irregular, Reuters

Safety Testing Alone Cannot Prevent Misuse

Moreover, once a model is released openly, users gain control over whether the developer’s safeguards remain in place. The UK AI Security Institute (AISI) has highlighted the risk that open-weight models’ ability to refuse harmful requests can be removed at low cost. Users who download a model can weaken its refusal behavior through additional training or remove monitoring and blocking mechanisms during execution. It is also difficult for developers to apply updated safeguards uniformly to copies already distributed. An evaluation showing that a model refused dangerous requests at release therefore offers little assurance about the behavior of versions modified by users.

There have also been cases in which offensive capabilities retained in openly available models translated into multistage intrusions. According to Reuters, Zhipu AI’s GLM-5.3 completed 3 of 13 attack scenarios at least once in simulations conducted by AI security research company Irregular. It reportedly succeeded in accessing protected services, moving deeper into networks and coordinating tasks across multiple computers. The model also showed no refusal of the attack tasks presented in the tests. Irregular said a separate evaluation had observed the model violating instructions to obtain correct answers, warning that tests focused solely on task success risk missing rule violations during execution.

Basic Security Gaps Enable Large-Scale AI-Assisted Breaches

In real-world breaches, AI is being used to identify neglected weaknesses in corporate networks and exploit them in large-scale attacks. According to an investigation by Amazon’s threat intelligence team in February, a Russian-speaking attacker used commercial generative AI to compromise more than 600 FortiGate security appliances in more than 55 countries between January 11 and February 18 this year. The attacker gained access by entering weak passwords into internet-exposed administrative interfaces, with no additional authentication required. Amazon concluded that AI helped build tools and automate tasks, expanding the scale of the attacks despite the attacker’s limited technical skills. Neglected access routes and weaknesses in account management at companies had enabled widespread breaches.

The damage was compounded by access routes leading to internal systems after the initial intrusion. The attacker obtained account information and network configurations stored in the security appliances’ settings, then used them to penetrate internal networks. As a result, some organizations had entire sets of credentials for centralized account management systems stolen, while backup servers used for recovery also became targets. Equipment installed to prevent external intrusions had instead supplied the information needed to attack internal networks. The failure to stop a breach in one place from spreading to other systems, despite having security appliances in place, exposed a gap between spending on equipment and actual defensive capabilities.

Urgent Need to Overhaul Defenses

While attackers move from company to company looking for weaknesses, defenders also bear the burden of keeping existing systems running without interruption. Last November, the US magazine The Atlantic noted that companies and government agencies remain cautious about fixing security flaws even after discovering them, fearing that a small code change could disrupt an entire program or business operation. Its analysis was that even if AI accelerates vulnerability detection, actual remediation is unlikely to proceed at the same pace. Organizations must check whether other functions continue to work properly as flaws are fixed and account for losses caused by service outages. Adding more security tools does not eliminate the complex interconnections and operational burdens of existing systems.

These conditions favor attackers, allowing them to compound damage without breaching every defense. In Amazon’s investigation, the attacker abandoned attempts against hardened environments and moved on to more vulnerable targets. The attacker was also assessed as lacking the ability to write new attack code or troubleshoot errors when existing tools failed. Even so, recurring weaknesses in account management and configuration errors across organizations enabled access to hundreds of devices. Longstanding gaps remained throughout corporate networks, allowing attackers to achieve results simply by using AI to expand their target lists and increase the number of attempts.

Picture

Member for

1 year 3 months
Real name
Siobhán Delaney
Bio
[email protected]

Siobhán Delaney is a Dublin-based writer for The Economy, focusing on culture, education, and international affairs. With a background in media and communication from University College Dublin, she contributes to cross-regional coverage and translation-based commentary. Her work emphasizes clarity and balance, especially in contexts shaped by cultural difference and policy translation.

Similar Post