In October 2024, Hugging Face and Protect AI teamed up to improve security for the growing AI community using models on the Hugging Face Hub. With a shared goal of democratizing AI and ensuring its safety, the collaboration leverages cutting-edge technology like Guardian’s scanning tools to safeguard against potential vulnerabilities in machine learning models. This partnership has already made significant strides in detecting and preventing security risks, providing more secure, trustworthy resources for developers. Now, after six months of continuous work, both companies have significantly advanced their AI model security efforts, scanning millions of models and adding new detection tools.
The Progress So Far: Major Milestones Achieved
Since the partnership began, Protect AI has expanded
- PAIT-ARV-100: Detects when an archive slip attempts to write to the file system at load time.
- PAIT-JOBLIB-101: Flags suspicious code execution within Joblib models during loading.
– PAIT-TF-200: Detects architectural backdoors within TensorFlow SavedModel.
- PAIT-LMAFL-300: Identifies when a Llamafile model can execute harmful code during inference.
These additions help address a variety of model file formats and obfuscation techniques, allowing Hugging Face users to get real-time alerts and detailed vulnerability reports. As of April 2025, Protect AI has scanned over 4.47 million model versions across 1.41 million repositories, identifying thousands of unsafe or suspicious issues, with 352,000 problems detected in over 51,000 models.
Strengthening the Zero Trust Approach to AI Security
A cornerstone of Protect
A Collaborative Effort: Insights from the Huntr Community
Protect AI’s partnership with the huntr community, the world’s largest AI/ML bug bounty program, has played a key role in uncovering new vulnerabilities. Since launching the program in October 2024, more than 200 reports have been received, adding invaluable data to Guardian’s detection system. This collaborative approach ensures the platform stays updated on the latest vulnerabilities and threats.
What Undercode Says:
The partnership between Protect AI and Hugging Face represents a pivotal step forward in the broader AI security landscape. As machine learning models become increasingly pervasive in various industries, ensuring their integrity is more crucial than ever. While open-source AI promises innovation, it also brings security concerns, as many models can be easily accessed and altered by malicious actors.
From a technical standpoint, the introduction of the four new threat detection modules is a game-changer. By identifying specific vulnerabilities like suspicious Joblib code execution or architectural backdoors in TensorFlow, Protect AI is making it easier for developers to spot and mitigate risks before they become full-blown security breaches. The PAIT-JOBLIB-101 module, in particular, is a notable improvement. Joblib is widely used for saving and loading Python objects, but its vulnerabilities have often been overlooked in traditional scanning methods. This new detection feature highlights the growing sophistication in AI security tools that can detect even the most subtle of threats.
Another standout feature of the Guardian platform is its ability to detect payload obfuscation—a technique used by cybercriminals to hide malicious code within seemingly innocent files. This is especially relevant as AI models often rely on compressed or serialized formats that can easily disguise payloads, which are then loaded by the model during runtime. By recognizing the use of techniques like compression, serialization, and encoding, Guardian ensures that even the most cleverly disguised attacks can be identified.
Moreover, the partnership’s Zero Trust approach continues to demonstrate why this model is so effective in AI security. Treating any arbitrary code execution as potentially harmful, regardless of intent, significantly reduces the chances of a successful attack. This method ensures that models are scrutinized under a rigorous lens, which is crucial in an environment where one wrong move could lead to devastating consequences.
The security improvements are not just theoretical but are backed by real-time numbers. With over 226 million requests served by Protect AI, the platform’s performance is robust enough to handle the high volume of scans needed to keep up with Hugging Face’s growing model repository. The 7.94 ms response time further demonstrates the system’s efficiency in identifying vulnerabilities in real-time.
Common Attack Themes and Their Implications
The reports from the huntr community have revealed several critical attack themes in AI/ML models:
– Library-dependent attack chains: These attacks leverage vulnerabilities in popular libraries, such as PyTorch and TensorFlow, to invoke harmful code during model execution. As these libraries are widely used, the scale of potential impact is substantial.
– Payload obfuscation: Techniques like compression, encoding, and serialization are used to hide malicious payloads. These attacks are harder to detect, as they rely on sophisticated methods to bypass traditional scanning methods.
– Framework-extensibility vulnerabilities: Issues arise when ML frameworks allow external code dependencies or custom layers that can be exploited. These weaknesses, such as those found in CVE-2025-1550 in Keras, show how flexible systems can also become easy targets.
These common attack vectors highlight the complexity and evolving nature of threats in the AI ecosystem. The detection of such advanced threats requires constant adaptation and innovation, something Protect AI and Hugging Face have committed to through regular updates to their threat detection capabilities.
Fact Checker Results:
- Real-time detection and updates: Protect AI’s platform scans millions of models, identifying a vast number of vulnerabilities and issuing alerts for real-time protection.
- Expanded detection coverage: New detection modules significantly enhance security for a wider range of model file formats, including previously overlooked ones like Joblib and Llamafile.
- Crowdsourced research: Collaboration with the huntr community has helped uncover new threats and refine Guardian’s detection capabilities, ensuring a proactive defense against emerging risks.
References:
Reported By: huggingface.co
Extra Source Hub:
https://www.linkedin.com
Wikipedia
Undercode AI
Image Source:
Pexels
Undercode AI DI v2
