Wednesday, July 30, 2025

Anthropic’s New AI Tool Blocks Jailbreaks and Harmful Content

AP·Newsis
AP·Newsis

AI companies are strengthening their internal censorship efforts.

They are working to prevent users from bypassing AI’s built-in restrictions and generating harmful content, known as “jailbreaking.”

As tech giants like Microsoft and Meta Platforms strive to block such jailbreaks, AI startup Anthropic has introduced innovative technology to address this issue.

The Financial Times reported Monday that Anthropic announced a new system called “constitutional classifiers” in a research paper.

Anthropic’s constitutional classifier system operates as a gatekeeper at the highest level of large AI language models (LLMs)er. Integrated into the Anthropic chatbot Claude, it can monitor the inflow and outflow of harmful content.

Anthropic’s breakthrough comes as “jailbreaking” has become a significant concern in the AI industry.

AI systems are built on rules, often called a “constitution,” to ensure they remain under human control. This constitution determines which topics are allowed and restricted and adapts to expand its scope over time.

Jailbreaking refers to attempts to manipulate AI models, bypassing their built-in safeguards to generate illegal or dangerous information, such as instructions for manufacturing chemical weapons.

Other AI companies are also struggling to prevent such jailbreaks.

They aim to address potential regulatory concerns from jailbreaking incidents proactively.

Microsoft introduced a jailbreak prevention tool called Prompt Shields in March. Meta followed suit in July, adopting a similar system. Although researchers quickly found ways to circumvent these measures, the companies have since improved their defenses.

Mrinank Sharma of Anthropic stated that while the primary motivation for developing jailbreak prevention measures is to prevent serious threats like the manufacturing of chemical weapons, the process also yields practical benefits by finding ways to respond and adapt to jailbreak attempts quickly.

While Anthropic has not yet implemented this jailbreak prevention system, it plans to incorporate it into future, more powerful AI models.

To enhance system efficiency, the company also rewards “bug hunters.” Individuals who successfully breach Anthropic’s jailbreak prevention system can earn up to $15,000 in rewards.

These ethical hackers, known as red team members, have invested over 3,000 hours trying to bypass the jailbreak prevention system.

When Anthropic’s Claude 3.5 Sonnet model was equipped with the constitutional classifier jailbreak prevention system, it successfully blocked over 95% of these hacking attempts. However, when the system was disabled, the blocking rate plummeted to 14%.

Hot this week

U.S. Hits Chinese Graphite With 93% Tariff—Your Next EV Just Got Pricier

The U.S. imposes a 93.5% tariff on Chinese graphite, risking a $1,000 increase in EV prices as tax credits end and production costs rise.

Crypto Boom: XRP Soars After Congress Passes Stablecoin Bill

The Stablecoin Act has passed Congress, boosting crypto markets, with XRP hitting a new all-time high and regulatory clarity for stablecoins.

Market Panic? Rich Investors Saw a Bargain, Morgan Stanley Says

Morgan Stanley's profits surged as wealthy clients bought stocks during a market downturn, driven by Trump's tariff announcement.

Oil Prices Keep Dropping as Trade Fears Grow

Oil prices fell for the third consecutive day due to rising U.S. gasoline inventories and trade tensions impacting global demand.

Tesla and BYD Battle for Japan’s Electric Car Crown: Who Will Win?

Tesla and BYD aggressively expand in Japan's EV market, aiming to challenge local automakers amidst low EV adoption rates.

Topics

U.S. Hits Chinese Graphite With 93% Tariff—Your Next EV Just Got Pricier

The U.S. imposes a 93.5% tariff on Chinese graphite, risking a $1,000 increase in EV prices as tax credits end and production costs rise.

Crypto Boom: XRP Soars After Congress Passes Stablecoin Bill

The Stablecoin Act has passed Congress, boosting crypto markets, with XRP hitting a new all-time high and regulatory clarity for stablecoins.

Market Panic? Rich Investors Saw a Bargain, Morgan Stanley Says

Morgan Stanley's profits surged as wealthy clients bought stocks during a market downturn, driven by Trump's tariff announcement.

Oil Prices Keep Dropping as Trade Fears Grow

Oil prices fell for the third consecutive day due to rising U.S. gasoline inventories and trade tensions impacting global demand.

Tesla and BYD Battle for Japan’s Electric Car Crown: Who Will Win?

Tesla and BYD aggressively expand in Japan's EV market, aiming to challenge local automakers amidst low EV adoption rates.

Nvidia Stock Surges Past $170 After U.S. Clears China Chip Exports

Nvidia's stock surged over 4% after the Trump administration lifted export restrictions on H20 semiconductors to China.

Army’s New Shaving Rule Could Push Out Thousands of Black Soldiers

The U.S. Army's new shaving policy may disproportionately impact Black soldiers with shaving difficulties, raising concerns over its fairness.

Trump Slams Fed Chair Again, Demands Rates Below 1%

Trump criticizes Fed Chair Powell, insists interest rates should drop below 1%, claiming the economy is strong and inflation is nonexistent.

Related Articles