Saturday, February 8, 2025

Anthropic’s New AI Tool Blocks Jailbreaks and Harmful Content

AP·Newsis
AP·Newsis

AI companies are strengthening their internal censorship efforts.

They are working to prevent users from bypassing AI’s built-in restrictions and generating harmful content, known as “jailbreaking.”

As tech giants like Microsoft and Meta Platforms strive to block such jailbreaks, AI startup Anthropic has introduced innovative technology to address this issue.

The Financial Times reported Monday that Anthropic announced a new system called “constitutional classifiers” in a research paper.

Anthropic’s constitutional classifier system operates as a gatekeeper at the highest level of large AI language models (LLMs)er. Integrated into the Anthropic chatbot Claude, it can monitor the inflow and outflow of harmful content.

Anthropic’s breakthrough comes as “jailbreaking” has become a significant concern in the AI industry.

AI systems are built on rules, often called a “constitution,” to ensure they remain under human control. This constitution determines which topics are allowed and restricted and adapts to expand its scope over time.

Jailbreaking refers to attempts to manipulate AI models, bypassing their built-in safeguards to generate illegal or dangerous information, such as instructions for manufacturing chemical weapons.

Other AI companies are also struggling to prevent such jailbreaks.

They aim to address potential regulatory concerns from jailbreaking incidents proactively.

Microsoft introduced a jailbreak prevention tool called Prompt Shields in March. Meta followed suit in July, adopting a similar system. Although researchers quickly found ways to circumvent these measures, the companies have since improved their defenses.

Mrinank Sharma of Anthropic stated that while the primary motivation for developing jailbreak prevention measures is to prevent serious threats like the manufacturing of chemical weapons, the process also yields practical benefits by finding ways to respond and adapt to jailbreak attempts quickly.

While Anthropic has not yet implemented this jailbreak prevention system, it plans to incorporate it into future, more powerful AI models.

To enhance system efficiency, the company also rewards “bug hunters.” Individuals who successfully breach Anthropic’s jailbreak prevention system can earn up to $15,000 in rewards.

These ethical hackers, known as red team members, have invested over 3,000 hours trying to bypass the jailbreak prevention system.

When Anthropic’s Claude 3.5 Sonnet model was equipped with the constitutional classifier jailbreak prevention system, it successfully blocked over 95% of these hacking attempts. However, when the system was disabled, the blocking rate plummeted to 14%.

Hot this week

After the Holidays, Beware: Intense Workouts Could Accelerate Aging

Excessive exercise can increase reactive oxygen species, accelerating aging, but hyperbaric oxygen therapy may help counteract this effect.

Nvidia Surges 5.4%, While Tesla Drops 3.6% in Latest Market Moves

Wall Street sees gains with Nvidia soaring while Tesla and Alphabet stocks tumble amid tariff concerns and disappointing earnings.

Michael Jordan’s Son Arrested for Drunk Driving and Drug Possession in Florida

Marcus Jordan, son of Michael Jordan, was arrested for DUI and cocaine possession after fleeing police in Florida.

Goldman Sachs Sees Little Disruption in Energy Prices from Tariffs on U.S. Goods

Goldman Sachs forecasts minimal impact on energy prices from China's tariffs on U.S. goods and Canadian crude oil this year.

Health Crisis Hits Former Chancellor Schröder as He Misses Key Nord Stream Testimony

Former German Chancellor Gerhard Schröder is hospitalized for burnout symptoms amid investigations into his role in the Nord Stream project.

Topics

After the Holidays, Beware: Intense Workouts Could Accelerate Aging

Excessive exercise can increase reactive oxygen species, accelerating aging, but hyperbaric oxygen therapy may help counteract this effect.

Nvidia Surges 5.4%, While Tesla Drops 3.6% in Latest Market Moves

Wall Street sees gains with Nvidia soaring while Tesla and Alphabet stocks tumble amid tariff concerns and disappointing earnings.

Michael Jordan’s Son Arrested for Drunk Driving and Drug Possession in Florida

Marcus Jordan, son of Michael Jordan, was arrested for DUI and cocaine possession after fleeing police in Florida.

Goldman Sachs Sees Little Disruption in Energy Prices from Tariffs on U.S. Goods

Goldman Sachs forecasts minimal impact on energy prices from China's tariffs on U.S. goods and Canadian crude oil this year.

Health Crisis Hits Former Chancellor Schröder as He Misses Key Nord Stream Testimony

Former German Chancellor Gerhard Schröder is hospitalized for burnout symptoms amid investigations into his role in the Nord Stream project.

Trade War Fears Subside as Wall Street Makes Strong Comeback

Wall Street bounced back after a three-day slump, driven by tech stocks and optimism over trade negotiations despite ongoing tariff concerns.

U.S. Keeps Close Watch on Korean Immigrants Amid Deportation Concerns

The U.S. State Department is providing consular assistance to Korean nationals arrested during Trump's crackdown on illegal immigration.

Ishiba and Trump Plan Talks on AI, Taiwan, and Semiconductors This Friday

Japanese PM Ishiba to discuss tech cooperation and Taiwan issues with Trump, emphasizing rule of law and bilateral ties expansion.

Related Articles