AI Briefing#llm-safety
The latest AI and developer news about #llm-safety, with the original source and a short summary.
The latest AI and developer news about #llm-safety, with the original source and a short summary.
Ontario's Auditor General audited 20 AI Scribe systems and found widespread errors.
MIT CSAIL has proposed RLCR to reduce the overconfidence problem in AI models.
AI safety policy is pointed out as strong at blocking catastrophic risks but inadequate at responding to users' mental health crises.
Introduced an LLM jailbreak technique that mixes LGBT identity and role-play.
OpenAI explained the cause behind the increase in goblin and gremlin references in ChatGPT.
OpenAI Codex's system prompt includes an instruction banning mentions of creatures like goblins.
Amazon and UIUC unveiled C3LLM, which statistically certifies conversational LLM risk.
It automatically finds behavioral differences between models, surfacing hidden risk signals.
AprielGuard, an 8B-parameter guardrail model designed for the security and safety of agentic workflows, has been released.
A framework is proposed to systematically evaluate the ability to effectively monitor **Chain-of-thought**, the reasoning process of AI models.
OpenAI published research on detecting scheming, the hidden intent of AI models, and reducing it by more than 30x.
Kakao has open-sourced the 'Kanana Safeguard' series, consisting of three AI guardrail models specialized for Korean user environments, under the Apache 2.0 license.
Anthropic lowered the universal jailbreak success rate to 4.4% with Constitutional Classifiers.
OpenAI has unveiled Deliberative alignment, a technique that helps models directly reason about and comply with safety specifications.
Google DeepMind and Hugging Face have released SynthID Text, which embeds and detects watermarks in AI-generated text.
Haize Labs has released a red teaming resistance benchmark that measures LLM vulnerability to sophisticated human attacks.
A new Safety Leaderboard based on DecodingTrust, which comprehensively evaluates LLMs' toxicity, bias, security, and more, has been released.
This covers the concepts and methodology of red-teaming to prevent harmful outputs from LLMs.