Building a Path Toward AI Accountability

Anthropic recently responded to the NTIA's Request for Comment on AI accountability, proposing accountability mechanisms for high-performance and general-purpose AI models.

The key recommendations are as follows:

• Support for evaluation research: Government support for AI model evaluation research should be expanded, and companies should be required to disclose evaluation methods and results to the extent that this protects their intellectual property. In the longer term, industry standards and benchmarks should be established in collaboration with NIST.
• Risk-responsive evaluations: Standardized capability evaluations targeting key risks such as deception and autonomy should be developed. Based on established risk thresholds, models that exceed the threshold and lack sufficient safety measures should have their deployment halted and regulators notified.
• Pre-registration of large-scale training runs: Before AI developers begin large-scale training, a confidential registry should be introduced requiring them to report model specifications, computing infrastructure, and safety plans to the government, in order to prepare for potential risks.
• Strengthening third-party auditor capacity: Flexible auditors with strong technical understanding and security awareness capable of protecting intellectual property and national security should be utilized.