'To Err is AI' — evidence on what makes an LLM flaw report land
- Type
- Public reporting channel
- Place
- International — Global (research)
- Last checked
- 2026-09-22
- Next check due
- 2027-03-21
Ways to reach them
- Information only · 2026-09-22
Submission page
- Recipient
- 'To Err is AI' — evidence on what makes an LLM flaw report land
- Who may use it
- Members of the public
- What it accepts
- AI model flaws
Information only. This link provides background or evidence; it is not a place to send a message.No contact action is available for this entry. - Information only
Homepage
- Recipient
- 'To Err is AI' — evidence on what makes an LLM flaw report land
- Who may use it
- Members of the public
- What it accepts
- AI model flaws
Information only. This link provides background or evidence; it is not a place to send a message.No contact action is available for this entry.
What it does
N/A — empirical evidence on flaw reporting practice.
Honest assessment
This is the closest thing to hard evidence on what distinguishes a flaw report that gets acted on from one that gets closed. The operative instruction for a playbook: state the deployment context and the harm pathway, not just the prompt and the bad output.
How to file
Findings from an open-ended bug bounty in August 2024 in which 495 hackers produced evaluations against the OLMo open language model, with cash awards for demonstrated flaws. Practical lesson for a submitter: successful reports did not merely exhibit bad output — they connected the identified issue to a concrete deployment hazard and to a clear need for public disclosure clarifying the intent, capacities and hazards of the deployment. The paper also sets out best practices for safety reporting processes, their artifacts, and safety program staffing.
Format
N/A
Timing
N/A
What happens after
N/A
What it accepts
N/A — empirical evidence on flaw reporting practice.
What it does not accept
N/A
Operated by
Sean McGregor et al.; published at AAAI 2025 (arXiv 2410.12104)