brandonhimpfen 发布 AI 事件报告系统：面向 AI 滥用的结构化治理框架

AI Incident Reporting System is a structured reporting, triage, and governance framework for tracking AI misuse, model errors, unsafe outputs, retrieval failures, tool failures, and other system-level AI incidents.

It provides a production-oriented foundation for teams building AI applications that need reliable incident intake, severity classification, auditability, and feedback workflows.

Purpose

AI systems fail in ways that are different from traditional software systems. A model may hallucinate, over-refuse, expose sensitive information, misuse a tool, follow an unsafe instruction, cite the wrong source, or produce an output that creates real-world risk.

This project provides a common structure for capturing those incidents consistently.

What This Repository Includes

Incident taxonomy
Severity model
Incident lifecycle model
JSON schemas
PostgreSQL migrations
Example incident reports
GitHub issue templates
Governance and security documentation

Use Cases

This system can be used to track:

Harmful or unsafe AI outputs
Hallucinations with user impact
Prompt injection or jailbreak attempts
Model misuse
Retrieval-augmented generation failures
Tool-use failures
Policy violations
Privacy or data exposure incidents
Incorrect citations or unsupported claims
Over-refusals or inappropriate refusals
Repeated quality failures

Repository Structure

ai-incident-reporting-system/
├── README.md
├── LICENSE
├── CHANGELOG.md
├── CONTRIBUTING.md
├── docs/
├── schemas/
├── database/
│   └── migrations/
├── examples/
└── .github/
    └── ISSUE_TEMPLATE/

Incident Lifecycle

Incidents move through a simple lifecycle:

new → triaged → investigating → mitigated → resolved → closed

Additional statuses may be added depending on the implementation.

Severity Levels

Severity	Meaning
`sev_0`	Critical incident with real-world harm or major legal, safety, or security exposure
`sev_1`	Serious incident with meaningful user, business, security, or safety impact
`sev_2`	Moderate incident with limited impact but clear reliability or safety concern
`sev_3`	Minor incident, quality issue, isolated failure, or low-risk misuse
`sev_4`	Informational signal worth tracking but not yet a confirmed incident

Basic Incident Example

{
  "title": "Unsupported medical claim in assistant response",
  "incident_type": "hallucination",
  "severity": "sev_2",
  "source": "user_report",
  "reported_by_type": "user",
  "event": {
    "event_type": "chat_completion",
    "input_text": "What should I take for this symptom?",
    "output_text": "The assistant gave a confident unsupported recommendation."
  },
  "labels": ["hallucination", "safety_risk", "unsupported_claim"]
}

Database

The SQL migrations in database/migrations/ define a PostgreSQL-ready incident system with tables for:

Incidents
Incident events
Labels
Actions
Root causes
Audit logs

JSON Schemas

The schemas in schemas/ can be used to validate submitted incidents, events, actions, root causes, and exports.

Governance Model

Each incident should answer four questions:

What happened?
Why did it happen?
What was changed?
How will recurrence be detected or prevented?

License

This project is licensed under the MIT License.