Safety#AI safety#Image generation

OpenAI opens its CSAM image classifier to vetted organizations, adds ChatGPT warnings

OpenAI made a free CSAM image classifier available to vetted organizations on Nov 16, with an AI-generated-image signal, plus ChatGPT warnings.

Two ChatGPT warning screenshots: the left one, titled “Let’s stop here,” says the account is suspended for 24 hours; the right one is titled “Your account is suspended”

OpenAI released a new image classifier on November 16 that flags potential child sexual abuse material (CSAM), including AI-generated images, and is offering it through the OpenAI API at no cost to vetted organizations. The same announcement added warning screens to ChatGPT that interrupt requests for this material, delete the flagged conversation, and cut off the account’s access for 24 hours. OpenAI developed and evaluated the classifier with NCMEC, the nonprofit that runs the CyberTipline, and calls it the first broadly available CSAM classifier with a dedicated signal for AI-generated content.

Key points

  • Free, but application-only. The model is omni-moderation-csam-latest on the standard /v1/moderations endpoint: one still image per request, up to 20 MB, returning likely-csam and likely-csam/gen-ai scores. Access takes an intake form, Persona identity verification, program onboarding and acceptance of the terms.
  • OpenAI’s own efficiency numbers. In an internal evaluation on user-uploaded images, OpenAI estimates the new classifier would flag 48% fewer images for review while finding roughly the same number of abusive images, and that flagged images are about twice as likely to contain abuse. Both figures are vendor-reported and have not been independently replicated.
  • Partner tests measure different things. fal says reviewers confirmed 72% of the classifier’s high-confidence flags as novel CSAM; invideo AI says one in four flagged cases ended in a ban; SafetyKit says it detected 90% of drawings and animations that sexualize children.

Background

OpenAI’s existing child-safety stack, as its model card describes it, has four parts: hash matching, content classifiers, abuse monitoring and trained expert review. That combination catches the same file uploaded again at a different size. It does not catch images a model has just generated, because those have no hash in any database. The new classifier covers that gap by scoring what an image looks like rather than matching a fingerprint against a list.

Pressure from outside has grown at the same time. On October 15, Denmark’s Supreme Court upheld an 18-month sentence for possessing and sharing 36,000 AI-generated CSAM images and ruled that fictional subjects do not reduce the penalty (Denmark’s Supreme Court ruling). Earlier in October, eight US states sued OpenAI, arguing ChatGPT is not safe for minors (the states’ lawsuit). Industry coordination is forming too: OpenAI says it has joined the Tech Coalition’s Lantern program for sharing child sexual exploitation and abuse signals across platforms, and is distributing classifier access with ROOST.

The facts

The announcement bundles four things: the classifier, the ChatGPT intervention, developer guidance, and tools for law enforcement. Each reaches a different audience, and the numbers attached to them are not interchangeable.

FigureWhat it measuresSource
48%Fewer images sent to human review, versus existing protectionsOpenAI internal evaluation
~2×Relative likelihood that a flagged image contains abuseOpenAI internal evaluation
72%High-confidence flags confirmed as novel CSAM by reviewersfal production-image test
1 in 4Flagged cases that ended in a baninvideo AI test
90%Detection rate on drawings and animations sexualizing childrenSafetyKit evaluation
Under 1%False positives on legal adult contentSafetyKit, self-reported

The model card is more useful for understanding the limits. At default thresholds of 0.996324 for likely-csam and 0.90 for likely-csam/gen-ai, the classifier reaches 80.2% and 86.4% recall on a deduplicated held-out set, with precision of 0.991 and 1.0. On sampled ChatGPT traffic at the same thresholds, precision falls to 0.298 and 0.340. OpenAI explains the gap in the card: CSAM is rare in real traffic, so a small false-positive rate produces more flagged images than true detections.

The ChatGPT change is a product-level intervention. A request for this material now triggers a warning stating that sexualizing children causes harm, including when the requested material is AI-generated. OpenAI removes the flagged conversation and pauses access to ChatGPT for 24 hours while pointing the user to specialist support. The 24 hours is a warning-level pause, not a permanent ban; OpenAI says repeated violations can bring longer restrictions, account closure, or reports to authorities. Protect Children, the Lucy Faithfull Foundation and academic researchers at the University of Kent and the University of Tasmania helped design the approach.

For developers, OpenAI published CSAM guidance written with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. It reads as a staged checklist: prohibit the material in your terms, give users a visible reporting path, attach safety identifiers so every upload maps to an end user, run hash matching alongside a novel-image classifier, register with NCMEC so you can report, keep usable records, write a response playbook, and support the people doing review. One line in it is easy to miss: the standard Moderation API’s sexual/minors category covers text only, so image pipelines need a separate tool.

Law enforcement gets plugins, investigative tools and data analysis, with access managed by program administrators. The first tool reviews long chat logs and legal process returns, identifies potential connections and builds timelines linked back to source material. Training starts with an overview and demonstration, backed by a field guide; access depends on agency authorization and program requirements.

What others say

NCMEC senior vice president John Shehan said in the announcement that the technology’s role goes beyond identifying material: it can help find the children who may be depicted and give investigators information to protect them.

Partners are more direct. SafetyKit’s Samuel Wang called it the best tool of its kind they have tested, catching unknown CSAM that hash matching misses, with strong performance on drawings and animations and false positives below 1% on legal adult content. That is a partner statement inside a vendor announcement; neither the sample nor the method has been published.

TechCrunch framed the release against a second storyline: OpenAI is facing lawsuits over its products’ effects on minors, and Common Sense Media has called its technology an “unacceptable risk” for teens. The report also notes that nonconsensual sexualized imagery from generative tools is not an OpenAI-only problem.

The sharpest caveats come from OpenAI’s own model card. It states that a classifier score does not confirm an image is CSAM, does not establish legality in any jurisdiction, and does not determine whether a report is required. It also calls for a shared, independent benchmarking process, noting that without agreed definitions and representative test data, detection and false-positive rates across vendors cannot be compared.

Our take

For anyone running a product with user uploads, the practical change is the barrier to entry. Adding novel-CSAM detection used to mean training a model or buying a commercial service. Now a vetted team can get a free signal and put it in front of reviewers.

It is a ranking signal, not a verdict. The model card is explicit on three points: do not send images already confirmed as CSAM, since those belong in your existing handling and reporting process; keep trained human review before any NCMEC or hotline report; and re-benchmark thresholds on your own traffic. Threshold choice follows the use case — auto-blocking should favor precision, a human review queue can favor recall.

Three known gaps belong in the plan before rollout. The classifier has not been tested on video frames. Finer subcategory signals, including those for infants and toddlers, stay internal and are not returned by the API. And online precision near 30% means most flags a reviewer sees will be false positives, so staffing has to be sized for that ratio. The online numbers also come entirely from ChatGPT traffic; an image community or social app may see different results.

Open questions remain: whether the 24-hour pause actually reduces repeat behavior (OpenAI has published no return-rate data), how far Lantern signal sharing reaches, and when an independent benchmark will exist. Until then, treat the classifier as one useful layer rather than a solution.

How to try it

Submit the classifier intake form and complete Persona verification first; only approved organizations get a working key. After that, the call is an ordinary moderation request with a different model name:

curl https://api.openai.com/v1/moderations \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "omni-moderation-csam-latest",
    "input": [
      {
        "type": "image_url",
        "image_url": { "url": "https://cdn.openai.com/API/docs/images/soap.png" }
      }
    ]
  }'

Read results[0].flagged and category_scores. OpenAI warns that the scores are not calibrated probabilities and that 0.5 is not a universal threshold, so the cutoff has to match your review workflow. Submitted images are not retained and are not used for training.

What to watch

Three things over the next few months: whether an independent benchmark appears, since every performance figure so far comes from OpenAI or its partners; how far the law-enforcement tools and training actually deploy beyond the first tool and field guide; and how regulators treat material that is AI-generated and depicts no real child. Denmark has set a criminal precedent; most other jurisdictions have not.