feat: flagProfanity for image + video, fix analyze() forwarding - #35
Merged
Conversation
…on_capped
The API added options.flag_profanity (free, deterministic, additive word-list
flag -- never affects is_bullying/unsafe/severity/risk_score/recommended_action)
and an account-level default_flag_profanity setting, plus escalation_capped on
the coded-term corroboration cap, but neither reached this SDK's types or
request body. A customer testing default_flag_profanity through the SDK had
no typed way to override it per-request and no `profanity` field on the
result, even though the API already supported both.
- DetectBullyingInput.flagProfanity / DetectUnsafeInput.flagProfanity: new
input properties, wired to options.flag_profanity in the request body.
Explicit true AND false both reach the API (not just truthy true) -- an
explicit false has to override the account default, same precedence the
API itself uses.
- BullyingResult.profanity / UnsafeResult.profanity: new response field,
{ detected, matches } | null.
- BullyingResult.escalation_capped / escalation_capped_reason and the same
on UnsafeResult: existed on the API since PR #117, never added here.
Not yet forwarded through the combined analyze() method (client-side fan-out
to detectBullying/detectUnsafe) -- only the two direct detection methods.
8 new tests asserting on the actual request body sent to fetch (mirrors
verdictOnlyIncludeEvidence.test.ts's pattern), not just the TS types.
115/115 tests passing, clean build. Version bump: 2.26.0 -> 2.27.0.
Extends the flagProfanity/profanity pair from 2.27.0 (detectBullying/ detectUnsafe) to analyzeImage, matching the API's new options.flag_profanity support on the image OCR text. Video was evaluated and left out -- the API has no OCR/text-extraction path on video to attach it to.
Same gap verdictOnly had in 2.19, fixed in 2.20 -- AnalyzeInput accepted flagProfanity but analyze() never passed it to the detectBullying/ detectUnsafe calls it fans out to. Didn't block anyone since the account default still applied, but per-request override wasn't reachable through the combined method.
categories was already computed server-side but only folded into the free-text rationale string -- no field existed to read it from. flagProfanity closes the evasion path left open by shipping profanity on analyzeImage alone: burning the same text into a video frame instead of a still previously never reached the word-list check. The API already OCRs text per frame for video (same vision call as image), it just wasn't aggregated or checked.
safenestdev
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes stacked on this branch, all part of the same
flagProfanityfeature line:flagProfanityonanalyzeImage,profanityonImageAnalysisResult(2.28.0, already published to npm) — matches Tuteliq/api#123 (merged, deployed).flagProfanitynow forwarded throughanalyze()(2.29.0, client-reported) — same class of gapverdictOnlyhad in 2.19, fixed in 2.20:AnalyzeInputacceptedflagProfanitybutanalyze()never passed it to thedetectBullying/detectUnsafecalls it fans out to.categories+flagProfanityonanalyzeVideo(2.30.0) — matches Tuteliq/api#125.categorieswas computed server-side but never exposed as its own field.flagProfanityextends the same word-list check already shipped on images to video.All three versions bumped, CHANGELOG updated. Not published to npm from here — 2.28.0/2.29.0 were already published directly from local builds; 2.30.0 is a manual step afterward.
Test plan
tsc --noEmitcleannpm run buildcleantests/flagProfanity.test.ts: image + videoflag_profanityforwarding (multipart FormData assertions) andprofanityfield passthrough;analyze()forwarding to both detectors +result.bullying/unsafe.profanitypassthrough