Content Safety Detection

    With the Content Safety Detection feature, AssemblyAI can detect if any of the following sensitive content is spoken in your audio/video files:

    Label Description Model Output Supported by Severity Scores
    Accidents Any man-made incident that happens unexpectedly and results in damage, injury, or death. accidents Yes
    Alcohol Content that discusses any alcoholic beverage or its consumption. alcohol Yes
    Company Financials Content that discusses any sensitive company financial information. financials No
    Crime Violence Content that discusses any type of criminal activity or extreme violence that is criminal in nature. crime_violence Yes
    Drugs Content that discusses illegal drugs or their usage. drugs Yes
    Gambling Includes gambling on casino-based games such as poker, slots, etc. as well as sports betting. gambling Yes
    Hate Speech Content that is a direct attack against people or groups based on their sexual orientation, gender identity, race, religion, ethnicity, national origin, disability, etc. hate_speech Yes
    Health Issues Content that discusses any medical or health-related problems. health_issues Yes
    Manga Mangas are comics or graphic novels originating from Japan with some of the more popular series being "Pokemon", "Naruto", "Dragon Ball Z", "One Punch Man", and "Sailor Moon". manga No
    Marijuana This category includes content that discusses marijuana or its usage. marijuana Yes
    Natural Disasters Phenomena that happens infrequently and results in damage, injury, or death. Such as hurricanes, tornadoes, earthquakes, volcano eruptions, and firestorms. disasters Yes
    Negative News News content with a negative sentiment which typically will occur in the third person as an unbiased recapping of events. negative_news No
    NSFW (Adult Content) Content considered "Not Safe for Work" and consists of content that a viewer would not want to be heard/seen in a public environment. nsfw No
    Pornography Content that discusses any sexual content or material. pornography Yes
    Profanity Any profanity or cursing. profanity Yes
    Sensitive Social Issues This category includes content that may be considered insensitive, irresponsible, or harmful to certain groups based on their beliefs, political affiliation, sexual orientation, or gender identity. sensitive_social_issues No
    Terrorism Includes terrorist acts as well as terrorist groups. Examples include bombings, mass shootings, and ISIS. Note that many texts corresponding to this topic may also be classified into the crime violence topic. terrorism Yes
    Tobacco Text that discusses tobacco and tobacco usage, including e-cigarettes, nicotine, vaping, and general discussions about smoking. tobacco Yes
    Weapons Text that discusses any type of weapon including guns, ammunition, shooting, knives, missiles, torpedoes, etc. weapons Yes

    Enabling Content Safety Detection when submitting files for transcription

    Simply include the content_safety parameter in your POST request, and set this parameter to true, as shown in the cURL request below.

    curl --request POST \
      --url https://api.assemblyai.com/v2/transcript \
      --header 'authorization: YOUR-API-TOKEN' \
      --header 'content-type: application/json' \
      --data '{"audio_url": "https://app.assemblyai.com/static/media/phone_demo_clip_1.wav", "content_safety": true}'

    Interpreting the Content Safety Detection response

    Once the transcription is complete, and you make a GET request to /v2/transcript/<id> to receive the transcription, there will be an additional key content_safety_labels:

    {
        # some keys have been hidden for readability
        ...
        "text": "foo bar...",    
        "id": "5551722-f677-48a6-9287-39c0aafd9ac1",
        "status": "completed",
        "content_safety_labels": {
            "status": "success", 
            "results": [
                {
                    "text": "...has led to a dramatic increase in fires and the disasters around the world have been increasing at an absolutely extraordinary. An unprecedented rate four times as many in the last 30 years...", 
                    "labels": [
                        {
                            "confidence": 0.9986903071403503, 
                            "severity": 0.11403293907642365,
                            "label": "disasters"
                        }
                    ], 
                    "timestamp": {
                        "start": 171710, 
                        "end": 200770
                    }
                }
            ],
            "summary": {
                "disasters": 0.89,
                "nsfw": 0.08,
                ...
            }
        },
        ...
    }

    Let's dig into the entire "content_safety_labels" response key below, to get a better understanding of what each value means.

    "content_safety_labels": {
        # will be "unavailable" in the rare chance that content safety results
        # were unavailable for this transcription
        "status": "success",
        # 'results' contains a list of each portion of text that was flagged
        # by the content safety model, along with the labels that were
        # assigned to the flag paragraph, the confidence score for each
        # label, the severity score (will be null/none if the label is not supported), 
        # and the timestamp for where the portion of text occurred in the source audio file
        "results": [
            {
                "text": "...has led to a dramatic increase in fires and the disasters around the world have been increasing at an absolutely extraordinary. An unprecedented rate four times as many in the last 30 years...", 
                "labels": [
                    {
                        "confidence": 0.9986903071403503,
                        "severity": 0.11403293907642365,
                        "label": "disasters"
                    }
                ], 
                "timestamp": {
                    "start": 171710, 
                    "end": 200770
                }
            }
        ],
        # for each content safety label detected, the 'summary' key will show
        # the relevancy for that label across the entire transcription
        # text; for example, if the "nsfw" label is detected only 1 time,
        # even with high confidence, in a 60 minute audio file, the
        # 'summary' key will show a low score, since the entire
        # transcription was not found to be "nsfw"
        "summary": {
            "disasters": 0.89,
            "nsfw": 0.08,
            ...
        }
    },

    Understanding Severity Scores and Confidence Scores

    Each label will be returned with a confidence score and a severity score. It is important to note that these two keys measure two very different things. The severity key will produce a score that shows how severe the flagged content is on a scale of 0–1. For example, a natural disaster with mass casualties would be a 1.0, whereas a wind storm that broke a lamppost would be a 0.1.

    In comparison, confidence displays how confident the model was in predicting the predicted label, on a scale of 0-1.

    We can break this down further by reviewing the following label:

    "labels": [
                        {
                            "confidence": 0.9986903071403503, 
                            "severity": 0.11403293907642365,
                            "label": "disasters"
                        }
                    ], 

    In the above example, our model is saying it is 99.87% confident that the content is about natural disasters; however, it is measured at a low severity of 0.1140. This means the model is very confident the content is about natural disasters, but the content was not severe in nature (ie, was likely about a mild natural disaster).

    Controlling the Threshold of Surfaced Results

    Depending on the content a developer is working with, they may need to control the confidence threshold of the response. By default, the content safety model will return any label and its context if the confidence of that particular passage is 50% or greater. If you wish to set a higher or lower threshold, add content_safety_confidence: {number} to your post request. This parameter will accept a number range of 25–100 inclusive.

    Here is a quick example of a request setting the threshold to 75%:

    curl --request POST \
      --url https://api.assemblyai.com/v2/transcript \
      --header 'authorization: YOUR-API-TOKEN' \
      --header 'content-type: application/json' \
      --data '{"audio_url": "https://app.assemblyai.com/static/media/phone_demo_clip_1.wav", "content_safety": true, "content_safety_confidence: 75}'