Azure Cognitive Search text translation skill 50k charactes limitation

Question

We are using Azure Cognitive Search to index various documents, e.g. Word or PDF files, which are stored in Azure Blob Storage. We would like to be able to translate the extracted content of non-English documents and store the translation result into a dedicated field in the index.

Currently the built-in Text Translation cognitive skill supports up to 50,000 characters on the input. The documents that we have could contain up to 1 MB of text. According to the documentation it's possible to split the text into chunks using the built-in Split Skill, however there's no skill that could merge the translated chunks back together. Our goal is to have all the extracted text translated and stored in one index field of type Edm.String, not an array.

Is there any way to translate large text blocks when indexing, other than creating a custom Cognitive Skill via Web API for that purpose?

score 2 · Accepted Answer · answered Apr 29 '20 at 01:10

Yes, the Merge Skill will actually do this. Define the skill in your skillset like the below. The "text" and "offsets" inputs to this skill are optional, and you can use "itemsToInsert" to specify the text you want to merge together (specify the appropriate source for your translation output). Use insertPreTag and insertPostTag if you want to insert perhaps a space before or after each merged section.

{
  "@odata.type": "#Microsoft.Skills.Text.MergeSkill",
  "description": "Merge text back together",
  "context": "/document",
  "insertPreTag": "",
  "insertPostTag": "",
  "inputs": [
    {
      "name": "itemsToInsert", 
      "source": "/document/translation_output/*/text"
    }
  ],
  "outputs": [
    {
      "name": "mergedText", 
      "targetName" : "merged_text_field_in_your_index"
    }
  ]
}

score 0 · Answer 2 · answered May 04 '20 at 10:34

Below is a snippet in C#, using Microsoft.Azure.Search classes. It follows the suggestion given by Jennifer in the reply above.

The skillset definition was tested to properly support translation of the text blocks bigger than 50k characters.

private static IList<Skill> GetSkills()
{
    var skills = new List<Skill>();
    skills.AddRange(new Skill[] {
        // ...some skills in the pipeline before translation
        new ConditionalSkill(
            name: "05-1-set-language-code-for-split",
            description: "Set compatible language code for split skill (e.g. 'ru' is not supported)",
            context: "/document",
            inputs: new []
            {
                new InputFieldMappingEntry(name: "condition", source: SplitLanguageExpression),
                new InputFieldMappingEntry(name: "whenTrue", source: "/document/language_code"),
                new InputFieldMappingEntry(name: "whenFalse", source: "= 'en'")
            },
            outputs: new [] { new OutputFieldMappingEntry(name: "output", targetName: "language_code_split") }
        ),
        new SplitSkill
        (
            name: "05-2-split-original-content",
            description: "Split original merged content into chunks for translation",
            defaultLanguageCode: SplitSkillLanguage.En,
            textSplitMode: TextSplitMode.Pages,
            maximumPageLength: 50000,
            context: "/document/merged_content_original",
            inputs: new []
            {
                new InputFieldMappingEntry(name: "text", source: "/document/merged_content_original"),
                new InputFieldMappingEntry(name: "languageCode", source: "/document/language_code_split")
            },
            outputs: new [] { new OutputFieldMappingEntry(name: "textItems", targetName: "pages") }
        ),
        new TextTranslationSkill
        (
            name: "05-3-translate-original-content-pages",
            description: "Translate original merged content chunks",
            defaultToLanguageCode: TextTranslationSkillLanguage.En,
            context: "/document/merged_content_original/pages/*",
            inputs: new []
            {
                new InputFieldMappingEntry(name: "text", source: "/document/merged_content_original/pages/*"),
                new InputFieldMappingEntry(name: "fromLanguageCode", source: "/document/language_code")
            },
            outputs: new [] { new OutputFieldMappingEntry(name: "translatedText", targetName: "translated_text") }
        ),
        new MergeSkill
        (
            name: "05-4-merge-translated-content-pages",
            description: "Merge translated content into one text string",
            context: "/document",
            insertPreTag: " ",
            insertPostTag: " ",
            inputs: new []
            {
                new InputFieldMappingEntry(name: "itemsToInsert", source: "/document/merged_content_original/pages/*/translated_text")
            },
            outputs: new [] { new OutputFieldMappingEntry(name: "mergedText", targetName: "merged_content_translated") }
        ),
        // ... some skills in the pipeline after translation
    });

    return skills;
}

private static string SplitLanguageExpression
{
    get
    {
        var values = Enum.GetValues(typeof(SplitSkillLanguage)).Cast<SplitSkillLanguage>();
        var parts = values.Select(v => "($(/document/language_code) == '" + v.ToString().ToLower() +"')");
        return "= " + string.Join(" || ", parts);
    }
}

Azure Cognitive Search text translation skill 50k charactes limitation

2 Answers2

Linked