Is there a way I can automate the creation of .json files used for language translations?

Question

I have files such as this that have translation keys and values:

locale-en.json
{
    "CHANGE_PASSWORD": "Change Password",
    "CONFIRM_PASSWORD":  "Confirm Password",
    "NEW_PASSWORD": "New Password"
}

locale-jp.json
{
    "CHANGE_PASSWORD": "パスワードを変更します",
    "CONFIRM_PASSWORD":  "パスワードを認証します",
    "NEW_PASSWORD": "新しいパスワード"
}

When I add a new translation key to the JSON file containing the English translations for example, I must remember to add that key and the associated translation to all the other JSON files. All the JSON files are also edited separately. The process is laborious and error prone.

Has anyone found a way to reduce the errors and to automate the process.

Ideally I would like to be able to run a script from Windows PowerShell that would change the files to this if an additional key was added to locale-en.json :

locale-en.json
{
    "CHANGE_PASSWORD": "Change Password",
    "CONFIRM_PASSWORD":  "Confirm Password",
    "NEW_PASSWORD": "New Password",
    "NEW_KEY": "New Key"
}

locale-jp.json
{
    "CHANGE_PASSWORD": "パスワードを変更します",
    "CONFIRM_PASSWORD":  "パスワードを認証します",
    "NEW_PASSWORD": "新しいパスワード",
    >>>"NEW_KEY": "New Key"
}

Answer 1

You could write something like this in powershell:

$masterFile = "locale-en.json"

function Get-LocaleMap($file){

    $map = @{}

    $localeJson = ConvertFrom-Json (gc $file -Raw)
    $localeJson | gm -MemberType NoteProperty | % {
        $map.Add($_.Name, ($localeJson | select -ExpandProperty $_.Name))
    }

    return $map
}

$masterLocale = Get-LocaleMap $masterFile

ls | ? { $_.Name -like "locale-*.json" -and $_.Name -ne $masterFile } | % {
    $locale = Get-LocaleMap $_.FullName
    $masterLocale.GetEnumerator() | % {
        if(!$locale.ContainsKey($_.Key)){
            $locale.Add($_.Key, $_.Value)
        }
    }

    ConvertTo-Json $locale | Out-File -FilePath $_.FullName -Force -Encoding utf8
}

It created a dictionary from your English json file. Then it looks up all other locale files and checks them for keys which are present in the English file but missing from them. Then it adds the missing keys and values and saves the locale files in Unicode.

Let me show you how you can do the same with old school Windows Scripting since you seem to prefer JavaScript:

var masterFile = "locale-en.json"
var fso = new ActiveXObject("Scripting.FileSystemObject");
var scriptPath = fso.GetParentFolderName(WScript.ScriptFullName);
var charSet = 'utf-8';
var f = fso.GetFolder(scriptPath);
var fc = new Enumerator(f.files);

function getLocaleMap(fileName){
    var path = scriptPath + '\\' + fileName;
    var stream = new ActiveXObject("ADODB.Stream"); // you cannot use fso for utf-8

    try{
        stream.CharSet = charSet;
        stream.Open();
        stream.LoadFromFile(path);
        var text = stream.ReadText();
        var json = {};
        eval('json = ' + text); // JSON.parse is not available in all versions
        return json;
    }
    finally{
        stream.Close();
    }
}

function saveAsUtf8(fileName, text){
    var path = scriptPath + '\\' + fileName;
    var stream = new ActiveXObject("ADODB.Stream"); 

    try{
        stream.CharSet = charSet;
        stream.Open();
        stream.Position = 0;
        stream.WriteText(text);
        stream.SaveToFile(path, 2); // overwrite
    }
    finally{
        stream.Close();
    }
}

var locales = [];
var masterMap = getLocaleMap(masterFile);

for (; !fc.atEnd(); fc.moveNext())
{
    var file = fc.item();
    var extension = file.Name.split('.').pop();
    if(extension != "json" || file.Name == masterFile){
       continue;
    }

    var map = getLocaleMap(file.Name);
    var newLocaleText = '{\r\n';
    var i = 0;

    for(var name in masterMap){
        var value = '';

        if(map[name]){
            value = map[name];
        }
        else{
            value = masterMap[name];
        }

        if(i > 0){
            newLocaleText += ",\r\n";
        }

        newLocaleText += "\t'" + name + "': '" + value + "'";
        i++;
    }

    newLocaleText += '\r\n}'

    saveAsUtf8(file.Name, newLocaleText);
}

You can run the javascript from command line like this:

Cscript.exe "C:\yourscript.js"

I hope it helps.

Answer 2

Is there a way I can automate the creation of .json files used for language translations?

YES, executing automatic tasks is exactly what automation tools like Grunt and Gulp where designed to do.

As you said, doing things manually is laborious and error prone, so Grunt/Gulp are the way to go.

With a simple Grunt/Gulp config, all the relevant .json files can be watched simultaneously: any key added to any of them will be instantly detected, and order the execution of the custom script of your choice.

---

HOW GRUNT/GULP CAN DO IT:

1. Grunt/Gulp will constantly watch all the relevant JSON files;
2. When a change is detected in a watched file, a custom script is run;
3. The custom script will read the changed file and retrieve the new key(s) and value(s);
4. The custom script will then be write to all the other relevant JSON files.

---

CONFIGURING GRUNT

To detect file changes automatically and execute myCustomScript, just use grunt-contrib-watch like so:

watch: {
  scripts: {
    files: ['**/*.locale.json'],
    tasks: ['myCustomScript'],
  },
}

---

CUSTOM SCRIPT TO ADD THE NEW KEY(S) TO THE RELEVANT .JSON FILES:

  grunt.event.on('watch', function(action, filepath) {
    // filepath is the path to the file where change is detected
    grunt.config.set('filepath', grunt.config.escape(filepath));
   });

  var myCustomScript=function(changedFile,keyFile){

     var project = grunt.file.readJSON(changedFile);
     //will store the file where changes were detected as a json object

     var keys=grunt.file.readJSON(keyFile);
     //will store keyFile as a json object

     //walk changedFile keys, and check is keys are in keyFile
     for (var key in project) {
       if (project.hasOwnProperty(key)) {
         if(!keys.hasOwnProperty(key)){
           //a new key was detected
           newKeyArray.push(key);
         }
       }
     }

  //should update all the other relevant JSON files with `grunt.file.write`, and add all the keys in newKeyArray:

  var filesToChangeArray=grunt.file.match('**/*.locale.json');
  //returns an array that contains all filepaths where change is desired
  filesToChangeArray.forEach(function(path){
    //walk newKeyArray to set addedContent string
    newKeyArray.forEach(function(key){
    addedContent+='"'+key+'":"to be set",';
    //this will write all the new keys, with a value of "to be set", to the addedContent string
    }
    grunt.file.write(path,addedContent);
    });
  }

---

Ideally I would like to be able to run a script from Windows PowerShell

Even though Grunt/Gulp are often used to execute custom files written in javaScript/nodejs, they are well able to order the execution of scripts written in other languages.

To execute a PowerShell script, you could use a Grunt plugin called grunt-shell, like so:

grunt.initConfig({
shell: {
    ps: {
        options: {
            stdout: true
        },
        command: 'powershell myScript.ps1'
    }
}
});

as detailed in this SO post.

So if PowerShell is your thing, you could have the best of both worlds:

• Easy detection with Grunt/Gulp watch;
• PowerShell script execution when change is detected.

However, you might as easily use Grunt/Gulp only for this: as Grunt/Gulp is already taking care of the detection in the background, all you need to do is have it run a custom script that reads your new keys (grunt.file.readJSON) and copies them (grunt.file.write) to the relevant files.

Answer 3

Automated the process using a javascript solution with nodejs via command line.

$ node localeUpdater.js

This will watch your default locale (locale-en.json) with any revisions made and update your whole locale file list as necessary.

1. create the necessary locale file list if not present then initialized it with default locale data
2. add new keys based on default locale
3. remove missing keys based on default locale

localeUpdater.js

var fs = require("fs");

var localeFileDefault = "locale-en.json";
var localeFileList = ["locale-jp.json", "locale-ph.json"];

fs.watchFile(localeFileDefault, function() {

  var localeDefault = readFile(localeFileDefault);
  var localeCurrent = null;
  var fileNameCurrent = null;

  for (var i in localeFileList) {
    fileNameCurrent = localeFileList[i];

    console.log("Adding new keys from default locale to file " + fileNameCurrent);
    localeCurrent = readFile(fileNameCurrent);
    for (var key in localeDefault) {
      if (!localeCurrent[key]) {
        console.log(key + " key added.");
        localeCurrent[key] = localeDefault[key];
      }
    }

    console.log("Removing keys not on default locale to file " + fileNameCurrent);
    for (var key in localeCurrent) {
      if (!localeDefault[key]) {
        console.log(key + " key removed.");
        delete localeCurrent[key];
      }
    }

    writeFile(fileNameCurrent, JSON.stringify(localeCurrent));
    console.log("File " + fileNameCurrent + " updated.");
  }

});

function readFile(fileName) {
  var result = null;
  if (fs.existsSync(fileName)) {
    result = fs.readFileSync(fileName, "utf8");
    result = result ? JSON.parse(result) : {};
  } else {
    writeFile(fileName, "{}");
    result = {};
  }
  return result;
}

function writeFile(fileName, content) {
  fs.writeFileSync(fileName, content, "utf8");
}

Answer 4

There are multiple safeguards you should put in place.

First off your translation function should have some safeguards. Something like:

function gettext(text) {
    if (manifest[text]) {
        return text;
    }

    return text;
}

I'm not sure how you register new strings, but we regex our code base for things like gettext('...') and then we compile a list of translations that way. A couple times a day we push that to a 3rd party translation company, which notices new strings. They populate new things and we pull content back. The "pull" involves a compilation to the different language files. The translation file compilation always falls back to english. In other words we download a file from the 3rd party and do something like:

_.map(strings, function(string) {
    return localeManifest[locale][text] || localeManifest['en_US'][text];
}

This ensures that even if the manifest for the locale doesn't contain the translation yet we still populate it with the English US version.