0

I am making a little django app to serve translations for my react frontend. The way it works is as follows:

  1. The frontend tries to find a translation using a key.
  2. If the translation for that key is not found, It sends a request to the backend with the missing key
  3. On the backend, the missing key is appended to a json file

Everything works just fine when the requests are sent one at a time (when one finishes, the other is sent). But when multiple requests are sent at the same time, everything breaks. The json file gets corrupted. It's like all the requests are changing the file at the same time which causes this to happen. I am not sure if that's the case because I think that the file can not be edited by two processes at the same time(correct me if I am wrong) but I don't receive such an error which indicates that the requests are handled one at a time according to this and this

Also, I tried something, which to my surprise worked, that is to add time.sleep(1) to the top of my api view. When I did this, everything worked as expected.

What is going on ?

Here is the code, just in case it matters:

@api_view(['POST'])
def save_missing_translation_keys(request, lng, ns):
    time.sleep(1)

    missing_trans_path = MISSING_TRANS_DIR / f'{lng}.json'

    # Read lng file and get current missing keys for given ns
    try:
        with open(missing_trans_path, 'r', encoding='utf-8') as missing_trans_file:
            if is_file_empty(missing_trans_path):
                missing_keys_dict = {}
            else:
                missing_keys_dict = json.load(missing_trans_file)
    except FileNotFoundError:
        missing_keys_dict = {}
    except Exception as e:
        # Even if file is not empty, we might not be able to parse it for some reason, so we log any errors in log file
        with open(MISSING_LOG_FILE, 'a', encoding='utf-8') as logFile:
            logFile.write(
                f'could not save missing keys {str(list(request.data.keys()))}\nnamespace {lng}/{ns} file can not be parsed because\n{str(e)}\n\n\n')
        raise e

    # Add new missing keys to the list above.
    ns_missing_keys = missing_keys_dict.get(ns, [])
    for missing_key in request.data.keys():
        if missing_key and isinstance(missing_key, str):
            ns_missing_keys.append(missing_key)
        else:
            raise ValueError('Missing key not allowed')

    missing_keys_dict.update({ns: list(set(ns_missing_keys))})

    # Write new missing keys to the file
    with open(missing_trans_path, 'w', encoding='utf-8') as missing_trans_file:
        json.dump(missing_keys_dict, missing_trans_file, ensure_ascii=False)

    return Response()
just human
  • 51
  • 1
  • 3
  • You have to make sure that no two threads or processes write to the same file at the same time. This is not a Django problem, the rules applies generally. You can use a lock (with a lockfile for production setups with multiple processes) or use sometime more thread-safe like a database with transportations. – Klaus D. Nov 08 '20 at 11:50
  • This app is not intended for production. I am making it so it could help me internationalize my frontend because the other solutions I see are expensive for me. I actually just tried using the database instead of files and It worked just fine. But I really want to know why it doesn't work with files. Can you tell me more about locking and thread safe apps (links) – just human Nov 08 '20 at 12:26
  • You should educate yourself on how to write thread-safe applications. The topic is too extensive to be explained here. – Klaus D. Nov 08 '20 at 12:30
  • Ok. I will look into that – just human Nov 08 '20 at 12:47

0 Answers0