Sort lines in txt based on double condition

Question

I have a large txt file whose structure looks like :

title_topic --> ~1900 different values
title_foreach_post --> can be the same with title_topic or []
post_number --> starts from null and goes up to n

I was wondering if there is a way to a) regroup and sort all lines that have the same title_topic and b) sort them afterwards based on logical number order (1,2..9,10,11, 12 etc.).

Thank you all in advance.

Example of txt file :

{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "null"}

{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "1."}

{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "10."}

{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "null"}

{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "1."}

{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "12."}

{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "11."}

{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "12."}

Output example:

Output example

Are you reading your data in a list? It looks like json data. — Tarik, Mar 28 '21 at 17:45
@Tarik https://jsonformatter.curiousconcept.com/ gives error MULTIPLE JSON ROOT ELEMENTS — pippo1980, Mar 28 '21 at 17:51
Thank you all for answering. It's a json file converted to txt, where each entry = 1 line — Artemis, Mar 28 '21 at 17:52
read it line by line appanding each line to a list. then work on element of the list. I am not expert but lots of hints on SO: https://stackoverflow.com/questions/72899/how-do-i-sort-a-list-of-dictionaries-by-a-value-of-the-dictionary — pippo1980, Mar 28 '21 at 17:57
adding an example output would help to understand your needs — pippo1980, Mar 28 '21 at 17:59
@pippo1980 Is it possible to use key with a double condition using key ? — Artemis, Mar 28 '21 at 18:04
'use key with a double condition using key ?' I dont understand it, sorry — pippo1980, Mar 28 '21 at 18:08
https://stackoverflow.com/questions/5212870/sorting-a-python-list-by-two-fields Sorting a Python list by two fields, https://docs.python.org/3/howto/sorting.html : Sorting HOW TO — pippo1980, Mar 28 '21 at 18:11
@pippo1980 I have to test it, but seems to do exactly what I want to achieve. Thank you — Artemis, Mar 28 '21 at 18:14

Arty · Accepted Answer · 2021-03-28T19:11:00.983

As you can see in code below I use sorted() built-in function to sort list of entries by two fields (e['title_topic'], e['post_number']).

Just for example I'm reading input file from text string using io.StringIO() to model file input, in real life you'll use regular file reading.

Try it online!

import io, json

text = r"""
{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "null"}
{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "1."}
{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "10."}
{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "null"}
{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "1."}
{"title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée", "title_foreach_post": ["\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "], "post_number": "12."}
{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "11."}
{"title_topic": "Résoudre un problème avec Go voyage", "title_foreach_post": ["\nRe: Résoudre un problème avec Go voyage "], "post_number": "12."}
"""

data = [json.loads(line) for line in io.StringIO(text) if line.strip()]
data = sorted(data, key = lambda e: (e['title_topic'],
    int(float(e['post_number'])) if e['post_number'] != 'null' else 0))
print(data)

Output:

[
    {
        "title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée",
        "title_foreach_post": [
            "\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "
        ],
        "post_number": "null"
    },
    {
        "title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée",
        "title_foreach_post": [
            "\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "
        ],
        "post_number": "1."
    },
    {
        "title_topic": "Loueur Goldcar ne rembourse pas une réservation annulée",
        "title_foreach_post": [
            "\nRe: Loueur Goldcar ne rembourse pas une réservation annulée "
        ],
        "post_number": "12."
    },
    {
        "title_topic": "Résoudre un problème avec Go voyage",
        "title_foreach_post": [
            "\nRe: Résoudre un problème avec Go voyage "
        ],
        "post_number": "null"
    },
    {
        "title_topic": "Résoudre un problème avec Go voyage",
        "title_foreach_post": [
            "\nRe: Résoudre un problème avec Go voyage "
        ],
        "post_number": "1."
    },
    {
        "title_topic": "Résoudre un problème avec Go voyage",
        "title_foreach_post": [
            "\nRe: Résoudre un problème avec Go voyage "
        ],
        "post_number": "10."
    },
    {
        "title_topic": "Résoudre un problème avec Go voyage",
        "title_foreach_post": [
            "\nRe: Résoudre un problème avec Go voyage "
        ],
        "post_number": "11."
    },
    {
        "title_topic": "Résoudre un problème avec Go voyage",
        "title_foreach_post": [
            "\nRe: Résoudre un problème avec Go voyage "
        ],
        "post_number": "12."
    }
]

Thank you. It's really close to what I want as an output. Given that post_number is a string, is there a way to achieve a logical order ? I want to avoid having and an order like 1,10, 2 and int() won't work because there string values : "null" and "1." that cannot be replaced or modified. int(e['post_number']) — Artemis, Mar 28 '21 at 19:00
@Artemis I modified my code to support null-handling and also convert to int. Please put a look. I handle null to be equal to 0 there, you can use any number for null instead. — Arty, Mar 28 '21 at 19:12
This is exactly what I need ! Thank you ! I will update the question and mark it as solved ! — Artemis, Mar 28 '21 at 19:16
@Arty how python chooses to sort between post numbers when I used for example: data = sorted(data, key = lambda e: (e['title_topic'], int(float(e['post_number'])) if e['post_number'] != 'null' else 12.)) ?? — pippo1980, Mar 29 '21 at 20:37
@pippo1980 In your code if post number is null then it is supposed to be 12. It means that null is just replaced with 12 when sorting. It means null entries will go at the very end if all your numbers are 1 2 3 ... 11. If you have bigger numbers then you have to use `max + 1` (max is max number) for value of null instead of 12. Also use `12` instead of `12.`, i.e. remove dot (`.`) at the end, because we have just integers, `12.` is floating point 12 — Arty, Mar 30 '21 at 09:06
@Arty was asking because I tried it and what happens is that the null line goes before the 12. line. I was wondering how sorted() handles keys with same values. In my case it seems that the one from the conditional statement goes before the other sorted keys. But I am note sure if it’s always like this or not and how sorted handles this stuff — pippo1980, Mar 30 '21 at 14:55
@pippo1980 Sorted does next steps 1) it compares two values `a` and `b`. 2) if `a` < `b` then in final array `a` goes somewhere to the left from `b`. 3) if `a` > `b` then `a` goes somewhere to the right from `b`. 4) if `a` == `b` then in sorted array initial relative order of `a` and `b` is preserved, meaning if `a` was left from `b` in initial array then in final array also `a` will be to the left of `b`, if `a` was to the right from `b` then final array preserves same ralative order. So in your case `12.` == `12` meaning that order is preserved same. Make null 13 to make it right from 12. — Arty, Mar 30 '21 at 15:09

Sort lines in txt based on double condition

Thank you all in advance.

1 Answers1