Merge specific arrays and sort them

Question

I have opened this large XML file and isolated the dates. I have converted each specific date into an array, but I want to place them into a single array and after that sort them.

Here is the code:

import numpy as np

with open('dblp-2020-04-01.xml','r' , encoding="ISO-8859-1") as f:
   for i, line in enumerate(f):
    if "<year>" in line:

        data = line[6:10]
        data_list = np.array([data])
        print(data_list)

The desired output is:

['2010']
['2002']
['1992']
['2002']
['1994']
  ...

Giorgos Myrianthous · Answer 1 · 2020-04-27T13:07:36.903

0

You need to create a np.array outside of your for-loop and finally append all the dates to it:

with open('dblp-2020-04-01.xml', 'r', encoding="ISO-8859-1") as f:
    data_list = np.array([])
    for i, line in enumerate(f):
        if "<year>" in line:
            data = line[6:10]
            data_list = np.append(data_list, data)
    print(data_list)

And the output will be

['2010', '2002', '1992', '2002', '1994']

Finally, you can sort your array using numpy.sort():

np.sort(data_list) # Ascending order
>>> ['1992', '1994', '2002', '2002', '2010']

UPDATE

Okay so given the scenario you are describing in the comments, I would say that the most efficient way to get a count per date from your XML data, is to load the XML into a pandas dataframe and finally use

df['yourDatesColumn'].groupby('yourDatesColumn').count()

or

df['yourDatesColumn'].value_counts()

in order to get the counts per date.

Alternatively, you can choose to create a pandas.Series object just for your date column (in case you don't want to load all the data to a pandas Dataframe).

edited Apr 27 '20 at 13:07

answered Apr 26 '20 at 13:03

Giorgos Myrianthous

36,235
20
134
156

Thanks a lot!Looks like though the date are way to many and it can't print them.Is there a way I can keep only the first 20 dates? – Apr 26 '20 at 13:37
@Costas At what point does it get stuck? Before or after sorting the dates? – Giorgos Myrianthous Apr 26 '20 at 13:38
@Costas Also, does it fail? It might be the case that your code needs some time to finish.. – Giorgos Myrianthous Apr 26 '20 at 13:39
before sorting them.(Let me mention the file is 3GB if that might help). – Apr 26 '20 at 13:42
@Costas Does it fail? – Giorgos Myrianthous Apr 26 '20 at 13:42
I dont think so.It doesnt show any unexpected error.It just keeps running. – Apr 26 '20 at 13:45
@Costas OK. So i think there’s nothing wrong other than it might take some time to complete. You can also verify this by adding a counter in your for-loop and printing it out at each iteration so you can keep track of the execution. – Giorgos Myrianthous Apr 26 '20 at 13:51
@Costas In case you want the first 20 dates, you can use inside your for-loop: `if i >= 20: break` – Giorgos Myrianthous Apr 26 '20 at 14:05
@Costas Any luck? – Giorgos Myrianthous Apr 27 '20 at 09:24
Hi.Looks like the data are approximately 6 million dates so it takes a lot of time to print them.I'll try to have at least a number of those dates. P.S.The whole idea of the code is to count the times each individual date is being repeated(for example for 2000 we have it 30000 time) – Apr 27 '20 at 12:30
I really dont want to use an Element Tree cause it will need DTD. – Apr 27 '20 at 14:32

Merge specific arrays and sort them

1 Answers1