Fastest way to iterate over a large list containing strings in Python?

Question

I have a large list that contains 30000 other lists. The large list contains a list of a string and two other lists. Here is an example:

large_list = [
    ...
    ['maj', [4, 7], ['3', '5']],
    ['maj7', [4, 7, 11], ['3', '5', '7']],
    ...
    ]

I want to iterate through this large_list and find the smaller list by looking for the name (maj, maj7, etc.).

What is the fastest way of doing this?

I had thought to put the list in a .txt file and reading line by line, but I don't really know if that's faster. All I know is it takes less of a toll on my IDE, as having a list of 30000 lines makes any coding sluggish.

where are the 3000000 lines coming from? how do they get into python? — Joran Beasley, May 14 '20 at 05:24
see if this helps--> [link](https://stackoverflow.com/q/10929724/11669010) — sage07, May 14 '20 at 05:27
you might want to look at the solution implemented here https://stackoverflow.com/a/39923218/12337794 — Bermylle Razon, May 14 '20 at 05:29
Would it be more efficient to declare/convert your structure as/to a dictionary, so you can do `large_dict['maj7']` to do the lookup. BTW, your syntax looks wrong - should there be a close square bracket at the end of the two sample lines? — Ken Y-N, May 14 '20 at 05:29
The lines came from other code that generated the correct attributes @Joran — RangeError, May 14 '20 at 06:00

Joran Beasley · Answer 1 · 2020-05-14T19:44:47.923

1

import pandas
# convert it to a dataframe
df = pandas.DataFrame(largeList)
#select all the rows where the zeroth element starts with "maj"
mask = df[0].str.startswith("maj")
print(df[mask])

edited May 14 '20 at 19:44

answered May 14 '20 at 05:25

Joran Beasley

110,522
12
160
179

I hadn't heard of DataFrame, but this looks promising. Thanks! – RangeError May 14 '20 at 06:06
2

Please put your answer always in context instead of just pasting code. See [here](https://stackoverflow.com/help/how-to-answer) for more details. – gehbiszumeis May 14 '20 at 12:52

score 0 · Answer 2 · answered May 14 '20 at 05:59

Hello Bud,

I am pretty new to Python, however, in this case, my best guess to find the smallest of them all sublist in this gigantic monster called large_list is by using the code shared below:

large_list=[['maj', 4, 7], ['3', '5'], ['maj9', 7], ['maj3', 100], ['maj4', 1360], ['maj7', 4, 7, 11], ['3', '5', '7']]
def getTheMinGuy(mylist):
    minimum = min(mylist)
    return print(f"The smaller sublist on this list is => {minimum}")
getTheMinGuy(large_list)

I certainly hope this helps finding the best solution for this issue, champion. Cheers!

how does this help him solve the problem? i dont think this code will do what you think it will do... he wants a sublist containing all the elements that start with maj or some variant of maj — Joran Beasley, May 14 '20 at 19:48

Fastest way to iterate over a large list containing strings in Python?

2 Answers2