0

I am wondering what I should do for the purpose of my project.

I am gonna operate on about 100 000 rows, every time.

what I wanted to do is to create an object "{}" and then, if I need to search for a value, just call it , for example

data['2018']['09']['Marketing']['AccountName']

the second option is to pull everyting into an array "[]" and in case I need to pull value, I will create a function to go through the array and sum numbers for specific parameters.

But don't know which method is faster. Will be thankful if you can shed some light on this

Thanks in advance,

akaribi
  • 57
  • 3
  • 1
    [In Python, when to use a Dictionary, List, or Set?](https://stackoverflow.com/questions/3489071/in-python-when-to-use-a-dictionary-list-or-set) – benvc Nov 09 '18 at 13:54
  • 2
    It depends on what your data looks like. Numpy and Pandas come to mind here – C.Nivs Nov 09 '18 at 13:54
  • The answer depends on what your data look like and how you use them. Now given the dataset size you probably want a proper sql database (sql database are not mere bit buckets, they're made for efficient querying with computations and aggregations) – bruno desthuilliers Nov 09 '18 at 14:18

1 Answers1

1

If performance (speed) is an issue, Python might not be the ideal choice...

Otherwise:

Might I suggest the use of a proper database, such as SQLLite (which comes shipped with Python). And maybe SQLAlchemy as an abstraction layer. (https://docs.sqlalchemy.org/en/latest/orm/tutorial.html)

After all, they were made exactly for this kind of tasks.

If that seems overkill: Have a look at Pandas.

Sebastian Loehner
  • 1,302
  • 7
  • 5
  • hey, thanks for the response. I haven't started working on this yet, just thinking what option I should use, which one is better. I am also aggregating values in database as much as possible ( MySQL) so all sql-related topics are already finished. When it comes to Pandas, why do you recommend it ? I didn't use it in past, with exception of parsing excel file. How does it help for my problem ? – akaribi Nov 09 '18 at 15:37