0

I have a CSV file which first line is the header.

Header1 | Header2 | ... | HeaderN
Cell11  | Cell12  | ... | Cell1N
    .         .      .      .
    .         .      .      .
    .         .      .      .
CellM1  | CellM2  | ... | CellMN

I would like to load the items into the local memory. Therefore I load the item as a list of dict, using the command dict(zip(self.header, item)). Then the item would look like this:

[[{Header1: Cell11}, {Header2: Cell12}... {HeaderN: Cell1N}],
 [{Header1: Cell21}, {Header2: Cell22}... {HeaderN: Cell2N}],
                       ...
 [{HeaderM: CellM1}, {Header2: CellM2}... {HeaderN: CellMN}]]

the headers are the keys and the key values are in the cell.

Since the searching efficiency is low therefore I would like to find another way to store it on ram. That way would lead me to find the items effectively.

Is it better to load the CSV file into built-in sqlite3?

Thanks in advance

Winston
  • 1,308
  • 5
  • 16
  • 34
  • 1
    For starters, a `DictReader` would make reading the CSV file much easier. Then please explain what kinds of searches you need to do and why the list of dicts is too slow for that. – Tim Pietzcker Sep 06 '13 at 09:08
  • I would like to do the operation similar to SQL command "SELECT * from table where HEADER1=values". If I need to do similar operation I need a for loop and check value in very row. – Winston Sep 06 '13 at 10:12
  • @TimPietzcker The search is low if the CSV file is large. Since it's running a for loop to look for specific header – Winston Sep 10 '13 at 02:20
  • @TimPietzcker I would like to load the CSV file into the ram database for easy access. I think the hard part will the table creation. Since the header is dynamic instead of static. – Winston Sep 10 '13 at 02:25
  • If the question is about time performance then you should measure it i.e., write both the code that use a list of dicts to search and the code that imports the data into sqlite db first, and compare the results. If you don't know *how* to import a csv file into sqlite db then ask. – jfs Sep 10 '13 at 06:07
  • I agree that the DictReader is better. But in my mind there are 2 ways to solve my problem. 1) Load the CSV into sqlite3 DB in RAM 2) Load it into a list of Dict (but not the same data structure as DictReader. I would expect [{Header1: Cell11, Cell21... ,CellM1}, {Header2: Cell12, Cell22... }... {HeaderN: Cell1N, Cell2N, ... CellMN}] – Winston Sep 11 '13 at 02:26

1 Answers1

0

This question is probably what you're looking for:

How can I pass my locals and access the variables directly from another function?

...which talks about using locals.update().

Generally, manually changing locals is advised against (security reasons mainly), but it depends on what you are using the code for.

Community
  • 1
  • 1
Dman2
  • 700
  • 4
  • 10