0

simple question: any ideas how this should be properly done?. I have 3 txt files with lots of information, I created a class that will be in charge of reading the data from the txt files and returning the data as a list of DTO components (Yes, the information can be bundle as such logic unit), depending on the txt file, after that the client will use a DAO and will use such a list and insert the data to a local database (sqlite). My concern is that having such a List could be memory demanding, should I avoid using such list and somehow insert this data using the dao object directly without bundling the data into a dto and finally such list?

Jhonathan Ibanez
  • 89
  • 1
  • 2
  • 10
  • How big will your files be? If they're going to be too big to fit in memory then you'll have to process them in chunks. – Mike B Feb 18 '14 at 20:50
  • Have you _tried_ it? It sounds like you're trying to perform optimization without even taking the "naive" approach. Sometimes, naive is good enough. If you _have_ tried it already and determined it to be too slow or demanding on the client machine, post some code. – Brian S Feb 18 '14 at 20:51

2 Answers2

0

You are asking a good question and partially answering it yourself. Yes, sure if you really have a lot of information you should not read all information from file and then store it in DB. You should read information chunk-by chunk or even if it is possible (from application point of view) line-by-line and store each line is DB.

In this case you will need memory for one line only at any time.

You can design the application as following.

  • File parser that returns Iterable<Row>
  • DB writer that accepts Iterable<Row> and stores rows in DB,
  • Manager that calls both.

In this case the logic responsible on reading and writing file will be encapsulated into certain modules and no extra memory consumption will be required.

AlexR
  • 114,158
  • 16
  • 130
  • 208
  • I was thinkg of using my DAO as my "Manager" in the sense they would "store rows in DB" (row actually being DTO in my first approach), would be the same that you suggest? – Jhonathan Ibanez Feb 18 '14 at 21:33
  • No, it is not, I read again and now I understand, sorry I cant upvote you, thanks a lot though. – Jhonathan Ibanez Feb 18 '14 at 21:44
0

do not return list, but an iterator like in this example: Iterating over the content of a text file line by line - is there a best practice? (vs. PMD's AssignmentInOperand)

you have to modify this iterator to return your DTO instead of String:

for(MyDTO line : new BufferedReaderIterator(br)){
    // do some work
}

Now you will iterate over file line by line, but you will return DTOs instead of returning lines. Such solution has small memory impact.

Community
  • 1
  • 1
Maciej Miklas
  • 3,305
  • 4
  • 27
  • 52
  • I was thinking this very same approach, I would make my client create some object or use certain method to create such a loop to finally insert data through such dto and the help of my dao. I think will use a combination of your answer and AlexR´s Sorry I can´t upvote you. – Jhonathan Ibanez Feb 18 '14 at 21:38
  • I did not see the first answer - it must have been posted in parallel. At least both ideas are the same. – Maciej Miklas Feb 19 '14 at 06:37