0

I've got a huge csv file that keeps increasing forever [although sometimes it gets reset], I know that's not good but unfortunately I can't change the design since it's another application that keeps adding stuff there.

I have split this file into new smaller files considering that new stuff is going to appear in that csv file every time.For example, one csv file for each 1000 values or something like that.

I'm thinking about writing a small program to do it and run it periodically via Windows Scheduled Tasks, is it the best way of fixing this problem? If so, Can you help me with the code [Java, VB, C#...]. If it's not the best solution, which path should I follow?

Cœur
  • 37,241
  • 25
  • 195
  • 267
D. Caan
  • 1,907
  • 6
  • 22
  • 36
  • 2
    Do you know *any* programming language? Then that's the language you should do this with. – Marko Topolnik Jan 20 '14 at 17:05
  • 1
    Is it important that the split files have the same number of lines in them? Perhaps you could just move the file every X minutes, like log rotation works. – Markus Jan 20 '14 at 17:06
  • Have a go (in any language you desire) and if you get stuck then come back and ask a specific question. If you don't know any languages then this isn't the right site - have a google for some basic tutorials and take it from there. – Qwerky Jan 20 '14 at 17:07
  • Thanks a lot guys for your replies. I'm sorry I haven't tried any code yet, actually I am looking for advice in what path to follow.@Markus Wüstenberg, there's no need in having the same number of lines, I just need the file split in multiple files automatically because I'll use another software to read these csv files and this software RENAMES the csv file to .OLD after reading it so I can't do it to the original file. Thanks again! :) – D. Caan Jan 20 '14 at 17:12
  • 1
    so the easiest way would be renaming on a regular basis (hourly,daily, weekly - depends on the amount of new data) A scheduled task would do that. – Stephan Jan 20 '14 at 17:18
  • @Stephan thanks for your reply. I'm sorry but I'm afraid I couldn't understand your solution. If I rename the ".OLD" file to csv again I'd be able to compare the processed csv file against the original one, with new entries. But how to do this? via batch file [cmd commands]? Thanks again – D. Caan Jan 20 '14 at 17:24
  • I hope, I get it right. I think this way: App1 generates `a.csv`. You rename/move it to another name/directory (`b.csv` (or `otherdir\a.csv`). App2 works with `b.csv` and renames it to `b.old` while App1 generates a new `a.csv`. Is this the way you want it? – Stephan Jan 20 '14 at 17:33
  • @Stephan App1 just adds new entries to the same csv – D. Caan Jan 20 '14 at 17:34
  • so I just have to generate a new csv with the differences between otherdir\a.csv and b.old then I'l have only the new stuff for the next time App2 reads it – D. Caan Jan 20 '14 at 17:36
  • I'm sorry for the confusing question I'm going to build a step-by-step of what I need.. Apologies to everyone and thanks for the answers! – D. Caan Jan 20 '14 at 17:43
  • 1
    would be the best. Still trying to reconstruct the way your data goes. App1 _should_ generate a new csv with the same name, if you "steal" it by moving or renaming. – Stephan Jan 20 '14 at 17:45

0 Answers0