1

I am using the below FOR-IN-DO syntax to read and get the last row of (>3GB) csv files

For /F "UseBackQ Delims==" %%A In (test.csv) Do Set "lastline=%%A"

The code works for small files. However, the FOR-IN-DO cannot open the large csv files. Is there a workaround for this problem?

phuclv
  • 37,963
  • 15
  • 156
  • 475
plm0998
  • 35
  • 5
  • 2
    it's bad idea to do such a thing in batch. It'll be extremely inefficient even if it's possible. Why not use a better scripting tool like powershell or python? – phuclv Apr 28 '21 at 10:18
  • 1
    it's also bad to use csv for huge data structures. There are many better binary formats for this purpose – phuclv Apr 28 '21 at 10:19
  • 2
    The `FOR` command reads the entire file into memory. And because cmd.exe is a 32bit program it is limited to a file size of 2,147,483,647 bytes. If you want to get the last line of a file I would recommend you use a tool designed for that purpose. Their are many ports of the command `TAIL` that work on Windows. I would assume you could even call out to [Powershell](https://stackoverflow.com/a/36507532/1417694) to accomplish this. – Squashman Apr 28 '21 at 12:48
  • 1
    @Squashman on 64-bit Windows there are both 64-bit and 32-bit cmd.exe in `%windir%\System32` and `%windir%\SysWOW64` respectively – phuclv Apr 28 '21 at 16:54
  • Thank you @phuclv . I will also try using batch to run powershell or vb script to do this. – plm0998 Apr 29 '21 at 01:50

1 Answers1

1

Depending upon the content of your source file, i.e. the CSV is not TAB delimited, or contain TAB characters you need to preserve, you may be able to do it like this:

Set "SourceFile=test.csv"
For /F %%G In ('%SystemRoot%\System32\find.exe /C /V "" 0^<"%SourceFile%"') Do Set /A "TotalLines=%%G-1"
For /F Delims^=^ EOL^= %%G In ('%SystemRoot%\System32\more.com +%TotalLines% 0^<"%SourceFile%"') Do Set "LastLine=%%G"

Based upon your comment, I'd offer the following modification of the above:

Set "SourceFile=%UserProfile%\Desktop\test.csv"
Set "DestinationDir=%UserProfile%\Documents"
Set "LineMatch=Specific String"
For /F %%G In ('%SystemRoot%\System32\find.exe /C /V "" 0^<"%SourceFile%"') Do Set /A "TotalLines=%%G-1"
For /F Delims^=^ EOL^= %%G In ('%SystemRoot%\System32\more.com +%TotalLines% 0^<"%SourceFile%"') Do If /I "%%G" == "%LineMatch%" Move /Y "%SourceFile%" "%DestinationDir%"
Compo
  • 36,585
  • 5
  • 27
  • 39
  • Please note that this example is to reproduce filling the content of a variable with the last line of the file, which replicated your own code `For /F "UseBackQ Delims==" %%A In (test.csv) Do Set "lastline=%%A"`. Anything regarding modification of the content of the last line of that file is a different question, and therefore not covered in my above response. – Compo Apr 28 '21 at 11:09
  • Thank you. I think that is good enough, what I want is to read the last line of the CSV, and if if "%lastline%" == "%specific_string%", I will move the CSV file to another directory. – plm0998 Apr 29 '21 at 01:44
  • I've added an additional example to my above answer based upon your comment @plm0998, I hope it helps. – Compo Apr 29 '21 at 08:10