0

I use a console app to work on large files. I use

IO.File.ReadAllBytes(OpenFileDlg.FileName)

but it loads all the large file on memory and stops when the memory is full , without reading it completely. I want to split the large file or read a part of it, "without loading all the file on memory, so it would work on small RAM"

Areeg Ghonaim
  • 13
  • 6
  • Do you know at what offset you need to begin to read? If so, use a Stream and [`Seek()`](https://learn.microsoft.com/en-us/dotnet/api/system.io.filestream.seek) that offset. – Jimi Oct 10 '20 at 18:39
  • no, I try to open large files not texts, there may be an HD video, so i don't have an offset – Areeg Ghonaim Oct 10 '20 at 18:46
  • Yes, well, it's the opposite. You don't have an offset if it's a text file, usually. Maybe you're looking for a string. If it's a - let's call it *binary* - file, then you can specify an offset. Are you saying that you don't know where to start to read? What is that you're actually trying to do with these files? – Jimi Oct 10 '20 at 18:48
  • converting the large file into hex then saving it. it works if the file is about 150mb or less only. – Areeg Ghonaim Oct 10 '20 at 18:51
  • What does *converting the large file into hex* mean? Are you saying that you want to convert your files to a Base64 string? Why? – Jimi Oct 10 '20 at 18:52
  • I already convert files from binary into Hex. but it works on memory, so this can not work on large files. – Areeg Ghonaim Oct 10 '20 at 18:56
  • *convert files from binary into Hex* doesn't mean anything. Are you converting byte arrays to a string representation? What's the use of it? Are you storing these things in a database? Something else? – Jimi Oct 10 '20 at 18:59
  • yes i convert byte arrays to a string. then i save this string in a file. the problem is: `IO.File.ReadAllBytes(filename)` It reads all the file on memory and this cannot work on large files in my small ram. so i want a way to split the file first without loading it on ram. – Areeg Ghonaim Oct 10 '20 at 19:03
  • All right, you're converting byte arrays to strings. Why? What's the reason and the destination of this operation (assume that, since I'm asking, it's something that counts)? Also, what kind of strings? Are those Base64 strings or something else (it also counts)? What will happen to these strings (still counts)? – Jimi Oct 10 '20 at 19:08
  • for encryption on a byte level – Areeg Ghonaim Oct 10 '20 at 19:35
  • To encrypt files, you write the content to a [CryptoStream](https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.cryptostream) with a standard provider, AES (RijndaelManaged) etc. In its simpler form, it's a sequential write to a Stream. No need to convert to string or to load into memory anything. – Jimi Oct 10 '20 at 19:46
  • my app already encrypts files – Areeg Ghonaim Oct 10 '20 at 20:08
  • Do you mean that you're already using a CryptoStream to generate encrypted files? So, what is your question about? – Jimi Oct 10 '20 at 20:10
  • no. i made something like it – Areeg Ghonaim Oct 10 '20 at 21:02

1 Answers1

1

Use FileStream to read in chunks of 4096 bytes then encrypt them (or do any necessary processing) and then write them to the new file stream, since it's streaming in/out it won't hog up all the memory trying to read/write everything at once, simple example:

Dim bytesRead As Integer
Dim buffer(4096) As Byte 'Default buffer size is 4096
Using inFile As New IO.FileStream("C:\\Temp\\inFile.bin", FileMode.Open)
    Using outFile As New IO.FileStream("C:\\Temp\\outFile.bin", FileMode.Append, FileAccess.Write)
        Do
            bytesRead = inFile.Read(buffer, 0, buffer.Length)
            If bytesRead > 0 Then
                ' Do encryption or any necessary processing of the bytes
                outFile.Write(buffer, 0, bytesRead)
            End If
        Loop While bytesRead > 0
    End Using
End Using
kshkarin
  • 574
  • 4
  • 9
  • ok, i tried it. It is a real life saver. but with one small problem. After completion, It added one full megabyte to a 33 mb file. when i tried it on a small 200 byte text file, It added 4 kb to it and had 3860 space at the end of the file . I don't know what is wrong. – Areeg Ghonaim Oct 11 '20 at 23:31
  • @AreegGhonaim had a typo in there, it should be `outFile.Write(buffer, 0, bytesRead)` check the edit – kshkarin Oct 11 '20 at 23:41
  • @AreegGhonaim Welcome to SO, please read here https://stackoverflow.com/help/someone-answers – kshkarin Oct 12 '20 at 08:25
  • hey can i use memory stream? i want this code to do that buffer depending on the available physical memory ,so that it would work faster if there r more rams? – Areeg Ghonaim Oct 17 '20 at 23:44
  • `FileStream` is already in memory and uses 4kb as the default buffer size, there's an overloaded constructor that allows to specify how large the buffer should be, but it's hard to tell if you'll see any benefits from increasing the buffer size, need to benchmark to determine the actual performance, please see here: https://stackoverflow.com/questions/1862982/c-sharp-filestream-optimal-buffer-size-for-writing-large-files – kshkarin Oct 18 '20 at 13:58