Reading/Writing Large Files VB.NET

Question

I am a new user and struggling with the below:

I would need to read the file and write back with some modifications, the issue is:

1. “ReadToEnd” works fine for smaller files [100MB approx] and everything works fine what I like to do. But for Bigger Size Files [300 MB +], it bombs out.
2. Then I tried “ReadLine” (Reading Line By Line) It works fine on the smaller or bigger files but it takes very very long to save back.

I have included both of the codes below "ReadToEnd" and "ReadLine" For testing you would need to create "100MB-File.txt" and "300MB-File.txt" files in c:\temp\ area.

I would really appreciate your help in this regard

'----------------Reading whole File ReadToEnd
Dim sr As New StreamReader("C:\temp\100MB-File.txt")
Dim path As String = "C:\temp\myFileNew1.txt"
Dim oneLine As String
oneLine = sr.ReadToEnd
Using sw As StreamWriter = File.CreateText(path)
    sw.WriteLine(oneLine)
End Using
sr.Close()

''---------------Reading Line by Line
Dim sr As New StreamReader("C:\temp\300MB-File.txt")
Dim path As String = "C:\temp\myFileNew2.txt"
Dim oneLine As String
oneLine = sr.ReadLine

Using sw As StreamWriter = File.CreateText(path)
    sw.WriteLine(oneLine)
End Using

Do Until sr.EndOfStream
    Console.WriteLine(oneLine)
    oneLine = sr.ReadLine()

    Using sw As StreamWriter = File.AppendText(path)
        sw.WriteLine(oneLine)
    End Using
Loop
sr.Close()

Answer 1

In addition to the Console.WriteLine mentioned in the comments, you are opening and closing the output file for each line you are writing. If you just open the file once, it should be much faster:

Dim sr As New StreamReader("C:\temp\300MB-File.txt")
Dim path As String = "C:\temp\myFileNew2.txt"
Dim oneLine As String
oneLine = sr.ReadLine

Using sw As StreamWriter = File.CreateText(path)
    sw.WriteLine(oneLine)
    Do Until sr.EndOfStream
        'Console.WriteLine(oneLine)
        oneLine = sr.ReadLine()
        sw.WriteLine(oneLine)
    Loop
End Using

sr.Close()

You could also simplify the code using File.ReadLines:

Dim inPath As String = "C:\temp\300MB-File.txt"
Dim outPath As String = "C:\temp\myFileNew2.txt"

Using sw As StreamWriter = File.CreateText(outPath)
    For Each oneLine As String In File.ReadLines(inPath)
        ' modify line here
        sw.WriteLine(oneLine)
    Next
End Using