C# - Writing to text file inside parallel.foreach loop

Question

I made a proxy checker, however, since I usnig Parallel.foreach loop, I get after X amount of time, an error and the program just crash.

it says that the writing is already used by another process, here is my code:

public static void Check()
    {
        Proxy.Load();
        ThreadPool.SetMinThreads(Program.Threads, Program.Threads); // For example 100, 100
        ThreadPool.SetMaxThreads(Program.Threads, Program.Threads); // For example 100, 100
        Parallel.ForEach(Proxy.Proxies, proxy =>
        {
            using (var request = new HttpRequest())
            {
                try
                {
                    request.UserAgent =
                        "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Viv/1.96.1147.52";
                    request.Proxy = HttpProxyClient.Parse(proxy);
                    request.ConnectTimeout = Program.TimeOut;
                    var Post = request.Post("http://209.250.244.126/judge/");
                    Console.ForegroundColor = ConsoleColor.Green;
                    Console.WriteLine($"[+] {proxy} - HTTP");
                    HTTP.Add(proxy);
                }
                catch
                {
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.WriteLine($"[-] {proxy}");
                }
            }
            // Write the Good proxies to the text file
            File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", HTTP);
        });
    }

What I need it to do:

I want to write all the Good proxies to the text file while the check is running, and I still want to use threads to make the check much faster

When two threads try to write to the file at the same time, then there is a problem, since one thread uses the file and the second thread will raise an exception. You need to make more files (for each thread), or usea different way of saving _(and locking file access is not the best way)_. Why all the threads write to the same file? The second run will remove the first result, the third remove second result etc. Remove writing when do you do not need the result file, or change the file name to be unique. — Julo, Sep 24 '18 at 13:34
And one more comment: do not use `Environment.CurrentDirectory + "/Good.txt"` and replace it with `Path.Combine(Environment.CurrentDirectory, "Good.txt")` _(without `/`)_. — Julo, Sep 24 '18 at 13:37
The short answer is that you should change `HTTP` to be a `ConcurrentBag`. Then **outside** the `Parallel.ForEach` you would do `File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", HTTP);` (or `File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", HTTP.ToList());` if the first one doesn't compile). — mjwills, Sep 24 '18 at 13:44
Additionally, using threads to make web requests isn't a great idea. You don't need them, as per https://stackoverflow.com/a/29214947/34092 . — mjwills, Sep 24 '18 at 13:46
@mjwills let's say I want to check 50,000 proxies, but I don't want to wait until the check is finished, I won't be able to do it because I have to wait until the check is done — John, Sep 24 '18 at 13:53
OK, I'd still suggest my original approach. Since it is simpler. But here is an alternative. Change `HTTP` to be a `BlockingCollection`. Create a new `Task` and `Run` it **prior** to the `Parallel.ForEach`. It will process the entries added to `HTTP` (inside a loop). You will call `File.WriteAllText` the first time and then `File.AppendAllText` for second (and later) iterations of the loop. The use of a `BlockingCollection` will allow the writing of the text file to occur, in a thread-safe way, independent of the http requests. — mjwills, Sep 24 '18 at 14:04
ok, Thank you :) I will read the link about avoid using threads — John, Sep 24 '18 at 14:10

score 0 · Answer 1 · answered Sep 25 '18 at 22:33

0

Try wrapping the file output line in a lock statement. This will restrict assess to the file to only one thread at a time.

answered Sep 25 '18 at 22:33

D Jordan

84
4

C# - Writing to text file inside parallel.foreach loop

1 Answers1