0

ThreadJobs has access to the same environment as it was started in.
But normally, PowerShell will respond with a syntax error when trying to change variables from the parent level.

The documentation MS Learn - about_Thread_Jobs has som insights but nothing I could find useful.

The example below illustrates the issue when trying to use plain PowerShell variables.

[Array]$Numbers = @()

foreach ($i in 0..11) {
    $Jobs = Start-ThreadJob {
        $Using:Numbers += $using:i
    }
}

$Jobs | Wait-Job | Remove-Job

$Numbers

ParserError: 
Line |
   6 |          $Using:Numbers += $using:i
     |          ~~~~~~~~~~~~~~~
     | The assignment expression is not valid. The input to an assignment operator must 
     | be an object that is able to accept assignments, such as a variable or a property.
Dennis
  • 871
  • 9
  • 29

2 Answers2

1

As the threads are run in parallell, there has to be some kind of method that prevents the parent object from getting corrupted or a ThreadJob from failing if two or more threads tries to perform operations at the exact same time on the object.

After wrestling several days with the concept of thread safe execution (and getting great help with the patience of Santiago Squarzon and others, my own conclusion is:

All the operation in the thread has to be made thread safe (hence the name).

  1. Don't try setting values with Using: if the objects are "plain" unless you can guarantee the value has been locked
  2. Even if using thread safe objects, don't try to both read and write of the value unless you can guarantee the value has been locked in between the two operations
  3. Only use the provided thread safe methods for manipulating data in the thread safe objects unless you can guarantee the value has been locked

In .Net, I found two thread safe classes you can work with

(There is one class name available per T-reference.)

None of the classes has a thread safe method for incrementing values.

So a thread safe ThreadJob in PowerShell 7.x, only adding new items to parent objects, might look like this

$SafeNumbers = [System.Collections.Concurrent.ConcurrentBag[object]]::new()

foreach ($i in 0..11) {
    $Thread = Start-ThreadJob {
        ($Using:SafeNumbers).Add($Using:i)
    }
}

Get-Job | Wait-Job | Remove-Job

$SafeNumbers.ToArray()

11
10
9
8
7
6
5
4
3
2
1
0

The order of the output is of course not guaranteed.

Dennis
  • 871
  • 9
  • 29
  • 1
    Now this is the right thing to do :) but now comes the time when you need to ask yourself: "Do I really need to add items from the threads themselves? Wouldn't it be easier to just produce the output from the threads and capture that output synchronously via `Receive-Job`"? That's exactly what the last example of my answer I linked you before was trying to explain ;) – Santiago Squarzon Jun 26 '23 at 18:41
  • 1
    Exactly :) I tried being clever and save me some job, which is never a good thing if Murphy has he's say. – Dennis Jun 26 '23 at 18:44
  • 1
    Since you seem to be keen on learning multithreading in powershell and you also seem to be stuck on windows powershell 5.1 (as I see no mention of `ForEach-Object -Parallel`) this might be helpful to you: https://stackoverflow.com/questions/74257556/is-there-an-easier-way-to-run-commands-in-parallel-while-keeping-it-efficient-in (the code in the GitHub repo is far better than the one showed in that answer if you decide to invest some time researching it) – Santiago Squarzon Jun 26 '23 at 18:53
  • `Start-ThreadJob` is native only to PowerShell 7.x + (you can of course install some modules for that in PoSH 5.1 as well). Pipes get's too hard to read when they get too big, so I try to stay away if I really don't need them for big objects. I try to focus on maintainable code :) – Dennis Jun 26 '23 at 19:19
0

After some searching, I found a work around where variable referrals apparently works...

But it will fail sooner or later, so DON´T use the example.

[Array]$Numbers = @()
$refNumbersVar = Get-Variable Numbers

foreach ($i in 0..11) {
    $Jobs = Start-ThreadJob {
        ($Using:refNumbersVar).Value += $using:i
    }
}

$Jobs | Wait-Job | Remove-Job

$Numbers

Edit: Warned all to not use this example.

Dennis
  • 871
  • 9
  • 29
  • 1
    This is not thread safe fyi – Santiago Squarzon Jun 22 '23 at 12:20
  • I was suspecting something like that, but I'm only adding values. Not changing or replacing. So that would still make it safe, right? – Dennis Jun 22 '23 at 12:24
  • Replaced [array] with a thread safe example. – Dennis Jun 22 '23 at 13:38
  • 1
    `ConcurrentDictionary` doesn't ensure thread safety for updating a single Value either. If you want thread safety for a single value you must use a locking mechanism like the one demonstrated here: https://stackoverflow.com/a/75252238/15339544 – Santiago Squarzon Jun 22 '23 at 15:06
  • 1
    https://github.com/MicrosoftDocs/PowerShell-Docs/issues/7378#issuecomment-1510052078 that comment demos how easy it is to make a sync hash or a concurrent dict fail while updating a single value. – Santiago Squarzon Jun 22 '23 at 15:28
  • But System.Collections.Concurrent.ConcurrentQueue is a thread-safe collection class. What's needed and what is actually the issue in this case? I really don't get why that example is a proof of failure? – Dennis Jun 22 '23 at 16:17
  • It's a proof of failure because it demonstrates what _could_ happen if you attempt to update a value from different threads at the same time. As you stated in the GH comment: "The only thing you need to care about is that two different threads doesn't try to update the value in the exact same moment, right?" correct, so, how do you ensure that 2 or more threads don't try to do such a thing? The answer is with a locking mechanism. ConcurrentDictionary ensures that reading and writing is thread safe but it doesnt ensure that enumerating is thread safe. – Santiago Squarzon Jun 22 '23 at 16:42
  • Read here carefully: https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2?view=net-7.0#thread-safety – Santiago Squarzon Jun 22 '23 at 16:42
  • As I read it, I can only use the specifically thread safe methods of the thread safe objects? So `+=` is not thread safe? – Dennis Jun 22 '23 at 17:03
  • Warned of using this answer. – Dennis Jun 26 '23 at 18:33