This is the famous problem of "ASCIIbetical" order versus "Natural" order as applied to powershell. To be able to sort in powershell the same way as explorer does, you can use this wrapper over StrCmpLogicalW API, that actually performs the natural sorting for Windows Explorer. This will require some plumbing though.
However, this article suggests that there is a three liner implementation of the sort in python. One would hope that Get-ChildItem cmdlet or at least File System Provider can have built-in natural sorting option. Unfortunately, they do not.
So here is the question, what is simplest implementation of this in Powershell? By simple I mean the least amount of code to write, and possibly no third-party/external scripts/components. Ideally I want a short Powershell function that would do the sorting for me.