Recreating a linux md5 checksum on Windows

Question

I'm performing some pretty straightforward checksums on our linux boxes, but I now need to recreate something similar for our windows users. To give me a single checksum, I just run:

md5sum *.txt | awk '{ print $1 }' | md5sum

I'm struggling to recreate this in Windows, either with a batch file or Powershell. The closest I've got is:

Get-ChildItem $path -Filter *.txt | 
Foreach-Object {
   $hash =  Get-FileHash -Algorithm MD5 -Path ($path + "\" + $_) | Select -ExpandProperty "Hash"
   $hash = $hash.tolower()  #Get-FileHash returns checksums in uppercase, linux in lower case (!)
   Write-host $hash
}

This will print the same checksum results for each file to the console as the linux command, but piping that back to Get-FileHash to get a single output that matches the linux equivalent is eluding me. Writing to a file gets me stuck with carriage return differences

Streaming as a string back to Get-FileHash doesn't return the same checksum:

$String = Get-FileHash -Algorithm MD5 -Path (Get-ChildItem -path $files -Recurse) | Select -ExpandProperty "Hash"
$stringAsStream = [System.IO.MemoryStream]::new()
$writer = [System.IO.StreamWriter]::new($stringAsStream)
$writer.write($stringAsStream)
Get-FileHash -Algorithm MD5 -InputStream $stringAsStream

Am I over-engineering this? I'm sure this shouldn't be this complicated! TIA

Answer 1

You need to reference the .Hash property on the returned object from Get-FileHash. If you want a similar view to md5hash, you can also use Select-Object to curate this:

# Get filehashes in $path with similar output to md5sum
$fileHashes = Get-ChildItem $path -File | Get-FileHash -Algorithm MD5

# Once you have the hashes, you can reference the properties as follows
# .Algorithm is the hashing algo
# .Hash is the actual file hash
# .Path is the full path to the file
foreach( $hash in $fileHashes ){
  "$($hash.Algorithm):$($hash.Hash) ($($hash.Path))"
}

For each file in $path, the above foreach loop will produce a line that similar to:

MD5:B4976887F256A26B59A9D97656BF2078 (C:\Users\username\dl\installer.msi)

The algorithm, hash, and filenames will obviously differ based on your selected hashing algorithm and filesystem.

Answer 2

The devil is in the details:

• (known already) Get-FileHash returns checksums in uppercase while Linux md5sum in lower case (!);
• The FileSystem provider's filter *.txt is not case sensitive in PowerShell while in Linux depends on the option nocaseglob. If set (shopt -s nocaseglob) then Bash matches filenames in a case-insensitive fashion when performing filename expansion. Otherwise (shopt -u nocaseglob), filename matching is case-sensitive;
• Order: Get-ChildItem output is ordered according to Unicode collation algorithm while in Linux *.txt filter is expanded in order of LC_COLLATE category (LC_COLLATE="C.UTF-8" on my system).

In the following (partially commented) script, three # Test blocks demonstrate my debugging steps to the final solution:

Function Get-StringHash {
    [OutputType([System.String])]
    param(
        # named or positional: a string
        [Parameter(Position=0)]
        [string]$InputObject
    )
    $stringAsStream = [System.IO.MemoryStream]::new()
    $writer = [System.IO.StreamWriter]::new($stringAsStream)
    $writer.write( $InputObject)
    $writer.Flush()
    $stringAsStream.Position = 0
    Get-FileHash -Algorithm MD5 -InputStream $stringAsStream |
        Select-Object -ExpandProperty Hash
    $writer.Close()
    $writer.Dispose()
    $stringAsStream.Close()
    $stringAsStream.Dispose()
}

function ConvertTo-Utf8String {
    [OutputType([System.String])]
    param(
        # named or positional: a string
        [Parameter(Position=0, Mandatory = $false)]
        [string]$InputObject = ''
    )
    begin {
        $InChars  = [char[]]$InputObject
        $InChLen  = $InChars.Count
        $AuxU_8 = [System.Collections.ArrayList]::new()
    }
    process {
        for ($ii= 0; $ii -lt $InChLen; $ii++) {
            if ( [char]::IsHighSurrogate( $InChars[$ii]) -and
                    ( 1 + $ii) -lt  $InChLen             -and
                    [char]::IsLowSurrogate( $InChars[1 + $ii]) ) {
                $s = [char]::ConvertFromUtf32(
                     [char]::ConvertToUtf32( $InChars[$ii], $InChars[1 + $ii]))
                $ii ++
            } else {
                $s = $InChars[$ii]
            }
            [void]$AuxU_8.Add( 
                ([System.Text.UTF32Encoding]::UTF8.GetBytes($s) | 
                    ForEach-Object { '{0:X2}' -f $_}) -join ''
            )
        }
    }
    end { $AuxU_8 -join '' }
}

# Set variables
$hashUbuntu = '5d944e44149fece685d3eb71fb94e71b'
$hashUbuntu   <# copied from 'Ubuntu 20.04 LTS' in Wsl2:
              cd `wslpath -a 'D:\\bat'`
              md5sum *.txt | awk '{ print $1 }' | md5sum | awk '{ print $1 }'
              <##>
$LF = [char]0x0A   # Line Feed (LF)
$path = 'D:\Bat'   # testing directory


$filenames = 'D:\bat\md5sum_Ubuntu_awk.lst'
<# obtained from 'Ubuntu 20.04 LTS' in Wsl2:
    cd `wslpath -a 'D:\\bat'`
    md5sum *.txt | awk '{ print $1 }' > md5sum_Ubuntu_awk.lst
    md5sum md5sum_Ubuntu_awk.lst | awk '{ print $1 }' # for reference
<##>

# Test #1: is `Get-FileHash` the same (beyond character case)?
$hashFile = Get-FileHash -Algorithm MD5 -Path $filenames |
                Select-Object -ExpandProperty Hash
$hashFile.ToLower() -ceq $hashUbuntu

# Test #2: is `$stringToHash` well-defined? is `Get-StringHash` the same?
$hashArray = Get-Content $filenames -Encoding UTF8
$stringToHash = ($hashArray -join $LF) + $LF
(Get-StringHash -InputObject $stringToHash) -eq $hashUbuntu 

# Test #3: another check: is `Get-StringHash` the same?
Push-Location -Path $path
$filesInBashOrder = bash.exe -c "ls -1 *.txt"
$hashArray = $filesInBashOrder |
    Foreach-Object {
        $hash = Get-FileHash -Algorithm MD5 -Path (
                        Join-Path -Path $path -ChildPath $_) |
                    Select-Object -ExpandProperty "Hash"
        $hash.tolower()
    }
$stringToHash = ($hashArray -join $LF) + $LF
(Get-StringHash -InputObject $stringToHash) -eq $hashUbuntu
Pop-Location

# Solution - ordinal order assuming `LC_COLLATE="C.UTF-8"` in Linux
Push-Location -Path $path
$hashArray = Get-ChildItem -Filter *.txt -Force -ErrorAction SilentlyContinue |
    Where-Object {$_.Name -clike "*.txt"} | # only if `shopt -u nocaseglob`
    Sort-Object -Property { (ConvertTo-Utf8String -InputObject $_.Name) } |
    Get-FileHash -Algorithm MD5 |
        Select-Object -ExpandProperty "Hash" |
    Foreach-Object {
        $_.ToLower()
    }
$stringToHash = ($hashArray -join $LF) + $LF
(Get-StringHash -InputObject $stringToHash).ToLower() -ceq $hashUbuntu
Pop-Location

Output (tested on 278 files): .\SO\69181414.ps1

5d944e44149fece685d3eb71fb94e71b
True
True
True
True