Error with hashtable join and covertFrom String

Question

I'm using the code referenced here: with the examples etc but I'm running into an error with this line:

$remotefilehash =  ($remotefiles -replace '^[a-f0-9]{32}(  )', '$0=  ' -join "`n") | ConvertFrom-StringData

https://stackoverflow.com/a/69782527/800592

here's the full error:

ConvertFrom-StringData : Data item '4826367e20469078693dced6dc642b96' in line '4826367e20469078693dced6dc642b96  =  My Videos/Drone/DJI-FPV/DJI_0003.MP4' is already defined. 
At P:\scripts\code\pcloud_sync.ps1:66 char:86
+ ... ace '^[a-f0-9]{32}(  )', '$0=  ' -join "`n") | ConvertFrom-StringData
+                                                    ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-StringData], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.ConvertFromStringDataCommand

Update2 I think my desire is to deduplicate my remoteFiles object which is simply an list/arry? of md5sum file path. since it seems like there are times where there is the same file in two different paths I want to de duplicate the remoteFiles obj on the md5sum

Answer 1

ConvertFrom-StringData parses each input string in the form of one or more <name>=<value> lines into a hashtable.

In doing so it doesn't tolerate duplicates; that is, if a previous <name> value is encountered again in a given input string, the statement-terminating error you saw occurs.

A simple repro:

# !! Error: 
# !!  "ConvertFrom-StringData: Data item 'Key' in line 'Key=b' is already defined"
ConvertFrom-StringData "Key=a`nKey=b"

Thus, the solution is to eliminate duplicates from your input string first - assuming that the <value> part among the lines with duplicate <name>s is also the same.

If not, you'll need a different data structure to represent your input data, which ConvertFrom-StringData cannot provide.

If you only care about the <name> parts - ignoring duplicates, irrespective of their <value>s - you can parse them into a [System.Collections.Generic.HashSet[T] instance; in your case:

$remotefileHashSet =
  [System.Collections.Generic.HashSet[string]] $remotefiles.ForEach({ (-split $_)[0] })

Note that hash sets are case-sensitive by default with respect to lookups; more work is needed if you want case-insensitivity: see this answer.