Merging hashtables in PowerShell: how?

Question

I am trying to merge two hashtables, overwriting key-value pairs in the first if the same key exists in the second.

To do this I wrote this function which first removes all key-value pairs in the first hastable if the same key exists in the second hashtable.

When I type this into PowerShell line by line it works. But when I run the entire function, PowerShell asks me to provide (what it considers) missing parameters to foreach-object.

function mergehashtables($htold, $htnew)
{
    $htold.getenumerator() | foreach-object
    {
        $key = $_.key
        if ($htnew.containskey($key))
        {
            $htold.remove($key)
        }
    }
    $htnew = $htold + $htnew
    return $htnew
}

Output:

PS C:\> mergehashtables $ht $ht2

cmdlet ForEach-Object at command pipeline position 1
Supply values for the following parameters:
Process[0]:

$ht and $ht2 are hashtables containing two key-value pairs each, one of them with the key "name" in both hashtables.

What am I doing wrong?

it is disappointing that this isn't built in, it is such a common need — Josh Petitt, Mar 13 '14 at 15:49
@JoshPetitt check out this library-it even handles recursive merges: https://github.com/chriskuech/functional — Dave Neeley, Jan 02 '20 at 23:09

iRon · Answer 1 · 2017-10-26T06:30:49.217

Merge-Hashtables

Instead of removing keys you might consider to simply overwrite them:

$h1 = @{a = 9; b = 8; c = 7}
$h2 = @{b = 6; c = 5; d = 4}
$h3 = @{c = 3; d = 2; e = 1}


Function Merge-Hashtables {
    $Output = @{}
    ForEach ($Hashtable in ($Input + $Args)) {
        If ($Hashtable -is [Hashtable]) {
            ForEach ($Key in $Hashtable.Keys) {$Output.$Key = $Hashtable.$Key}
        }
    }
    $Output
}

For this cmdlet you can use several syntaxes and you are not limited to two input tables: Using the pipeline: $h1, $h2, $h3 | Merge-Hashtables
Using arguments: Merge-Hashtables $h1 $h2 $h3
Or a combination: $h1 | Merge-Hashtables $h2 $h3
All above examples return the same hash table:

Name                           Value
----                           -----
e                              1
d                              2
b                              6
c                              3
a                              9

If there are any duplicate keys in the supplied hash tables, the value of the last hash table is taken.

(Added 2017-07-09)

Merge-Hashtables version 2

In general, I prefer more global functions which can be customized with parameters to specific needs as in the original question: "overwriting key-value pairs in the first if the same key exists in the second". Why letting the last one overrule and not the first? Why removing anything at all? Maybe someone else want to merge or join the values or get the largest value or just the average...
The version below does no longer support supplying hash tables as arguments (you can only pipe hash tables to the function) but has a parameter that lets you decide how to treat the value array in duplicate entries by operating the value array assigned to the hash key presented in the current object ($_).

Function

Function Merge-Hashtables([ScriptBlock]$Operator) {
    $Output = @{}
    ForEach ($Hashtable in $Input) {
        If ($Hashtable -is [Hashtable]) {
            ForEach ($Key in $Hashtable.Keys) {$Output.$Key = If ($Output.ContainsKey($Key)) {@($Output.$Key) + $Hashtable.$Key} Else  {$Hashtable.$Key}}
        }
    }
    If ($Operator) {ForEach ($Key in @($Output.Keys)) {$_ = @($Output.$Key); $Output.$Key = Invoke-Command $Operator}}
    $Output
}

Syntax

HashTable[] <Hashtables> | Merge-Hashtables [-Operator <ScriptBlock>]

Default By default, all values from duplicated hash table entries will added to an array:

PS C:\> $h1, $h2, $h3 | Merge-Hashtables

Name                           Value
----                           -----
e                              1
d                              {4, 2}
b                              {8, 6}
c                              {7, 5, 3}
a                              9

Examples To get the same result as version 1 (using the last values) use the command: $h1, $h2, $h3 | Merge-Hashtables {$_[-1]}. If you would like to use the first values instead, the command is: $h1, $h2, $h3 | Merge-Hashtables {$_[0]} or the largest values: $h1, $h2, $h3 | Merge-Hashtables {($_ | Measure-Object -Maximum).Maximum}.

More examples:

PS C:\> $h1, $h2, $h3 | Merge-Hashtables {($_ | Measure-Object -Average).Average} # Take the average values"

Name                           Value
----                           -----
e                              1
d                              3
b                              7
c                              5
a                              9


PS C:\> $h1, $h2, $h3 | Merge-Hashtables {$_ -Join ""} # Join the values together

Name                           Value
----                           -----
e                              1
d                              42
b                              86
c                              753
a                              9


PS C:\> $h1, $h2, $h3 | Merge-Hashtables {$_ | Sort-Object} # Sort the values list

Name                           Value
----                           -----
e                              1
d                              {2, 4}
b                              {6, 8}
c                              {3, 5, 7}
a                              9

way cleaner and more versatile that other solutions. Special appreciation for `$Input + $Args` technique, I haven't seen it before and it looks promising for extra-small utility functions which can process pipeline and array arguments — maoizm, Jun 30 '17 at 09:28
@iRon, very interesting technique. Thanks for sharing. I've made a change to allow ordering. Rather than detract from your excellent answer and post my own answer, I've created a Gist: https://gist.github.com/arcotek-ltd/41f2326aca65d0dca7d7e669bde871d9. By all means, add it to your official answer, if you want. — woter324, Apr 17 '20 at 13:26
@woter324, thanks for the feedback and suggestion. You might consider to make your script more [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) by using the IDirectory interface like: `If ($Hashtable -is [System.Collections.IDictionary])` and the common `.Contains()` method but in general, note that the user might not expect a `[Ordered]` type as output (even it quiet easily converts into a `[HashTable]`). Another improvement to these scripts is [using the `get_Keys()` method rather than the `Keys` property](https://stackoverflow.com/q/60810458/1701026). — iRon, Apr 17 '20 at 14:37

score 17 · Accepted Answer · answered Jan 10 '12 at 08:45

17

I see two problems:

The open brace should be on the same line as Foreach-object
You shouldn't modify a collection while enumerating through a collection

The example below illustrates how to fix both issues:

function mergehashtables($htold, $htnew)
{
    $keys = $htold.getenumerator() | foreach-object {$_.key}
    $keys | foreach-object {
        $key = $_
        if ($htnew.containskey($key))
        {
            $htold.remove($key)
        }
    }
    $htnew = $htold + $htnew
    return $htnew
}

answered Jan 10 '12 at 08:45

jon Z

15,838
1
33
35

I actually noticed the thing with modifying the collection and had added a third hashtable in another try. But the foreach-object problem remained. It didn't occur to me that the opening bracket had to be on the same line as for foreach-object because I was used to putting it on the next line. – Andrew J. Brehm Jan 10 '12 at 09:17
I found I had to create a third hashtable in the function like this: $htstatic = @{}; $htstatic += $htold; This gives me a new hashtable with the same contents as htold's to enumerate. Alternatively, I guess I can just remove from the other hashtable. – Andrew J. Brehm Jan 10 '12 at 09:28

score 9 · Answer 3 · edited Jul 15 '19 at 19:04

Not a new answer, this is functionally the same as @Josh-Petitt with improvements.

In this answer:

Merge-HashTable uses the correct PowerShell syntax if you want to drop this into a module
Wasn't idempotent. I added cloning of the HashTable input, otherwise your input was clobbered, not an intention
added a proper example of usage

function Merge-HashTable {
    param(
        [hashtable] $default, # Your original set
        [hashtable] $uppend # The set you want to update/append to the original set
    )

    # Clone for idempotence
    $default1 = $default.Clone();

    # We need to remove any key-value pairs in $default1 that we will
    # be replacing with key-value pairs from $uppend
    foreach ($key in $uppend.Keys) {
        if ($default1.ContainsKey($key)) {
            $default1.Remove($key);
        }
    }

    # Union both sets
    return $default1 + $uppend;
}

# Real-life example of dealing with IIS AppPool parameters
$defaults = @{
    enable32BitAppOnWin64 = $false;
    runtime = "v4.0";
    pipeline = 1;
    idleTimeout = "1.00:00:00";
} ;
$options1 = @{ pipeline = 0; };
$options2 = @{ enable32BitAppOnWin64 = $true; pipeline = 0; };

$results1 = Merge-HashTable -default $defaults -uppend $options1;
# Name                           Value
# ----                           -----
# enable32BitAppOnWin64          False
# runtime                        v4.0
# idleTimeout                    1.00:00:00
# pipeline                       0

$results2 = Merge-HashTable -default $defaults -uppend $options2;
# Name                           Value
# ----                           -----
# idleTimeout                    1.00:00:00
# runtime                        v4.0
# enable32BitAppOnWin64          True
# pipeline                       0

Thanks sonjz, definitely an improvement! – Josh Petitt Apr 02 '15 at 14:28 — Josh Petitt, Apr 02 '15 at 14:28

AndrIV · Answer 4 · 2019-03-10T18:14:20.627

In case you want to merge the whole hashtable tree

function Join-HashTableTree {
    param (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [hashtable]
        $SourceHashtable,

        [Parameter(Mandatory = $true, Position = 0)]
        [hashtable]
        $JoinedHashtable
    )

    $output = $SourceHashtable.Clone()

    foreach ($key in $JoinedHashtable.Keys) {
        $oldValue = $output[$key]
        $newValue = $JoinedHashtable[$key]

        $output[$key] =
        if ($oldValue -is [hashtable] -and $newValue -is [hashtable]) { $oldValue | ~+ $newValue }
        elseif ($oldValue -is [array] -and $newValue -is [array]) { $oldValue + $newValue }
        else { $newValue }
    }

    $output;
}

Then, it can be used like this:

Set-Alias -Name '~+' -Value Join-HashTableTree -Option AllScope

@{
    a = 1;
    b = @{
        ba = 2;
        bb = 3
    };
    c = @{
        val = 'value1';
        arr = @(
            'Foo'
        )
    }
} |

~+ @{
    b = @{
        bb = 33;
        bc = 'hello'
    };
    c = @{
        arr = @(
            'Bar'
        )
    };
    d = @(
        42
    )
} |

ConvertTo-Json

It will produce the following output:

{
  "a": 1,
  "d": 42,
  "c": {
    "val": "value1",
    "arr": [
      "Foo",
      "Bar"
    ]
  },
  "b": {
    "bb": 33,
    "ba": 2,
    "bc": "hello"
  }
}

score 5 · Answer 5 · edited Jul 15 '19 at 18:58

5

I just needed to do this and found this works:

$HT += $HT2

The contents of $HT2 get added to the contents of $HT.

edited Jul 15 '19 at 18:58

Peter Mortensen

30,738
21
105
131

answered Sep 08 '13 at 15:09

Scriptimus Prime

71
1
2

8

That works except when $HT2 has a key that already exists in $HT. Then the merge throws an exception. – Tony L. Nov 01 '13 at 15:57

Andy Arismendi · Answer 6 · 2012-01-10T08:46:36.350

4

The open brace has to be on the same line as ForEach-Object or you have to use the line continuation character (backtick).

This is the case because the code within { ... } is really the value for the -Process parameter of ForEach-Object cmdlet.

-Process <ScriptBlock[]> 
Specifies the script block that is applied to each incoming object.

This will get you past the current issue at hand.

edited Jan 10 '12 at 08:46

answered Jan 10 '12 at 08:41

Andy Arismendi

50,577
16
107
124

+1 for being direct and providing help with the current problem rather than simply providing a solution. To your a answer, does `ScriptBlock[]` mean that you can provide multiple script blocks? I need to look into that. – Shibumi Nov 01 '13 at 16:18

score 3 · Answer 7 · edited Jan 02 '20 at 07:18

3

I think the most compact code to merge (without overwriting existing keys) would be this:

function Merge-Hashtables($htold, $htnew)
{
   $htnew.keys | where {$_ -notin $htold.keys} | foreach {$htold[$_] = $htnew[$_]}
}

I borrowed it from Union and Intersection of Hashtables in PowerShell

edited Jan 02 '20 at 07:18

Dave Neeley

3,526
1
24
42

answered Nov 16 '16 at 11:41

Mehrdad Mirreza

984
12
20

score 3 · Answer 8 · edited Dec 17 '21 at 21:07

I wanted to point out that one should not reference base properties of the hashtable indiscriminately in generic functions, as they may have been overridden (or overloaded) by items of the hashtable.

For instance, the hashtable $hash=@{'keys'='lots of them'} will have the base hashtable property, Keys overridden by the item keys, and thus doing a foreach ($key in $hash.Keys) will instead enumerate the hashed item keys's value, instead of the base property Keys.

Instead the method GetEnumerator or the keys property of the PSBase property, which cannot be overridden, should be used in functions that may have no idea if the base properties have been overridden.

Thus, Jon Z's answer is the best.

score 2 · Answer 9 · edited Jul 15 '19 at 19:10

2

To 'inherit' key-values from parent hashtable ($htOld) to child hashtables($htNew), without modifying values of already existing keys in the child hashtables,

function MergeHashtable($htOld, $htNew)
{
    $htOld.Keys | %{
        if (!$htNew.ContainsKey($_)) {
            $htNew[$_] = $htOld[$_];
        }
    }
    return $htNew;
}

Please note that this will modify the $htNew object.

edited Jul 15 '19 at 19:10

Peter Mortensen

30,738
21
105
131

answered Apr 09 '18 at 07:17

Sen Jacob

3,384
3
35
61

score 1 · Answer 10 · edited Jul 15 '19 at 18:59

1

I just wanted to expand or simplify on jon Z's answer. There just seems to be too many lines and missed opportunities to use Where-Object. Here is my simplified version:

Function merge_hashtables($htold, $htnew) {
    $htold.Keys | ? { $htnew.ContainsKey($_) } | % {
        $htold.Remove($_)
    }
    $htold += $htnew
    return $htold
}

edited Jul 15 '19 at 18:59

Peter Mortensen

30,738
21
105
131

answered Nov 01 '13 at 16:10

Tony L.

7,988
5
24
28

Josh Petitt · Answer 11 · 2014-03-13T21:47:29.627

1

Here is a function version that doesn't use the pipeline (not that the pipeline is bad, just another way to do it). It also returns a merged hashtable and leaves the original unchanged.

function MergeHashtable($a, $b)
{
    foreach ($k in $b.keys)
    {
        if ($a.containskey($k))
        {
            $a.remove($k)
        }
    }

    return $a + $b
}

edited Mar 13 '14 at 21:47

answered Mar 13 '14 at 21:32

Josh Petitt

9,371
12
56
104

See improvements by sonjz – Josh Petitt Apr 02 '15 at 14:29

Merging hashtables in PowerShell: how?

11 Answers11

Merge-Hashtables

Merge-Hashtables version 2

In case you want to merge the whole hashtable tree

Linked