Azure's New-AzureRmHDInsightCluster command does not accept documented ComponentVersion option

Question

According to MS's docs for New-AzureRmHDInsightCluster, it should accept -ComponentVersion as an option:

$httpCredential = New-Object System.Management.Automation.PSCredential ($httpUserName, $clusterpassword)

$sparkConfig = New-Object "System.Collections.Generic.Dictionary``2[System.String,System.String]"
$sparkConfig.Add("spark", "2.1")

New-AzureRmHDInsightCluster `
    -ClusterName mycluster `
    -ComponentVersion $sparkConfig `
    -ClusterSizeInNodes 4 `
    -HttpCredential $httpCredential `
    -Location "Central US" `
    -OSType Linux `
    -ResourceGroupName tstcluster

However, this command results in:

##[error]A parameter cannot be found that matches parameter name 'ComponentVersion'.

Is there a way to select the Spark version required? We used to use:

Add-AzureRmHDInsightComponentVersion -Config $config -ComponentName "Spark" -ComponentVersion "2.1"

But that's now rejected:

##[error]The term 'Add-AzureRmHDInsightComponentVersion' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

EDIT:

A couple of factors that were key to solving this: First, this script is run by an Azure PowerShell Script task in an Azure Dev Ops pipeline. Second, the PowerShell version used to run this script was 1.*.

Joy Wang · Answer 1 · 2018-09-20T04:41:51.500

0

Try the command below, it works fine on my side. Make sure you have installed the AzureRM.HDInsight powershell module, you could check it with Get-Module AzureRM.HDInsight.

Sample command:

$httpUserName = "joyhd"
$clusterpassword = ConvertTo-SecureString "<password>" -AsPlainText -Force 
$httpCredential = New-Object System.Management.Automation.PSCredential($httpUserName, $clusterpassword)    
$SshCredential = New-Object System.Management.Automation.PSCredential($httpUserName, $clusterpassword)
$storageAccountName = "<storageAccountName>"
$storageAccountKey = "xxxxxxx"
$storageContainer = "testhd"

New-AzureRmHDInsightClusterConfig `
            | Add-AzureRmHDInsightComponentVersion `
                -ComponentName "Spark" `
                -ComponentVersion "2.1" `
            | New-AzureRmHDInsightCluster `
    -ClusterName joytesthd `
    -ClusterType "Spark" `
    -ClusterSizeInNodes 4 `
    -HttpCredential $httpCredential `
    -Location "eastus" `
    -OSType Linux `
    -ResourceGroupName joywebapp `
    -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
    -DefaultStorageAccountKey $storageAccountKey `
    -DefaultStorageContainer $storageContainer `
    -SshCredential $SshCredential

Result:

edited Sep 20 '18 at 04:41

answered Sep 20 '18 at 04:36

Joy Wang

39,905
3
30
54

I've tried that. In our case, this is not something that runs locally, but runs under Azure DevOps. The script is run on a vm created and configured by Microsoft. – Don Branson Sep 20 '18 at 06:55
@DonBranson May be you could include the environment in your question. – Joy Wang Sep 20 '18 at 07:18
Yes, it seems I need to. Perhaps those vms are controllable in some fashion? Or does MS just have a stock template for making them? I'll have to dig into that to find out. It seems like AzureRM.HDInsight must be installed there, just a version that doesn't recognize the ComponentVersion option for commands that are documented as supporting it. – Don Branson Sep 20 '18 at 07:25
@DonBranson Here is a [post](https://stackoverflow.com/questions/36954237/using-powershellget-on-vsts-hosted-agents), may be helpful. – Joy Wang Sep 20 '18 at 07:51

score 0 · Answer 2 · answered Sep 25 '18 at 19:34

MS was not able to explain why a script that worked in Azure Dev Ops (then VSTS) stopped working. That is, why Add-AzureRmHDInsightComponentVersion was supported in January 2018 but not in September of that same year.

Their solution was to select the latest version of PowerShell available in a pipeline (3.*) and set the Preferred Azure PowerShell Version to 3.8.0.

Making those changes made the existing script workable again.

Azure's New-AzureRmHDInsightCluster command does not accept documented ComponentVersion option

2 Answers2