How to get the Azure Data Factory parameters into the ARM template parameters file (ARMTemplateParametersForFactory.json) after publishing

Question

I am trying to create my Azure DevOps release pipeline for Azure Data Factory.

I have followed the rather cryptic guide from Microsoft (https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment ) regarding adding additional parameters to the ARM template that gets generated when you do a publish (https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template )

Created a arm-template-parameters-definition.json file in the route of the master branch. When I do a publish, the ARMTemplateParametersForFactory.json in the adf_publish branch remains completely unchanged. I have tried many configurations.

I have defined some Pipeline Parameters in Data Factory and want them to be configurable in my deployment pipeline. Seems like an obvious requirement to me.

Have I missed something fundamental? Help please!

The JSON is as follows:

{
    "Microsoft.DataFactory/factories/pipelines": {
        "*": {
            "properties": {
                "parameters": {
                        "*": "="                
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/integrationRuntimes": {
        "*": "="
    },
    "Microsoft.DataFactory/factories/triggers": {},
    "Microsoft.DataFactory/factories/linkedServices": {},
    "Microsoft.DataFactory/factories/datasets": {}
}

why do you think it must change? it shouldnt. it should use these parameters to deploy whatever you need. its not going to update contents of that file in the repo (why would it?) — 4c74356b41, Dec 07 '18 at 05:50
@4c74356b41 Yes it should when you publish and it does, but not consistently. I suggest you read the link posted in the question. — NER1808, Dec 10 '18 at 13:28

score 18 · Accepted Answer · edited Aug 20 '19 at 13:42

I've been struggling with this for a few days and did not found a lot of info, so here what I've found out. You have to put the arm-template-parameters-definition.json in the configured root folder of your collaboration branch:

So in my example, it has to look like this:

If you work in a separate branch, you can test your configuration by downloading the arm templates from the data factory. When you make a change in the parameters-definition you have to reload your browser screen (f5) to refresh the configuration.

If you really want to parameterize all of the parameters in all of the pipelines, the following should work:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters":{
            "*":{
                "defaultValue":"="
            }
        }
    }
}

I prefer specifying the parameters that I want to parameterize:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters":{
            "LogicApp_RemoveFileFromADLSURL":{
                "defaultValue":"=:-LogicApp_RemoveFileFromADLSURL:"
            },
            "LogicApp_RemoveBlob":{
                "defaultValue":"=:-LogicApp_RemoveBlob:"
            }
        }
    }
}

Added a small clarification below if you don't use standard layout folders. HTH — MarkD, Nov 16 '19 at 00:13
Just a note in case it helps anyone else...I'm not sure if the names should have a colon at the end - i.e. I think LogicApp_RemoveFileFromADLSURL: should be LogicApp_RemoveFileFromADLSURL Having the colon crashed my automated deployment scripts. — mjharper, Jul 07 '21 at 12:49

score 6 · Answer 2 · answered Nov 16 '19 at 00:15

Just to clarify on top of Simon's great answer. If you have non standard git hierarchy (i.e. you move the root to a sub-folder like I have done below with "Source"), it can be confusing when the doc refers to the "repo root". Hopefully this diagram helps.

score 4 · Answer 3 · edited Mar 20 '19 at 15:43

You've got the right idea, but the arm-template-parameters-definition.json file needs to follow the hierarchy of the element you want to parameterize.

Here is my pipeline activity I want to parameterize. The "url" should change based on the environment it's deployed in

{
    "name": "[concat(parameters('factoryName'), '/ExecuteSPForNetPriceExpiringContractsReport')]",
    "type": "Microsoft.DataFactory/factories/pipelines",
    "apiVersion": "2018-06-01",
    "properties": {
        "description": "",
        "activities": [
            {
                "name": "NetPriceExpiringContractsReport",
                "description": "Passing values to the Logic App to generate the CSV file.",
                "type": "WebActivity",
                "typeProperties": {
                    "url": "[parameters('ExecuteSPForNetPriceExpiringContractsReport_properties_1_typeProperties')]",
                    "method": "POST",
                    "headers": {
                        "Content-Type": "application/json"
                    },
                    "body": {
                        "resultSet": "@activity('NetPriceExpiringContractsReportLookup').output"
                    }
                }
            }
        ]
    }
}

Here is the arm-template-parameters-definition.json file that turns that URL into a parameter.

{
   "Microsoft.DataFactory/factories/pipelines": {
        "properties": {
            "activities": [{
                "typeProperties": {
                    "url": "-::string"
                }
            }]
        }
    },
    "Microsoft.DataFactory/factories/integrationRuntimes": {},
    "Microsoft.DataFactory/factories/triggers": {},
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": "="
    },
    "Microsoft.DataFactory/factories/datasets": {
        "*": "="
    }
}

So basically in the pipelines of the ARM template, it looks for properties -> activities -> typeProperties -> url in the JSON and parameterizes it.

I'm trying to get back to an adf_publish template with no parameterisation but it always parameterises all of my linkedServices connection string no matter what I do. Do you know why that is? — Nick.Mc, May 13 '19 at 13:30
OK in the doco it says "By default, all secure strings, such as Key Vault secrets, and secure strings, such as connection strings, keys, and tokens, are parameterized.".. so that's why I can't unparameterize them. — Nick.Mc, May 20 '19 at 08:04

score 0 · Answer 4 · answered Jan 25 '19 at 00:30

0

Here are the necessary steps to clear up confusion:

Add the arm-template-parameters-definition.json to your master branch.
Close and re-open your Dev ADF portal
Do a new Publish

Your ARMTemplateParametersForFactory.json will then be updated.

answered Jan 25 '19 at 00:30

4gatch

1
1

score 0 · Answer 5 · answered Feb 21 '19 at 08:55

I have experienced similar problems with the ARMTemplateParametersForFactory.json file not being updated whenever I publish and have changed the arm-template-parameters-definition.json.

I figured that I can force update the Publish branch by doing the following:

Update the custom parameter definition file as you wish.
Delete ARMTemplateParametersForFactory.json from the Publish branch.
Refresh (F5) the Data Factory portal.
Publish.

The easiest way to validate your custom parameter .json syntax seems to be by exporting the ARM template, just as Simon mentioned.

score 0 · Answer 6 · answered Apr 27 '23 at 18:32

0

I had this issue because of CORS extension in Chrome, which did not allow PATCH method used by ADF. Changing settings for the extension made publish process work fine.

answered Apr 27 '23 at 18:32

Антон Колесник

3
1

How to get the Azure Data Factory parameters into the ARM template parameters file (ARMTemplateParametersForFactory.json) after publishing

6 Answers6

Linked