Triggering a Data Factory Pipeline through javascript

Question

I have made this pipeline in Azure Data Factory that copies data from a Azure Storage Table to a Azure SQL database Table.

The Azure storage table is given data from a Javascript chatbot that records answers and stores them in the table. I would like to trigger the CopyTabletoSQL through my javascript app once all of the answers have been recorded.

This is my CopyTableToSQL pipeline object.

{
"name": "CopyTabletoSQL",
"type": "Copy",
"policy": {
    "timeout": "7.00:00:00",
    "retry": 0,
    "retryIntervalInSeconds": 30,
    "secureOutput": false
},
"typeProperties": {
    "source": {
        "type": "AzureTableSource"
    },
    "sink": {
        "type": "SqlSink",
        "writeBatchSize": 10000
    },
    "enableStaging": false,
    "dataIntegrationUnits": 0
},
"inputs": [
    {
        "referenceName": "tableInputDataset",
        "type": "DatasetReference"
    }
],
"outputs": [
    {
        "referenceName": "OutputSqlTable1",
        "type": "DatasetReference"
    }
]
}

Is there any way to have this execute from a javascript app? The docoumentation (https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers) only mentions .net, Powershell, REST API and Python SDK but nothing for node.js

I have managed to use https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/createrun to create a run and try it and it was succesfull. POST -URL- Authorization: Bearer -random symbols- Content-type: application/json How would I go about implementing this into my javascript app so that I can call on it at the end of a run of my app? Many thanks! — BergBerg, Jul 13 '18 at 08:48

score 1 · Answer 1 · answered Jul 13 '18 at 08:34

1

Azure Data Factory nodejs sdk is not supported so far. Based on your description, you already have created pipeline in your adf account.You could use Azure Data Factory Pipelines - Create Run REST api to execute it.

In this process, you need to generate Authorization token in Headers. You could refer to ADAL nodejs sdk to generate token.

Before that, you need to grant adf permission to your ad app.

Hope it helps you.

answered Jul 13 '18 at 08:34

Jay Gong

23,163
2
27
32

I have managed to use https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/createrun to create a run and try it and it was succesfull. POST -URL- Authorization: Bearer -random symbols- Content-type: application/json How would I go about implementing this into my javascript app so that I can call on it at the end of a run of my app? Many thanks! – BergBerg Jul 13 '18 at 08:47
@BergBerg You could send post request with Authorization token in your js app. About generating token, please refer to the sdk mentioned in my above answer. – Jay Gong Jul 13 '18 at 08:50
Thank you for all your help/trouble! I found this: https://pastebin.com/urtabdvL from here https://stackoverflow.com/questions/14873443/sending-an-http-post-using-javascript-triggered-event Everything seems to be fine however, I am not sure what to write in `var postData = "Some data";` I'm guessing it is somehow related to the JSON body? also would I need to add `Authorization: Bearer -random symbols-` somehow? I will look at the SDK you mentioned regarding how to generate a token! Many Thanks! – BergBerg Jul 13 '18 at 08:59
For the post data, you could just reference what UI sent out. I think an empty object {} should work if you don’t need pass parameter value. – Fang Liu Jul 13 '18 at 12:01
For some added context: This is how my app looks like right now, https://pastebin.com/fbp3ddzD . I've built it following guides and how to's from the microsoft documents and some googleing/ stackoverflow. It is a simple chatbot that asks 3 questions, records the answers and then I have a pipeline that copies the answers from my table to a SQL database. Everything works fine, the bot asks and records the answers, and the pipeline copies them to my SQL database, I'm looking for a way to trigger the copy from table to database automaticly once a conversation have been finished. – BergBerg Jul 13 '18 at 14:17

score 0 · Answer 2 · answered Jul 12 '18 at 14:05

0

You could call rest api in JavaScript.

answered Jul 12 '18 at 14:05

Fang Liu

2,325
2
13
18

Triggering a Data Factory Pipeline through javascript

2 Answers2