Questions tagged [azure-mapping-data-flow]
38 questions
2
votes
0 answers
Long Azure Data Factory mapping data flow "file system init duration"
I have an ADF mapping data flow that uses an ADLS gen2 source with a large number of small, say 10kB, files. 98% of the flows time is spent in "file system init duration" in this source. I can't seem to find any documentation on what may affect…
ebclark
- 45
- 4
1
vote
2 answers
ADF Column Name Validation and Data Validation
I am trying to add some validation to my ADF pipeline. Is there a way to achieve the following validation in ADF?
Validate column header and return error message. There is a list of required column names that I need to check against the raw Excel…
Candice
- 31
- 5
1
vote
1 answer
Check if a Column Exists and Add it When Doesn't Exist
In the Azure Data Factory data mapping flow, is there a way to check if a column Date exists in the input file? If true, select the Date column, if not then create a Date column but leave the column blank in the output?
I tried with conditional…
Candice
- 31
- 5
1
vote
1 answer
Azure data factory - custom mapping for Rest service
So, I am creating a Copy activity that reads from SQL Server table and have to send the data to an API end point with the PATCH request.
API provider specified that the body must be in the form…
Dmitriy Ryabin
- 363
- 3
- 16
1
vote
1 answer
Azure data factory - mapping data flows regex implementation to format a number
I am creating a mapping data flow where I have a phone number column which can contain values like
(555) 555-1234 or
(555)555-1234 or
555555-1234
I want to extract numbers from this value. How can that be done. I have tried the below function with…
user961
- 453
- 6
- 20
1
vote
3 answers
how can i parse a nested json file in Azure Data Factory?
I have multiple json files in datalake which look like below:
The complex type also have arrays embedded in it.
How can i flatten this json to csv file by either using copy activity or mapping data flows ?
Thanks
azuresnowflake1
- 135
- 1
- 10
1
vote
1 answer
Mapping Data Flows Dynamic Column Updates
I have a text input source. This has over 100 columns so I won't show all of them here - a cut-down view of the data would be:
CustomerNo
DOB
DOD
Status
01418495
01/02/1940
NULL
1
01418496
01/01/1930
NULL
1
The users want to be able to…
SmallFry
- 143
- 2
- 9
0
votes
0 answers
Azure Dataflow Overwrite XML Schema that Uses Namespaces Causes Issues in Dervied Column
In most cases where we're overwriting an XML schema in Azure Dataflows we're having no problems. In a recent file we recieved, we started having trouble where if we make any changes to the schema, e.g. simply changing a node to an array, downstream…
Jacques
- 6,936
- 8
- 43
- 102
0
votes
1 answer
Add cross apply in an Azure Data Factory Data Flow
How do I simulate the cross apply in an Azure Data Factory Data flow?
SELECT
wo.OrderId,
max(wo.OrderVersionId) as OrderVersionId,
max(tss.FromDate) as TSSFromDate,
tss.ToDate as TSSToDate
FROM dayshift.OrderMain wo CROSS APPLY (
…
datainstints
- 5
- 2
0
votes
1 answer
azure adf mapping data flow expression with case statement on cached sink
I am using Azure Data Factory Mapping Data Flows.
I am trying to write an equivalent of sql 'case' statement to get a value in a derived column based on cashed sink.
My cashed sink input is a query: 'select ID from dbo.aaa'. It returns all ID as…
Andrii Rozrutskyi
- 25
- 3
0
votes
1 answer
use array as parameter in mapping Data Flow in Azure Data Factory
I am working with Change Data Capture (CDC) feature in mapping Data Flows in ADF. The feature picks up all new, deleted, or amended rows in the source table as expected.
Nevertheless, it provides no indication what exactly happened to returned row:…
Andrii Rozrutskyi
- 25
- 3
0
votes
1 answer
Converting datediff to Azure Mapping Data flows expression language
I need to convert this sql snippet to Azure Data Flow expression language:
"case when Trans_StatusId <> 1 then 0 else datediff(day, AppliedDate, getdate()) end"
I tried "iif(Trans_StatusId != 1, 0, subDays(currentTimestamp(),…
datainstints
- 5
- 2
0
votes
2 answers
Add Isnull join condition to Azure data factory data flow
I have this sql script : I need to add the "ON UpdatedWOV.Id IS NULL
AND CTH.Id = PT.TranId" to a dataflow. My challenge is specifically how to add "ON UpdatedWOV IS NULL"
please note that there are other join conditions that doesn't require the…
datainstints
- 5
- 2
0
votes
0 answers
Application Map# Create custom Map in azure portal?
is it possible create custom Application map path?
I have 100 + microservice and functions jobs and service bus and lots other components which connect to each other based on business flow.
I want to create my own business flow with specific…
Maulik Dave
- 81
- 9
0
votes
2 answers
Azure Data Flows - Accessing deeply nested values in an XML file in
I'm working on a data flow to ingest XML files and trying to pull a deeply nested node's value up into another part of the flow. E.g.
1234
1/2/2023T15:33:22
....
…
Jacques
- 6,936
- 8
- 43
- 102