How to copy files to different directories based on the file name in ADFv2

Question

I'm trying to copy files to some folder location in ADLS. Here i'm trying to copy based on the file name. Here my source and destination is ADLS only.

For Ex.

    If filename is ABC.csv then copy to ABC folder

    If filename is DEF.csv then copy to DEF folder

    If filename is XYZ.csv then copy to XYZ folder and so on.

Please help me achieving this using azure data factory v2.

I know this is easy task in SSIS. But in ADFv2 i tried using Foreach loop to iterate thru all the files (i have only 3 files) and inside foreach loop there is IF condition to check the file name. Not sure how to change the destination folder location in the run time.

Hauke Mallow · Answer 1 · 2019-06-18T05:41:34.903

2

You could achieve this using dynamic content in the filepath of the sink. So instead using an if-then-else-logic to define the target folder, build it dynamically depending on the filename.

In the following example the target folder is constructed dynamically with the trigger starttime:

 "folder": {
        "name": "Target"
    },
    "type": "AzureDataLakeStoreFile",
    "typeProperties": {
        "fileName": "",
        "folderPath": {
            "value": "@CONCAT('/tenant02/data/raw/corporate/hmaserver/Ingest_Date=', formatDateTime(trigger().startTime, 'yyyy-MM-dd/'))",
            "type": "Expression"
        }

In the Data Factory V2 UI edit the sink data set to do this:

Add the dynamic content in the connection:

You could do this similar, e. g. replacing the file extension:

@replace(item().name, '.csv', '')

Another option could be to using a Databricks activity for this. See my following Answer for a complex copy logic in a Python Notebook.

edited Jun 18 '19 at 05:41

answered Jun 18 '19 at 05:05

Hauke Mallow

2,887
3
11
29

I believe, the above code you mentioned works only to copy files to somedetination folder. But here i need to check the filename before copying like below Step 1: Check filename, if it is ABC.csv then Step 2: Copy to ABC.csv file to ABC folder Step 3: Check for the next filename and so on till all files are copied. I have only 5 files to copy like this. so i tried taking array variable having list of files and foreach loop to iterate thru files. could you please help. Appreciate if you can provide your thoughts in UI with 1 or 2 screenshots. – user3588007 Jun 18 '19 at 05:31
The idea is to build the target directory name depending on the filename, instead using an if then else logic. Will add the screenshots... – Hauke Mallow Jun 18 '19 at 05:31

score 0 · Answer 2 · answered Jun 20 '19 at 17:48

You can using If Condition activity in Azure Data Factory like bellow:

Define an If Condition to check if file name is ABC.

If true, define an activity to copy it to folder ABC.
If false, insert inside ifFalseActivities another If Condition to check if file name is DEF then write activity for copying file to destination folder based on result of this activity.

If you can define the destination folder based on file name before you pass the data to ADF (such as you define the file name, destination folder returned from stored procedure) then inside Lookup activity, you can using dynamic content for File path and file name.

How to copy files to different directories based on the file name in ADFv2

2 Answers2