1

Scenario

We have azure blob storage container with following folder structure. • 20190601-20190630

Basically, this folder will contain daily CSV files for the given month.

This folder structure is dynamic. So, in the next month, folder 20190701-20190731 will be populated with daily CSV files.

Problem

On daily basis, need to move these files from azure blob storage to azure data lake using azure data factory (v2).

How to specify folder structure (dynamically) in the Input Dataset (Azure Blob Storage) in Azure Data Factory(V2)?

Example: 20190601-20190630/*.CSV for the month June 2019

Basically, StartDateOfMonth and EndDateOfMonth are dynamic.

Thanks in Advance

Mangesh T.
  • 23
  • 4

1 Answers1

0

You could configure your dataset folder path like:

   "folderPath": {
                        "value": "@concat( 
                               formatDateTime(pipeline().parameters.scheduledRunTimeStart, 'yyyyMMdd'), 
                               '-',
                               formatDateTime(pipeline().parameters.scheduledRunTimeEnd, 'yyyyMMdd')
                               , '/'
                        "type": "Expression"
    }

And pass the parameters into dataset:

"parameters": {
    "scheduledRunTimeStart": {
        "type": "String"
    },
    "scheduledRunTimeEnd": {
        "type": "String"
    }
}
Jay Gong
  • 23,163
  • 2
  • 27
  • 32
  • Passing the date time parameter is correct but we need to pass the **StartDateOfMonth** and **EndDateOfMonth**. The parent folder structure will be as follows For June month - 20190601-2019063 For July month - 20190701-20190731 For August month - 20190801-20190830 – Mangesh T. Jul 01 '19 at 05:04
  • @MangeshT. You could format the date string first follow your rules ,then pass them into pipeline as parameters. Fox example,use java Calendar jar sdk:https://stackoverflow.com/questions/14241836/get-first-date-of-current-month-in-java – Jay Gong Jul 02 '19 at 02:00