1

I am trying to create an Event Trigger in Azure Data Factory on Blob Created in my Azure Storage Container. The challenge I am facing in doing so is what if I receive multiple files in one single event say 10 files.

Now what happens is the event is fired 10 times and all these 10 files are executed at-least 100 times by the Data Factory.

Edit -- My pipeline does xml to json transformation.

enter image description here

The error I get

enter image description here

enter image description here

The issue I am facing is similar to Azure Data Factory - Event based triggers on multiple files/blobs

amit agarwal
  • 63
  • 2
  • 17
  • Hey @amitagarwal, can you please clarify your ask here? And also is there any fixed number for the files which you would receive? or timings – Nandan Feb 22 '21 at 11:50
  • Can you not just capture the metadata of the files within the container and then execute whatever process is required on each file via a foreach loop? – iamdave Feb 22 '21 at 11:50
  • In theory, creating a blob will execute a pipeline run. Can you show us your pipeline logic? – Joseph Xu Feb 23 '21 at 05:25
  • @Josephxu -- please check my edited question. – amit agarwal Feb 23 '21 at 07:42
  • Hi @amit agarwal, so when a xml file was uploaded to the container, it will fires a pipeline run to convert xml file into json file? – Joseph Xu Feb 23 '21 at 07:49
  • Yes... When I trigger it manually obviously it runs only once. But when I set it to event based trigger it triggers multilple times since multiple blob files are created. – amit agarwal Feb 23 '21 at 07:54
  • per my experience, it most like a bug. Sorry we can't give more useful helps for you. I'd like to suggest you to ask Azure support(it takes money and time but can get the root cause). – Joseph Xu Feb 25 '21 at 06:02

1 Answers1

0

I've created a simple test. When I uploaded 10 xml files, it fires

  1. Set Wildcard file path as *.xml. enter image description here

  2. Then I created a event trigger. enter image description here

  3. When I uploaded 10 xml files at one time via Azure storage Explorer. I can see 10 pipeline runs. enter image description here And 10 blob in the output container. enter image description here

So what are the differences between yours and mine?

Joseph Xu
  • 5,607
  • 2
  • 5
  • 15
  • When you upload 10 files those are one by one uploaded. But what if those 10 files are uploaded in one shot. I Suggest instead of uploading try pasting 100 files in the container in one shot and see the difference. As 10 would be too short to see the difference. – amit agarwal Feb 23 '21 at 09:07
  • Do you mean like this [screenshot](https://i.stack.imgur.com/NkPiq.png). The result is also 10 times。 – Joseph Xu Feb 23 '21 at 09:07
  • No... I use Azure Storage Explorer. Just copy 100 files from one container and paste it to other container where the event trigger is pointed. – amit agarwal Feb 23 '21 at 09:13
  • So said, I also did as you said, it works fine. :( – Joseph Xu Feb 23 '21 at 09:29
  • 1
    Please refer the errors that I am getting in the screenshot above. – amit agarwal Feb 23 '21 at 10:46