0

I have written console application to monitor/analyze the files in Azure data lake store. I have created an application in Azure active directory to access the azure resources. I have followed all the steps given here to give access to application on azure data lake store. I have provided access to the parent and all the childern folders/files of data lake store.

Now, I am able to access the files through my code. I am trying to get the modification time and expiration time of a file produced by USQL job in data lake store by using DataLakeStoreFileSystemManagement Client in the code. I am using Microsoft provided .NET API for data lake analytics & data lake store.

I am getting all that information for the files for which i have provided access to. But when the usql job adds a new folder/file in the azure data lake store then i am not getting the modification time and expiration in my code. Instead i am getting exception of Forbidden Error 403.

the Usql jobs creates plenty of folders everyday and i just can't go there and provide access manually to all newly created files and folder. It should inherit the access role for newly created folders/files.

what should I do ? or Is that a bug in Azure data lake store? Please Help.

Jai
  • 416
  • 6
  • 20

2 Answers2

0

You are not allowed to call REST end points from within user code in U-SQL (reasons are explained here). The DataLakeStoreFileSystem Management client is attempting to (recursively) call into ADL through REST end points and is being blocked by the container boundary protection. So the 403 (Forbidden) is by design.

We are working on adding file properties to our U-SQL APIs as meta properties in one of the upcoming refreshes. Would that help?

Michael Rys
  • 6,684
  • 15
  • 23
  • Do you mean to say that i have to provide access manually to newly added file/folders till the API update is available? – Jai Jan 30 '18 at 10:35
  • just to clarify, i am not trying to access the expiration and modification time in Usql. I have a console application which gathers all the data of a files which are stored in ADL. These files are generated by various usqljob on daily basis and to access those files i have to provide access to Application which we create in AAD. i have given access to that application to all the parent and child files/folders and i am successfully getting all that information in my console app by using DataLakeStoreFileSystemmgmt client . But for new files then i am getting 403 error for modification time. – Jai Jan 30 '18 at 11:40
  • Thanks for the clarification. That is a different scenario from what I understood. In that case let me get someone from the store API team to investigate. Assuming you have the right permissions, you should be able to get the information. – Michael Rys Jan 30 '18 at 21:31
  • Thanks, That would be helpfull. – Jai Jan 31 '18 at 05:22
0

We need more information to debug the issue you are facing. Please file a support ticket (from the Azure Portal) and then email me the ticket number (cpalmer@microsoft.com). In the support ticket, identify the ADLS account name, timestamp of the access, name of the file/folder you were accessing that got 403, (approximate) timestamp when you believe you created that file/folder.

CP_MSFT
  • 51
  • 1