6

I want to execute a query in azure data lake daily. Can we schedule a U-SQL query in azure data lake?

Jai
  • 416
  • 6
  • 20

2 Answers2

2

Currently, there is no built-in way inside Data Lake Analytics to schedule a U-SQL job. Instead, you can use other services or tools to perform the scheduling. A popular one for Azure customers is Azure Data Factory.

Simple scheduling of U-SQL jobs inside Data Lake Analytics is something we are considering adding as a native capability.

saveenr
  • 8,439
  • 3
  • 19
  • 20
  • Thanks for your response. I want to apply few transformation on daily basis on data which is present in azure data lake. I want to generate a new file out of it. But if I use ADF job I won't be able to apply the transformations. Am I right? – Jai Dec 16 '16 at 05:23
  • Jai, you can very easily run a Data Lake Analytics job in a Data Factory pipeline. https://azure.microsoft.com/en-us/blog/creating-big-data-pipelines-using-azure-data-lake-and-azure-data-factory/ – Alexandre Gattiker Dec 16 '16 at 09:21
  • I know that I can schedule a Data Lake Analytics job which will take the data from Azure Data lake and copy it somewhere. I want to schedule a job which takes data from azure data lake and apply some transformation on top of it and create a new file in the same azure data lake which contains the transformed data. But as per my understanding of ADF, I don't think that I can apply any transformation on data while pulling it from or pushing it to Azure data lake. Because the only way, we can apply the transformation on data in Azure Data Lake is by using U-SQL which I can't use in ADF pipeline. – Jai Dec 16 '16 at 11:51
  • 2
    You can use U-SQL in an ADF pipeline. See: https://learn.microsoft.com/en-us/azure/data-factory/data-factory-usql-activity – saveenr Dec 16 '16 at 14:06
2

There's two ways to execute a query in azure data lake daily:

  1. Using ADF and Store the U-SQL script in Blob Storage and reference it via a Blob Storage linked service.
  2. Create a SSIS Package using visual studio then import this package in SqlServer Agent serves Job . see Schedule U-SQL jobs
Mümin
  • 309
  • 1
  • 20