-1

i am new to python and i want to move data files from different on premises data sources to azure data lake storage and i want to validate these data files for different rules of validation before the get moved using azure data factory pipeline and azure databrick

I am aware how to create custome python copy activity in databricks for running python script

i need help with writing the script for validation

Mohit Chauhan
  • 1
  • 1
    Welcome to StackOverflow. [On topic](https://stackoverflow.com/help/on-topic), [how to ask](https://stackoverflow.com/help/how-to-ask), and ... [the perfect question](https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/) apply here. StackOverflow is a knowledge base for *specific* programming problems -- not a coding or tutorial resource. This posting is too broad for Stack Overflow. For generic help, one generally uses tutorial materials and examples from the documentation for that module or package. – Prune Feb 05 '20 at 16:42

1 Answers1

0

You might want to have a look at Data Factory to move files between different storages. It's pretty straightforward and scalable and you can even create Data Flow pipelines based on a Spark Cluster without having to write a single line of code.

Axel R.
  • 1,141
  • 7
  • 22