0

In Azure mapping dataflow we now have option to save files in delta format. But that is only available when we select inline dataset (without data bricks subscription). And when the sink dataset is inline dataset, it does not allow to set partition based on any column.

I can write pyspark code to rewrite the delta table with required partition. But that would incur additional cost.

What could be work arounds for getting good performance on delta data?

Shadman Sadekeen
  • 1
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Oct 18 '21 at 10:09

1 Answers1

0

There was a UI issue that was recently fixed by the engineering team. Until this reflects at your end.

You could do the following as a workaround :

Option 1 :

You can change the type of sink to something else, like a delimited text sink, and you should then see the key columns in Key partitioning. Then, switch the Sink type back to Delta.

Reference : https://learn.microsoft.com/en-us/answers/questions/599075/index.html

Option 2: You could enable the partitioning at the source end.

enter image description here

The partitioned data was flowing as a stream. I was able to achieve the partitioned data as a result

Satya V
  • 3,811
  • 1
  • 6
  • 9