We have a bunch of Dataflows (Gen1) set up in a Microsoft Fabric (Preview) premium workspace (these are mostly api connections to source operational data, with some Power Query transformations).
I want to query the Dataflow directly within a Python or R script in a Notebook, but it doesn't seem possible.
It seems like I can only connect to Lakehouses within my Notebooks, meaning I'll have to load the data from the Dataflow into a Lakehouse first, adding a layer of complexity. Alternatively, I can fetch the data directly from the API's within the Notebook, but then I'll have to do a whole bunch of additional transformations that have already been done in the Dataflows using Power Query.
Feel like I'm missing something obvious, as we can connect to the Dataflows with PowerBI, just not within a Notebook.
What I tried:
In Microsoft Fabric (Preview), using the Data Science persona:
- New -> Notebook (Preview)
The Lakehouse Explorer appears as default on the left, with the Notebook on the right. In the Lakehouse Explorer, I can add new or existing Lakehouses, but no options to reference Dataflows.
I would hope to see some options of connecting directly to a Dataflow, so I can add the tables created in the Dataflow as a Dataframe (in either R or Python).
Note that in Gen2 dataflows, there is an option to load your data directly in the lakehouse, which would mitigate this issue to some extent, but we've found Gen2 to be a bit unstable so we're not keen on using that at the moment.
