Stack Overflow
Stack Overflow

Questions tagged [databricks-repos]

For questions around Databricks Repos for Git integration.

This tag is for questions around Databricks Repos - development best practices, code organization, integration with CI/CD, deployment, etc.

58 questions
5
votes
2 answers

Azure Databricks - Sync repo files automatically

In Azure Databricks, I am able to add different git repos in repos filed manually through console. But not able to find any way to automatically sync new files if I committed anything in source repos. Is there any service in Azure which can do this…
Shubham Sharma
  • 313
  • 3
  • 10
5
votes
1 answer

Azure databricks repos: how to pull via API

I am using repos for Azure DevOps to connect Azure databricks to my respositories in DevOps. I need to pull automatically from Azure DevOps pipelines. For that I tried using databricks API to pull, but referring to this link there is no method for…
E. Erfan
  • 1,239
  • 19
  • 37
5
votes
3 answers

Run a notebook from another notebook in a Repo Databricks

I have a notebook with functions in a repo folder that I am trying to run in another notebook. Normally I can run it as such: %run /Users/name/project/file_name So I cloned the two files (function_notebook, processed_notebook) into a Repo in…
Roger Steinberg
  • 1,554
  • 2
  • 18
  • 46
4
votes
1 answer

Using the Databricks Repos API to bring Repo in top-level production folder to latest version

I am having an issue with Github Actions workflow using the Databricks Repos API. We want the API call in the Git Action to bring the Repo in our Databricks Repos Top-level folder to the latest version on a merge into the main branch. The Github…
Jorge
  • 392
  • 3
  • 14
4
votes
1 answer

How to sharing code between two projects on Azure Databricks

I have two ML projects on Azure Databricks that work almost the same except that they are for different clients. Essentially I want to use some management system so I can share and reuse the same code across different projects. (i.e. python files…
Jing Lin
  • 41
  • 1
4
votes
1 answer

How do I access Databricks Repos metadata?

Is there a way to access data such as Repo url and Branch name inside a notebook within a Repo? Perhaps something in dbutils.
Stanislav Žoldak
  • 95
  • 1
  • 5
3
votes
1 answer

databricks use service principal + remote repository in workflow/job

I would like to create a databricks job where the owner is set to a service principal. Main reason is that if people leave company there is nothing to change and no job breaks due to failing PATs or similar. The problem is when doing so, that the…
thompson
  • 179
  • 1
  • 12
3
votes
2 answers

Azure Databricks API, cannot add repos using service principal and API calls

I need to add Azure DevOps repos to azure databricks repo by using databricks API at this link. I am using a service principal credentials for this. The service principal is already added as admin user to databricks. With my service principal I can…
E. Erfan
  • 1,239
  • 19
  • 37
3
votes
1 answer

How can I deploy arbitrary files from an Azure git repo to a Databricks workspace?

Databricks recently added support for "files in repos" which is a neat feature. It gives a lot more flexibility to the projects, since we can now add .json config files and even write custom python modules that exists solely in our closed…
Bjarne Thorsted
  • 117
  • 1
  • 11
3
votes
2 answers

Cannot push Databricks Repo to GitHub Repository

I want to create an Azure Databricks repository that is linked with my Github repository. This is what I did: Create new GitHub repository with Readme.md Create authentication token and add it to Databricks In databricks, enable all-file sync for…
erocoar
  • 5,723
  • 3
  • 23
  • 45
2
votes
2 answers

How to import own modules from repo on Databricks?

I have connected a Github repository to my Databricks workspace, and am trying to import a module that's in this repo into a notebook also within the repo. The structure is as such: Repo_Name Checks.py Test.ipynb The path to this repo is in my…
MKJ
  • 120
  • 8
2
votes
2 answers

Programmatic checkout of Databricks Repos branch

I have an integration test that compares the output from running the same scripts from 2 different branches (ie, master and a feature branch). Currently this test kicks off from my local machine, but I'd like to migrate it to a Databricks job, and…
stevemn
  • 23
  • 2
2
votes
1 answer

How to checkout a specific repo tag?

I've created a repo in Databricks. I would like to work on a specific tagged version of the repo, however, I can't see options in the UI to specify the branch. Next I created a notebook and attempted to run %sh git tag fatal: not a git repository…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
2
votes
1 answer

How does importing a module in another file in a Databricks Repos work?

I am using databricks repos I have two files, My function in a file called func.py in another folder called folder1 def lower_events(df): return df.withColumn("event",f.lower(f.col("event"))) My main notebook in which I am calling the…
Shubh
  • 585
  • 9
  • 29
2
votes
1 answer

Databricks REST API call for updating branch error : User Settings > Git Integration to set up an Azure DevOps personal access token

I am getting below error for updating the repo to a different branch using databricks rest api as mentioned at https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/update-repo . I have authenticated using service principal and…
rpshgupta
  • 135
  • 1
  • 8
1
2 3 4