15

I want to install multiple private Github repositories as python packages into my repository using poetry. This works fine when running poetry install locally as I have my public SSH key added to Github which allows poetry to access the private repos. The problem is that I want to install these same private packages in my CI/CD pipeline and for that I need to add one deploy key per repo. The correct deploy key needs to be used towards the correct repository and for that to work, I need to setup a config with some aliases on the following format (I haven't actually gotten so far as to know whether this will actually work or not):

// /.ssh/config
Host repo-1.github.com
 IdentityFile ~/.ssh/repo-1-deploy-key
Host repo-2.github.com
 IdentityFile ~/.ssh/repo-2-deploy-key

Where repo-1 and repo-2 are the names of the private repositories I need to install. When running locally the pyproject.toml packages need to be setup in the following format:

// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git@github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git@github.com/equinor/repo-2.git" }
...

As this will allow the developer to install the packages without any configuration (given that they have access). For the CI/CD pipeline however, the URLs need to match the alias in the SSH config file and therefore it needs to look something like this:

// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git@repo-1.github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git@repo-2.github.com/equinor/repo-2.git" }
...

Now, where I seem to be stuck at is on how I can include two different git paths in the same pyproject file? I tried the following:

//pyproject.toml

[tool.poetry.dependencies]
repo-1 = { git = "ssh://git@repo-1.github.com/equinor/repo-1.git", optional=true }
repo-2 = { git = "ssh://git@repo-2.github.com/equinor/repo-2.git", optional=true }

[tool.poetry.dev-dependencies]
repo-1 = { git = "ssh://git@repo-1.github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git@repo-2.github.com/equinor/repo-2.git" }

[tool.poetry.extras]
cicd_modules = ["repo-1", "repo-2"]

So that I could run poetry install locally and it would use the dev dependencies and poetry install --no-dev --extras cicd_modules in the CI/CD pipeline to use the aliased paths. Sadly, this gives me a CalledProcessError as it seems like poetry tries to install the optional package despite the optional flag set to true.

What am I doing wrong here, am I using the optional flag incorrectly somehow? Is there a better way to solve this? In summary I just want to be able to install multiple private repositories as packages using poetry and Github deploy keys in a CI/CD pipeline without breaking local install behaviour, if it's in this way or another better way, I don't really have a strong opinion on.

Joachim
  • 567
  • 1
  • 7
  • 20
  • Did you managed to solve this issue? I ran into it myself when using Github Actions. The ssh-agent action has support for Deploy Keys, but it doesn't work with Poetry - https://github.com/webfactory/ssh-agent#support-for-github-deploy-keys – Martynas Jurkus Jun 22 '22 at 12:38

2 Answers2

0

I think the problem here is that you are inventing new hosts (e.g., repo-1.github.com) but those hostnames don't actually exist, and you are not mapping them to the correct existing hostname (github.com).

Your .ssh/config should look like:

Host repo-1.github.com
 Hostname github.com
 IdentityFile ~/.ssh/repo-1-deploy-key
Host repo-2.github.com
 Hostname github.com
 IdentityFile ~/.ssh/repo-2-deploy-keyHost repo-0

Then, you can use repo-1.github.com and repo-2.github.com as if they were real hosts, and they will be routed to github.com:

...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git@repo-1.github.com/equinor/repo-1.git" }
repo-2 = { git = "ssh://git@repo-2.github.com/equinor/repo-2.git" }
...
Sam
  • 101
  • 1
  • 5
  • Are you forced to use ssh? If not check if poetry has an equivalent to: https://docs.readthedocs.io/en/stable/guides/private-python-packages.html – Pablo Henkowski Aug 29 '23 at 17:52
0

If you are running in the CI/CD pipeline a subtly different configuration, consider using a shell script to modify the text of the pyproject.toml file. In this case I might use a sed one-liner to replace ssh://git@repo-1.github.com with https://PRIVATE_TOKEN:PASSWORD@github.com.

sed -i "s/ssh://git@repo-1.github.com/https://$CI_CD_USER:$CI_CD_TOKEN@github.com/g" pyproject.toml

And add the appropriate environment variable values for CI_CD_USER and CI_CD_TOKEN to your configuration.

Dr.Ransom
  • 118
  • 2
  • 8