Copy projects from git submodule to ADO repository

Question

I have an ADO repository which includes a GitHub repository as a git submodule.

Here is an example of .csproj file in ADO repo:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\..\..\Repositories.ABC\Repositories.ABC.csproj" />
    <ProjectReference Include="..\..\..\Repositories.XYZ\Repositories.XYZ.csproj" />
  </ItemGroup>
</Project>

Project references mentioned in .csproj - Respositories.ABC and Repositories.XYZ are a part of GitHub repository (git submodule). Is there a way to copy the projects - Respositories.ABC and Repositories.XYZ from git submodule to ADO repository in the specified path so that build will not fail?

Here is my project structure:

Here is my submodule structure:

score 0 · Answer 1 · answered Nov 25 '20 at 02:18

Yeah, you should enable getting submodules:

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

However if you use private repos for submodules and do not keep them in Azure Repos you may check rather - Alternative to using the Checkout submodules option

In some cases you can't use the Checkout submodules option. You might have a scenario where a different set of credentials are needed to access the submodules. This can happen, for example, if your main repository and submodule repositories aren't stored in the same Azure DevOps organization, or if your job access token does not have access to the repository in a different project.

If you can't use the Checkout submodules option, then you can instead use a custom script step to fetch submodules. First, get a personal access token (PAT) and prefix it with pat:. Next, base64-encode this prefixed string to create a basic auth token. Finally, add this script to your pipeline:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: Bearer <BASE64_ENCODED_STRING>" submodule update --init --recursive

Thank you! To get a clarification - if I have an ADO repo which includes another ADO repo as git submodule, I can use the first steps you mentioned correct? — user989988, Nov 25 '20 at 03:03
Generally speaking yes. If they are on the same project then for sure. There could be some edge case when it is not possible using this method, but you should give a try. — Krzysztof Madej, Nov 25 '20 at 03:13
Thank you! Where should I be adding the checkout step that you suggested above? Would that be in a yaml file within main ADO repo or in a yaml file within submodule? Also, let's say path to one of the projects in submodule is /Service/Project/Repositories.XYZ. What should I specify as path in the checkout step? Please let me know. — user989988, Nov 25 '20 at 03:17
It should be your first step in steps in your YAML file. By default it gets code to `C:\agent\_work\1` but it depends where normally submodule is in your main repo. Please ad this step `- pwsh: ls '$(System.DefaultWorkingDirectory)'` after checkout to see how your root folder look like. — Krzysztof Madej, Nov 25 '20 at 03:37

Leo Liu · Answer 2 · 2020-11-26T02:10:54.167

0

Is there a way to copy the projects - Respositories.ABC and Repositories.XYZ from git submodule to ADO repository in the specified path so that build will not fail?

I am afraid there is no such way to copy the Github submodule to ADO repository in the specified path.

When we add the submodules to ADO repository, it will create two files in the ADO repository .gitmodules and project name file:

If we check out the submodules, git will save the project to the project name folder based on the path in the .gitmodules. Since the files .gitmodules and project name file are add in the source control, it checkout by default. If we want to change the path, we have to change the path manually after checkout.

To make sure the build successfully, we just need to make sure the path of ProjectReference in the .csproj file match the file structure in the repo.

Update:

If the directory hierarchy where your Cleaner.csproj file is located is like this Service\Project\Hosts\Cleaner\Function\Cleaner.csproj, the path should be:

<ProjectReference Include="..\..\..\..\..\crypto-project\Service.xxxx\Repositories.ABC\Repositories.ABC.csproj" />

The path should be the relative position of Cleaner.csproj and .gitmodules file, a ..\ represents the upper level directory, and the path behind ..\ should be crypto-project + each level of directory folder.

edited Nov 26 '20 at 02:10

answered Nov 25 '20 at 09:37

Leo Liu

71,098
10
114
135

Thank you! Could you please give an example of how the path of ProjectReference should change in Main.csproj file? – user989988 Nov 26 '20 at 00:34
Thank you! In my example, .csproj file in ADO repo has 2 project references - Repositories.ABC.csproj & Repositories.XYZ.csproj (which are in git submodule). How do update the path? Please give an example. – user989988 Nov 26 '20 at 01:13
@user989988, the path related your file structure in the repo, would you please share the file structure in the repo like me in the answer (including mian.csproj and Repositories.ABC & Repositories.XYZ files). – Leo Liu Nov 26 '20 at 01:25
@user989988, Is the `.gitmodules` file in the Cleaner folder? If yes, then the path should be ``. The path should be the relative position of `Cleaner.csproj` and `.gitmodules` file, a ..\ represents the upper level directory, and the path behind ..\ is `crypto-project` + each level of directory folder. – Leo Liu Nov 26 '20 at 01:50
No it's in the root folder - I have added one more image to make project structure more clear. Also, could you please add your answer as a part of solution instead of comment to be more clear? Thank you! – user989988 Nov 26 '20 at 02:06
@user989988, I have updated the answer, please check if it helps you. – Leo Liu Nov 26 '20 at 02:11
Than you! yaml file should have a step - checkout: self submodules: true correct? – user989988 Nov 26 '20 at 02:15
@user989988, Right! But you may encounter authentication problem. Just do and check the result and let me know for free. – Leo Liu Nov 26 '20 at 02:17
1

I updated project reference: however it gives error: Unable to find project information for 'C:\Users\\Desktop\submodules\\crypto-project\Service\Project\Repositories.ABC\Repositories.ABC.csproj'. If you are using Visual Studio, this may be because the project is unloaded or not part of the current solution so run a restore from the command-line. Otherwise, the project file may be invalid or missing targets required for restore. – user989988 Nov 26 '20 at 02:35
@user989988, If I build with Visual Studio task, I get the same error as you. You could try to use MSBuild to build the main.csproj file directly, I could build it successful on my side. But we seem to have overlooked an important foundation, add a ProjectReference to a project that is not in the same solution is not recommended. https://stackoverflow.com/questions/26314094/how-to-reference-a-project-out-of-solution and https://stackoverflow.com/questions/11601950/adding-a-projectreference-to-a-project-that-is-not-in-the-same-solution – Leo Liu Nov 26 '20 at 03:10
1

I see - instead of updating project references in .csproj, is there a different approach for a success local build maybe without using git submodule? – user989988 Nov 26 '20 at 04:33
@user989988, If you do not have any specific restrictions on the referenced projects, my personal suggestion is to add these two projects to your current solution, and then directly use the project reference, everything will become simple and proficient – Leo Liu Nov 26 '20 at 04:37

Copy projects from git submodule to ADO repository

2 Answers2