I encountered a really annoying issue when trying to execute dotnet builds within a container for a standard CI/CD flow. The issues seem to occur 'randomly' (on average about 4/10 builds fail when ran from Jenkins without any changes made to the code or config in between). The issues I'm getting (It seems a bit random which one occurs when - stacktrace below):
- MSB4018: CreateAppHost task failed unexpectedly - occurs most frequently
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: The "CreateAppHost" task failed unexpectedly. [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: System.IO.IOException: The process cannot access the file '/opt/app-root/<project_path>/Pse.Spr.Reports.Web/obj/Release/netcoreapp3.1/Pse.Spr.Reports.Web' because it is being used by another process. [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at System.IO.FileStream.Init(FileMode mode, FileShare share, String originalPath) in /_/src/System.Private.CoreLib/shared/System/IO/FileStream.Unix.cs:line 120 [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) in /_/src/System.Private.CoreLib/shared/System/IO/FileStream.cs:line 252 [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/System.IO.FileSystem/src/System/IO/FileSystem.Unix.cs:line 29 [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.NET.HostModel.AppHost.BinaryUtils.CopyFile(String sourcePath, String destinationPath) [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.NET.HostModel.AppHost.HostWriter.CreateAppHost(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean windowsGraphicalUserInterface, String assemblyToCopyResorcesFrom) [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.NET.Build.Tasks.CreateAppHost.ExecuteCore() [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/opt/app-root/<project_path>/<project_csproj>]
- MSB4018: GenerateDepsFile task failed unexpectedly
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: The "GenerateDepsFile" task failed unexpectedly. [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: System.IO.IOException: The process cannot access the file '/opt/app-root/<project_release_path><*.deps.json>' because it is being used by another process. [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at System.IO.FileStream.Init(FileMode mode, FileShare share, String originalPath) in /_/src/System.Private.CoreLib/shared/System/IO/FileStream.Unix.cs:line 120 [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) in /_/src/System.Private.CoreLib/shared/System/IO/FileStream.cs:line 252 [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at Microsoft.NET.Build.Tasks.GenerateDepsFile.WriteDepsFile(String depsFilePath) [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/opt/app-root/<project_path>/<project_csproj>]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.103/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(192,5): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)
- NETSDK1029: Unable to use .../apphost as application host executable as it does not contain the expected placeholder byte sequence - my personal favorite :)
/usr/lib64/dotnet/sdk/3.1.117/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(393,5): error NETSDK1029: Unable to use '/usr/lib64/dotnet/packs/Microsoft.NETCore.App.Host.rhel.8-x64/3.1.17/runtimes/rhel.8-x64/native/apphost' as application host executable as it does not contain the expected placeholder byte sequence '63-33-61-62-38-66-66-31-33-37-32-30-65-38-61-64-39-30-34-37-64-64-33-39-34-36-36-62-33-63-38-39-37-34-65-35-39-32-63-32-66-61-33-38-33-64-34-61-33-39-36-30-37-31-34-63-61-65-66-30-63-34-66-32' that would mark where the application name would be written. [/opt/app-root/<project_csproj>]
About now I think I've gone through all of the search results and none of the answers has helped so far - so maybe here I'll get lucky (and hopefully help someone in the process).
Technical data:
Jenkins agent
Running on RHEL 8.4 with Podman v. 3.2.3
Container data
OS - RHEL7.9 and RHEL8.4 (I tried both)
Is run with a mounted volume from the Jenkins agent (the directory with the source code from cloned from the repository).
Basic Dockerfile setup where I use a RHEL8 base image from our internal repository - the RHEL7 is a modified by installing scl-utils and such but the general idea is the same
ARG rhel_version=7.9
ARG build_version=1.0.0
FROM <baseRHELimage>
ENV HOME=/opt/app-root/ \
DOTNET_APP_PATH=/opt/app-root/
#DOTNET_RUNNING_IN_CONTAINER=true
# Don't download/extract docs for nuget packages
ENV NUGET_XMLDOC_MODE=skip
RUN <setup yum repos>
dnf install -y dotnet-sdk-3.1 &&\
dnf update -y; dnf clean all
.NET data
Using dotnet core 3.1 SDK
The issues occur when I try to run a dotnet build. The Jenkinsfile is firstly removing all bin/ and obj/ directories in the first stage and then invoking the build for the application.
The most promising things I tried:
- Some people suggested that the folders bin/ and obj/ needed to be fully deleted - this is happening in the first stage prior to the run
- I tried splitting up the dotnet publish command into:
dotnet restore
dotnet build --no-restore
dotnet publish --no-build
... keeps throwing one of those 3 errors randomly across builds directly withing dotnet build
I tried invoking that dotnet build with different flags like 'no-parallel' thinking maybe threads are somehow the root cause (I am getting a lot of 'process cannot access the file because it is being used by another process' warnings - but these happen for the successful builds as well - the default dotnet retries seem to be handling those
Anyone any idea what I might be missing? The CI/CD pipeline was running previously on a RHEL7.7 agent directly (no containers used). Now we want to move to containerized builds with newer versions of RedHat - but these randomly reoccuring issues are blocking me so if anyone has any suggestions - greatly appreciated ;)
