Wix: Installer slow because file in use

Question

I use wix toolset to install an ASP.NET Core application as a windows service. When doing a major upgrade, the installer can take very long (5 mins) while it usually is a couple of seconds. This is because the InstallValidate action gives errors because files are in use by the service, I noticed that if I stop the service before starting the installer, it always runs smoothly. I use the following code to install and start/stop the service: (this is generated using heat)

<Component Id="ApiEndpoint.exe" Guid="*">
                <File Id="ApiEndpoint.exe" KeyPath="yes" Source="$(var.publishDir)\ApiEndpoint.exe" />
                <wix:ServiceInstall Id="ApiEndpointInstall" DisplayName="ApiEndpoint" Name="ApiEndpoint" ErrorControl="normal" Start="auto" Type="ownProcess" Vital="yes" xmlns:wix="http://schemas.microsoft.com/wix/2006/wi" xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
                    <util:ServiceConfig RestartServiceDelayInSeconds="60" ResetPeriodInDay="1" FirstFailureActionType="restart" SecondFailureActionType="restart" ThirdFailureActionType="restart" />
                </wix:ServiceInstall>
                <wix:ServiceControl Id="ApiEndpointControl" Name="ApiEndpoint" Start="install" Stop="both" Remove="uninstall" xmlns:wix="http://schemas.microsoft.com/wix/2006/wi" xmlns:util="http://schemas.microsoft.com/wix/UtilExtension" />
            </Component>

Shouldn't the files in use by the ApiEndpoint be excluded because the service will be stopped anyway? Is there a way I can stop the service before the InstallValidate, I tried this with a custom action, but this wasn't possible because no admin privileges are present at that point.

I also tried to ignore the exe file from heat and make a component for it myself, like below, this results in the same issue.

    <Fragment xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
        <ComponentGroup Id="ExeComponents" Directory="INSTALLFOLDER">
            <Component Win64="yes" Id="ExeComponent" Guid="*">
                <File Id="ExecFile" Source="$(exeSource)" KeyPath="yes" />
                <ServiceInstall Id="ApiEndpointInstall" DisplayName="ApiEndpoint" Name="ApiEndpoint" ErrorControl="normal" Start="auto" Type="ownProcess" Vital="yes">
                    <util:ServiceConfig RestartServiceDelayInSeconds='60' ResetPeriodInDays='1' FirstFailureActionType='restart' SecondFailureActionType='restart' ThirdFailureActionType='restart' />
                    <util:PermissionEx
                    User="Everyone"
                    GenericAll="yes"
                    ServiceChangeConfig="yes"
                    ServiceEnumerateDependents="yes"
                    ChangePermission="yes"
                    ServiceInterrogate="yes"
                    ServicePauseContinue="yes"
                    ServiceQueryConfig="yes"
                    ServiceQueryStatus="yes"
                    ServiceStart="yes"
                    ServiceStop="yes" />
                </ServiceInstall>
                <ServiceControl Id="ApiEndpointControl" Name="ApiEndpoint" Start="install" Stop="both" Remove="uninstall" />
            </Component>
        </ComponentGroup>
    </Fragment>

I'm very new at wix so I'm not sure if I'm missing something obvious or if this is just a limit of Wix.

I can see that the service stops eventually when the files are being copied, but it just seems like it is stopped either too late, or the installer should ignore the files in use by the service that will be stopped anyway.