Cannot deploy to app service deployment slot with private endpoint using azure pipeline (Error Code: ERROR_COULD_NOT_CONNECT_TO_REMOTESVC)

Question

I have an app service with a production (default) and test slot, and each slot has a private endpoint preventing public access to the apps. I'm able to use an azure pipeline to deploy to the production slot but I receive an error like below when I try to deploy same setup to test slot

Also I've tried setting up this application setting in the Azure App Service. "WEBSITE_WEBDEPLOY_USE_SCM=true" but still the issue is same.