Azure web app for containers (Spring Boot 3 Java 17 using JDBC) connection issues w/ Microsoft SQL Server Hybrid Connection

Question

Overview

I have deployed a Spring Boot API from my private container registry to an Azure Web App for containers instance. I have setup a hybrid connection via Azure Hybrid Relay w/ a HCM hosted on a Windows 2022 server VM that has a connection to a Microsoft SQL Server 2014.

Connection string: jdbc:sqlserver://<hostname>:<port>;databaseName=<db_name>

Hybrid Connection: <hostname>:<port>

When to container runs in the Web App on deployment, the run eventually errors out.

ERROR - Container didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.

Given the output logs, The container gets stuck on the HikariPool and no connection to the database is established.

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.2)

2023-08-31T11:20:16.250-04:00  INFO 14128 --- [           main] Application      : Starting Application using Java 17.0.6 with PID 14128 
2023-08-31T11:20:16.252-04:00  INFO 14128 --- [           main] Application      : The following 1 profile is active: "dev"
2023-08-31T11:20:17.085-04:00  INFO 14128 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2023-08-31T11:20:17.110-04:00  INFO 14128 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 16 ms. Found 0 JPA repository interfaces.
2023-08-31T11:20:17.656-04:00  INFO 14128 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8000 (http)
2023-08-31T11:20:17.665-04:00  INFO 14128 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-08-31T11:20:17.665-04:00  INFO 14128 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.11]
2023-08-31T11:20:17.739-04:00  INFO 14128 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-08-31T11:20:17.741-04:00  INFO 14128 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1447 ms
2023-08-31T11:20:17.888-04:00  INFO 14128 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...

The metric indicates that on the initial start up of the web application. The sender client connection fails

Metrics

Spring Boot Application Configuration

The azure deployment active profile uses managed Identity retrieves the database credentials from a specified azure keyvault (spring-cloud-azure-starter-keyvault)

Note: This configuration works in local testing using Intellij Azure Tools to replicate the managed identity

server:
  port: 8000

spring:
  cloud:
    azure:
      credential:
        managed-identity-enabled: true
      keyvault:
        secret:
          property-source-enabled: true
          property-sources:
            - endpoint: https://${keyvault}.vault.azure.net/
              retry:
                mode: exponential
                exponential:
                  max-retries: 4
                  base-delay: PT0.0801S
                  max-delay: PT9S
  datasource:
    url: ${db-url}
    username: ${db-user}
    password: ${db-pass}
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver

HCM

All Configurations show a status of connected both in the HCM and in the Azure Portal

HCM Setup

Container Configuration

FROM <java-17-image>

EXPOSE 8000

ENV PROFILE=<default_profile>

WORKDIR /opt/app

COPY ../../target/<application_name>.jar app.jar

ENTRYPOINT [ "java", "-jar", "-Dspring.profiles.active=${PROFILE}", "app.jar"]

Note Not running container as root user (excluded from example)

Troubleshooting steps thus far

Test Managed Identity

I ran the application with the azure deployment active profile locally, utilizing Intellij Azure Tools. The configuration was able to access credentials from the keyvault and establish a connection to the database using the same connection string.

Check HCM Logs

I checked the event viewer on the Windows 2022 Server VM running the HCM to confirm there were no connection issues with it

HCM Events

Establish Windows VM Host can reach the database

Made sure that the VM was able to access the database host

$ Test-NetConnection <host> -Port <port>

Check TCP connection in Web App Hudu Console

Made sure I can access to hybrid connection host from the Hudu bash console

$ tcpping <host> <port>