Spring JPA with Hibernate transactions deadlock

Question

I have a Spring (core) 5.3.18 application configured with a mixed XML/annotation-driven approach. To access the backing MariaDB, I use Hibernate core 5.6.7.Final and JPA (javax.persistence-api:2.2) with an HikariCP 4.0.3 connection pool. The data source is an HikariDataSource with a default connection pool size of 10, and with the leakDetectionThreshold set to 6 seconds (transactions are really short). The configured JpaTransactionManager fom spring uses as entityManagerFactory a LocalContainerEntityManagerFactoryBean that configures Hibernate via the HibernateJpaVendorAdapter.

At runtime, with just one thread performing DB operations everything works fine. When multiple threads start requiring the DB at the same time though, threads get stuck on what seems like a starvation condition, all waiting to retrieve a connection. HikariCP reports the following leak, reported 10 times for all of the connections available in the pool:

java.lang.Exception: Apparent connection leak detected
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...

In my tests, for N threads running apparently I needed a connection pool size of exactly N*2 to avoid this behaviour, which led me to think that by some mistake (or expected behaviour unknown to me) the application I set up consumes two connections for every transaction, instead of just one. This would explain why not even one request succeeds when all threads are sent requests at the same time, but it's just my guess: each of them acquires the first connection object at some point, and then when they try to acquire the second they all get stuck at the same time.

I really can't figure out what's happening behind Spring's and JPA's magic though. In my understanding of the documentation, a public method of a @Transactional class be wrapped in a spring proxy that gets the connection just before the transaction occurs, and closes it (actually causing the connection to return to the pool instead of bein phisically closed, see also When are connections returned to the connection pool with Spring JPA (Hibernate) Entity Manager?) after the transactin is committed/rolled-back.

Something's amiss, though. I would really appreciate any help or hint about what to do, I've been stuck on this for ages.

Below is the XML spring configuration. There are no additional persistence.xml nor hibernate.properties/cfg.xml.

<bean id="dataSourceHikari" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">
    <property name="driverClassName" value="org.mariadb.jdbc.Driver" />
    <property name="jdbcUrl" value="${db.url}" />
    <property name="username" value="${db.user}" />
    <property name="password" value="${db.password}" />
    <property name="dataSourceProperties">
        <props>
            <prop key="autoReconnect">true</prop>
            <prop key="zeroDateTimeBehavior">convertToNull</prop>
        </props>
    </property>
    <property name="validationTimeout" value="3000" />
    <property name="readOnly" value="false" />
    <property name="connectionTimeout" value="60000" />
    <property name="maxLifetime" value="60000" />
    <property name="maximumPoolSize" value="${db.maxPoolSize:10}" />
    <property name="leakDetectionThreshold" value="6000" />
</bean>


<tx:annotation-driven transaction-manager="transactionManager" />

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="dataSource" ref="dataSourceHikari" />
    <property name="packagesToScan" value="my.application.package" />
    <property name="jpaVendorAdapter">
        <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
    </property>
    <property name="jpaProperties">
        <props>
            <prop key="hibernate.dialect">org.hibernate.dialect.MySQL5InnoDBDialect</prop>
        </props>
    </property>
</bean>

<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
    <property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>

My DB layer instead looks like this below. Each application thread simply invokes DBLayerClass#createSession(String) on the @Autowired DBLayerClass myDBObj once for every incoming request.

import javax.persistence.EntityManager;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
/* other stuff */

@Component
@Transactional(readOnly = false, rollbackFor = {RuntimeException.class, MyCustomExceptions.class})
public class DBLayerClass {

    @PersistenceContext
    private EntityManager entityManager;

    public Session createSession(String sessionId) throws MyCustomExceptions {
        try {
            if (getSessionById(sessionId) != null)
                throw new MyCustomExceptions("...");
            Session session = new Session(sessionId);
            entityManager.persist(session);
            return session;
        } catch (EntityExistsException e) {
            throw new MyCustomExceptions("...", e);
        } catch (PersistenceException e) {
            throw new MyCustomExceptions("...", e);
        }
    }

    private Session getSessionById(String sessionId) throws MyCustomExceptions {
        try {
            CriteriaBuilder cb = entityManager.getCriteriaBuilder();

            CriteriaQuery<Session> cq = cb.createQuery(Session.class);
            Root<Session> from = cq.from(Session.class);

            cq.where(cb.equal(from.get("sessionId"), sessionId));

            TypedQuery<Session> q = entityManager.createQuery(cq);

            return q.getSingleResult();
        } catch (NoResultException e) {
            return null;
        } catch (PersistenceException e) {
            throw new MyCustomExceptions("...", e);
        }
    }
}

The fields on my @Entity classes use @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) annotations for Long primary keys, and other regular annotations such as @Column or @Temporal. Most fancy ones are collections with @OneToMany.

I re-wrote a simpler and basic test scenario, made to start a bunch of worker threads which keep sending db requests in a loop. A handful of createSession(...) might work at first, but the test starves soon enough and the above leaks are reported by HikariCP.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/META-INF/spring/dao-test.xml" })
public class MyTestClass {

    @Autowired
    private DBLayerClass db;

    @Test
    public void testConcurrentUsage() throws Exception {
        Callable<Exception> c = new Callable<Exception>() {
            private AtomicLong nextId = new AtomicLong(0);

            @Override
            public Exception call() throws Exception {
                try {
                    long id;
                    while ((id = nextId.incrementAndGet()) < 100L) {
                        db.createSession(String.format("session-%d", id));
                    }
                    return null;
                } catch (Exception e) {
                    return e;
                }
            }
        };

        final int nThreads = 30;
        Thread[] threads = new Thread[nThreads];
        ArrayList<Future<Exception>> threadResults =  new ArrayList<>(nThreads);
        for (int i = 0; i < threads.length; i++) {
            FutureTask<Exception> threadResult = new FutureTask<>(c);
            threadResults.add(threadResult);
            threads[i] = new Thread(threadResult);
            threads[i].start();
        }

        for (Future<Exception> result : threadResults) {
            Exception e = result.get();
            if (e != null) {
                for (Thread thread : threads) {
                    thread.stop();
                }
                throw e;
            }
        }
    }

Finally, these below are the dependencies:

<dependency>
  <groupId>org.mariadb.jdbc</groupId>
  <artifactId>mariadb-java-client</artifactId>
  <version>3.0.4</version>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-core</artifactId>
  <version>5.3.18</version>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context</artifactId>
  <version>5.3.18</version>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-beans</artifactId>
  <version>5.3.18</version>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-test</artifactId>
  <version>5.3.18</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-tx</artifactId>
  <version>5.3.18</version>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-orm</artifactId>
  <version>5.3.18</version>
</dependency>
<dependency>
  <groupId>org.hibernate</groupId>
  <artifactId>hibernate-core</artifactId>
  <version>5.6.7.Final</version>
</dependency>
<dependency>
  <groupId>javax.persistence</groupId>
  <artifactId>javax.persistence-api</artifactId>
  <version>2.2</version>
</dependency>
<dependency>
  <groupId>org.hibernate.validator</groupId>
  <artifactId>hibernate-validator</artifactId>
  <version>6.0.23.Final</version>
</dependency>
<dependency>
  <groupId>com.zaxxer</groupId>
  <artifactId>HikariCP</artifactId>
  <version>4.0.3</version>
</dependency>