5

I'm trying to achieve Batch Insert using JpaRepository, but it seems that it doesn't work even though I'm using the recommended properties. This is my code:

Entity - Book.java:

@Entity(name = "books")
@Table(name = "books")
public class Book {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    private Long id;

    private String title;

    private String author;

    private String edition;

    private String status;

    @Column(unique = true)
    private String isbn;

    @JsonIgnore
    @OneToMany(cascade = CascadeType.ALL,mappedBy = "book", fetch = FetchType.LAZY)
    private List<Image> images = new ArrayList<>();

// Getters and Setters omitted

}

Service - BookServiceImpl


@Service
public class BookServiceImpl implements BookService {

    @Autowired
    private BookRepository bookRepository;

    @Override
    public List<Book> storeBooks(List<Book> books) {

        return bookRepository.saveAll(books);
    }
}

Properties - application.properties:

spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localhost/bookdb?reWriteBatchedInserts=true
spring.datasource.username=**
spring.datasource.password=**

spring.jpa.hibernate.ddl-auto=create
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect
spring.jpa.properties.hibernate.jdbc.batch_size=100
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.order_updates=true
spring.jpa.properties.hibernate.generate_statistics=true

SQL Log after inserting:

Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: select nextval ('hibernate_sequence')
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
Hibernate: insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)
2021-03-07 09:57:50.163  INFO 7800 --- [nio-8080-exec-1] i.StatisticalLoggingSessionEventListener : Session Metrics {
    2883700 nanoseconds spent acquiring 1 JDBC connections;
    0 nanoseconds spent releasing 0 JDBC connections;
    9612998 nanoseconds spent preparing 10 JDBC statements;
    23803401 nanoseconds spent executing 9 JDBC statements;
    23764601 nanoseconds spent executing 1 JDBC batches;
    0 nanoseconds spent performing 0 L2C puts;
    0 nanoseconds spent performing 0 L2C hits;
    0 nanoseconds spent performing 0 L2C misses;
    275826200 nanoseconds spent executing 1 flushes (flushing a total of 9 entities and 9 collections);
    0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)

I don't know if the problem is with the logs or something, but I implemented everything as recommended...

Cœur
  • 37,241
  • 25
  • 195
  • 267
Danik_Help
  • 195
  • 3
  • 12
  • 1
    Have you defined a transaction somewhere? Perhaps the inserts have not been committed. – Gordon Tytler Mar 07 '21 at 08:32
  • Does this help you? https://stackoverflow.com/a/50882952/12312156 – Faramarz Afzali Mar 07 '21 at 08:35
  • The problem was with the logger, the default hibernate logger, because by default it doesn't show if the SQL Inserts are batched – Danik_Help Mar 07 '21 at 08:42
  • 2
    `23764601 nanoseconds spent executing 1 JDBC batches;` --> this shows u its executed in batches.https://www.baeldung.com/spring-data-jpa-batch-inserts#console refer to this link. – Raghu Dinka Vijaykumar Mar 07 '21 at 08:44
  • @RaghuDinkaVijaykumar yes, i realized now that the batch insert was already working, the problem is that Hibernate Default logging doesn't show if the SQL Inserts are batched or not, so the solution was to implement BeanPostProcessor and add two dependencies, SLF4J and datasource proxy – Danik_Help Mar 07 '21 at 08:52
  • @Danik_Help Please edit your question. to reflect what you're asking. Batch Insert not working doesn't sound right. – Raghu Dinka Vijaykumar Mar 07 '21 at 08:59
  • @RaghuDinkaVijaykumar okay – Danik_Help Mar 07 '21 at 09:00

1 Answers1

8

I realized that the batch insert was already working, the problem is that Hibernate Default Logging doesn't show if the SQL Inserts are batched or not, so the solution was to implement BeanPostProcessor and add two dependencies, SLF4J and DataSource Proxy.


@Component
public class DatasourceProxyBeanPostProcessor implements BeanPostProcessor {

    @Override
    public Object postProcessBeforeInitialization(final Object bean, final String beanName) throws BeansException {
        return bean;
    }

    @Override
    public Object postProcessAfterInitialization(final Object bean, final String beanName) throws BeansException {
        if (bean instanceof DataSource) {
            ProxyFactory factory = new ProxyFactory(bean);
            factory.setProxyTargetClass(true);
            factory.addAdvice(new ProxyDataSourceInterceptor((DataSource) bean));
            return factory.getProxy();
        }
        return bean;
    }

    private static class ProxyDataSourceInterceptor implements MethodInterceptor {
        private final DataSource dataSource;

        public ProxyDataSourceInterceptor(final DataSource dataSource) {
            super();
            this.dataSource = ProxyDataSourceBuilder.create(dataSource).countQuery().logQueryBySlf4j(SLF4JLogLevel.INFO).build();
        }

        @Override
        public Object invoke(final MethodInvocation invocation) throws Throwable {
            Method proxyMethod = ReflectionUtils.findMethod(dataSource.getClass(), invocation.getMethod().getName());
            if (proxyMethod != null) {
                return proxyMethod.invoke(dataSource, invocation.getArguments());
            }
            return invocation.proceed();
        }
    }
}

so i updated my pom.xml

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
        <!-- https://mvnrepository.com/artifact/net.ttddyy/datasource-proxy -->
        <dependency>
            <groupId>net.ttddyy</groupId>
            <artifactId>datasource-proxy</artifactId>
            <version>1.7</version>
        </dependency>

Then i tested it again and i got this from SLF4J:


2021-03-07 10:48:46.075  INFO 14044 --- [nio-8080-exec-5] n.t.d.l.l.SLF4JQueryLoggingListener      : Name:, Connection:6, Time:4, Success:True, Type:Prepared, Batch:True, QuerySize:1, BatchSize:16, Query:["insert into books (author, edition, isbn, status, title, id) values (?, ?, ?, ?, ?, ?)"], Params:[(Author 2,Edition 2,978-472-592-193-7,Owned,Book 2,33),(Author 2,Edition 2,978-412-592-193-7,Owned,Book 2,34),(Author 2,Edition 2,978-473-592-193-7,Owned,Book 2,35),(Author 2,Edition 2,978-472-552-193-7,Owned,Book 2,36),(Author 2,Edition 2,978-472-092-193-7,Owned,Book 2,37),(Author 2,Edition 2,978-402-592-193-7,Owned,Book 2,38),(Author 2,Edition 2,178-472-592-193-7,Owned,Book 2,39),(Author 2,Edition 2,278-472-592-193-7,Owned,Book 2,40),(Author 2,Edition 2,978-472-592-472-7,Owned,Book 2,41),(Author 2,Edition 2,592-472-592-123-7,Owned,Book 2,42),(Author 2,Edition 2,562-472-592-123-9,Owned,Book 2,43),(Author 2,Edition 2,978-123-562-123-9,Owned,Book 2,44),(Author 2,Edition 2,472-472-582-123-9,Owned,Book 2,45),(Author 2,Edition 2,222-472-592-123-9,Owned,Book 2,46),(Author 2,Edition 2,978-222-123-123-9,Owned,Book 2,47),(Author 2,Edition 2,978-433-502-123-9,Owned,Book 2,48)]

Sources:

Danik_Help
  • 195
  • 3
  • 12
  • I am having similar issue but even after adding above code its not working – Abu Talha Siddiqi Jan 11 '23 at 12:55
  • It's important to note that even after config datasource proxy, it will not log correct sql statement, I would recommend to enable mysql log to see the correct sql query. in my case, I missed "reWriteBatchedInserts=true" and proxy logs shows batch insert (which is not the case when I check mysql log) – ExploreEv Jul 17 '23 at 16:39