Can I use Spring Data saveAll() method and Hibernate 5 batch?

Question

I have a Spring boot application with application.yml:

spring:
    datasource:
        driver-class-name: org.postgresql.Driver
        url: *my_url*
        password: *my_pass*
        username: *my_username*
    jpa:
        properties:
            hibernate:
                jdbc:
                    batch_size: 15
                    #order_inserts: true
                    #order_updates: true
                    #batch_versioned_data: true

When I try to save 200 000 entities using the method saveAll(Iterable<S> entities), it saves all 200 000 entities at the same time, but I want to save batches of 15 entities at a time.

Is it possible to use Spring Data's SimpleJpaRepository and Hibernate's batch?

https://frightanic.com/software-development/jpa-batch-inserts/ — Cepr0, Sep 04 '17 at 17:15
@11thdimension I think no ) https://www.google.com.ua/search?q=batch+insert+in+spring+data+jpa — Cepr0, Sep 04 '17 at 17:57
@Cepr0 It's from 5 years ago, guy is using Session to do manage transactions. It's old. — 11thdimension, Sep 04 '17 at 17:59
@11thdimension https://vladmihalcea.com/2017/04/25/the-best-way-to-do-batch-processing-with-jpa-and-hibernate/ 2017-04-25 - the same approach. — Cepr0, Sep 04 '17 at 18:01
@11thdimension you are not attentive, see last paragraph: **JPA batch inserts with Spring Data JPA**. why don't you offer a better approach? ) — Cepr0, Sep 04 '17 at 18:03
@Cepr0, It doesn't work with the Spring Data, as it won't allow us to manage transactions manually. A better way will be to use `JdbcTemplate` to manage batch operations. — 11thdimension, Sep 04 '17 at 19:01
@11thdimension What do you say about my attempt (see my answer)? )) — Cepr0, Sep 04 '17 at 20:48
@Cepr0 It won't work, as processBatch in same class, so Spring can not create a proxy for Transaction. — 11thdimension, Sep 05 '17 at 12:17
If there's no proxy then there's no transaction, meaning if something fails then Spring can not recover. — 11thdimension, Sep 05 '17 at 12:27

score 2 · Accepted Answer · answered Sep 04 '17 at 20:45

My approach ))

@Service
public class BulkService {

    @PersistenceContext
    private EntityManager em;

    // @Value("${spring.jpa.properties.hibernate.jdbc.batch_size}")
    private int batchSize = 20;

    private List<Entity> savedEntities;

    public Collection<Entity> bulkSave(List<Entity> entities) {
        int size = entities.size();
        savedEntities = new ArrayList<>(size);

        try {
            for (int i = 0; i < size; i += batchSize) {
                int toIndex = i + (((i + batchSize) < size) ? batchSize : size - i);
                processBatch(entities.subList(i, toIndex));
                em.flush();
                em.clear();
            }
        } catch (Exception ignored) {
            // or do something...  
        }
        return savedEntities;
    }

    @Transactional
    protected void processBatch(List<Entity> batch) {

        for (Entity t : batch) {
            Entity result;
            if (t.getId() == null) {
                em.persist(t);
                result = t;
            } else {
                result = em.merge(t);
            }
            savedEntities.add(result);
        }
    }
}

Working example and test

Can I use Spring Data saveAll() method and Hibernate 5 batch?

1 Answers1