Out of memoty when convert entities to json - Process 100.000 records

Question

Currently I'm facing an issue when convert entities to JSON

@Entity
class EntityA {
   @OneToMany(mappedBy = "parent",cascade = CascadeType.ALL)
   EntityB entityB;
   @OneToMany(mappedBy = "parent",cascade = CascadeType.ALL)
   EntityC entityC;
}

In DB I have 100k Entities A and each entity A has around ~500 entities B/C

I need to build a service to do following

a. Convert 100k entities A to json String ( which must include data Entity B and C)
b. Save json String to a file
c. Delete 100k entities A ( also B,C)

I use JPA pagination to get 1000 entities A each time like this

interface EntityARepository extends JpaRepository<EntityA, String>, JpaSpecificationExecutor<EntityA> {
   Slice<EntityA> findAll(Pageable pageable);
}

And in Service I implement like this

@Service
class EntityToJsonService {
    ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();
    @Autowired
    EntityARepository entityARepository;
    
    @Transactional
    public void convertAndProcess() throws Exception {
        Slice<EntityA> slice = null;      
        Pageable pageable = PageRequest.of(0,100);
        List<String> deletedIds = new ArrayList<>();
        while(true) {
            slice = entityARepository.findAll(pageable);
            log.info("slice ekyc info - page number {}, numberOfElements: {}, size: {}",
                    slice.getNumber(), slice.getNumberOfElements(), slice.getSize());           
            for (EntityA entityA : slice.getContent()) {
                //convert to json 
                byte[] data = ow.writeValueAsBytes(entityA);
                Files.write(Paths.get("path"), data);
                deletedIds.add(e.getId());
            }
            if (!slice.hasNext()) {
                break;
            }
            pageable = slice.nextPageable();
            entityManager.flush();
            entityManager.clear();
        }

        //delete data after converted 
        deleteEntityAAndChildrenRecords(deletedIds);
    }
}

Although I do paging and load 100 entity A each time , I will get out of memory issue when running above service ( when process to page ~700).

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3664)
    at java.lang.String.<init>(String.java:207)
    at java.lang.StringBuilder.toString(StringBuilder.java:407)
    at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:426)
    at com.fasterxml.jackson.core.io.SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:83)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3410)
    at com.vladmihalcea.hibernate.type.util.ObjectMapperWrapper.toString(ObjectMapperWrapper.java:58)

Does anyone got any idea to implement the service to handle this case Thanks

So for each EntityA, a separate file is to be created, right? Try putting ```ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();``` in the loop. I.e. create a new object for each EntityA object. I haven't used, and not 100% sure, but seems like, ```ow.writeValueAsBytes(entityA)``` keeps on appending to the existing value, and it keeps on growing. — Ishan, Apr 25 '23 at 13:59
Each entityA will be converted to a json file ( which contains all associated EntityB, EntityC) — chinh, Apr 25 '23 at 14:22
So, either ObjectWriter keeps on appending all the EntityAs processed one by one and keeps growing in size and on page ~700, it grows so large that out of memory error is raised. OR, maybe some EntityA object on that page itself has too many EntityB and EntityCs that it itself cannot be converted and stored in memory. Much more than your expected "around ~500 entities B/C". First try moving the ObjectWriter as an instance member to a local variable inside the loop. Create new for each. If that also does not work, then the 2nd problem could be there. — Ishan, Apr 25 '23 at 14:26
It doesn't work just like that. I mean any solution to address problem , the Out of memory error because JPA hold too much entities in memory cache. — chinh, Apr 25 '23 at 14:37
if `.clear()` doesn't work, then the problem is with your pagination. could you please share `Slice findAll(Pageable)` implementation? also have you tried to analyse heap dump? — ursa, Apr 25 '23 at 15:06
Does this answer your question? [java.lang.OutOfMemoryError: Java heap space with JSON conversion](https://stackoverflow.com/questions/34657372/java-lang-outofmemoryerror-java-heap-space-with-json-conversion) — aled, Apr 25 '23 at 16:29
@ursa, @Query("SELECT er from EntityA a where a.companyId is not null") Slice findAll(Pageable pageable); — chinh, Apr 27 '23 at 04:46

Out of memoty when convert entities to json - Process 100.000 records

0 Answers0