1

I want to insert a list of Urls in a database under the condition that it doesn't exist in or an other table.
My current attempt is very slow every request can take 100ms - 500ms.

for(URL url : urls) {
 if (this.urlToEditService.findByUrl(newUrl).size() == 0 &&
     this.rawPageService.findByUrl(newUrl).size() == 0) {

      UrlToEdit urlToEdit = new UrlToEdit(newUrl);
      urlToEditService.save(urlToEdit);
 }
}

This is my save method.

public boolean save(UrlToEdit urlToEdit) {
        Transaction transaction = null;
        try (Session session = factory.openSession()) {
            transaction = session.beginTransaction();
            session.save(urlToEdit);
            transaction.commit();
        } catch (Exception e) {
            if (transaction != null) {
                transaction.rollback();
            }
            e.printStackTrace();
            return false;
        }

        return true;
}

Is there a way to insert all with one transaction? I could Imagine that the open and close part is the boodleneck.

EDIT1: I changed the code a bit so I would save in one transaction based on this answer:

List<URLToEdit> list = new ArrayList();
for(URL url : urls) {
 if (this.urlToEditService.findByUrl(newUrl).size() == 0 &&
     this.rawPageService.findByUrl(newUrl).size() == 0) {

      list.add( new UrlToEdit(newUrl));
 }
}

public boolean save(List<URLToEdit> urlToEditList) {
        Transaction transaction = null;
        
        int counter = 0;

        try (Session session = factory.openSession()) {
            transaction = session.beginTransaction();
            for (UrlToEdit urlToEdit : urlToEditList) {
                     session.save(urlToEdit);
                     counter++;

                     if(counter > 20) { 
                         counter = 0;
                         session.flush();
                         session.clear();
                     }
            }
            transaction.commit();
        } catch (Exception e) {
            if (transaction != null) {
                transaction.rollback();
            }
            e.printStackTrace();
            return false;
        }

        return true;
}

But the speed doesn't increase. So the slow part is still the exists check.

Meeresgott
  • 431
  • 3
  • 17

0 Answers0