I want to insert a list of Urls in a database under the condition that it doesn't exist in or an other table.
My current attempt is very slow every request can take 100ms - 500ms.
for(URL url : urls) {
if (this.urlToEditService.findByUrl(newUrl).size() == 0 &&
this.rawPageService.findByUrl(newUrl).size() == 0) {
UrlToEdit urlToEdit = new UrlToEdit(newUrl);
urlToEditService.save(urlToEdit);
}
}
This is my save method.
public boolean save(UrlToEdit urlToEdit) {
Transaction transaction = null;
try (Session session = factory.openSession()) {
transaction = session.beginTransaction();
session.save(urlToEdit);
transaction.commit();
} catch (Exception e) {
if (transaction != null) {
transaction.rollback();
}
e.printStackTrace();
return false;
}
return true;
}
Is there a way to insert all with one transaction? I could Imagine that the open and close part is the boodleneck.
EDIT1: I changed the code a bit so I would save in one transaction based on this answer:
List<URLToEdit> list = new ArrayList();
for(URL url : urls) {
if (this.urlToEditService.findByUrl(newUrl).size() == 0 &&
this.rawPageService.findByUrl(newUrl).size() == 0) {
list.add( new UrlToEdit(newUrl));
}
}
public boolean save(List<URLToEdit> urlToEditList) {
Transaction transaction = null;
int counter = 0;
try (Session session = factory.openSession()) {
transaction = session.beginTransaction();
for (UrlToEdit urlToEdit : urlToEditList) {
session.save(urlToEdit);
counter++;
if(counter > 20) {
counter = 0;
session.flush();
session.clear();
}
}
transaction.commit();
} catch (Exception e) {
if (transaction != null) {
transaction.rollback();
}
e.printStackTrace();
return false;
}
return true;
}
But the speed doesn't increase. So the slow part is still the exists check.