0

I am trying to send a large file(1GB) to virtuoso local server using jena and its taking a lot of time according to my current implementation

VirtGraph graph = new VirtGraph("graphName", "jdbc:virtuoso://localhost:1111", "dba", "dba");
final String filename = Paths.get("ttlFilePath").toString();
CollectorStreamTriples inputStream = new CollectorStreamTriples();
RDFParser.source(filename).parse(inputStream);
long start1 = System.nanoTime();
for (Triple triple : inputStream.getCollected()) {
    graph.add(triple);
    System.out.println(triple);
}

Is thier any way i can drastically reduce the time taken by maybe doing some batch processing or dividing my file into smaller pieces?

Naman Sharma
  • 45
  • 6
  • Send the file directly to Virtuoso using the SPARQl Graph Store Protocol. There is no need to add it via VirtGraph which is for fine-grained manipulation of the remote graph not bulk operations. For normal SPARQL operations communicate directly the server. – AndyS Aug 08 '22 at 09:45
  • No need to collect the triple - send to the graph directly `RDFParser.source(filename).parse(graph)` -- this might give a small improvement. – AndyS Aug 08 '22 at 09:47
  • Does VirtGraph support transaction control? Otherwise you are possibly doing "autocommit" on each triple. Or a bulk update handler? (Note: this is from reading the documentation not use) – AndyS Aug 08 '22 at 10:22
  • By SPARQL graph store protocol are you suggesting the HTTP post call to send the file to server, if yes then i faced an issue there as well, link to that issue is here https://stackoverflow.com/questions/72958941/upload-a-large-ttl-file-to-virtuoso-graph – Naman Sharma Aug 08 '22 at 11:44
  • 1
    Yes - spec: https://www.w3.org/TR/sparql11-http-rdf-update/ - If you use "curl -XPOST",then you should have no problems with the file size. – AndyS Aug 08 '22 at 16:04

1 Answers1

2

Our RDFLoader Jena example is a Virtuoso Jena sample of uploading RDF data to Virtuoso DBMS. It includes support of transactions and deadlock handler.

TallTed
  • 9,069
  • 2
  • 22
  • 37
Sergey Malinin
  • 166
  • 3