Marklogic : Query response time is very high

Question

I have around 15000 records in xml format in a uri, say:"documents/products/specs/*.xml". Each xml is of around 25 Kilo Bytes in size. I connected to this marklogic server uisng a remote Apache Tomcat Server that has a XCC client (Java), which tries to execute a AdHocQuery which resembles something like this:

let $a := cts:uri-match('documents/products/specs/*.xml')
          for $xml in $a
          return fn:doc($a)

(for loop is implemented in java).

This works fine. But for records with larger count, say for 15000 record, it takes 60 minutes, when all the server and internet speed are very good. (Total size of all document in the uri will be around 20 MB which should not take more than 20 Minutes).

Is there any workaround ?

15,000 * 25 = 375,000 not 20-MB. – mblakele Feb 21 '13 at 17:07 — mblakele, Feb 21 '13 at 17:07

score 2 · Answer 1 · edited Mar 12 '13 at 18:30

2

Try this:

cts:search(
    fn:doc(),
    cts:document-query(
        cts:uri-match('documents/products/specs/*.xml')
    ), "unfiltered"
)

edited Mar 12 '13 at 18:30

Jules

14,200
13
56
101

answered Mar 12 '13 at 18:02

Charles Foster

338
3
5

score 1 · Answer 2 · answered Feb 21 '13 at 11:57

1

What you are doing is requesting the full body of ALL documents. This is not a typical query, rather it is a DB dump. The query you show will buffer up all this data, then send it through tomcat which again buffers up all the data and then send it to you. This is a large dataset to be sending in one request.

What is the intent of your query ? If you want to get all documents you should either dump them out using a program like mlcp or fetch them in smaller batches by first collecting the URI's then fetching the documents. This can be speeded up substantially by doing the document fetching in parallel. You can see examples of Java source in xmlsh that shows how to fetch documents in XCC in parallel

http://xmlsh.svn.sourceforge.net/viewvc/xmlsh/extensions/marklogic/src/org/xmlsh/marklogic/get.java?revision=792&view=markup

My guess (correct me if I am wrong) is you are just experimenting and dont actually need all the docs. In which case a more realistic query should be tried.

answered Feb 21 '13 at 11:57

DALDEI

3,722
13
9

the requirement is to make ms-excel compatible csv files representing all files in the directory. – Ranjan Sarma Feb 21 '13 at 12:31
So ask that question: "how do I best make an ms-excel compatible csv file representing all documents in a MarkLogic directory?" – mblakele Feb 21 '13 at 17:03
@mblakele : http://stackoverflow.com/questions/15016876/ms-excel-compatible-csv-file-representing-all-documents-in-a-marklogic-directory – Ranjan Sarma Feb 22 '13 at 04:33
do you want the document names or the content to be placed in the csv? – Eric Bloch Feb 24 '13 at 01:03

score 0 · Answer 3 · answered Mar 01 '13 at 20:54

The reason that the query is taking so long is because the Marklogic server is reading form the disk, for most of those files. Unless you have a really big tree-cache size. What you need to do is lower the scope of your query. Maybe add some indexes to the files.

All that said if all you want to do is ETL that data out then you might want to batch the request.

Marklogic : Query response time is very high

3 Answers3

Linked