1

I'm currently using Saxon to process Xquery in our .NET application. We're working with really big XML files (~2GB). When running the Xquery against one of these files using the Saxon binary file directly, the time it takes to complete the evaluation is around 2 minutes, but when doing the evaluation from my C# application the time elapsed increases to around 10 minutes, and I haven't yet been able to identify what I'm doing wrong.

This is what I'm doing when I run the Xquery using the Saxon binary file through the command line:

Query.exe -config:config.xml -q:XQueryTest.txt

These are the contents of the config.xml:

<configuration xmlns="http://saxon.sf.net/ns/configuration" edition="HE">
  <xquery defaultElementNamespace="http://www.irs.gov/efile"/>
</configuration>

And XQueryTest.txt contains the Xquery we are going to process. When running the Xquery from the command line, we modify it to indicate the file we will run it against, using the doc() function. Here is a sample line:

for 
    $ReturnData at $currentReturnDataPos in if(exists(doc("2GB.XML")/Return/ReturnData)) then doc("2GB.XML")/Return/ReturnData else element{'ReturnData'} {''}

As mentioned above, running this command, takes about 2 minutes to complete.

Now these is what I'm doing in my .NET application to make this same evaluation. 

Processor processor = new Processor();
DocumentBuilder documentBuilder = processor.NewDocumentBuilder();
documentBuilder.IsLineNumbering = true;
documentBuilder.WhitespacePolicy = WhitespacePolicy.PreserveAll;
XQueryCompiler compiler = processor.NewXQueryCompiler();

string query = BuildXqueryString();

if (!String.IsNullOrEmpty(query))
{
    XQueryExecutable executable = compiler.Compile(query);
    XQueryEvaluator evaluator = executable.Load();

    using (XmlReader myReader = XmlReader.Create(@"C:\Users\Administrator\Desktop\2GB.xml"))
    {
        evaluator.ContextItem = documentBuilder.Build(myReader);
    }

    var evaluations = evaluator.Evaluate();
}

The issue we have is in this line: evaluator.ContextItem = documentBuilder.Build(myReader). Which is not even the evaluation, but just the loading of the file. This line takes just too much time to execute, and I need to know if that is expected, or if there's a way to increase its speed. I have used all the different overloads of the Build() method and they all take a lot of time to complete, way more than the 2 minutes that the execution takes when executing from the command line.

Regarding using the streaming capacity of Saxon to read the file by parts, because of the Xqueries we generate, that is not an option, as the Xquery can combine information in any part of the XML.

Ricardo
  • 43
  • 5
  • http://stackoverflow.com/questions/2415434/the-limitation-on-the-size-of-net-array shows that the maximum size of any allocated object in .NET is 2GB. Perhaps Saxon is running into a .NET memory issue and having to take extra steps that the binary may not have to take? Perhaps looking at resources used would show the binary is able to use additional memory? Sorry I don't have anything else or specific experience with Saxon. Cheers! – chryosolo Apr 28 '16 at 03:59

2 Answers2

1

We have seen a similar 5-to-1 ratio between Saxon on the Java platform and Saxon on the .NET platform in some cases, and we haven't got to the bottom of why it happens despite extensive investigation. Part of the reason is that it seems to be inconsistent. When we first shipped Saxon on .NET using the IKVMC cross-compiler, the ratio was much better, with only about a 25% overhead on .NET, but there seem to have been a number of changes in technology since then: Java VMs have got faster, IKVMC has switched from using the GNU Classpath library to OpenJDK, and .NET itself hasn't stood still.

It's new to me, though, that the same code should run much faster from the .NET command line than it runs from the .NET API.

The big difference here is that when you run from the command line, Saxon builds the document using the Apache Xerces parser (converted to .NET code using IKVMC), whereas when you use DocumentBuilder.build() in the way shown, you are using Microsoft's XmlReader.

I would expect the document building to run fastest when you supply a (file system) URI, but I can't say I've measured it. It might be worth doing some experiments (perhaps with smaller files) and showing us the results. Alternatively, have you tried using the doc() method from your application, rather than building the document first?

Michael Kay
  • 156,231
  • 11
  • 92
  • 164
  • We also thought that using an URI would get us better results, but that was not the case at all, it was still very slow. We tried using your final recommendation (why didn't we think of that?), and indeed the speed was greatly increased. It is still a little slower (a little over 3 minutes), but nowhere near the previous 10 minutes, which is great! I think we never thought about this approach because we were thinking that the correct implementation should first load the XML and then run the generic Xquery against it, instead of hard coding the XML path inside the Xquery. Thanks!! – Ricardo Apr 29 '16 at 01:05
  • Thanks for the info. There are some strange effects going on here which I don't understand: we must see if we can reproduce this. – Michael Kay Apr 29 '16 at 09:10
0

The slow performance is caused by using the .NET XmlReader to do the parsing. The Push/Pull SAX eventing handling with the .NET XML parser and the Saxon receiver is much slower than using the JAXP xerces parser directly, which is supplied within Saxon.

To force the JAXP parser, you can do the following should work:

evaluator.ContextItem = documentBuilder.Build(new Uri("file:///C:\Users\Administrator\Desktop\2GB.xml"));

ond1
  • 691
  • 6
  • 10