How to validate very large XML files?

Question

How can I validate a large XML file (>100mb)? I try to open it with IE, FX & GC and it either crashes or doesn't do anything.

possible duplicate of [Text editor to open big (giant, huge, large) text files](http://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files) — Jehof, Sep 23 '11 at 11:33
@Jehof Nope, not at all. A text editor is different from a validator. — phihag, Sep 23 '11 at 13:19
@phihag to be fair, the OP *did* talk about opening the file in IE and Firefox, so it's unclear whether the OP means to visually manually validate or programatically validate. — dj_segfault, Sep 23 '11 at 14:13

Ciro Santilli OurBigBook.com · Answer 1 · 2018-05-31T20:25:41.663

13

xmllint --stream

Worked on a 1.2Gb file with memory limited to 500Mb:

ulimit -Sv 500000
xmllint --stream a.xml

Without --stream, Linux kills the process, and without ulimit, my computer jams.

I was not able however to get output from --xpath when using --stream: How to do command line XPath queries in huge XML files?

Tested in Ubuntu 14.04, xmllint version 20901.

edited May 31 '18 at 20:25

answered Oct 07 '15 at 12:45

Ciro Santilli OurBigBook.com

347,512
102
1,199
985

1

It's worth nothing that `xmllint` is cross platform. I use it on Windows. I confirm that the `--stream` option works there well too. I didn't even need to set a memory limit to process 3.5GB. However .net library seems to be 2x faster. – Jarekczek Mar 11 '16 at 07:02
1

@Jarekczek thanks for letting me know! You don't need the `ulimit` in Linux either with `--stream`, I'm just showing people how not to brick their machines / test that it actually does not use much memory ;-) – Ciro Santilli OurBigBook.com Mar 11 '16 at 07:13
1

To maybe help with Google searches: this is the answer you need if running xmllint just stops with "Killed" when validation a huge file. – Legolas Jul 15 '22 at 14:20

score 8 · Answer 2 · answered Sep 23 '11 at 13:22

8

You can try using a command-line validator, for example xmlstarlet:

$ xmlstarlet validate bigfile.xml

answered Sep 23 '11 at 13:22

phihag

278,196
72
453
469

It may work, but that project appears to be dead, so don't expect updates. See http://xmlstar.sourceforge.net/ – dj_segfault Sep 23 '11 at 14:14
+1 Since this is the only one which does not require Windows. And it's free :) – Haralan Dobrev Apr 25 '13 at 12:10
`sel` however does not work :-( http://stackoverflow.com/questions/33653844/xmlstarlet-sel-on-large-file – Ciro Santilli OurBigBook.com Nov 19 '15 at 06:22
`sel` is for selection. Use `validate` to validate, not `sel`. – phihag Nov 19 '15 at 10:43

score 6 · Answer 3 · edited Jun 10 '14 at 17:12

6

The only tool I know that combines a large file viewer and an XML validator for huge files is XML ValidatorBuddy . The file viewer doesn't load the complete file at once but it is possible to scroll and there is also XML syntax-coloring applied. The validation uses the SAX parser from Xerces and your document with >100mb shouldn't be a problem.

edited Jun 10 '14 at 17:12

Clemens

1,744
11
20

answered Nov 02 '12 at 16:07

lichtfusion

106
1
3

score 3 · Answer 4 · answered Dec 06 '12 at 08:15

3

Oxygen XML has a HUGE FILE support that does validation

http://www.oxygenxml.com/#14.1Huge_XML_Files_Support

answered Dec 06 '12 at 08:15

innovimax

440
5
8

score 2 · Answer 5 · edited Oct 07 '15 at 12:26

2

You can also use the XML Tools Plugin in Nodepad++, it has a function "Check XML Syntax now". It's simple to download and if you don't use Notepad++ already, it's a good reason to start!

edited Oct 07 '15 at 12:26

Ciro Santilli OurBigBook.com

347,512
102
1,199
985

answered Dec 23 '14 at 12:39

Mariaki

31
1

score 1 · Answer 6 · answered Mar 26 '21 at 07:36

1

The following command worked for me xmllint --huge

answered Mar 26 '21 at 07:36

Mobeen Shaikh Mob App Dev.

51
1
4

score 0 · Answer 7 · answered Dec 15 '14 at 17:44

0

Windows Version of XML Starlet:

> xml val <xmlfile.xml>

answered Dec 15 '14 at 17:44

Andre Mesquita

91
1
4

score 0 · Answer 8 · answered Mar 16 '17 at 11:21

0

Liquid Studio Community Edition contains a Large File Editor which can also be used to validate XML files. Its not really got an upper limit on the size of the files you can open Terra-byte files open instantly on low spec machines, and its free.

answered Mar 16 '17 at 11:21

Sprotty

5,676
3
33
52

score 0 · Answer 9 · answered Sep 23 '11 at 14:17

In Java, and I'm sure in other languages, there are solutions for reading in an entire XML file and processing it as a complete DOM, and solutions that process the XML as a stream in an event-driven way. You would want the second kind of solution, which never loads the entire file in memory. See SAX for a Java solution to the problem.

score 0 · Answer 10 · answered Oct 06 '11 at 08:41

You could try the EditiX XML editor.

If you load your document into EditiX and there are problems with the XML, eg. mismatched opening and closing tags, the editor will still load the file and in the bottom right corner of the screen you'll see a number displayed in red eg. a red "5" means there are five errors in the document.

I've not tried a 100mb document but I've done over 15mb and it seemed quite happy.

There's a free version.

score 0 · Answer 11 · answered Oct 06 '11 at 13:41

0

in addition to dj_segfault's comment on phihag's answer, xmlstarlet is fortunately NOT dead. They've just released Version 1.3

If you want a decent commandlinetool that can manipulate xml, xmlstarlet is perfect (and pretty fast).

answered Oct 06 '11 at 13:41

raincrumb

409
4
5

1

IMHO, ***This should be a comment*** – Kiquenet Mar 06 '17 at 20:43

score -1 · Answer 12 · edited May 23 '17 at 12:17

-1

On Windows you can write a simple application based on .net platform. The System.Xml.XmlReader class is capable of validating huge files. An example is in this answer: Validating an XML against referenced XSD in C#

edited May 23 '17 at 12:17

Community

1
1

answered Mar 10 '16 at 22:05

Jarekczek

7,456
3
46
66

Contextually, a "write your own xml validating program in .NET" answer doesn't make much sense when there are already programs for all OSes to do this. – pydsigner Mar 10 '16 at 23:05
Packaging a third party application along with your program is troublesome. Using a built-in operating system library, what I suggest, is much easier. Since stackoverflow is for programmers, I hope such suggestions will always come up. -1 doesn't worry me, but please don't make this answer invisible. – Jarekczek Mar 11 '16 at 06:56
if your first thought was to validate XML in a web browser, as the OP's was, you're almost certainly not trying to ship a solution with a program. Your answer doesn't apply well to the situation. – pydsigner Mar 11 '16 at 07:20

How to validate very large XML files?

12 Answers12

Linked