Extraction of data from a simple XML file

Question

I've a XML file with the contents:

<?xml version="1.0" encoding="utf-8"?>
<job xmlns="http://www.sample.com/">programming</job>

I need a way to extract what is in the <job..> </job> tags, programmin in this case. This should be done on linux command prompt, using grep/sed/awk.

Answer 1

Do you really have to use only those tools? They're not designed for XML processing, and although it's possible to get something that works OK most of the time, it will fail on edge cases, like encoding, line breaks, etc.

I recommend xml_grep:

xml_grep 'job' jobs.xml --text_only

Which gives the output:

programming

On ubuntu/debian, xml_grep is in the xml-twig-tools package.

Answer 2

 grep '<job' file_name | cut -f2 -d">"|cut -f1 -d"<"

Answer 3

Using xmlstarlet:

echo '<job xmlns="http://www.sample.com/">programming</job>' | \
   xmlstarlet sel -N var="http://www.sample.com/" -t -m "//var:job" -v '.'

Answer 4

Please don't use line and regex based parsing on XML. It is a bad idea. You can have semantically identical XML with different formatting, and regex and line based parsing simply cannot cope with it.

Things like unary tags and variable line wrapping - these snippets 'say' the same thing:

<root>
  <sometag val1="fish" val2="carrot" val3="narf"></sometag>
</root>


<root>
  <sometag
      val1="fish"
      val2="carrot"
      val3="narf"></sometag>
</root>

<root
><sometag
val1="fish"
val2="carrot"
val3="narf"
></sometag></root>

<root><sometag val1="fish" val2="carrot" val3="narf"/></root>

Hopefully this makes it clear why making a regex/line based parser is difficult? Fortunately, you don't need to. Many scripting languages have at least one, sometimes more parser options.

As a previous poster has alluded to - xml_grep is available. That's actually a tool based off the XML::Twig perl library. However what it does is use 'xpath expressions' to find something, and differentiates between document structure, attributes and 'content'.

E.g.:

xml_grep 'job' jobs.xml --text_only

However in the interest of making better answers, here's a couple of examples of 'roll your own' based on your source data:

First way:

Use twig handlers that catches elements of a particular type and acts on them. The advantage of doing it this way is it parses the XML 'as you go', and lets you modify it in flight if you need to. This is particularly useful for discarding 'processed' XML when you're working with large files, using purge or flush:

#!/usr/bin/perl

use strict;
use warnings;

use XML::Twig;

XML::Twig->new(
    twig_handlers => {
        'job' => sub { print $_ ->text }
    }
    )->parse( <> );

Which will use <> to take input (piped in, or specified via commandline ./myscript somefile.xml) and process it - each job element, it'll extract and print any text associated. (You might want print $_ -> text,"\n" to insert a linefeed).

Because it's matching on 'job' elements, it'll also match on nested job elements:

<job>programming
    <job>anotherjob</job>
</job>

Will match twice, but print some of the output twice too. You can however, match on /job instead if you prefer. Usefully - this lets you e.g. print and delete an element or copy and paste one modifying the XML structure.

Alternatively - parse first, and 'print' based on structure:

my $twig = XML::Twig->new( )->parse( <> );
print $twig -> root -> text;

As job is your root element, all we need do is print the text of it.

But we can be a bit more discerning, and look for job or /job and print that specifically instead:

my $twig = XML::Twig->new( )->parse( <> );
print $twig -> findnodes('/job',0)->text;

You can use XML::Twigs pretty_print option to reformat your XML too:

XML::Twig->new( 'pretty_print' => 'indented_a' )->parse( <> ) -> print;

There's a variety of output format options, but for simpler XML (like yours) most will look pretty similar.

Answer 5

just use awk, no need other external tools. Below works if your desired tags appears in multitine.

$ cat file
test
<job xmlns="http://www.sample.com/">programming</job>
<job xmlns="http://www.sample.com/">
programming</job>

$ awk -vRS="</job>" '{gsub(/.*<job.*>/,"");print}' file
programming

programming

Answer 6

Using sed command:

Example:

$ cat file.xml
<note>
        <to>Tove</to>
                <from>Jani</from>
                <heading>Reminder</heading>
        <body>Don't forget me this weekend!</body>
</note>

$ cat file.xml | sed -ne '/<heading>/s#\s*<[^>]*>\s*##gp'
Reminder

Explanation:

cat file.xml | sed -ne '/<pattern_to_find>/s#\s*<[^>]*>\s*##gp'

n - suppress printing all lines
e - script

/<pattern_to_find>/ - finds lines that contain specified pattern what could be e.g.<heading>

next is substitution part s///pthat removes everything except desired value where / is replaced with # for better readability:

s#\s*<[^>]*>\s*##gp
\s* - includes white-spaces if exist (same at the end)
<[^>]*> represents <xml_tag> as non-greedy regex alternative cause <.*?> does not work for sed
g - substitutes everything e.g. closing xml </xml_tag> tag

Answer 7

Assuming same line, input from stdin:

sed -ne '/<\/job>/ { s/<[^>]*>\(.*\)<\/job>/\1/; p }'

notes: -n stops it outputting everything automatically; -e means it's a one-liner (aot a script) /<\/job> acts like a grep; s strips the opentag + attributes and endtag; ; is a new statement; p prints; {} makes the grep apply to both statements, as one.

Answer 8

How about:

cat a.xml | grep '<job' | cut -d '>' -f 2 | cut -d '<' -f 1

Answer 9

A bit late to the show.

xmlcutty cuts out nodes from XML:

$ cat file.xml
<?xml version="1.0" encoding="utf-8"?>
<job xmlns="http://www.sample.com/">programming</job>
<job xmlns="http://www.sample.com/">designing</job>
<job xmlns="http://www.sample.com/">managing</job>
<job xmlns="http://www.sample.com/">teaching</job>

The path argument names the path to the element you want to cut out. In this case, since we are not interested in the tags at all, we rename the tag to \n, so we get a nice list:

$ xmlcutty -path /job -rename '\n' file.xml
programming
designing
managing
teaching

Note, that the XML was not valid to begin with (no root element). xmlcutty can work with slightly broken XML, too.

Answer 10

yourxmlfile.xml

<item> 
  <title>15:54:57 - George:</title>
  <description>Diane DeConn? You saw Diane DeConn!</description> 
</item> 
<item> 
  <title>15:55:17 - Jerry:</title> 
  <description>Something huh?</description>
</item>

grep 'title' yourxmlfile.xml

  <title>15:54:57 - George:</title>
  <title>15:55:17 - Jerry:</title>

grep 'title' yourxmlfile.xml | awk -F">" '{print $2}'

  15:54:57 - George:</title
  15:55:17 - Jerry:</title

grep 'title' yourxmlfile.xml | awk -F">" '{print $2}' | awk -F"<" '{print $1}'

  15:54:57 - George:
  15:55:17 - Jerry:

Answer 11

Use xml2 to use line-oriented tools with XML

Example:

xml2 <foo.xml | sed -n 's#.*/job=##p'

Output:

programming

Where to get xml2

The xml2 command often can be installed using your system's package manager (for example, apt install xml2). It can also be downloaded from https://github.com/cryptorick/xml2.

xml2 Documentation

• Introduction
• Reference
• Examples

Why xml2 is needed

Naive use of grep, sed, and awk is brittle. Consider the following XML file which would break such solutions:

<?xml version="1.0" encoding="utf-8"?>
<root>
  <job xmlns=
       "http://www.people.com/"
       val1="fish" val2="carrot"
       val3="narf"
       >teaching<!-- A comment about the </job> tag --></job>
</root>

Why not xml_grep and friends

Most of the robust answers to this question suggest using tools, such as xml_grep, which search using the XPath syntax. XPath is designed especially for searching XML documents and is a fine solution if you already know XPath or don't know anything else.

However, if you just need to search XML files and know the standard UNIX tools, it may not be worth your time to learn XPath which has limited utility beyond XML. Fortunately, xml2 provides an easy way to leverage the power of UNIX and regular expressions by converting the XML syntax to a "flat file" format in which each record is on a single line.

Example xml2 output

For example, running xml2 < foo.xml on the following file:

<?xml version="1.0" encoding="utf-8"?>
<root>
  <job xmlns="http://www.sample.com/">programming</job>
  <job xmlns="http://www.supple.com/">designing</job>
  <job xmlns="http://www.simple.com/">managing</job>
  <job xmlns=
       "http://www.people.com/"
       val1="fish" val2="carrot"
       val3="narf"
       >teaching<!-- A comment about the </job> tag --></job>
</root>

would output the following text file:

/root/job/@xmlns=http://www.sample.com/
/root/job=programming
/root/job
/root/job/@xmlns=http://www.supple.com/
/root/job=designing
/root/job
/root/job/@xmlns=http://www.simple.com/
/root/job=managing
/root/job
/root/job/@xmlns=http://www.people.com/
/root/job/@val1=fish
/root/job/@val2=carrot
/root/job/@val3=narf
/root/job=teaching
/root/job/!= A comment about the </job> tag 

As you can see, the peculiarities of the XML file have been made normalized and the output can be easily parsed by grep, sed, or awk. In particular, the command xml2 <foo.xml | sed -n 's#.*/job=##p' outputs:

programming
designing
managing
teaching

Side note: I added a <root> node to make the file valid XML although xml2 works fine either way.

Limitations

While xml2 is very handy for search and replace, if you are going to be doing a lot of work with XML, you'll probably want to learn XPath and XSLT which can perform more powerful hierarchical transformations.