Python script not appending my csv with parsed xml data, no errors

Question

I am trying to parse xml data from the National Vulnerability Database (NVD) in order to isolate vulnerabilities what are remotely executable (Access Vector = Network).

I have followed other questions here to parse xml and append a csv with the data I need. I am not receiving any error messages, but my cvs is not appended with any data. Is this because my 'findall' method is not finding anything? I am not receiving any error messages. Any help is much appreciated!

The information in the tree that I need is 1) the entry ID attribute and 2) the text in the cvss:access-vector element:

<nvd>
    <entry id="CVE-2018-0001">
        <vuln:cvss>
            <cvss:base_metrics>
                <cvss:access-vector>NETWORK</cvss:access-vector>
            </cvss:base_metrics>
        </vuln:cvss>
    </entry>
</nvd>

Location of data feed: https://nvd.nist.gov/vuln/data-feeds#XML_FEED
Name of specific file used for testing: nvdcve-2.0-2018.xml

Snippet of my xml data (this contains the root, one child and all the descendants of that child):

<?xml version='1.0' encoding='UTF-8'?>
<nvd xmlns:vuln="http://scap.nist.gov/schema/vulnerability/0.4" xmlns:patch="http://scap.nist.gov/schema/patch/0.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:cpe-lang="http://cpe.mitre.org/language/2.0" xmlns="http://scap.nist.gov/schema/feed/vulnerability/2.0" xmlns:scap-core="http://scap.nist.gov/schema/scap-core/0.1" xmlns:cvss="http://scap.nist.gov/schema/cvss-v2/0.2" nvd_xml_version="2.0" pub_date="2018-03-07T03:00:00" xsi:schemaLocation="http://scap.nist.gov/schema/patch/0.1 https://scap.nist.gov/schema/nvd/patch_0.1.xsd http://scap.nist.gov/schema/feed/vulnerability/2.0 https://scap.nist.gov/schema/nvd/nvd-cve-feed_2.0.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd">
  <entry id="CVE-2018-0001">
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d10"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d15"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d20"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d25"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d30"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d35"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d40"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d45"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d50"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d55"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d60"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.1x46:d65"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3x48:d10"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3x48:d15"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3x48:d20"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3x48:d25"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3x48:d30"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x49:d10"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x49:d20"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x49:d30"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d20"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d21"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d25"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d30"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d32"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d33"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d34"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d60"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d61"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d62"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1x53:d63"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r1"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r2"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r3"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r4"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r8"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1:r9"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r1"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r2"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r3"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r4"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r5"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r7"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.2:r8"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1:r1"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:15.1:r2"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r1"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r10"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r2"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r3"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r4"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r5"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r6"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r7"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r8"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:12.3:r9"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d10"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d15"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d16"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d25"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d26"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d27"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d35"/>
        <cpe-lang:fact-ref name="cpe:/o:juniper:junos:14.1x53:d50"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-software-list>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d10</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d15</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d20</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d25</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d30</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d35</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d40</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d45</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d50</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d55</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d60</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.1x46:d65</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r1</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r10</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r2</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r3</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r4</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r5</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r6</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r7</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r8</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3:r9</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3x48:d10</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3x48:d15</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3x48:d20</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3x48:d25</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:12.3x48:d30</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r1</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r2</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r3</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r4</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r8</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1:r9</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d10</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d15</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d16</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d25</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d26</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d27</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d35</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.1x53:d50</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r1</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r2</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r3</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r4</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r5</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r7</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:14.2:r8</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1:r1</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1:r2</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x49:d10</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x49:d20</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x49:d30</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d20</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d21</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d25</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d30</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d32</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d33</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d34</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d60</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d61</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d62</vuln:product>
      <vuln:product>cpe:/o:juniper:junos:15.1x53:d63</vuln:product>
    </vuln:vulnerable-software-list>
    <vuln:cve-id>CVE-2018-0001</vuln:cve-id>
    <vuln:published-datetime>2018-01-10T17:29:00.930-05:00</vuln:published-datetime>
    <vuln:last-modified-datetime>2018-02-22T21:29:02.140-05:00</vuln:last-modified-datetime>
    <vuln:cvss>
      <cvss:base_metrics>
        <cvss:score>7.5</cvss:score>
        <cvss:access-vector>NETWORK</cvss:access-vector>
        <cvss:access-complexity>LOW</cvss:access-complexity>
        <cvss:authentication>NONE</cvss:authentication>
        <cvss:confidentiality-impact>PARTIAL</cvss:confidentiality-impact>
        <cvss:integrity-impact>PARTIAL</cvss:integrity-impact>
        <cvss:availability-impact>PARTIAL</cvss:availability-impact>
        <cvss:source>http://nvd.nist.gov</cvss:source>
        <cvss:generated-on-datetime>2018-01-30T17:21:59.327-05:00</cvss:generated-on-datetime>
      </cvss:base_metrics>
    </vuln:cvss>
    <vuln:cwe id="CWE-416"/>
    <vuln:references xml:lang="en" reference_type="UNKNOWN">
      <vuln:source>BID</vuln:source>
      <vuln:reference href="http://www.securityfocus.com/bid/103092" xml:lang="en">103092</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>SECTRACK</vuln:source>
      <vuln:reference href="http://www.securitytracker.com/id/1040180" xml:lang="en">1040180</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>CONFIRM</vuln:source>
      <vuln:reference href="https://kb.juniper.net/JSA10828" xml:lang="en">https://kb.juniper.net/JSA10828</vuln:reference>
    </vuln:references>
    <vuln:summary>A remote, unauthenticated attacker may be able to execute code by exploiting a use-after-free defect found in older versions of PHP through injection of crafted data via specific PHP URLs within the context of the J-Web process. Affected releases are Juniper Networks Junos OS: 12.1X46 versions prior to 12.1X46-D67; 12.3 versions prior to 12.3R12-S5; 12.3X48 versions prior to 12.3X48-D35; 14.1 versions prior to 14.1R8-S5, 14.1R9; 14.1X53 versions prior to 14.1X53-D44, 14.1X53-D50; 14.2 versions prior to 14.2R7-S7, 14.2R8; 15.1 versions prior to 15.1R3; 15.1X49 versions prior to 15.1X49-D30; 15.1X53 versions prior to 15.1X53-D70.</vuln:summary>
  </entry>
</nvd>

Python:

#Some sources I'm trying to follow
#https://www.youtube.com/watch?v=OdQ7xdXxeUA
#http://blog.appliedinformaticsinc.com/how-to-parse-and-convert-xml-to-csv-using-python/

#Step1: import lib
import csv
import xml.etree.cElementTree as ET

#Step 2: Parse xml document and get root
tree = ET.parse('nvdcve-2.0-2018.xml')
root = tree.getroot()

#Step 3: Create and open csv file
xml_data_to_csv = open('Out1.csv','w')

#Step 4: Create a list for the column headers
list_head=[]

#Step 5: Create variable to write to csv
Csv_writer=csv.writer(xml_data_to_csv)

#Step 6: Loop for each node
count=0
for element in root.findall('entry'):
    List_nodes=[]

    #Get head by tag
    if count == 0:
        av = element.findall('.//access-vector').tag
        list_head.append(av)

        count=+1

    #Write List_nodes to csv
    Csv_writer.writerow(List_nodes)

#Close csv file
xml_data_to_csv.close()

score 0 · Answer 1 · answered Mar 26 '18 at 18:07

0

This is indeed a bit confusing but it is possible, below code should get you the items you need (csv related code removed).

import xml.etree.ElementTree as ET
root = ET.parse('nvdcve-2.0-2018.xml').getroot()

cvss = {'cvss':'http://scap.nist.gov/schema/cvss-v2/0.2'}
namespace = {"namespace":"http://scap.nist.gov/schema/feed/vulnerability/2.0"}

for entry in root.findall('.//namespace:entry',namespace):
    entry_id = entry.get('id')
    print(entry_id)
    for access_vector in entry.findall('.//cvss:access-vector', cvss):
        value = access_vector.text
        print(value)

prints

CVE-2018-0001
NETWORK

answered Mar 26 '18 at 18:07

johnII

1,423
1
14
20

Thanks! I will try this as well as the answer below. – lanceDamage Mar 26 '18 at 19:24
This answered my question regarding isolating the data. In this answer and the one below, my csv is still not being appended with the data. – lanceDamage Mar 26 '18 at 19:48

score 0 · Answer 2 · answered Mar 26 '18 at 18:09

You are right in that it is your findall() command. I believe your problem is that the xml document has numerous namespaces prefixes: vuln, patch, cpe, scap, cvss.

xmlns:vuln="http://scap.nist.gov/schema/vulnerability/0.4"
xmlns:patch="http://scap.nist.gov/schema/patch/0.1" 
xmlns:cpe-lang="http://cpe.mitre.org/language/2.0"
xmlns:scap-core="http://scap.nist.gov/schema/scap-core/0.1" 
xmlns:cvss="http://scap.nist.gov/schema/cvss-v2/0.2"

To have your search expression work properly, you need to tell if to use the namespaces in the search. This question goes into the details. You want to modify your code something like this.

for element in root.findall('entry'):
    List_nodes=[]

    #Get head by tag
    if count == 0:
        namespaces = {'cvss':"http://scap.nist.gov/schema/cvss-v2/0.2"} 
        # add more as needed
        av = element.findall('.//cvss:access-vector',namespaces).tag
        list_head.append(av)

Thanks! I like the efficient answer here integrating with my loop, but my csv is still not being appended. I'm wondering if I should be using iterfind() instead of findall(). — lanceDamage, Mar 26 '18 at 19:44
This answered my question as well regarding how to isolate the data. The csv is still not being written. I now think that perhaps something is wrong with the way I am creating and appending the csv. I will write another question for that specifically. I would upvote both of these, but I don't have enough points yet. — lanceDamage, Mar 26 '18 at 20:10

Python script not appending my csv with parsed xml data, no errors

2 Answers2