0

I have a dataset from HMDB the Saliva Metabolites data.

This data is an XML file. What I want to do is to convert this XML file to a list of lists (nested lists) in Python, however, I don't want all the nodes in the list.

EDITED: AND THIS IS EXAMPLE OF PARTIAL DATA FOR ONE METABOLITE

<?xml version="1.0" encoding="UTF-8"?>
<hmdb xmlns="http://www.hmdb.ca">
<metabolite>
  <version>4.0</version>
  <creation_date>2005-11-16 15:48:42 UTC</creation_date>
  <update_date>2019-01-11 19:13:56 UTC</update_date>
  <accession>HMDB0000001</accession>
  <status>quantified</status>
  <secondary_accessions>
    <accession>HMDB00001</accession>
    <accession>HMDB0004935</accession>
    <accession>HMDB0006703</accession>
    <accession>HMDB0006704</accession>
    <accession>HMDB04935</accession>
    <accession>HMDB06703</accession>
    <accession>HMDB06704</accession>
  </secondary_accessions>
  <name>1-Methylhistidine</name>
  <cs_description>1-Methylhistidine, also known as 1-mhis, belongs to the class of organic compounds known as histidine and derivatives. Histidine and derivatives are compounds containing cysteine or a derivative thereof resulting from reaction of cysteine at the amino group or the carboxy group, or from the replacement of any hydrogen of glycine by a heteroatom. 1-Methylhistidine has been found in human muscle and skeletal muscle tissues, and has also been detected in most biofluids, including cerebrospinal fluid, saliva, blood, and feces. Within the cell, 1-methylhistidine is primarily located in the cytoplasm. 1-Methylhistidine participates in a number of enzymatic reactions. In particular, 1-Methylhistidine and Beta-alanine can be converted into anserine; which is catalyzed by the enzyme carnosine synthase 1. In addition, Beta-Alanine and 1-methylhistidine can be biosynthesized from anserine; which is mediated by the enzyme cytosolic non-specific dipeptidase. In humans, 1-methylhistidine is involved in the histidine metabolism pathway. 1-Methylhistidine is also involved in the metabolic disorder called the histidinemia pathway.</cs_description>
  <description>One-methylhistidine (1-MHis) is derived mainly from the anserine of dietary flesh sources, especially poultry. The enzyme, carnosinase, splits anserine into b-alanine and 1-MHis. High levels of 1-MHis tend to inhibit the enzyme carnosinase and increase anserine levels. Conversely, genetic variants with deficient carnosinase activity in plasma show increased 1-MHis excretions when they consume a high meat diet. Reduced serum carnosinase activity is also found in patients with Parkinson's disease and multiple sclerosis and patients following a cerebrovascular accident. Vitamin E deficiency can lead to 1-methylhistidinuria from increased oxidative effects in skeletal muscle. 1-Methylhistidine is a biomarker for the consumption of meat, especially red meat.</description>
  <synonyms>
    <synonym>(2S)-2-amino-3-(1-Methyl-1H-imidazol-4-yl)propanoic acid</synonym>
    <synonym>1-Methylhistidine</synonym>
    <synonym>Pi-methylhistidine</synonym>
    <synonym>(2S)-2-amino-3-(1-Methyl-1H-imidazol-4-yl)propanoate</synonym>
    <synonym>1 Methylhistidine</synonym>
    <synonym>1-Methyl histidine</synonym>
  </synonyms>
  <chemical_formula>C7H11N3O2</chemical_formula>
  <smiles>CN1C=NC(C[C@H](N)C(O)=O)=C1</smiles>
  <inchikey>BRMWTNUJHUMWMS-LURJTMIESA-N</inchikey>
<diseases>
    <disease>
      <name>Kidney disease</name>
      <omim_id/>
      <references>
        <reference>
          <reference_text>McGregor DO, Dellow WJ, Lever M, George PM, Robson RA, Chambers ST: Dimethylglycine accumulates in uremia and predicts elevated plasma homocysteine concentrations. Kidney Int. 2001 Jun;59(6):2267-72.</reference_text>
          <pubmed_id>11380830</pubmed_id>
        </reference>
        <reference>
          <reference_text>Ehrenpreis ED, Salvino M, Craig RM: Improving the serum D-xylose test for the identification of patients with small intestinal malabsorption. J Clin Gastroenterol. 2001 Jul;33(1):36-40.</reference_text>
          <pubmed_id>11418788</pubmed_id>
        </reference>
        <reference>
         </reference>
      </references>
    </disease>
    <disease>

Importing the file:

import xml.etree.ElementTree as et

data1 = et.parse('D:/path/to/Tal/my/HMDB/DataSets/saliva_metabolites/saliva_metabolites.xml')
root = data1.getroot()

Now, not sure how to select specific nodes. Meaning, my goal is to create a list of metabolites and each metabolite from the list will contain a list of nodes (say, <accession>, <name>, <synonyms> and <diseases_name>)

In turn, those elements will contain another list (say, inside <synonyms> there will be a list of values names, or inside <diseases_name> will be the list of names of diseases and each disease will contain a list of pub_id values).

# To access the 4'th node of the first metabolit

>>  root[0][3].text
'HMDB0000001'

where root[0][3] represents the <accession> node.

Tried to run loop with print so i'll understand the output of the loop but recieved list of None

for node in root:
    print(node.find('accession'))
None
None
None
None
None
.
.
.

Also tried

>> root.findall('./metabolite/accession')
[]

But received empty brackets

for list of synonyms of the first metbolite i tried:

>> root[0][9].text
'\n    '

# This gave the first value of synonyms
root[0][9][0].text
'\n    '

I used those questions to find an answer:

  1. How do I parse XML in Python?
  2. how to create a list of elements from an XML file in python
  3. Python: XML file to pandas dataframe
  4. Convert XML into Lists of Tags and Values with Python
  5. Generating nested lists from XML doc

Any hints, ideas would be a help, thank you for your time

TaL
  • 173
  • 2
  • 15

1 Answers1

0

You are ignoring the namespace in the XML.

<hmdb xmlns="http://www.hmdb.ca">

means that there is no <hmdb> element. There is a <hmdb> in the http://www.hmdb.ca namespace. And since it's the default namespace for this element, all descendant elements are in the same namespace, unless they override that.

So this

root.findall('./metabolite/accession')

will not return anything because you're searching in the wrong namespace.

Let's search in the http://www.hmdb.ca namespace by giving it the handle h, for convenience:

ns = {
    "h": "http://www.hmdb.ca"
}

accession = root.findall('./h:metabolite/h:accession', ns)
print(accession)

This finds one element (see how it explicitly denotes the namespace when you print it):

[<Element '{http://www.hmdb.ca}accession' at 0x03E6E7B0>]

You can use the same explicit syntax in ElementTree, but it gets unwieldy very quickly:

t.findall('./{http://www.hmdb.ca}metabolite/{http://www.hmdb.ca}accession')

The shorter (and standard) syntax with the prefix: is a lot nicer to work with.

Tomalak
  • 332,285
  • 67
  • 532
  • 628
  • Or use a wildcard for the namespace (Python 3.8): https://stackoverflow.com/a/62117710/407651 – mzjn Aug 26 '20 at 07:47
  • ...that's the dirty way of dealing with it. I prefer to not be sloppy. – Tomalak Aug 26 '20 at 08:17
  • @Tomalak! Thank you for your time! When ` print(accession)` I don't see the *value* of accession but its location - `[]` how come? And another question if ok, How can I move forward from in creating lists of lists ( Nested lists) ? Thanks again – TaL Aug 26 '20 at 08:18
  • @TaL `.findall()` gives you a list of elements (in this case with length 1). Each element has several properties, the `text` it contains is only one of them, so you have to take one more step to extract it. I think you will find your own way of nesting loops so you get the nested lists you have in mind, you don't need me to spell that out for you. – Tomalak Aug 26 '20 at 08:35