Ways to get external data into own mongodb (different structures)?

Question

I'm having an external Export of XML data which I want to have in my mongoDB. Because the data in the export doesnt fit my needs when it comes to the structure, I'd like to kind of "map" it.

Exmaple of kind of the data structure I'm getting from the XML:

<?xml version="1.0" encoding="utf-8" ?>
<ogr:FeatureCollection
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation=""
     xmlns:ogr="http://ogr.maptools.org/"
     xmlns:gml="http://www.opengis.net/gml">
  <gml:boundedBy>
    <gml:Box>
      <gml:coord><gml:X>X.XXX</gml:X><gml:Y>Y.YYY</gml:Y></gml:coord>
      <gml:coord><gml:X>X.XXX</gml:X><gml:Y>Y.YYY</gml:Y></gml:coord>
    </gml:Box>
  </gml:boundedBy>                                                                               
  <gml:featureMember>
    <ogr:POI_TypeXYZ fid="POI_XYZ">
      <ogr:geometryProperty><gml:Point srsName="XYZ"><gml:coordinates>SAMPLE_DATA</gml:coordinates></gml:Point></ogr:geometryProperty>
      <ogr:ID>SAMPLE_DATA</ogr:ID>
      <ogr:TITLE>SAMPLE_DATA</ogr:TITLE>
      <ogr:STREET>SAMPLE_DATA</ogr:STREET>
      <ogr:NUM>SAMPLE_DATA</ogr:NUM>
      <ogr:PHONE>SAMPLE_DATA</ogr:PHONE>
    </ogr:POI_TypeXYZ>
  </gml:featureMember>
  <gml:featureMember>
    <ogr:POI_TypeXYZ fid="POI_XYZ">
      <ogr:geometryProperty><gml:Point srsName="XYZ"><gml:coordinates>SAMPLE_DATA</gml:coordinates></gml:Point></ogr:geometryProperty>
      <ogr:ID>SAMPLE_DATA</ogr:ID>
      <ogr:TITLE>SAMPLE_DATA</ogr:TITLE>
      <ogr:STREET>SAMPLE_DATA</ogr:STREET>
      <ogr:NUM>SAMPLE_DATA</ogr:NUM>
      <ogr:PHONE>SAMPLE_DATA</ogr:PHONE>
    </ogr:POI_TypeXYZ>
  </gml:featureMember>
</ogr:FeatureCollection>

And for example I'd rather have different names for the fields etc. and have full control over that data.

Are there any tools that could help? Or what is the general strategy to handle such things?

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

0

It looks like a 3 step process:

Convert xml to JSON.
Change the name of the fields. refer here to know how to change the key of JSON object.(You have full control over data here)
Convert JSON object to BSON and store it into Mongodb.

import json

data = json.loads(document)

mongocollection.insert(data)

edited Jun 20 '20 at 09:12

Community

1
1

answered Dec 05 '17 at 15:15

Anirudh Bagri

2,346
1
21
33

I tried converting my JSON to BSON (just to test it with an online coverter) but how do I import a BSON file? mongoimport seems not to work. – SVARTBERG Dec 06 '17 at 11:47
what do you mean by import a BSON file? what are you trying to import? – Anirudh Bagri Dec 06 '17 at 11:56
Not that familiar with MongoDB yet, how am I supposed to store BSON (and how can I convert it from JSON) into my MongoDB? – SVARTBERG Dec 06 '17 at 15:05
You don't need to worry about changing JSON to Bson, MongoDriver will take care of it. Check my edit. If this solves your problem, please mark it as correct answer – Anirudh Bagri Dec 06 '17 at 15:22
Gonna try asap. Thx. Question: Is "import json" == cli -> mongoimport? And what do you mean with data = json.loads(document)? – SVARTBERG Dec 06 '17 at 15:27
document is the JSON string. json.loads() will convert the json document into Python object – Anirudh Bagri Dec 06 '17 at 15:51
No, import json is to use json libraries. For mongo you need to install PyMongo. Follow this document. (https://docs.mongodb.com/getting-started/python/client/) – Anirudh Bagri Dec 06 '17 at 15:52
So is this still the same if I use nodejs to import? I'm not sure if I get what you say. – SVARTBERG Dec 07 '17 at 11:37
You need to add libraries to use JSON and Mongo. JavaScript has library for both. Please mark answer as correct if it helped you. – Anirudh Bagri Dec 07 '17 at 11:46
Yeah thats clear. When I asked, I just used JS to convert the XML data to a JSON file I needed an wrote it to disk (output.json). And then I tried to import that file via mongoimport.exe. I'd really like to accept your answer but to be honest it rather confused me a bit. Maybe we talked past each other. – SVARTBERG Dec 07 '17 at 11:54
Would it be helpful if I therefore update my question? Because currently I have the import ready but overwriting it currently is a problem. – SVARTBERG Dec 07 '17 at 12:00
So I asked a new question because the circumstances changed a bit: https://stackoverflow.com/questions/47697822/correctly-inserting-and-or-updating-many-datasets-to-mongodb-using-mongoose – SVARTBERG Dec 07 '17 at 14:59

Ways to get external data into own mongodb (different structures)?

1 Answers1