Converting nested JSON array to simple JSON in Python

Question

I have a JSON file with nested JSON objects. I want to index this JSON file into my Elasticsearch via Logstash, but Elasticsearch doesn't support nested JSON (yet). Is there a way in python to convert these nested objects to simple JSON, that is searchable?

The JSON I'm using is created by a python script that uses Nmap and determine hosts on a network. At this moment the ports array cannot be search, which is the whole purpose of my project. This is because a JSON array is indexed as one field into the database instead their own JSON fields.

Sample:

{
    "host": {
        "status": {
            "_state": "up",
            "_reason": "echo-reply",
            "_reason_ttl": "60"
        },
        "address": {
            "_addr": "xxx.xxx.xxx.xxx",
            "_addrtype": "ipv4"
        },
        "hostnames": {
            "hostname": {
                "_name": "xxxxxx.com",
                "_type": "PTR"
            }
        },
        "ports": {
            "extraports": {
                "extrareasons": {
                    "_reason": "no-responses",
                    "_count": "994"
                },
                "_state": "filtered",
                "_count": "994"
            },
            "port": [
                {
                    "state": {
                        "_state": "closed",
                        "_reason": "reset",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "_name": "ftp-data",
                        "_method": "table",
                        "_conf": "3"
                    },
                    "_protocol": "tcp",
                    "_portid": "20"
                },
                {
                    "state": {
                        "_state": "open",
                        "_reason": "syn-ack",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "cpe": "cpe:/a:vsftpd:vsftpd",
                        "_name": "ftp",
                        "_product": "vsftpd",
                        "_version": "2.0.8 or later",
                        "_hostname": "Welcome",
                        "_method": "probed",
                        "_conf": "10"
                    },
                    "_protocol": "tcp",
                    "_portid": "21"
                },
                {
                    "state": {
                        "_state": "open",
                        "_reason": "syn-ack",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "cpe": "cpe:/a:openbsd:openssh:5.3",
                        "_name": "ssh",
                        "_product": "OpenSSH",
                        "_version": "5.3",
                        "_extrainfo": "protocol 2.0",
                        "_method": "probed",
                        "_conf": "10"
                    },
                    "_protocol": "tcp",
                    "_portid": "22"
                },
                {
                    "state": {
                        "_state": "open",
                        "_reason": "syn-ack",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "cpe": "cpe:/a:apache:http_server:2.2.15",
                        "_name": "http",
                        "_product": "Apache httpd",
                        "_version": "2.2.15",
                        "_extrainfo": "(CentOS)",
                        "_method": "probed",
                        "_conf": "10"
                    },
                    "_protocol": "tcp",
                    "_portid": "80"
                },
                {
                    "state": {
                        "_state": "open",
                        "_reason": "syn-ack",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "cpe": "cpe:/a:apache:http_server:2.2.15",
                        "_name": "http",
                        "_product": "Apache httpd",
                        "_version": "2.2.15",
                        "_extrainfo": "(CentOS)",
                        "_tunnel": "ssl",
                        "_method": "probed",
                        "_conf": "10"
                    },
                    "_protocol": "tcp",
                    "_portid": "443"
                },
                {
                    "state": {
                        "_state": "open",
                        "_reason": "syn-ack",
                        "_reason_ttl": "60"
                    },
                    "service": {
                        "_name": "rsync",
                        "_extrainfo": "protocol version 31",
                        "_method": "probed",
                        "_conf": "10"
                    },
                    "_protocol": "tcp",
                    "_portid": "873"
                }
            ]
        },
        "times": {
            "_srtt": "1392",
            "_rttvar": "258",
            "_to": "50000"
        },
        "_starttime": "1527320392",
        "_endtime": "1527320668"
    }
}

What's the expected output? What have you tried, and what exactly is the problem with it? — jonrsharpe, May 26 '18 at 14:06
Try https://stackoverflow.com/questions/6027558/flatten-nested-python-dictionaries-compressing-keys — OneCricketeer, May 26 '18 at 14:09

score 0 · Answer 1 · answered May 26 '18 at 14:10

0

Dont't use nested keyword in elasticsearch mapping

Better way you could do is, defined entire json structure into mapping

Then search in elasticsearch you may get it.

answered May 26 '18 at 14:10

Narendra Prasath

1,501
1
10
20

Converting nested JSON array to simple JSON in Python

1 Answers1