1

I've downloaded some JSON data from Shodan, and only want to retain some fields from it. To explore what I want, I'm running the following, which works-

shodan parse --fields ip,port --separator , "data.json.gz"

However, I now want to output/ export the data; I'm trying to run the following -

shodan parse --fields ip,port -O "data_processed.json.gz" "data.json.gz"

It's requiring me to specify a filter parameter, which I don't need. If I do add an empty filter as so, it tells me data_processes.json.gz doesn't exist.

shodan parse --fields ip,port -f -O "data_processed.json.gz" "data.json.gz"

I'm a bit stumped on how to export only certain fields of my data; how do I go about doing so?

JsDart
  • 183
  • 13

1 Answers1

2

If you only want to output those 2 properties then you can simply pipe them to a file:

shodan parse --fields ip,port --separator , data.json.gz > data_processed.csv

A few things to keep in mind:

  1. You probably want to export the ip_str property as it's a more user-friendly version of the IP address. The ip property is a numeric version of the IP address and aimed at users storing the information in a database.
  2. You can convert your data file into Excel or CSV format using the shodan convert command. For example: shodan convert data.json.gz csv See here for a quick guide: https://help.shodan.io/guides/how-to-convert-to-excel
achillean
  • 520
  • 4
  • 10
  • That worked, thanks! I also appreciate the heads up about the csv- however, I'm losing data on vulnerabilities when I do convert it; but that's probably to do with how the data is inherently stored within the files. My team and I are still playing around with the data, but yeah; thanks once again! – JsDart Mar 02 '19 at 00:40
  • Make sure you're running the latest version of the CLI as that was a bug we recently fixed (https://github.com/achillean/shodan-python/issues/85). If you export to CSV using the latest version of the library/ CLI you will see the vulnerability information as well. However, you will still lose a few other properties as it's difficult to flatten all the information that's stored in JSON. – achillean Mar 02 '19 at 01:58