How to parse JSON in Nim (problem with long integers)?

Question

I'm working on a piece of code in Nim that pulls a JSON object from the Shodan API. Here's the full JSON string from Shodan:

{"city": "Alverca", "region_code": "14", "os": null, "tags": ["self-signed"], "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "area_code": null, "dma_code": null, "last_update": "2019-11-01T17:56:18.470438", "country_code3": "PRT", "country_name": "Portugal", "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "postal_code": "2619-510", "longitude": -9.038600000000002, "country_code": "PT", "ip_str": "85.139.242.90", "latitude": 38.899, "org": "ZON Tv Cabo", "data": [{"_shodan": {"id": null, "options": {}, "ptr": true, "module": "telnet", "crawler": "82488cbcb7dd25da13f728d04775390417d9ee4e"}, "hash": 1329569225, "os": null, "opts": {"telnet": {"will": ["SGA", "STATUS", "ECHO"], "do": ["TTYPE", "TSPEED", "XDISPLOC", "NEW_ENVIRON", "ECHO", "NAWS", "LFLOW"], "dont": [], "wont": []}}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "port": 23, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-11-01T17:56:18.470438", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "\r\nBODET PUNCHING BOARD\r\nLinux/ppc 2.4.20_mvl31-BODET_V1.1B2\r\n\r\nWelcome to 172.17.30.99\r\nFri Nov  1 17:53:58 2019\r\nTech-code: ", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}, {"_shodan": {"id": "7afc2cf1-2b4a-4074-9343-cd576d240364", "options": {}, "ptr": true, "module": "https", "crawler": "0636e1e6dd371760aeaf808ed839236e73a9e74d"}, "hash": 1484578305, "os": null, "tags": ["self-signed"], "opts": {"vulns": [], "heartbleed": "2019/10/31 06:28:03 85.139.242.90:443 - SAFE\n"}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "http": {"html_hash": -468632088, "robots_hash": null, "redirects": [], "securitytxt": null, "title": "", "sitemap_hash": null, "robots": null, "favicon": null, "host": "85.139.242.90", "html": "<!DOCTYPE html>\n<html>\n<head>\n<meta charset=\"UTF-8\">\n<title></title>\n</head>\n<body>\n<script>location.href = \"./home/index.html\";</script>\n</body>\n</html>", "location": "/", "components": {}, "server": null, "sitemap": null, "securitytxt_hash": null}, "port": 443, "ssl": {"dhparams": null, "tlsext": [{"id": 65281, "name": "renegotiation_info"}], "versions": ["TLSv1", "-SSLv2", "-SSLv3", "TLSv1.1", "TLSv1.2", "-TLSv1.3"], "acceptable_cas": [], "cert": {"sig_alg": "sha256WithRSAEncryption", "issued": "20170520021607Z", "expires": "20250806021607Z", "expired": false, "version": 2, "extensions": [{"data": "\\x03\\x02\\x01\\xa6", "name": "keyUsage"}, {"critical": true, "data": "0\\x03\\x01\\x01\\xff", "name": "basicConstraints"}, {"data": "\\x16\\x1fSelf Signed Certificate(System)", "name": "nsComment"}, {"data": "0\\x10\\x82\\x0eXC8f45bb.local", "name": "subjectAltName"}], "fingerprint": {"sha256": "317aadb5fb5ddaf97232cdfb8c4a8da23d2f3f11f7229f028235f6545d08ef1f", "sha1": "3d2a2dcdb25b76b3ddddc740c2e4660ff07009d5"}, "serial": 46474876880932987910930945182556062189, "subject": {"CN": "XC-8F45BB"}, "pubkey": {"type": "rsa", "bits": 2048}, "issuer": {"CN": "XC-8F45BB"}}, "cipher": {"version": "TLSv1/SSLv3", "bits": 256, "name": "AES256-SHA256"}, "chain": ["-----BEGIN CERTIFICATE-----\nMIIDHTCCAgWgAwIBAgIQIva8VyQosXPBl/OnC/WB7TANBgkqhkiG9w0BAQsFADAU\nMRIwEAYDVQQDEwlYQy04RjQ1QkIwHhcNMTcwNTIwMDIxNjA3WhcNMjUwODA2MDIx\nNjA3WjAUMRIwEAYDVQQDEwlYQy04RjQ1QkIwggEiMA0GCSqGSIb3DQEBAQUAA4IB\nDwAwggEKAoIBAQC7ypFTTvDMJ0wYR0LGFOOJf/g6CyRFqJvAmtY0SZKw8EOXC365\n+ajGtJQ0qcsOqmFEFUmC5J0dUsuljbkqECx9cnVtXLtUUQ8pPfTz7Tphz+0zB/KS\nbG7NdrjWbHhVikPLCMrna6cxbI+d1vWA9NoLty02x1fpR8MH9SEqHlO89KbPaDwo\nmw6gjwNS+ImBnF6yzfslUQkcR3J3KGfCrNWsP+mYl7yx4+Awk3wI6vwkUpWmJX+T\nTEUV8rrTSyrHocc7hDYTN/bg5FgUsMLwuuHkEg+JzBTEmdVp0mI0Liq9B/hoVpKz\niX1si/yYkdqKQgNP4SALOqFdmB0+nkqN7rYzAgMBAAGjazBpMAsGA1UdDwQEAwIB\npjAPBgNVHRMBAf8EBTADAQH/MC4GCWCGSAGG+EIBDQQhFh9TZWxmIFNpZ25lZCBD\nZXJ0aWZpY2F0ZShTeXN0ZW0pMBkGA1UdEQQSMBCCDlhDOGY0NWJiLmxvY2FsMA0G\nCSqGSIb3DQEBCwUAA4IBAQCC8CGt0dtiRn6e79Rtjpr383RJdk2d8VfFbQSWj0Ct\nzZUdgktJiQR9+cNKYCoHvJ8E4mm1sb+Wgz2/CrP+7J8ZNRsb8UOabwrREeBvz0wl\nwiIwmrnuCYKZ8AMIEI4f3BmXVSz5baIFTHWWCuS22np5jz8bpYYKLIK4Pc6r+sEf\nfhd7H6YAPEPqAMlC/UTicDmXHKqKbLFDTHNyKiouO3DGFqpNDd4zOWsyDrHkbl91\nVAk6xEPha5Y0QyIlpkfcIAG0e/VxgzMxfiGPSV2kxgaVq+wbNq9T61GsXZ4ZD00L\nj8Q+YW28opH0OZe1h1V8uTytGnKnt295Z1X7hEae04XQ\n-----END CERTIFICATE-----\n"], "alpn": []}, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-10-31T05:27:57.891394", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 153\r\nX-Frame-Options: SAMEORIGIN\r\nX-Content-Type-Options: nosniff\r\nX-XSS-Protection: 1; mode=block\r\n\r\n", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}, {"_shodan": {"id": "921aea7c-4258-40f4-90b0-73088269f39b", "options": {}, "ptr": true, "module": "rsync", "crawler": "339d3eded941e01ca426596e93f3fdf4c9346ccd"}, "product": "rsyncd", "hash": 1601166835, "version": "26", "opts": {}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "os": null, "rsync": {"authentication": false, "modules": {"punching": "Punching home", "root": "Root filesystem"}}, "port": 873, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-10-30T12:11:50.048579", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "@RSYNCD: 26\nroot           \tRoot filesystem\npunching       \tPunching home\n@RSYNCD: EXIT\n", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}, {"_shodan": {"id": null, "options": {}, "ptr": true, "module": "whois", "crawler": "122dd688b363c3b45b0e7582622da1e725444808"}, "hash": -1288910848, "os": null, "opts": {}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "port": 43, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-10-28T18:52:53.093633", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "676478697\n", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}, {"_shodan": {"id": "99dd6dfe-b491-4691-8b62-c8957bb045e2", "options": {}, "ptr": true, "module": "http-simple-new", "crawler": "122dd688b363c3b45b0e7582622da1e725444808"}, "hash": 1240885964, "os": null, "opts": {}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "http": {"html_hash": 1670855880, "robots_hash": null, "redirects": [], "securitytxt": null, "title": "Identification", "sitemap_hash": null, "robots": null, "favicon": null, "host": "85.139.242.90", "html": "<html>\r<head>\r<title>Identification</title>\r<meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>\r<script>\rvar clicable = false;\rdocument.oncontextmenu = menuContextuelHandler;\rfunction menuContextuelHandler(){event.srcElement.click();return false;}\rfunction valider(arg){\rif(clicable){\rdocument.getElementById('nomMethode').value=arg;\rdocument.forms[0].submit();}clicable=false;}\rfunction loadBody(){clicable = true;\rtry{init();}catch(e){};}\rfunction doBlink(elt){\rwindow.setInterval(function(){showHide(elt);}, 1000)}\rfunction showHide(elt){if (elt){\relt.style.visibility = (elt.style.visibility == \"hidden\") ? \"visible\" : \"hidden\";}}\r</script>\r</head>\r<body onload='loadBody();'  onclick='return clicable;' id='corps'>\r<form action=Login.do method=post name='formulaire'>\r<input type='hidden' id='nomMethode' name='nomMethode' value='MainPage'>\r<input type='hidden' id='sessionId' name='sessionId' value='1571047569808'>\r<table style='border:1px solid #000000;width:100%;text-align:center'>\r<tr><td style='width:20%;text-align:left'><b>&nbsp;&nbsp;19/10/2019 10:36:05</b>\r</td><td style='width:60%;text-align:center'><i>\rKelio visio : <b><font color=blue>Kelio Visio Lavradio</font></b><font color=green> 85.139.242.90</font></i></td><td style='width:20%;text-align:right'><img src='bodet.png' align=right></td></tr><tr><td colspan=3 style='width:100%;text-align:center'><h2>Identification\r</h2></td></tr></table>\r<table style=\"width:50%\"><tr><td style=\"width:50%;text-align:center\">\r</td><td style=\"width:50%;text-align:center\">\r</td><td style=\"width:50%;text-align:center\">\r</td></tr></table><br>\r<br><br><br><br><br><br>\r<div style=\"width:60%;text-align:right\">\r<h2><img src=\"password.png\">\r&nbsp;&nbsp;Login:&nbsp;&nbsp;&nbsp;&nbsp;<input type=\"password\" name=\"password\"/>\r<script type='text/javascript'>document.formulaire.password.focus();</script>\r<input type=submit name=\"OK\" value=\"OK\" onClick=javascript:valider(\"MainPage\"); style=\"color:#000000;background-color:#CCCCCC\">\r</h2></div>\r</form>\r<br><br><br><table width=100% border=0><tr><td><h6>\r</h6></td></tr></table>\r</body>\r</html>\r", "location": "/", "components": {}, "server": null, "sitemap": null, "securitytxt_hash": null}, "port": 8008, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-10-19T09:36:09.751093", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 2117\r\n\r\n", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}, {"_shodan": {"id": null, "options": {}, "ptr": true, "module": "line-printer-daemon", "crawler": "f7946cbe2dc20c40fcbcb81ad90aa01731b690ab"}, "hash": -372273874, "os": null, "opts": {}, "ip": 1435234906, "isp": "Nos Comunicacoes, S.A.", "port": 515, "hostnames": ["a85-139-242-90.static.cpe.netcabo.pt"], "location": {"city": "Alverca", "region_code": "14", "area_code": null, "longitude": -9.038600000000002, "country_code3": "PRT", "country_name": "Portugal", "postal_code": "2619-510", "dma_code": null, "country_code": "PT", "latitude": 38.899}, "timestamp": "2019-10-13T12:06:45.139731", "domains": ["netcabo.pt"], "org": "ZON Tv Cabo", "data": "no entries\n", "asn": "AS2860", "transport": "tcp", "ip_str": "85.139.242.90"}], "asn": "AS2860", "ports": [23, 443, 873, 43, 8008, 515]}

All of the code dealing with the API interface is working fine, but I'm having trouble parsing the resulting JSON objects. When the objects are simple, Nim's parser works fine, but when I tried to parse the above JSON, I get an error. The Nim code used to parse the JSON is:

let jsonRsp = parseJson(rspJson)

And, the error produced by the compiler is:

/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(870) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(862) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(820) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(829) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(820) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(820) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(820) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/json.nim(797) parseJson
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/strutils.nim(1107) parseBiggestInt
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/parseutils.nim(447) parseBiggestInt
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/parseutils.nim(423) rawParseInt
/home/nxl4/.choosenim/toolchains/nim-1.0.2/lib/pure/parseutils.nim(401) integerOutOfRangeError
Error: unhandled exception: Parsed integer outside of valid range [ValueError]

I get what the error is saying, that one of the integers is too long for the parser. Since I can't change the data (it is whatever the API spits out), I'm looking to see if anyone has strategies for parsing this kind of JSON data in Nim. Apart from the compiler's complaint, every other JSON validator shows the string as being valid JSON.

score 4 · Accepted Answer · answered Nov 03 '19 at 12:30

This is the offending field:

"serial": 46474876880932987910930945182556062189

which is larger than 2^64. This is tricky, see JSON integers: limit on size

I fed your example JSON through a three different JSON formatters/validators and, while it passes validation, validators also converts the integer above to a floating point value thus losing significant digits in the process. i.e. the formatted/validated result is not the same as the original.

On Safari and Firefox JS console:

JSON.parse("{\"serial\": 46474876880932987910930945182556062189}") {serial: 4.647487688093299e+37}

So some parsers silently convert that large integer to a different number. My immediate reaction to this behaviour is that silently losing precision is worse than reporting an error. I see three issues here:

Parsers silently losing precision.
JSON is emitted without taking into account that even parsers in popular web-browsers cannot parse it without loss of precision.
Nim's JSON parser probably does not implement support for arbitrary large integers.

The first one is the worst one of the three IMO but it is not going away. Shodan API's interoperability could be improved by emitting the serial number as a string instead of a large integer. And you could report the issue on Nim's issue tracker for consideration. Python's JSON parser for instance parses that particular integer without loss of precision.

Thanks very much for the detailed response! The more I researched this, the more it looks like there's an issue with Nim's parser. So, I'm going to open up an issue on the tracker and see if it can be addressed that way. — nxl4, Nov 03 '19 at 12:42

How to parse JSON in Nim (problem with long integers)?

1 Answers1