14

I have a problem I'd like to solve to not have to spend a lot of manual work to analyze as an alternative.

I have 2 JSON objects (returned from different web service API or HTTP responses). There is intersecting data between the 2 JSON objects, and they share similar JSON structure, but not identical. One JSON (the smaller one) is like a subset of the bigger JSON object.

I want to find all the interesecting data between the two objects. Actually, I'm more interested in the shared parameters/properties within the object, not really the actual values of the parameters/properties of each object. Because I want to eventually use data from one JSON output to construct the other JSON as input to an API call. Unfortunately, I don't have the documentation that defines the JSON for each API. :(

What makes this tougher is the JSON objects are huge. One spans a page if you print it out via Windows Notepad. The other spans 37 pages. The APIs return the JSON output compressed as a single line. Normal text compare doesn't do much, I'd have to reformat manually or w/ script to break up object w/ newlines, etc. for a text compare to work well. Tried with Beyond Compare tool.

I could do manual search/grep but that's a pain to cycle through all the parameters inside the smaller JSON. Could write code to do it but I'd also have to spend time to do that, and test if the code works also. Or maybe there's some ready made code already for that...

Or can look for JSON diff type tools. Searched for some. Came across these:

https://github.com/samsonjs/json-diff or https://tlrobinson.net/projects/javascript-fun/jsondiff

https://github.com/andreyvit/json-diff

both failed to do what I wanted. Presumably the JSON is either too complex or too large to process.

Any thoughts on best solution? Or might the best solution for now be manual analysis w/ grep for each parameter/property?

In terms of a code solution, any language will do. I just need a parser or diff tool that will do what I want.

Sorry, can't share the JSON data structure with you either, it may be considered confidential.

John Vandenberg
  • 474
  • 6
  • 16
David
  • 3,223
  • 3
  • 29
  • 41
  • You aware that you can easily pretty-print the JSON data with newlines and indentation? – I Hate Lazy Oct 09 '12 at 02:02
  • FWIW: 37 display pages of JSON text is far from "huge"; I wouldn't event consider it "large", but rather just "not tiny". – Lawrence Dol Oct 09 '12 at 02:57
  • @user1689607, how do you pretty-print JSON data? I'm not a ajax/jquery/js web developer, haven't done web dev w/ latest technologies, only did non ajax stuff years back. I just test apps nowadays. – David Oct 09 '12 at 23:27
  • @Software Monkey, true. But compared to the examples people give when talking about or demoing JSON diff, that is huge. Same w/ XML. Be nice if people provided complex larger structure examples as well. – David Oct 09 '12 at 23:28
  • @David: In a JavaScript environment, first parse the JSON into JavaScript objects, then stringify it using the indentation parameter. `var parsed = JSON.parse(jsonString); jsonString = JSON.stringify(parsed, null, 4); console.log(jsonString);` The third argument to `JSON.stringify` lets you set the indentation size or characters, so it'll print out nicely with newlines and indentation. – I Hate Lazy Oct 09 '12 at 23:38

3 Answers3

14

Beyond Compare works well, if you set up a JSON file format in it to use Python to pretty-print the JSON. Sample setup for Windows:

  1. Install Python 2.7.
  2. In Beyond Compare, go under Tools, under File Formats.
  3. Click New. Choose Text Format. Enter "JSON" as a name.
  4. Under the General tab:
    • Mask: *.json
  5. Under the Conversion tab:
    • Conversion: External program (Unicode filenames)
    • Loading: c:\Python27\python.exe -m json.tool %s %t
      • Note, that second parameter in the command line must be %t, if you enter two %ss you will suffer data loss.
  6. Click Save.
Community
  • 1
  • 1
Josh Kelley
  • 56,064
  • 19
  • 146
  • 246
  • 1
    I had to enter "c:\Python27\python.exe -m json.tool %s %t" for the loading path (%t instead of %s for the latter variable). – Claus Conrad Sep 23 '14 at 16:13
  • Note that if using a newer Python version (3.7.2 in my case), it installs by default in `C:\Users\\AppData\Local\Programs\Python\Python37-32` – JYelton Mar 20 '19 at 17:19
  • 1
    Also note that adding `--sort-keys` to the command line arguments may be needed if you're comparing JSON from two sources where the properties are not in the same sequence. Thus, my Beyond Compare "Loading" setting is: `C:\Users\\AppData\Local\Programs\Python\Python37-32\python.exe -m json.tool %s %t --sort-keys`. – JYelton Mar 22 '19 at 22:11
3

Jeremy Simmons has created a better File Format package Posted on forum: "JsonFileFormat.bcpkg" for BEYOND COMPARE that does not require python or so to be installed.

Just download the file and open it with BC and you are good to go. So, its much more simpler.

JSON File Format

I needed a file format for JSON files.

I wanted to pretty-print & sort my JSON to make comparison easy.

I have attached my bcpackage with my completed JSON File Format.

The formatting is done via jq - http://stedolan.github.io/jq/

Props to Stephen Dolan for the utility https://github.com/stedolan.

I have sent a message to the folks at Scooter Software asking them to include it in the page with additional formats.

If you're interested in seeing it on there, I'm sure a quick reply to the thread with an up-vote would help them see the value posting it. Attached Files Attached Files File Type: bcpkg JsonFileFormat.bcpkg (449.8 KB, 58 views)

Community
  • 1
  • 1
Alex S
  • 242
  • 2
  • 17
  • Hmm...I wonder what version this works with or the exact install steps. On a Mac with BC4. On Tools > Import Settings, select the downloaded pkg file, then next screen is to pick file formats to import & it is blank, and can't proceed, unless I check checkbox for "delete all existing file formats", but I don't suppose that is good idea, I stopped there. – David Aug 13 '15 at 22:24
  • @David - I am using this on BC3 + Windows 8.1 x 64. The issue is that it uses a Windows .exe, so I am not sure how to put it on a Mac. Although an OSX version of the App exists as well. You might want to try this on a Windows to test it out and/ or contact the author of the package on how to do it for Mac. My thought is it should not be that complex to do the same on Mac - But, since I dont have one I cant figure that out for you. – Alex S Aug 14 '15 at 09:24
  • @David - I posted a BC based solution for SQLite as well - here: http://sqa.stackexchange.com/a/14283/13028 - This one shows how to configure file types on BC. Both of these are User contributed File Extensions for BC. One is pre-packaged, the other has a How To. Let me know if that works out for you. If not, I'll try to help you with this when I have more time PS: Again, I don't have a Mac so I couldn't test it for sure. – Alex S Aug 14 '15 at 09:29
  • The issue with Beyond Compare is that it doesn't match the content very well. I use jq outside of BC to sort two "catalogs" so all keys appear in same order etc, but with just one key value different, BC can misalign a large portion. It is a PITA to try and use align diff in BC to correct since it will re-evaluate and possibly undo other alignment fixes already done. I asked in BC forums here: https://www.scootersoftware.com/vbulletin/forum/beyond-compare-3-discussion/text-compare-2-way/13568-comparing-formatted-json-how-to-improve-the-brace-matching but still have not found a good solution. – user107172 Jan 24 '19 at 23:29
0

I have a small GPL project that would do the trick for simple JSON. I have not added support for nested entities as it is more of a simple ObjectDB solution and not actually JSON (Despite the fact it was clearly inspired by it.

Long and short the API is pretty simple. Make a new group, populate it, and then pull a subset via whatever logical parameters you need.

https://github.com/danielbchapman/groups

The API is used basically like ->

SubGroup items = group
                  .notEqual("field", "value")
                  .lessThan("field2", 50); //...etc...

There's actually support for basic unions and joins which would do pretty much what you want.

Long and short you probably want a Set as your data-type. Considering your comparisons are probably complex you need a more complex set of methods.

My only caution is that it is GPL. If your data is confidential, odds are you may not be interested in that license.

Daniel B. Chapman
  • 4,647
  • 32
  • 42