6

We have a situation with a large number of JSON objects which adhere to a JSON Schema.

The issue we have is that the JSON Schema is under constant development so regular changes are made to the schema (adding/removing fields etc) thus invalidating all the JSON data.

We need a tool to automatically migrate the data based on changes to the schema, or it could be looked at as a cleanup/correction task - automatically (without any manual processing) cleaning up columns and adding new ones.

Are there any open source tools in this space which could help, or any recommendations for suggested approaches?

For example I've looked into https://github.com/bazaarvoice/jolt and https://stedolan.github.io/jq/ but they don't take JSON Schema into account.

Paul Odeon
  • 4,407
  • 1
  • 37
  • 37
  • You're asking for something to be automated which needs manual intervention. If you want to do this automagically, you can't care much about your data. Also, what is "the data"? What format? What database? What language do you use to interact with the data? What sort of data is it? – Relequestual Mar 26 '18 at 12:07
  • The data is recursive json, but I don't think it's particularly relevant to the question, we are happy to lose the data as long as it is normalised to the schema. – Paul Odeon Mar 26 '18 at 13:49
  • If you ask me how to get from A to B, I'm going to give you walking directions, and then you tell me your car doesn't go that way. You're asking a question without giving enough context for anyone to be confident to answer. – Relequestual Mar 26 '18 at 13:54
  • Your question is off topic because you are asking for a library recommendation. Please see point 4 of https://stackoverflow.com/help/on-topic – Relequestual Mar 26 '18 at 13:59
  • I think this question is actually relevant, and i'd like to have suggestions on this too. @PaulOdeon is asking for library recommendations or suggested approaches. – Ricovitch Oct 03 '18 at 13:53
  • 4
    I think this is a perfectly sensible question especially for cases where data is store statically as JSON while there is a need for schema evolution. I'm always baffled when people proceed to grill the OP as though it were a taboo to ask. What you're basically after is the equivalent of XSLT for JSON. You might consider converting your JSON to XML for which you have an XSLT then after the transformation convert back to JSON. There are many tools out there as evidence from https://stackoverflow.com/questions/1618038/xslt-equivalent-for-json – polarise Feb 04 '21 at 09:50

1 Answers1

0

I was searching for a similar thing, but when I think about it, there are few cases that you need migration in regular relational databases:

  • when you need to add a new column - this case you can change the schema itself without any problem.
  • when you need to rename a column or change the type of column: in json schema draft-7 you can benefit from anyOf
  • when you are changing the whole data modeling: in this case for json-schema you can use versioning, different schemas for different versions.
Ramtin M. Seraj
  • 686
  • 7
  • 17