2

What I am trying to achieve is:

  • We have a JSON Object Stream coming in.
  • I want to reconstruct a Java Object from the JSON.
  • I want to apply transformations (like change format of strings and reject an object if some condition is met like 'isToBeConsumed' == false and possible flatten the heirarchy from nesting to linear)
  • I want to be able to store the values of the JSON objects into the database.

When a new type of JSON Object Stream comes in, I am OK to run CREATE TABLE for each type of Stream but i do not want to do any code change.

In other words, I want to be able to define a Java Class in a Config file.

{"studentName":"string", "studentRollNumber":"integer", "studentGrade":"integer", "studentPhoneNumber":"string"}

Now if a 2nd stream comes in like:

{"teacherName":"string", "teacherEmployeeId":"integer", "teacherPhoneNumber":"integer", "teacherAddress":"string"}

i do not want to do a code change.

I am ok to restrict the possibilities to 4 basic data types.

I might have rules like

{"excludeIf": { "columnName": "grade", "condition": "equals", "value":"10"}

But lets not worry about the exclude rules as of now and focus on how to reconstruct a JAVA class from Config File and then Deserialize a JSON stream into an object of that class.

I saw that Gson 2.0+ gives a feature to convert a generic Map https://stackoverflow.com/a/20442943/429218

I am looking for more options to Dynamically Construct a Class in Java and Deserialize a JSON stream into it.

Thanks

Community
  • 1
  • 1
rjdthegreat
  • 204
  • 3
  • 10
  • Dynamic create tables for each type? Create a Java class for each different json input? It seems a **TERRIBLE** design here... Maybe if you explain what you want to do you will get better possibilities. – Jorge Campos Jan 12 '16 at 04:49
  • I am going to have 3 type of streams coming in.. Students Info Stream, Teachers Info Stream and Marks Info Steam. I might have a 4th type coming in few months later like School Events Stream. I want to build a system where I have an onboarding process for this new stream (where i will be engaged manually and I will create the database tables and change some configs) But i do not want to make a code change for incorporating this 4th Type of Stream. – rjdthegreat Jan 12 '16 at 04:56
  • Why would you do this dynamically if you already KNOW what you will gona receive?? This is a terrible idea and will be impossible to maintain and will have a terrible code. Imagine no you have three types of data, but your code will support anything given your rules. Every thing in your system would have to be dynamic based on the objects you receive. Please, don't do that. Create your domain models, serialize your objects and work with, if you have to change the code later so be it. It will be HUNDRED THOUSANDS times better. This is an honest advice. Please reconsider! – Jorge Campos Jan 12 '16 at 05:11
  • The student and teacher stream are placeholders, I have absolutely no idea what kind of data is supposed to come in. I am going to release my applications as a Platform and I need to be data agnostic. It is not an option to restrict myself to specific Data Model. Is it hard to imagine (read as Implement) a system where i have complete control over what i can send in, define rules when i want to exclude a particular object/row and persist it so that i run SQL on it later? – rjdthegreat Jan 12 '16 at 05:22
  • Ok, Let's imagine you were able to create such a system (that will be able to receive ANY data and accept it and control it in tables and classes)! What you gone do with it? Lets say after hundreds of different types? I'm POSITIVELY ABSOLUTELY SURE that you don't think this through enough to think on this kind of solution. There is no such a thing as a system that receives ANY type of data and create dynamic java classes for it, it would be impossible to manage and to create another functionalities based on data that you don't know! – Jorge Campos Jan 12 '16 at 05:27
  • Suggestion: take a look at nosql databases that support storing schema-less data sources. – Farshid Zaker Jan 12 '16 at 05:31
  • As a platform, a better approach would be to first let users define their domain model then provide a generic api that lets data be mapped to it, but you would not create new Java classes instead you would use data transformation pipeline that is built to process arbitrary types of data. – John Scattergood Jan 12 '16 at 05:31
  • @FarshidZaker: Thanks for the suggestion but it is imperative to use a SQL datastore because the user is familiar with SQL and wants to run queries on the data. – rjdthegreat Jan 12 '16 at 05:33
  • @JohnScattergood: I see what you are saying, How do you think should we let users define their models? Does the way mentioned in the question seem reasonable? Is there any standard way to define domain model. I see that you say we do not need Java Classes but can you help me imagine what kind of Data Transformation Constructs (Processors) you are referring to. – rjdthegreat Jan 12 '16 at 05:36
  • @JorgeCampos: Hi Jorge, Thanks for your comments. I feel you and I understand why you think this is a bad idea but can you think of something in which we can achieve this data agnostic nature without having to create dynamic java classes. I hear you and that takes me to the definition of a class. A class represents a real world pattern. I cannot have any real world crap if i do not know what it is. Lets rule out classes. Can you suggest a generic way to achieve data-agnostic nature? – rjdthegreat Jan 12 '16 at 05:39
  • @rjdthegreat How dynamic does it need to be? You you don't mind restarting you could use "entity definition tables (entity and entity fields)" to define the user data and then procedurally generate a hibernate XML file and let the system create the classes on start up. If it has to be completely dynamic then you could use an EAV model: https://en.m.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model. Most ESBs provide processor pipelines: Mule and Apache ServiceMix are examples – John Scattergood Jan 12 '16 at 06:07
  • Even with a dynamic data it would not be a good idea, but lets forgot about this and think in a possible solution. I would create a model that will be a set of key,values like a configuration. Thinking in database tables it would be something like: `entity{id, description}; entity_fields{ id, fieldName, dataType }; source{id, name (where the data came from)}; fields_value{ id, idField, value(this one would have to be string and inside your system you would convert it based on datatype); }` – Jorge Campos Jan 12 '16 at 11:30
  • The problem with this is to identify data with the same type from same source, your system would have to check entity, fields and the source to not duplicate data. That is another problem if such scenario, think about two domains like `{"studentName":"string", "studentRollNumber":"integer"}` and `{"stuName":"string", "stuRollNumber":"integer"}` how would your system know that they are the same? They should have. But, automatically it is impossible. It would be an AI to process and understand such scenarios. – Jorge Campos Jan 12 '16 at 11:33
  • @JorgeCampos: Thanks for your response. – rjdthegreat Jan 12 '16 at 18:26
  • @JorgeCampos: I see your point. It seems like v r driving towards something acceptable+achievable. We can eliminate the concept of source. One type of Entity = One source. Also, all the data received is immutable, No updates or No duplicates. I need to persist each and every record that comes from the stream. I may be testing ur patience but do u think it is somewhat acceptable to not have different java classes for each entity type but have class Entity { Map keyValues}; Entity e = {"name": "ABC", "rollNo": 1},. and persist it in database as Student(name,rollNo,..) – rjdthegreat Jan 12 '16 at 18:38
  • I think that it would be better, yes. Way better than to compile a class to represent it. Think of this situation: You have your system that is a blackBox that has the class DoSomething. At start DoSomething does not know about the new classes, then you source will send an object that you will convert into ObjThing, now you need that DoSomething understand ObjThing in order to do that something. How would you do that? Your entire system would be based on Reflections and Dynamic class loaders. That is the MAIN problem. You would have to think what would be possible at any possible scenario – Jorge Campos Jan 12 '16 at 19:33
  • which is humanly impossible. Having the so called classes as data would be WAY more flexible and understandable since you KNOW what expect and your DoSomething class would have to process only the data. – Jorge Campos Jan 12 '16 at 19:34

1 Answers1

0

This seems like in runtime, application doesn't know the JSON format and yet need to find the values and keys to create tables and put data in db. Please refer to Parsing Json in Java without knowing structure. This will give you an idea of doing it in generic way. However, you need to write more code with if statements to create tables automatically if application finds something new and non existing. However, this is a terrible design as stated by @Jorge Campos. In Real world application will have a contract i.e. client and server. Doing in generic way impacts performance and lot of cpu utilization for garbage collection.

Community
  • 1
  • 1
Anvesh Vejandla
  • 221
  • 1
  • 2
  • 11
  • 1
    The student and teacher stream are placeholders, I have absolutely no idea what kind of data is supposed to come in. I am going to release my applications as a Platform and I need to be data agnostic. It is not an option to restrict myself to specific Data Model. Is it hard to imagine (read as Implement) a system where i have complete control over what i can send in, define rules when i want to exclude a particular object/row and persist it so that i run SQL on it later? – rjdthegreat Jan 12 '16 at 05:23
  • As stated above this can be implemented. Refer to the answers in the link. You just need to tweak your code a little. Try to make a sample and see how that works. The above link pretty much tells you how to get keys and values from the json and then you need to do define your use case with those. – Anvesh Vejandla Jan 12 '16 at 05:28
  • There is also possibility for this. AnveshVejandla , instead of saying that the application need to find the keys, let's assume all the keys which ever needed are provided as a configuration as said by rjdthegreat instead of providing keys which are not necessary. – npradeep357 Oct 12 '16 at 03:26