1

I'm designing an interface which takes in data from multiple upstream systems. The interface would receive huge amounts of data (~90k).

All the upstream systems send data in different formats as shown below:

For e.g., SYSTEM:

  • A as XML

  • B as JSON

  • C as CSV

  • Z as {Key1}=Value1

DESIGN-1

A possible solution is to convert all different formats to JSON.

Now there are two major challenges here that I need to address:

  1. Need the data to reach the downstream system as quickly as possible.

  2. Accommodate the addition of new sources, data types with minimal code changes.

For XML, marshalling can help, for CSVs, can create POJOs and map data to them.

DESIGN-2

Store all data as key-value pairs in HashMap.

In this approach, new fields would come in as new keys instead of having to add a new mapping in code as mentioned in DESIGN-1. Less parsing and mappings would make it faster

Can anybody suggest me a better way to do this? Please keep in minds that points 1 and 2 in design 1 are very important.

İsmail Y.
  • 3,579
  • 5
  • 21
  • 29
vermachint
  • 93
  • 1
  • 9
  • Convert all data to a common format. JSON is fine. Key / Value maps seems unlikely to be sufficient to represent most XML and JSON data. – DwB Feb 01 '17 at 18:10
  • Hey @vermac, devermac, whatever... When we spend time to help you, do you think that it is fair to [delete your post](https://stackoverflow.com/questions/52679673/numpy-dot-product-of-a-4d-array-with-its-transpose-fails/52682167#52682167)? Undelete it and tell us what you finally did! – keepAlive Oct 06 '18 at 19:23
  • There are no problem with being wrong. Be fair. – keepAlive Oct 06 '18 at 19:30
  • @Kanak Hey, apologies for deleting the post. I've undeleted it. I'll get back on it once I have my thoughts cleared out. Was pretty confused myself. – vermachint Oct 06 '18 at 19:36
  • No problem. Natural to be confused. Know that you can also be banned when deleting posts to which others have answered. Please read [this](https://meta.stackoverflow.com/questions/255583/what-can-i-do-when-getting-we-are-no-longer-accepting-questions-answers-from-th&ved=2ahUKEwiY0uvnyPLdAhXIBSwKHR9xBocQjjgwAXoECAkQAQ&usg=AOvVaw37hgzm4YzP4kHjMZoV1yL4). Furthermore your question is anything but trivial. I am going to upvote it to maximize your chances of getting other answers. – keepAlive Oct 06 '18 at 19:47

0 Answers0