1

I have the following problem to solve.

I need to write a java program that:

  1. reads JSON object j1,j2,...,jn from a web service.
  2. does some number crunching on each object to come up with j1',j2',...,jn'
  3. Sends objects j1',j2',...,jn' to a web service.

The computational, space requirements for steps 1,2, and 3 can vary at any given time.

For example:

  1. The time it takes to process a JSON object at step 2 can vary depending on the contents of the JSON Object.
  2. The rate of objects being produced by the webservice in step 1 can go up or down with time.
  3. The consuming web service in step 3 can get backlogged.

To address the above design concerns want to implement the following architecture:

enter image description here

  1. Read JSON objects from the external webservice and place them on a Q
  2. An automatically size adjusting worker thread pool that consumes JSON objects from the Q and processes them. After processing them, places the resulting objects on the second Q
  3. An automatically size adjusting worker thread pool that consumes JSON objects from the second Q to send them to the consuming webservice.

Question:

I am curious if there is framework which I can use to solve this problem?

Notes:

  1. I can solve this using a range of components like custom Queues, Threadpools using the concurrency package -- however I'm looking for a solution that allows the writing of such a solution.
  2. This is not going to live inside a container. This will be a Java process where the entry point is public static void main(String args[])
  3. However if there is a container suited to this paradigm I would like to learn about it.
  4. I could split this into multiple processes, however I'd like to keep it very simple and in a single process.

Thanks.

Thanks.

user1172468
  • 5,306
  • 6
  • 35
  • 62

2 Answers2

1

I'm not exactly sure what the end question is for your post, but you have a reasonable design concept. One question I have for you is what environment are you in? Are you in a JavaEE container or just a simple standalone application?

If you are in a container, it would make more sense to have Message Driven Beans processing off of the JMS queues than having a pool of worker threads.

If in your own container, it would make more sense for you to manage the thread pool yourself. With that said, I would also consider having separate applications running that pull the work off of the queues which would lead to a better scaling architecture for you. If the need ever came up, you could add more machines with more workers pointing at the one queue.

Jeff Richley
  • 1
  • 2
  • 8
  • Many thanks @Jeff I'll edit the question and refine it based on your input but basically: 1. No Container 2. Want to keep is simple: one single process – user1172468 Oct 23 '12 at 18:59
  • 1
    To my knowledge, there isn't any one stop shop frameworks but you can link a few together. You can use a framework such as [Apache Axis](http://axis.apache.org/axis/) for consuming and pushing to web services and the [Apache Commons Pooling](http://commons.apache.org/pool/) for the pooling for threads and automatically growing. – Jeff Richley Oct 23 '12 at 19:13
1

try Apache camel or Spring Integration to wire things up. These are kind of integration frameworks, will ease your interaction with webservices. What you need to do is define a route from webservice 1 -> number cruncher -> web service 2. routing and conversion required in between can be handled by the framework itself

you'd implement your cruncher as a camel processor. parallelizing your cruncher may be achieved via SEDA; Camel has a component for this pattern. Another alternate would be AsyncProcessor

I'd say you first take a look at the principles behind frameworks like camel. The abstractions they create are very relevant for the problem in hand.

Community
  • 1
  • 1
mzzzzb
  • 1,422
  • 19
  • 38
  • Mmm, @mzzzzb, how do you think Twitter Storm would fit the bill? – user1172468 Oct 23 '12 at 21:09
  • @user1172468, im afraid i dont know anything about Storm. just looked into it and seems interesting, not in a position to advise. one more thing to learn :) – mzzzzb Oct 24 '12 at 02:53