1

I am encountering a situation at work similar to the thread over here. The difference is that we are not using a web service. But rather we are using a web app. We have a requirement that lets a user download 8000 records with at least 30 columns which is written to an excel file via jxl.

This is a long running process that is not being done asynchronously for reasons I don't know. It also has a huge memory footprint ~ 500 - 800 MB. To top it all of, it takes on average 2 minutes and 10 seconds to finish.

What we are currently doing is delegating this requirement out of process from the application server via jms. The app server sends the request to the broker and then a consumer gets the request from the queue, processes it and sends back the url of the excel file.

I have some qualms about this because I have read been reading about JMS and most of the usecases suggested involve asynchronous requests so that users won't have to wait for a long time like for instance sending email, sending a request for approval, invoicing. There are lot of examples in this thread and the use cases suggested can be done asynchronously. So our current solution sounds like a hack rather than a true solution.

Any suggestions, patterns that can be done to further improve this process?

EDIT: I unfortunately can't do away with the "synchronicity" if you may call it that of the feature because it's a business requirement. So I am looking for answers/tweaks/tricks/patterns that can enhance the performance and lessen the heap memory usage of the process instead.

Community
  • 1
  • 1
Jeune
  • 3,498
  • 5
  • 43
  • 54

2 Answers2

4

Did I understand this right?

  • The user makes a request. During this request
    • a JMS message is sent
    • an Excel file created
    • a JMS message returned
    • and then the URL is sent to the user?

If so, I agree, that's an awful abuse of the asynchroneous nature of JMS.

What I would do is:

Immediately show the user a result page with a text like your request is being processed and then

  • either use AJAX to poll the server for the result URL (possibly showing the user a status bar)
  • or send the URL to the client via email
Sean Patrick Floyd
  • 292,901
  • 67
  • 465
  • 588
  • +1, OP really doesn't want this to be synchronous. More than two minutes is much too long for a web server thread to be active on a single request. – Qwerky Nov 30 '10 at 10:26
  • Yes you understood it correctly but I was wondering if you can suggest alternatives to doing this asynchronously... – Jeune Nov 30 '10 at 11:08
  • If the process takes 2 minutes, anything else would be awful. Anything taking more than 30 seconds should be asynchronous, and even that's a lot. – Sean Patrick Floyd Nov 30 '10 at 11:10
  • I want to convince our business analysts to do this. Any tips? – Jeune Nov 30 '10 at 11:16
  • 2
    Your business analysts shouldn't be dictating system design. If they need to know why tell them that at the very least it ties up server threads creating a limit to the number of concurrent users. – Qwerky Nov 30 '10 at 11:46
  • Well with the processing happening in another heap space away from the app server's heap space, we have solved the tying up of server threads. What I probably meant about the business analysts is that the use case of the feature indicates the when the user clicks a button to do this whole process, the user has to wait for the whole process to finish whether it takes 30 seconds or two minutes. So clearly it's a business requirement that the customer himself indicated. – Jeune Nov 30 '10 at 12:05
1

I'm not gonna suggest what you can do with your JMS backend - its probably not used well or maybe it is - I'm not sure.

We implemented something similar and here is what we ended up with (our backend implmentation was completely different, for a different use case, but the "async user exp" matches) :

  1. Allow user to submit a request.
  2. Once (1) is received, submit the job to the backend (JMS etc...) and tie it to a job id
  3. The response of (1) is a this (permanent or temporary) handle - the job id, for example
  4. The backend implementation should end by setting the status of the job
  5. Allow users to poll for the status of the job (via the handle you gave them)
  6. If the job status is "in progress", tell them so (with probably an estimated time for completion - so that they dont unnecessarily waste time polling & burdening you)
  7. If the status is "done successfully", report that to the user, with the other data (url etc...) the user would need to access the result
Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
madhurtanwani
  • 1,199
  • 7
  • 13
  • polling need not be a webserivce per se - as suggested in another answers, it could be an end point, you would be polling on, via AJAX to find the status of the job – madhurtanwani Nov 30 '10 at 11:36
  • actually a very good implementation of this is in Yahoo's Delicious page for "importing bookmark files" : https://secure.delicious.com/settings/bookmarks/import – madhurtanwani Nov 30 '10 at 11:37