3

My company has a 3rd party web service we are designing a front end for. The "objects" used by this web service are very large (and variable depending on the number of sub-entities created). The web service does not expose methods to commit/load sub-entities, only the full object hierarchy.

The UI itself is split into many sub screens, and master/detail views to be able to efficiently/easily edit the large amount of data.

The issue is where to store all the data you aren't currently looking at.

Doing the web service commit takes up to 30 seconds for large records, so it is not feasible to use the web service for the intermittent data storage.

As mentioned, the data can be quite large (multiple megabytes, to possibly a gigabyte of data in the edge cases)

This is for ASP.Net 4.0, with a MS-SQL back-end, and a 3rd party SOAP web service.

Changing the web service contract is not an option.

Here are some ideas we have had. Help me choose, or identify something better!

1) The initial developer threw serialized XML into the session (on-machine session). I quickly identified this as problem, as using session in this way could drastically affect performance due to the memory taken up, as well as load balancing issues etc.

2) Session server - possible, but requiring additional hardware purchase for this is probably not an option

3) SQL session - Performance of storing such a large object?

4) Write XML to disk/share (zipped?)

5) Write XML to SQL (zipped?)

6) SQL relational database - create tables for each entity type. This would allow individual sub-entity load/save which could be great on performance. We are worried about brittleness and maintenance since we do not control the 3rd party service (although we have that issue anyway really, since the GUI also would be brittle)

7) viewstate - way too big, perf hit

Jason Coyne
  • 6,509
  • 8
  • 40
  • 70
  • 1
    maybe you should have a look to json objects http://www.json.org/ it is replacing XML these days. because of the simple notation compared to XML the data volume will decrease too – kasper Taeymans Sep 14 '11 at 14:09
  • That is a good suggestion, however we have to maintain XML serialization in order to get the web service to work, so JSON would mean maintaining two sets of hydration logic. (even though the XML part is mostly auto-generated). – Jason Coyne Sep 14 '11 at 14:13
  • In your title you said "session" data, which leads me to believe that this information is user-specific in some way. Is that correct? – James Johnson Sep 14 '11 at 14:21
  • Yes, all of this data is per user per transaction – Jason Coyne Sep 14 '11 at 14:22

1 Answers1

6

Option 6 is probably your best bet. It sounds like you have to deal with maintaining changes to the service's data structure anyway. You might as well come up with a schema where you can "pull" the data from the service to your data store, let users play with it, then "push" the data back when they're done.

You might also be interested in learning about non-SQL data stores. A schema-less or document-based data store might reduce your maintenance costs while still allowing you to store the entire model away and retrieve pieces of it based on queries after the fact.

StriplingWarrior
  • 151,543
  • 27
  • 246
  • 315