2

I am creating a silverlight pivot collection with 31K items (and images), however when I'm using the DeepZoomTools library to create the deep zoom images; it takes hours and hours (and hasn't actually completed even one).

Is there a multi-threaded way or distributed way in which collections could be created?

1 Answers1

0

It is a time intensive process to be sure. Does your individual data points change often? What we have found in nearly all of our projects that the image for an individual item almost never changes. This allows you to streamline the process a little bit.

What I do in a case like this is to initially process the entire dataset. Then the next time I run the process, I only update the images that have been added or modified. As I said, in almost all of my cases this solved the problem you are running into. In fact, when it works, I will plug my card generation into whatever business applications that are running and generate/modify a card when data is added/changed in the system. This removes the need for batch processing altogether after your initial build.

If that will not work for you, take a look at the code for PAuthor. It is using DeepZoomTools and does so in a multi-threaded way. You should be able to find the code you are looking for there. PAuthor - CodePlex

Let me know if you have more specifics about your specific needs and we can see if we can come up with something.

Tony Champion - CDS
  • 865
  • 5
  • 5
  • In this particular case the data doesn't change. It just takes a long long time. I had to reduce the MAXLEVEL to 5 on the collection so that it finishes in a reasonable amount of time; however the final image looks pixilated. I want to be able to do 500K elements, of data that doesn't change often (that's the goal). –  Nov 16 '11 at 17:57
  • No way around 500k elements taking forever to run. You are going to have to build a dynamic .cxml generation though. PV won't handle more than about 3k items at a time. So you are going to have to prefilter some how (ie searching, linking, etc) – Tony Champion - CDS Nov 17 '11 at 04:58
  • I've pushed pivot viewer to 15K elements before. It more about the client pc that's consuming it. The generation of the data is my hurdle –  Nov 17 '11 at 19:05
  • Do you have to regenerate the entire collection each day or are you just adding/updating a small set? – Tony Champion - CDS Nov 17 '11 at 20:53
  • Regeneration, because any number of records can be mass edited or updated (data cleaning). After that it will be small adding/updating. It' a little vague to say that I don't know how long till it's a small updates but, I just don't know. Further, I just finished testing a revised version of the 31K collection, and it turns out that the images are not getting clearer in silverlight, but they are working in Microsoft Live Labs Pivot (http://research.microsoft.com/en-us/downloads/dd4a479f-92d6-496f-867d-666c87fbaada/default.aspx) –  Nov 17 '11 at 22:20
  • You will notice some differences between the two in cases like that. ARe the images changes in mass edits or is it just the metadata? – Tony Champion - CDS Nov 18 '11 at 12:55
  • the images are representations of the textual information; they change with the metadata. –  Nov 18 '11 at 15:47
  • Gotcha. Ok. I go back to my original answer. Download PAuthor from CodePlex. In the PAuthorLib project there is a ParallelDeepZoomCreator class in the Imaging folder. I would start with that. You can add in your process for generating your image and then DeepZoom it. For that many images, I would create the image using GDI+. It will be much quicker than using HTML templates in PAuthor. You can get better performance than DeepZoomTools.dll (which PAuthor uses) if you write it yourself for your specific needs, but it is a great place to start. – Tony Champion - CDS Nov 18 '11 at 20:02
  • Alright, so expand on Pauthor's Parallel Creator, and create a system of distributed image collection creation. –  Nov 18 '11 at 20:14
  • Is it possible to create each image as a separate deep zoom image? –  Nov 27 '11 at 18:15
  • Yes and PV supports having each card a deep zoom image. Makes for some interesting experiences. – Tony Champion - CDS Nov 28 '11 at 05:54
  • So I can create the images separately and then weave a single Dzc file, or would they have to be separate dzc files? –  Nov 28 '11 at 13:18
  • Same as the normal dzc. You just need to extend the item maps to the higher image sizes. – Tony Champion - CDS Nov 28 '11 at 18:33
  • I got it! Pivot JIT Server! (http://www.silverlight.net/learn/data-networking/pivot-viewer/download-just-in-time-tools#). I'm going to create a version that runs on Windows HPC; solution found. –  Dec 01 '11 at 13:48