5

Here is the situation:

I have a huge data set that I need quick access to. It's a financial data set so basically the way it's set up is that at each point in time, you have data for thousands of stocks. Now, loading this data into the program takes a while (20-40 seconds) and I would like to avoid having to do this every time I make changes in the code.

Basically, I had an idea and I want to know if it makes sense / if it is optimal. I was thinking of setting up some kind of server that has all the data loaded and all the object definitions, and then from a second program/client, I would like to send a predefined type of class (a strategy class) to the server, have the server run the code, and then have the server send me back the results of the test.

I am creating trading strategies in this case, so I have an abstract class that defines what a "strategy" needs. I have a market simulator that calls the derived "strategy" every time the date changes, then the strategy buys or sells stocks, and then stores the profits from the strategy in some result object. So I want to be able to code the strategy part, send it over, and receive the result part. I want to avoid loading all the data every time I change params or raw code in my strategy object.

I hope this makes sense to someone and I am sorry if it isn't very clear. Please let me know if this is possible, and then, how would I go about googling for this?? I don't even know what to search for here.

John Saunders
  • 160,644
  • 26
  • 247
  • 397
philmo
  • 205
  • 3
  • 7
  • Off-topic; the 20-40 seconds load time; what is it loading in this case? I might be able to offer you some pointers to reduce this time significantly – Marc Gravell Feb 17 '11 at 15:46
  • Well basically the way the data is formatted is as such. I have a csv file where each row has a data point. A data point consists of date,ticker,price,idNum to keep it simple. So what I did was I created serialized files for each column, and when to program opens, it loops through the columns and creates individual "Stock" objects which are then held in a "Stocks" object (essentially a dictionary with an int key and Stock value). Any suggestions? – philmo Feb 17 '11 at 15:54

2 Answers2

4

I'd define an Interface that the server will use to invoke your strategies, then implement each strategy in a separate assembly.

The server would then load each assembly in a separate AppDomain and execute it there. This will give the server process some protection against bugs in the strategy implementation, but more importantly will allow it to unload the assembly.

Chris Wenham
  • 23,679
  • 13
  • 59
  • 69
  • Ok I will look into this. I think this is what I'm looking for. I googled loading assemblies in seperate AppDomains and it looks promising. – philmo Feb 17 '11 at 15:48
  • Hey Lawrence I have a question for you. I am trying to do the following. I code my strategies in a different project, then compile the file. I then want to import it into the "server" program, run a given strategy, and repeat. I am getting file access issues however. I cannot compile & copy the file (.dll) when my "server" is running. I looked into shadow files but that doesn't seem to be helping. Any ideas?? Thanks in advance. – philmo Feb 17 '11 at 20:35
  • It's important than the strategy assembly's DLL is ONLY loaded in a new AppDomain on the server, and that you never use any Reflection method in the server's main AppDomain to peer into that assembly, because doing so will lock the .dll file. Second, when you want to overwrite the .dll with a newer version, the server has to unload the child AppDomain first in order to unlock the file. – Chris Wenham Feb 17 '11 at 20:42
  • Hey Lawrence, since it was a little off topic, I started a new thread. I posted the code I am working on (mock up). I am wondering if you could take a quick look at it and let me know where I went wrong. Thanks again. Here is the link: http://stackoverflow.com/questions/5034654/c-help-needed-with-unloading-dlls-from-appdomain-still-not-working-even-with – philmo Feb 17 '11 at 21:11
0

This sounds like a potential security nightmare, but if that's what you want...

Consider sending a serialized workflow to the server for execution. That's just a block of XML.

John Saunders
  • 160,644
  • 26
  • 247
  • 397
  • Acutally this is solely used on my machine so I'm not concerned about security. I am just testing different strategies and I want to avoid reloading the data every time. This is not being distributed in any way. – philmo Feb 17 '11 at 15:47