2

I have a scientific application that I built in Python (the application's 'critical areas' are optimized with Cython, for increased speed).

Every instance of the application is given a text file (with parameters) an an input. The application reads the parameters from the text file and, using data that is stored in the hard drive, runs and outputs the results of the calculations back into the hard drive. Every instance requires about 600MB of memory over the course of its operation.

Currently, I am running the app on my laptop (Intel Core 2 Duo, T7500, @2.2Ghz, 2GB RAM). Every 'instance run' on my laptop takes about 3 hours to complete.

Due to the needs of the project, there is a need for me to run 10000 instances. Obviously, it would take forever to do so on my laptop, hence the need for more computing power.

Knowing that every such an instance is independent from another, what would be the cost of a server that can run say 10 instances at the same time (I am on a budget...)? Can you recommend on a configuration?

Currently, I am using Windows XP but ideally, I would be happy to have the server installed with unix (ubuntu).

nmichaels
  • 49,466
  • 12
  • 107
  • 135
Shalom Rav
  • 81
  • 4
  • Before you throw money at the problem, is there a way in which you could reduce the run time? Have you profiled your code? Can it be run in parallel? If and only if the answer is "I am absolutely positive that my code is as good as it gets" then you can throw money at it, but consider that 99% of the cases, the code itself can be greatly improved. – Escualo Sep 15 '10 at 19:01

2 Answers2

1

You can spin up Amazon EC2 standard instances (1.7GB / 1 slow core) for $0.085 per hour, or 23GB / 8core "cluster compute" instances for $1.60 an hour.

"One EC2 Compute Unit equals 1.0-1.2 GHz 2007 Xeon processor."

According to the tool, 10,000 "High-CPU Medium" instances with 5 EC2 Compute Units and 1.7GB each, for 3 hours, is $5100. This doesn't include the cost of getting your source data in and results out.

You can also bid on idle server time. The current "spot" price is about 1/3 the "on-demand" price, and fluctuates with demand. If you put in a low bid, your job may be interrupted by demand.

It is interesting to compare the cost of electricity for running your server/cooling with the cost of an Amazon instance. Commerical electricity rates are about 7.5¢/kWh here..

Joe Koberg
  • 25,416
  • 6
  • 48
  • 54
  • Joe, thank you, Can you explain why choose few "cluster compute" instances rather than many 'small instances'? small instances cost much less... – Shalom Rav Sep 15 '10 at 20:41
  • Well that just sets the high end of the scale if you are doing parallel cluster computation (cluster instances have 10GB interconnect)... And I felt the High CPU matched the limiting factor of your laptop configuration (core2 duo @ 2.2GHz) ... I presume you aren't memory constrained on your laptop. – Joe Koberg Sep 16 '10 at 01:48
  • I was just estimating 10,000 of the nearest equivalent to your laptop for 3 hours (10k High CPU instances simultaneously for 3 hours) – Joe Koberg Sep 16 '10 at 01:51
0

http://calculator.s3.amazonaws.com/calc5.html

It'd probably cost you about 15 dollars and you cut run it on a massive cluster of computers if the problem is easily parallelized.

Novikov
  • 4,399
  • 3
  • 28
  • 36
  • I'm sorry, I completely missed the part about 10,000 instances... This will be one of the cases where any and all further optimizations are worth it but either way it won't be very cheap ($2000 best case?) – Novikov Sep 15 '10 at 18:53