Stack Overflow
Stack Overflow

Questions tagged [starcluster]

An open source cluster-computing toolkit for Amazon’s Elastic Compute Cloud (EC2).

StarCluster is an open source cluster-computing toolkit mainly developed in for Amazon’s Elastic Compute Cloud () released under the LGPL license.

StarCluster has been designed to automate and simplify the process of building, configuring, and managing clusters of virtual machines on Amazon’s EC2 cloud. StarCluster allows anyone to easily create a cluster computing environment in the cloud suited for distributed and parallel computing applications and systems.

52 questions
7
votes
2 answers

Using snow (and snowfall) with AWS for parallel processing in R

In relation to my earlier similar SO question , I tried using snow/snowfall on AWS for parallel computing. What I did was: In the sfInit() function, I provided the public DNS to socketHosts parameter like so sfInit(parallel=TRUE,socketHosts…
harshsinghal
  • 3,720
  • 8
  • 35
  • 32
6
votes
1 answer

Alternatives to StarCluster?

StarCluster is a well known toolkit for Amazon EC2. However, it was developed based on Python 2, which is going out if date. And it is not compatible with Python 3.x. So I'd like to know is there any alternative to StarCluster? I have searched in…
Lingze
  • 61
  • 2
6
votes
0 answers

Is there a Starcluster equivalent for Google Compute Engine (GCE) yet?

Does anyone know if there is a Starcluster equivalent for GCE? I have been quite happy using Starcluster with EC2 for embarrisingly parallel jobs. Now I want to try out GCE. I would be happy to contribute to whatever projects might be in the…
Jose Cortez
  • 320
  • 3
  • 10
5
votes
1 answer

How to run StarCluster with AWS IAM Role

How can I modify StarCluster config or code to include IAM Role? So that when I do this: starcluster start mycluster The IAM Role can be automatically activated in EC2 instances (master and nodes). I did try to look around but of no avail info.
neversaint
  • 60,904
  • 137
  • 310
  • 477
5
votes
3 answers

Send input to command line prompt from Python program

I believe this to be a very simple question, but I have been failing to find a simple answer. I am running a python program that terminates an AWS cluster (using starcluster). I am just calling a command from my python program using subprocess,…
lbrendanl
  • 2,626
  • 4
  • 33
  • 54
5
votes
3 answers

Tools for setting up and running a grid job on Google Compute Engine?

I have the need to set up and run "embarrassingly" parallel jobs on Google Compute Engine. I am looking for tools to facilitate this. On EC2, I was using MIT's Starcluster to set up the cluster, and then just submitting the job to SGE. Are there…
Jose Cortez
  • 320
  • 3
  • 10
5
votes
1 answer

cluster computing using starcluster and ipython on AWS

I'm trying to experiment some with cluster computing on AWS. I'm completely new at this and having some issues. I'm trying to follow the tutorial found here: http://star.mit.edu/cluster/docs/latest/plugins/ipython.html#using-the-ipython-cluster. I…
user1074057
  • 1,772
  • 5
  • 20
  • 30
5
votes
1 answer

Deploying scientific python algorithm on Amazon ec2

I have a Python scientific model that calls some C code and uses numpy, scipy, and many geographic analysis modules. I would like to deploy it on EC2 but I don't know much about EC2 yet. I have checked that I could use the StarCluster package to…
PeterS
  • 51
  • 3
4
votes
3 answers

Ensuring one Job Per Node on StarCluster / SunGridEngine (SGE)

When qsubing jobs on a StarCluster / SGE cluster, is there an easy way to ensure that each node receives at most one job at a time? I am having issues where multiple jobs end up on the same node leading to out of memory (OOM) issues. I tried using…
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
3
votes
1 answer

How to mount S3 bucket to StarCluster

Under the standard single instance I would mount the S3-Bucket with the following command using Goofy (or alternatively s3fs) my-ec2$ ./go/bin/goofys mybucket my_mountdir How can I do that under StarCluster? Does it allow S3 mounting at all? Or is…
neversaint
  • 60,904
  • 137
  • 310
  • 477
3
votes
2 answers

IAM Permissions Required by StarCluster

I am following the StarCluster configuration instructions and I would like to create a new user for StarCluster to use. My question is what are the minimal set of IAM permissions that StarCluster requires to operate? Is the AmazonEC2FullAccess…
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
3
votes
1 answer

Changing number of available slots in running SGE Instance

I would like to change the number of slots on a running SGE instance. The instance was started by StarCluster. I tried following this page and running: ubuntu@master: $ qconf -mattr exechost complex_values slots=1 master ubuntu@master modified…
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
3
votes
0 answers

IOError error message in IPython notebook when using StarCluster

I am running a small StarCluster and using it to run an IPython Notebook. Every time I have an error in the code I'm writing in the notebook, I get the following error message added onto the end of the notebook's output: Traceback (most recent call…
Maximilian
  • 7,512
  • 3
  • 50
  • 63
3
votes
1 answer

starcluster easy_install windows

I am trying to setup a R cluster on AWS using StarCluster and lateron BioCoductor AMI. The objective is to run some parallel computations on multiple nodes using snow. However I am already stuck at the beginning in installing StarCluster on WINDOWS…
user2825843
  • 31
  • 2
3
votes
1 answer

Parallel processing of small functions in the cloud

I'm having a few million/billion (10^9) data-input-sets, that need to be processed. They are quiet small < 1kB. And they need about 1 second to be processed. I have read a lot about Apache Hadoop, Map Reduce and StarCluster. But I am not sure what…
Mark
  • 7,507
  • 12
  • 52
  • 88
1
2 3 4