60

If I set threadsafe: true in my app.yaml file, what are the rules that govern when a new instance will be created to serve a request, versus when a new thread will be created on an existing instance?

If I have an app which performs something computationally intensive on each request, does multi-threading buy me anything? In other words, is an instance a multi-core instance or a single core?

Or, are new threads only spun up when existing threads are waiting on IO?

Moishe Lettvin
  • 8,462
  • 1
  • 26
  • 40
  • 1
    [This answer](http://stackoverflow.com/a/11443482/236564) might help answer your first question. Takashi states that AE instance are limited to 10 concurrent threads, so it appears the 11th thread would start a new instance. – Kyle Finley Jul 18 '12 at 06:01
  • 25
    +1 for publicly asking this on SO and not by email to your coworkers. – systempuntoout Jul 18 '12 at 07:33

3 Answers3

37

The following set of rules are currently used to determine if a given instance can accept a new request:

if processing more than N concurrent requests (today N=10): false
elif exceeding the soft memory limit: false
elif exceeding the instance class CPU limit: false
elif warming up: false
else true

The following of total CPU/core limits currently apply to each instance classes:

CLASS 1: 600MHz 1 core
CLASS 2: 1.2GHz 1 core
CLASS 4: 2.4GHz 1 core
CLASS 8: 4.8GHz 2 core

So only a B8 instance can process up to 2 fully CPU bound requests in parallel.

Setting threadsafe: true (Python) or <threadsafe>true</threadsafe> (Java) for instances classes < 8 would not allow more than one CPU bound requests to be processed in parallel on a single instance.

If you are not fully CPU bound or doing I/O, the Python and Java runtime will spawn new threads for handling new request up to 10 concurrent requests with threadsafe: true

Also note that even though the Go runtime is single threaded, it does support concurrent requests: It will spawn 1 goroutine per requests and yield control between goroutines while they are performing I/O.

Fred Sauer
  • 1,012
  • 9
  • 20
proppy
  • 10,495
  • 5
  • 37
  • 66
  • This applies only to backends, right? For frontend instances, I see new instances spin up when none of the above rules are triggered. – Kris Giesing Aug 09 '12 at 14:33
  • 1
    @KrisGiesing that also applies to front end but the decision of spawning a new instance for handling new requests versus waiting in the queue, also depends on your performance settings. – proppy Aug 09 '12 at 14:38
  • @proppy thank you for answering this. If new instances are spun up based on CPU and memory, why does the concurrent request limit still exist? I though the rationale behind the N=10 was to prevent out of memory errors. Could the N=10 check be removed, or is there another reason for it? – Kyle Finley Aug 09 '12 at 15:40
  • @KrisGiesing there is a feature request about that, let's move the discussion here: http://code.google.com/p/googleappengine/issues/detail?id=7927 – proppy Aug 10 '12 at 12:50
  • @proppy I don't think that issue is relevant to me. I'm talking about this: http://code.google.com/p/googleappengine/issues/detail?id=7910 – Kris Giesing Aug 11 '12 at 13:44
  • @proppy if your last comment was directed at me - I've posted the question as a comment to that issue. Thank you. – Kyle Finley Aug 13 '12 at 18:31
  • @proppy could you please post a link from the app engine documentation, that show instance cores number ! thanks – david Dec 11 '14 at 12:55
1

Read the next messages from link which was suggested by Kyle Finley

Jeff Schnitzer: Is there still a hard limit of 10 threads?

Yes, but probably not for the reason you expect. The primary issue we run into is memory management. If we raised the default to 100, many apps would then see out-of-memory deaths (more than they do now), and these deaths show up differently for python/java/go. The right path forward is more intelligent algorithms wrt memory, providing configurability, and so on. This is an example of the kinds of projects we work on for the scheduler, but as with any team we have to prioritize our projects. I'd recommend filing this (or any other desired scheduler enhancements) on the public issue tracker so they can get feedback/data/votes.

maxiperez
  • 1,470
  • 2
  • 20
  • 40
  • I think that's part of the answer, but it doesn't address the multi/single core aspect of instances. For anyone interested in raising this concurrent request limit please star [Allow configurable limit of concurrent requests per instance](http://code.google.com/p/googleappengine/issues/detail?id=7927) – Kyle Finley Aug 07 '12 at 15:17
  • It also doesn't address at all whether an instance needs to actually have 10 active threads before a new instance is created. – Wooble Aug 07 '12 at 18:27
0

If I set threadsafe: true in my app.yaml file, what are the rules that govern when a new instance will be created to serve a request, versus when a new thread will be created on an existing instance?

Like people are saying here, if a previous instance is already using 10 threads, a new instance with a new thread would be initiated. A new thread will be created if all other threads are busy, they must be either waiting for some response or with computing results.

If I have an app which performs something computationally intensive on each request, does multi-threading buy me anything? In other words, is an instance a multi-core instance or a single core?

Now this question is very controversial. Everyone knows the answer but still they are skeptical. Multi-threading can never buy you any good if your task is based on just computations unless you're using a multi-core processor, don't ask me why a multi-core processor will help better, you know the answer. Now google app engine is not sophisticated enough to decide that when new threads should be dispatched to the other processor/core(if it exists), only new instances are dispatched to the other core/processor. Want your thread to run in the other core/processor? Well, throw some skills there and booya! Remember, it's upto you to decide if threads should run in other cores/processors, the engine can not take the responsibility for such because this could lead to so many confusions, the engine is not God. In short, by default the instance is single core, the engine can't decide for you when it should go multi-core.

Or, are new threads only spun up when existing threads are waiting on IO?

The first part of my answer clears this out. Yes, they only spun up when existing threads are busy, this is how threadsafe works, to prevent deadlocks.

Now I can tell you this all, from my personal experience, I worked on the app engine for many months and programmed/debugged/tested apps that were highly dependent on the threadsafe architecture. If you want I can add references(I don't have references, just personal experience, but I'm ready to search and put things on the table for you), but I don't think they are needed in this case, threadsafe works in obvious ways which I have validated myself.

Derpy Derp
  • 429
  • 1
  • 5
  • 12