48

I have two integers in my program; let's call them "a" and "b". I would like to add them together and get another integer as a result. These are regular Python int objects. I'm wondering; how do I add them together with Twisted? Is there a special performAsynchronousAddition function somewhere? Do I need a Deferred? What about the reactor? Is the reactor involved?

Glyph
  • 31,152
  • 11
  • 87
  • 129
  • 4
    Why wouldn't you use `a + b`? – Gabe Sep 06 '11 at 00:14
  • 15
    Won't "`a + b`" block? – Glyph Sep 06 '11 at 04:11
  • 4
    Since `a + b` is not I/O, it will not block. It will take some time to execute, but so will any other operation that you do. – Gabe Sep 06 '11 at 04:12
  • 4
    To be clear a+b does block. What if a and b are sufficiently huge integers ? Keep in mind that non-blocking very often means "blocking for a very very small time". – Jeremy Oct 09 '13 at 10:20
  • 2
    I think your use of "block" here is inconsistent with general usage. Normally, a process is described as "blocked" when it becomes non-runnable, waiting for some external event (such as the end of a network transfer). `a+b` as a normal arithmetic operation will never block - it will just use CPU until it's completed, so the process/thread that runs it will remain runnable throughout. – holdenweb May 24 '17 at 10:34
  • 1
    It's true that the idiomatic usage of the term "block" often means "stuck on CPU", but when making formal distinctions it's good to use another term ("stuck", or "hung") to indicate paused waiting on CPU rather than paused waiting on I/O. – Glyph May 26 '17 at 00:30

3 Answers3

43

OK, to be clear.

Twisted doesn't do anything about cpu bound tasks and for good reason. there's no way to make a compute bound job go any quicker by reordering subtasks; the only thing you could possibly do is add more compute resources; and even that wouldn't work out in python because of a subtlety of its implementation.

Twisted offers special semantics and event loop handling in case the program would become "stuck" waiting for something outside if its control; most normally a process running on another machine and communicating with your twisted process over a network connection. Since you would be waiting anyways, twisted gives you a mechanism to get more things done in the meantime. That is to say, twisted provides concurrency for I/O Bound tasks

tl;dr: twisted is for network code. Everything else is just normal python.

SingleNegationElimination
  • 151,563
  • 33
  • 264
  • 304
  • 5
    (Worth noting though is that with `spawnProcess`, Twisted does provide a *little* support for CPU-bound tasks, by converting them into I/O bound tasks and then communicating with them the same way. But this is a good explanation nonetheless.) – Glyph Sep 06 '11 at 04:16
  • 5
    As a friend and neighbor, I'm begging you: Please do not spawn a process for the purpose of adding two integers. – SingleNegationElimination Sep 06 '11 at 04:49
  • 26
    Tempting as it is to troll you further, I should note at this point that I'm the original author of Twisted and that this question was written as a sort of experiment. We get a lot of questions of the flavor: "How do I call function `X`, with Twisted?", where the answer ("`X()`") is so obvious that we have trouble communicating it. Many programmers are too used to using idioms like "`time.sleep()`" to mean "... and then do a bunch of I/O and/or computation ... " that they can't or won't flesh out their examples further, when the distinction between I/O and computation is really critical. – Glyph Sep 06 '11 at 07:45
  • 2
    I actually quite liked your answer and will probably use elements from it in the future. I was surprised to get such useful feedback so quickly ;). – Glyph Sep 06 '11 at 07:48
9

How about this:

c = a + b

That should work, and it doesn't need to be done asynchronously (it's pretty fast).

Keith
  • 42,110
  • 11
  • 57
  • 76
-9

Good question, and Twisted (or Python) should have a way to at least spawn "a + b" of to several cores (on my 8 core i7).

Unfortunately the Python GIL prevents this from happening, meaning that you will have to wait, not only for the CPU bound task, but for one core doing the job, while the seven others core are doing nothing.

Note: Maybe a better example would be "a() + b()", or even "fact(sqrt(a()**b())" etc. but the important fact is that above operation will lock one core and the GIL pretty much prevents Python for doing anything else during that operation, which could be several ms...

renejsum
  • 69
  • 7
  • -1: whether or not `a + b` runs on a single or multiple CPUs depends on what `a` and `b` are e.g., it is possible to release GIL if `a` and `b` are numpy arrays. The task may be embarrassingly parallel i.e., it is trivial to distribute it among several CPUs. The result can be slower than sequential variant but it can be done easily. – jfs Dec 21 '13 at 07:39
  • It is not possible to distribute processing across several cores in the same CPU and process, because of the GIL. So even if I have 8 cores in my brand new MBP and I started 8 threads to distribute a calculation, the performance will degrade a factor 1000 or more. – renejsum Dec 28 '13 at 22:38
  • it is possible to distribute processing across several CPU cores in the same process because GIL can be released in C extensions such as `numpy`, `lxml`, `regex`, etc. – jfs Dec 29 '13 at 05:26
  • I understand that there are workarounds, but if you just wanted to start 8 threads in "pure" Python and have them crunch ahead one on each core, possible coordinating progress through some locks you are out of luck... – renejsum Dec 30 '13 at 01:11
  • It's probably a bit late, but have a look at Pony language, it has an `actor` keyword, which means async class/object. Pony will start a thread for each core on CPU, potentially making the same program running on a 16 core CPU (almost) twice as fast as running on a 8-core. I wish I could do the same in Python, but the GIL prevent's it. (It does work in GIL'less Jython and IronPython though) – renejsum Jan 11 '18 at 06:12