62

How can I translate the following code from Java to Python?

AtomicInteger cont = new AtomicInteger(0);

int value = cont.getAndIncrement();
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
user2983041
  • 1,761
  • 4
  • 19
  • 31

6 Answers6

52

Most likely with an threading.Lock around any usage of that value. There's no atomic modification in Python unless you use pypy (if you do, have a look at __pypy__.thread.atomic in stm version).

viraptor
  • 33,322
  • 10
  • 107
  • 191
  • 2
    I use acquire() and release() of lock an it work, but I think that is not efficient like atomic class in Java. – user2983041 May 08 '14 at 18:42
  • 22
    `with your_lock: variable += 1` is probably shorter. – viraptor May 08 '14 at 21:08
  • 4
    @viraptor, don't forget to take `value` from `variable` while the lock is still held – Michael Mol Jun 17 '16 at 12:33
  • 1
    FYI: suspending and resuming threads is expensive, hence the implementation of atomics in java, which do a repeated compare and swap operation instead of locking: https://www.baeldung.com/java-atomic-variables – lyjackal Apr 13 '19 at 14:42
33

itertools.count returns an iterator which will perform the equivalent to getAndIncrement() on each iteration.

Example:

import itertools
cont = itertools.count()
value = next(cont)
Will Manley
  • 2,340
  • 22
  • 17
  • 11
    At first glance this doesn't appear to be thread safe, is it? – Collin Nov 21 '14 at 14:35
  • 8
    The GIL guarantees it is because `cont.next()` is a single operation that both increments and returns the value. So it will be safe on CPython and PyPy. I don't know about Jython. – Will Manley Nov 21 '14 at 15:38
  • 33
    Ignoring for the moment my hesitation in using the ticking time-bomb that is relying on the GIL for synchronization, what makes you say that is definitively a single operation? It's a single _function call_, but is it a single bytecode operation? – Collin Nov 21 '14 at 16:11
  • 23
    [Yes](https://github.com/akheron/cpython/blob/d04cd9a21a22a56ec3cb577229808c019217b8e0/Modules/itertoolsmodule.c#L3999) – Will Manley Nov 24 '14 at 11:29
  • @Collin The itertools.count of CPython is implemented in C and does not release GIL. – Jiangge Zhang Jun 12 '16 at 09:55
  • 16
    @JianggeZhang I still agree with Colin's hesitation. In many contexts, it's not wise to rely on a _contingency_ for synchronization: i.e., implicit details of how it's _currently_ implemented that go beyond the "contract" and intent of the method. It's tantamount to relying on a side effect. For some contexts, this might be fine, but imo it's still iffy in general – Turix Jul 08 '16 at 23:18
  • 1
    @WillManley where do you see it's an atomic operation there's multiple operations there - what am I missing? – Guy Jul 13 '16 at 12:48
  • 2
    It's multiple operations in C, but it's a single Python bytecode operation over which the GIL will be held. – Will Manley Jul 13 '16 at 16:44
  • 24
    @WillManley it relies on an implementation detail of CPython (and PyPy), it's not a good practice to rely on implementation details. Moreover it would not be safe on Jython, see: http://www.jython.org/jythonbook/en/1.0/Concurrency.html#no-global-interpreter-lock and: https://bitbucket.org/jython/jython/src/91083509a11cdeadc9407b4d9a1ece2b8ffc45ce/src/org/python/modules/itertools/count.java?at=default&fileviewer=file-view-default#count.java-109 Please update the post. – Stan Prokop Jan 19 '17 at 20:33
  • 1
    @WillManley, I believe the code should read value = next(cont) rather than value = cont.next() (unless this is a Python 2 vs 3 issue). If you agree, please could you update your answer? – Eric McLachlan Dec 02 '19 at 12:55
21

This will perform the same function, although its not lockless as the name 'AtomicInteger' would imply.

Note other methods are also not strictly lockless -- they rely on the GIL and are not portable between python interpreters.

class AtomicInteger():
    def __init__(self, value=0):
        self._value = int(value)
        self._lock = threading.Lock()
        
    def inc(self, d=1):
        with self._lock:
            self._value += int(d)
            return self._value

    def dec(self, d=1):
        return self.inc(-d)    

    @property
    def value(self):
        with self._lock:
            return self._value

    @value.setter
    def value(self, v):
        with self._lock:
            self._value = int(v)
            return self._value
user48956
  • 14,850
  • 19
  • 93
  • 154
  • Why lock in value()... isnt retrieval an atomic op by itself?? – Tomer W Jun 07 '20 at 12:30
  • That sounds right. Although, you’d also need to check that value remains an int throughout, and not promoted to a long, imaginary or some kind of non atomic object. (E.g, what would happen if you supplied MyLoggingInt to the constructor) – user48956 Jun 07 '20 at 14:36
  • Isnt all those just Read reference? You eitger get the old value, or the newer. (If not, python is really messed up)... is it? – Tomer W Jun 08 '20 at 05:49
  • Actually, I'm not sure you can assume that. Is self.x. atomic. It certainly not *always* atomic (beceause `__getattr__`, `__getattribute__` may be implemented in subclasses. (Also, I've added int(x) to protect cases where the input is not an immutable, threadsafe object). – user48956 Nov 06 '20 at 19:59
  • @yahya - did you profile it? – user48956 Jun 13 '22 at 16:58
  • @Yahya Do you agree that its functionally atomic, if not atomic at the hardware level? – user48956 Sep 08 '22 at 16:15
  • 1
    @user48956 After another test, I found out it's working.. it was merely the platform I'm working on sluggish. +1 from me! – Yahya Sep 08 '22 at 16:43
15

Using the atomics library, the same could would be written in Python as:

import atomics


a = atomics.atomic(width=4, atype=atomics.INT)

value = a.fetch_inc()

This method is strictly lock-free.

Note: I am the author of this library

doodspav
  • 290
  • 3
  • 11
  • It looks like you are recommending your own library. Please be aware that in case of self-promotion, ["you must disclose your affiliation in your answers"](https://stackoverflow.com/help/promotion). – MisterMiyagi Nov 10 '21 at 17:03
  • My apologies, I'll do that now (including other comments) – doodspav Nov 10 '21 at 17:06
  • 1
    No worries, just wanted to let you know. Nice work, by the way. – MisterMiyagi Nov 10 '21 at 17:06
  • Hi. I am just a passerby here, and luckly just found your answer. It is great that it was posted just a few hours ago. So, let me ask you: What is `width=4`? It was not clear for me by reading the docs. Also, why should I care about the width at all? Further, the docs should include a GitHub link for contributing and also present simple cases understandable by newbies like the one in this answer instead of jumping right to complicated stuff like testing its correctness in multiple threads with heavy concurrency or memory mapping access. – Victor Stafusa - BozoNaCadeia Nov 11 '21 at 08:12
  • Also, instead of `atomics.atomic(width=4, atype=atomics.INT)`, it would be much more natural and straightforward if the API provides something as `atomics.atomic_int()` or `atomics.create_atomic_int()` or perhaps `atomics.new_atomic_int()`, which would simply call `atomics.atomic(width=4, atype=atomics.INT)`. Also, I suggest you to completely eliminate the `width` if possible or at least provide a default value (likely 4) for it. – Victor Stafusa - BozoNaCadeia Nov 11 '21 at 08:17
  • 1
    Anyway, congratulations for providing such library. – Victor Stafusa - BozoNaCadeia Nov 11 '21 at 08:17
  • Thank you! You're absolutely right that the docs could use a fair amount of work, right now I've just put them in the README to have *something* (better than nothing). I plan on writing proper docs over the next week or two. (1/3) – doodspav Nov 11 '21 at 08:44
  • The `width` param is there to specify the width of the underlying buffer in bytes. This library uses hardware atomic instructions provided by the `patomic` library, which operate on a fixed set of widths. This library might, for example, only provide support for `8` byte atomic operations on a specific platform (if the underlying API only provides that), and so creating an atomic object with a `4` byte width would raise `UnsupportedWidthException`. Additionally, the width sets the limit on the values representable by the type (so max value of `width=4` is different to `width=8`). (2/3) – doodspav Nov 11 '21 at 08:49
  • I do agree that some good defaults would be helpful, however I did need some way to make it clear that the atomic integer types are not arbitrary precision. I think that for now I'm going to focus on improving the documentation for what exists before modifying the library or adding new features. (3/3) – doodspav Nov 11 '21 at 08:52
1

8 years and still no full example code for the threading.Lock option without using any external library... Here it comes:

import threading

i = 0
lock = threading.Lock()

# Worker thread for increasing count
class CounterThread(threading.Thread):
    def __init__(self):
        super(CounterThread, self).__init__()
        
    def run(self):
        lock.acquire()
        global i
        i = i + 1
        lock.release()


threads = []
for a in range(0, 10000):
    th = CounterThread()
    th.start()
    threads.append(th)

for thread in threads:
    thread.join()

global i
print(i)
Mike B
  • 2,136
  • 2
  • 12
  • 31
0

Python atomic for shared data types.

https://sharedatomic.top

The module can be used for atomic operations under multiple processs and multiple threads conditions. High performance python! High concurrency, High performance!

atomic api Example with multiprocessing and multiple threads:

You need the following steps to utilize the module:

  1. create function used by child processes, refer to UIntAPIs, IntAPIs, BytearrayAPIs, StringAPIs, SetAPIs, ListAPIs, in each process, you can create multiple threads.

     def process_run(a):
       def subthread_run(a):
         a.array_sub_and_fetch(b'\x0F')
    
       threadlist = []
       for t in range(5000):
           threadlist.append(Thread(target=subthread_run, args=(a,)))
    
       for t in range(5000):
           threadlist[t].start()
    
       for t in range(5000):
           threadlist[t].join()
    
  2. create the shared bytearray

    a = atomic_bytearray(b'ab', length=7, paddingdirection='r', paddingbytes=b'012', mode='m')
    
  3. start processes / threads to utilize the shared bytearray

     processlist = []
    
     for p in range(2):
    
       processlist.append(Process(target=process_run, args=(a,)))
    
     for p in range(2):
    
       processlist[p].start()
    
     for p in range(2):
    
       processlist[p].join()
    
     assert a.value == int.to_bytes(27411031864108609, length=8, byteorder='big')