Reading a 64 bit variable that is updated by an ISR

Question

I am not finding much material on non-atomic operations.

Suppose I have a 32 bit processor and I want to keep count of microseconds in a 64 bit variable. An interrupt will update the variable every microsecond. The scheduler is non-preemptive. There will be a function to clear the variable and another to read it. Since it is a 32 bit processor then access will be non-atomic. Is there a “standard” or idiomatic way of handling this so that the reader function will not get a half-updated value?

Answer 1

Is there a “standard” or idiomatic way of handling this so that the reader function will not get a half-updated value?

What you need to do is use what I call "atomic access guards", or "interrupt guards". This is an area of interest of mine that I have spent a ton of time learning about and using in microcontrollers of various types.

@chux - Reinstate Monica, is correct, but here's some additional clarity I want to make:

For reading from volatile variables, make copies in order to read quickly:

Minimize time with the interrupts off by quickly copying out the variable, then using the copy in your calculation:

// ==========
// Do this:
// ==========

// global volatile variables for use in ISRs
volatile uint64_t u1;
volatile uint64_t u2;
volatile uint64_t u3;

int main()
{
    // main loop
    while (true)
    {
        uint64_t u1_copy;
        uint64_t u2_copy;
        uint64_t u3_copy;

        // use atomic access guards to copy out the volatile variables
        // 1. Save the current interrupt state
        const uint32_t INTERRUPT_STATE_BAK = INTERRUPT_STATE_REGISTER;
        // 2. Turn interrupts off
        interrupts_off();
        // copy your volatile variables out
        u1_copy = u1;
        u2_copy = u2;
        u3_copy = u3;
        // 3. Restore the interrupt state to what it was before disabling it.
        // This leaves interrupts disabled if they were previously disabled
        // (ex: inside an ISR where interrupts get disabled by default as it
        // enters--not all ISRs are this way, but many are, depending on your
        // device), and it re-enables interrupts if they were previously
        // enabled. Restoring interrupt state rather than enabling interrupts
        // is the right way to do it, and it enables this atomic access guard
        // style to be used both inside inside **and** outside ISRs.
        INTERRUPT_STATE_REGISTER = INTERRUPT_STATE_BAK;

        // Now use your copied variables in any calculations
    }
}

// ==========
// NOT this!
// ==========

volatile uint64_t u1;
volatile uint64_t u2;
volatile uint64_t u3;

int main()
{
    // main loop
    while (true)
    {
        // 1. Save the current interrupt state
        const uint32_t INTERRUPT_STATE_BAK = INTERRUPT_STATE_REGISTER;
        // 2. Turn interrupts off
        interrupts_off();

        // Now use your volatile variables in any long calculations
        // - This is not as good as using copies! This would leave interrupts
        //   off for an unnecessarily long time, introducing a ton of jitter
        //   into your measurements and code.

        // 3. Restore the interrupt state to what it was before disabling it.
        INTERRUPT_STATE_REGISTER = INTERRUPT_STATE_BAK;

    }
}

For writing to volatile variables, write quickly:

Minimize time with the interrupts off by quickly only disabling them while updating the volatile variables:

// global volatile variables for use in ISRs
volatile uint64_t u1;
volatile uint64_t u2;
volatile uint64_t u3;

int main()
{
    // main loop
    while (true)
    {
        // Do calculations here, **outside** the atomic access interrupt guards

        const uint32_t INTERRUPT_STATE_BAK = INTERRUPT_STATE_REGISTER;
        interrupts_off();
        // quickly update your variables and exit the guards
        u1 = 1234;
        u2 = 2345;
        u3 = 3456;
        INTERRUPT_STATE_REGISTER = INTERRUPT_STATE_BAK;
    }
}

Alternative: lock-free atomic reads via a repeat read loop: doAtomicRead(): ensure atomic reads withOUT turning interrupts off!

An alternative to using atomic access guards, as shown above, is to read the variable repeatedly until it doesn't change, indicating that the variable was not updated mid-read after you read only some bytes of it.

Here is that approach. @Brendan and @chux-ReinstateMonica and I discussed some ideas of it under @chux-ReinstateMonica's answer.

#include <stdint.h>  // UINT64_MAX

#define MAX_NUM_ATOMIC_READ_ATTEMPTS 3

// errors
#define ATOMIC_READ_FAILED (UINT64_MAX)

/// @brief          Use a repeat-read loop to do atomic-access reads of a 
///     volatile variable, rather than using atomic access guards which
///     disable interrupts.
uint64_t doAtomicRead(const volatile uint64_t* val)
{
    uint64_t val_copy;
    uint64_t val_copy_atomic = ATOMIC_READ_FAILED;
    
    for (size_t i = 0; i < MAX_NUM_ATOMIC_READ_ATTEMPTS; i++)
    {
        val_copy = *val; 
        if (val_copy == *val)
        {
            val_copy_atomic = val_copy;
            break;
        }
    }

    return val_copy_atomic;
}

If you want to understand deeper, here is the same doAtomicRead() function again, but this time with extensive explanatory comments. I also show a commented-out slight variation to it which may be helpful in some cases, as explained in the comments.

/// @brief          Use a repeat-read loop to do atomic-access reads of a 
///     volatile variable, rather than using atomic access guards which
///     disable interrupts.
///
/// @param[in]      val             Ptr to a volatile variable which is updated
///                                 by an ISR and needs to be read atomically.
/// @return         A copy of an atomic read of the passed-in variable, 
///     if successful, or sentinel value ATOMIC_READ_FAILED if the max number
///     of attempts to do the atomic read was exceeded.
uint64_t doAtomicRead(const volatile uint64_t* val)
{
    uint64_t val_copy;
    uint64_t val_copy_atomic = ATOMIC_READ_FAILED;
    
    // In case we get interrupted during this code block, and `val` gets updated
    // in that interrupt's ISR, try `MAX_NUM_ATOMIC_READ_ATTEMPTS` times to get
    // an atomic read of `val`.
    for (size_t i = 0; i < MAX_NUM_ATOMIC_READ_ATTEMPTS; i++)
    {
        val_copy = *val; 

        // An interrupt could have fired mid-read while doing the **non-atomic**
        // read above, updating the 64-bit value in the ISR and resulting in
        // 32-bits of the old value in the 64-bit variable being wrong now
        // (since the whole 64-bit value has just been updated with a new
        // value), so verify the read above with a new read again.
        // 
        // Caveat: 
        //
        // Note that this method is **not _always_** foolproof, as technically
        // the interrupt could fire off and run again during this 2nd read,
        // causing a very rare edge-case where the exact same incorrect value
        // gets read again, resulting in a false positive where it assigns an
        // erroneous value to `val_copy_atomic`! HOWEVER, that is for **you or
        // I** to design and decide as the architect. 
        //
        // Is it _possible_ for the ISR to really fire off again immediately
        // after returning? Or, would that never happen because we are
        // guaranteed some minimum time gap between interrupts? If the former,
        // you should read the variable again a 3rd or 4th time by uncommenting
        // the extra code block below in order to check for consistency and
        // minimize the chance of an erroneous `val_copy_atomic` value. If the
        // latter, however, and you know the ISR won't fire off again for at
        // least some minimum time value which is large enough for this 2nd
        // read to occur **first**, **before** the ISR gets run for the 2nd
        // time, then you can safely say that this 2nd read is sufficient, and
        // you are done.
        if (val_copy == *val)
        {
            val_copy_atomic = val_copy;
            break;
        }

        // Optionally delete the "if" statement just above and do this instead.
        // Refer to the long "caveat" note above to see if this might be
        // necessary. It is only necessary if your ISR might fire back-to-back
        // with essentially zero time delay between each interrupt.
        // for (size_t j = 0; j < 4; j++)
        // {
        //     if (val_copy == *val)
        //     {
        //         val_copy_atomic = val_copy;
        //         break;
        //     }
        // }
    }

    return val_copy_atomic;
}

The above could be optimized to only obtain a new reading of *val only once per iteration, instead of twice, by adding one extra read before the start of the loop, and reading only once in the loop, like this:

[This is my favorite version:]

uint64_t doAtomicRead(const volatile uint64_t* val)
{
    uint64_t val_copy_new;
    uint64_t val_copy_old = *val;
    uint64_t val_copy_atomic = ATOMIC_READ_FAILED;
    
    for (size_t i = 0; i < MAX_NUM_ATOMIC_READ_ATTEMPTS; i++)
    {
        val_copy_new = *val; 
        if (val_copy_new == val_copy_old)
        {
            // no change in the new reading, so we can assume the read was not 
            // interrupted during the first reading
            val_copy_atomic = val_copy_new;
            break;
        }
        // update the old reading, to compare it with the new reading in the
        // next iteration
        val_copy_old = val_copy_new;  
    }

    return val_copy_atomic;
}

General usage Example of doAtomicRead():

// global volatile variable shared between ISRs and main code
volatile uint64_t u1;

// Inside your function: "atomically" read and copy the volatile variable
uint64_t u1_copy = doAtomicRead(&u1);
if (u1_copy == ATOMIC_READ_FAILED)
{
    printf("Failed to atomically read variable `u1`.\n");

    // Now do whatever is appropriate for error handling; examples: 
    goto done;
    // OR:
    return;
    // etc.
}

This requires the writer to be atomic with respect to any readers, which is true, for example, in the case of a single writer writing to this variable. This write might occur inside an ISR, for example. We're detecting only torn reads (due to the reader being interrupted) and retrying. If the 64-bit value was ever already in a torn written state in memory when this reader ran, the reader could erroneously see it as valid.

A SeqLock doesn't have that limitation, so is useful for multi-core cases. But, if you don't need that (ex: you have a single-core microcontroller), it is probably less efficient, and the doAtomicRead() trick works just fine.

For the special edge case of a monotonically-incrementing counter (not for a variable which can be updated with any value, such as a variable storing a sensor reading!), as Brendan suggested here you only need to re-read the most-significant half of the 64-bit value and check that it didn't change. So, to (probably) slightly improve the efficiency of the doAtomicRead() function above, update it to do that. The only possible tearing (unless you miss 2^32 counts) is when the low half wraps and the high half gets incremented. This is like checking the whole thing, but retries will be even less frequent.

Going further on this topic of atomic access guards, disabling interrupts, etc.

1. My c/containers_ring_buffer_FIFO_GREAT.c demo from my eRCaGuy_hello_world repo. Description of this code example from my comments at the top of this file:

   Demonstrate a basic, efficient lock-free SPSC (Single-Producer Single-Consumer) ring buffer FIFO queue in C (that also runs in C++).

   This queue is intended to work lock-free in a SPSC context only, such as on a bare-metal microcontroller where an ISR needs to send data to the main loop, for example.

2. [Peter Cordes's answer on the SeqLock ("sequence lock") pattern] Implementing 64 bit atomic counter with 32 bit atomics

3. [my answer] C++ decrementing an element of a single-byte (volatile) array is not atomic! WHY? (Also: how do I force atomicity in Atmel AVR mcus/Arduino)

4. My long and detailed answer on Which Arduinos support ATOMIC_BLOCK? and:

   1. How are the ATOMIC_BLOCK macros implemented in C with the gcc compiler, and where can I see their source code?, and
   2. How could you implement the ATOMIC_BLOCK functionality in Arduino in C++ (as opposed to avrlibc's gcc C version)?
   3. I explain in detail how this really clever atomic access guard macro works in C via gcc extensions, and how it could easily be implemented in C++:

      ATOMIC_BLOCK(ATOMIC_RESTORESTATE)
      {
          my_var_copy = my_var;
      }
      

5. [my Q&A] Which variable types/sizes are atomic on STM32 microcontrollers?

   1. Not all variables need atomic access guards for simple reads and writes (for increment/decrement they ALWAYS do!--see my first link in this list above!), as some variables have naturally atomic reads and writes for a given architecture.
      1. For 8-bit AVR microcontrollers (like ATmega328 on Arduino Uno): 8-bit variables have naturally atomic reads and writes.
      2. For 32-bit STM32 microcontrollers, all non-struct (simple) types 32-bits and smaller have naturally atomic reads and writes. See my answer above for details and source documentation and proof.

6. Techniques to disable interrupts on STM32 mcus: https://stm32f4-discovery.net/2015/06/how-to-properly-enabledisable-interrupts-in-arm-cortex-m/

7. [my answer] global volatile variable not being updated in ISR: How to recognize and fix race conditions in Arduino by using atomic access guards:

8. [my answer] What are the various ways to disable and re-enable interrupts in STM32 microcontrollers in order to implement atomic access guards?

9. https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock

Answer 2

Within the ISR, a subsequent interrupt is usually prevented (unless a higher priority, but then the count is usually not touched there) so simply count_of_microseconds++;

Outside the ISR, to access (read or write) the count_of_microseconds you need interrupt protection or atomic access.

When atomic not available^*1 but interpret control is available:

volatile uint64_t count_of_microseconds;
...
saved_interrupt_state();
disable_interrupts();
uint64_t my_count64 = count_of_microseconds;
restore_interrupt_state();
// now use my_count64 

else use

atomic_ullong count_of_microseconds;
...
unsigned long long my_count64 = count_of_microseconds;
// now use my_count64 

See How to use atomic variables in C?

Since C89, use volatile with count_of_microseconds.

---

[Update]

Regardless of the approach used (this answer or others) in the non-ISR code to read/write the counter, I recommend to enclose the read/write code in a helper function to isolate this critical set of operations.

---

^*1 <stdatomic.h> available since C11 and __STDC_NO_ATOMICS__ not deifned.

#if __STDC_VERSION__ >= 201112
#ifndef __STDC_NO_ATOMICS__
#include <stdatomic.h>
#endif
#endif

Answer 3

I am glad to hear the that the read twice method is workable. I had doubts, don't know why. In the meantime I came up with this:

struct
{
    uint64_t ticks;
    bool toggle;
} timeKeeper = {0};

void timeISR()
{
    ticks++;
    toggle = !toggle;
}

uint64_t getTicks()
{
    uint64_t temp = 0;
    bool startToggle = false;
    
    do
    {
        startToggle = timeKeeper.toggle;
        temp = timekeeper.ticks;
    } while (startToggle != timeKeeper.toggle);
        
    return temp;
}