Why won't AnyEvent::child callbacks ever run if interval timer events are always ready?

Question

Update this issue can be resolved using the fixes present in https://github.com/zbentley/AnyEvent-Impl-Perl-Improved/tree/io-starvation

Context:

I am integrating AnyEvent with some otherwise-synchronous code. The synchronous code needs to install some watchers (on timers, child processes, and files), wait for at least one watcher to complete, do some synchronous/blocking/legacy stuff, and repeat.

I am using the pure-perl AnyEvent::Loop-based event loop, which is good enough for my purposes at this point; most of what I need it for is signal/process/timer tracking.

The problem:

If I have a callback that can block the event loop for a moment, child-process-exit events/callbacks never fire. The simplest example I could make watches a child process and runs an interval timer. The interval timer does something blocking before it finishes:

use AnyEvent;

# Start a timer that, every 0.5 seconds, sleeps for 1 second, then prints "timer":
my $w2 = AnyEvent->timer(
    after => 0,
    interval => 0.5,
    cb => sub {
        sleep 1; # Simulated blocking operation. If this is removed, everything works.
        say "timer";
    },
);

# Fork off a pid that waits for 1 second and then exits:
my $pid = fork();
if ( $pid == 0 ) {
    sleep 1;
    exit;
}

# Print "child" when the child process exits:
my $w1 = AnyEvent->child(
    pid => $pid,
    cb => sub {
        say "child";
    },
);

AnyEvent->condvar->recv;

This code leaves the child process zombied, and prints "timer" over and over, for "ever" (I ran it for several minutes). If the sleep 1 call is removed from the callback for the timer, the code works correctly and the child process watcher fires as expected.

I'd expect the child watcher to run eventually (at some point after the child exited, and any interval events in the event queue ran, blocked, and finished), but it does not.

The sleep 1 could be any blocking operation. It can be replaced with a busy-wait or any other thing that takes long enough. It doesn't even need to take a second; it appears to only need to be a) running during the child-exit event/SIGCHLD delivery, and b) result in the interval always being due to run according to the wallclock.

Questions:

Why isn't AnyEvent ever running my child-process watcher callback?

How can I multiplex child-process-exit events with interval events that may block for so long that the next interval becomes due?

What I've tried:

My theory is that timer events which become "ready" due to time spent outside of the event loop can indefinitely pre-empt other types of ready events (like child process watchers) somewhere inside AnyEvent. I've tried a few things:

• Using AnyEvent::Strict doesn't surface any errors or change behavior in any way.
• Partial solution: Removing the interval event at any point does make the child process watcher fire (as if there's some internal event polling/queue population done inside AnyEvent that only happens if there are no timer events already "ready" according to the wallclock). Drawbacks: in the general case that doesn't work, since I'd have to know when my child process had exited to know when to postpone my intervals, which is tautological.
• Partial solution: Unlike child-process watchers, other interval timers seem to be able to multiplex with each other just fine, so I can install a manual call to waitpid in another interval timer to check for and reap children. Drawbacks: child-waiting can be artificially delayed (my use case involves lots of frequent process creation/destruction), any AnyEvent::child watchers that are installed and fire successfully will auto-reap the child and not tell my interval/waitpid timer, requiring orchestration, and it just generally feels like I'm misusing AnyEvent.

Answer 1

The interval is the time between the start of each timer callback, i.e. not the time between the end of a callback and the start of the next callback. You setup a timer with interval 0.5 and the action for the timer is to sleep one second. This means that once the timer is triggered it will be triggered immediately again and again because the interval is always over after the timer returned.

Thus depending on the implementation of the event loop it might happen that no other events will be processed because it is busy running the same timer over and over. I don't know which underlying event loop you are using (check $AnyEvent::MODEL) but if you look at the source code of AnyEvent::Loop (the loop for the pure Perl implementation, i.e. model is AnyEvent::Impl::Perl) you will find the following code:

   if (@timer && $timer[0][0] <= $MNOW) {
      do {
         my $timer = shift @timer;
         $timer->[1] && $timer->[1]($timer);
      } while @timer && $timer[0][0] <= $MNOW;

As you can see it will be busy executing timers as long as there are timers which need to run. And with your setup of the interval (0.5) and the behavior of the timer (sleep one second) there will always be a timer which needs to be executed.

If you instead change your timer so that there is actual room for the processing of other events by setting the interval to be larger than the blocking time (like 2 seconds instead of 0.5) everything works fine:

...
interval => 2,
cb => sub {
    sleep 1; # Simulated blocking operation. Sleep less than the interval!!
    say "timer";


...
timer
child
timer
timer

Answer 2

Update this issue can be resolved using the fixes present in https://github.com/zbentley/AnyEvent-Impl-Perl-Improved/tree/io-starvation

@steffen-ulrich's answer is correct, but points out a very flawed behavior in AnyEvent: since there is no underlying event queue, certain kinds of events that always report "ready" can indefinitely pre-empt others.

Here is a workaround:

For interval timers that are always "ready" due to a blocking operation that happens outside of the event loop, it is possible to prevent starvation by chaining interval invocations onto the next run of the event loop, like this:

use AnyEvent;

sub deferred_interval {
    my %args = @_;
    # Some silly wrangling to emulate AnyEvent's normal
    # "watchers are uninstalled when they are destroyed" behavior:
    ${$args{reference}} = 1;
    $args{oldref} //= delete($args{reference});
    return unless ${$args{oldref}};

    AnyEvent::postpone {
        ${$args{oldref}} = AnyEvent->timer(
            after => delete($args{after}) // $args{interval},
            cb => sub {
                $args{cb}->(@_);
                deferred_interval(%args);
            }
        );
    };

    return ${$args{oldref}};
}

# Start a timer that, at most once every 0.5 seconds, sleeps
# for 1 second, and then prints "timer":
my $w1; $w1 = deferred_interval(
    after => 0.1,
    reference => \$w2,  
    interval => 0.5,
    cb => sub {
        sleep 1; # Simulated blocking operation.
        say "timer";
    },
);

# Fork off a pid that waits for 1 second and then exits:
my $pid = fork();
if ( $pid == 0 ) {
    sleep 1;
    exit;
}

# Print "child" when the child process exits:
my $w1 = AnyEvent->child(
    pid => $pid,
    cb => sub {
        say "child";
    },
);

AnyEvent->condvar->recv;

Using that code, the child process watcher will fire more or less on time, and the interval will keep firing. The tradeoff is that each interval timer will only start after each blocking callback finishes. Given an interval time of I and a blocking-callback runtime of B, this approach will fire an interval event roughly every I + B seconds, and the previous approach from the question will take min(I,B) seconds (at the expense of potential starvation).

I think that a lot of the headaches here could be avoided if AnyEvent had a backing queue (many common event loops take this approach to prevent situations exactly like this one), or if the implementation of AnyEvent::postpone installed a "NextTick"-like event emitter to be fired only after all other emitters had been checked for events.