Airflow task: OSError: [Errno 23] Too many open files in system

Question

Does anyone has had this error with the S3PrefixSensor?

OSError: [Errno 23] Too many open files in system: '/usr/local/lib/python3.6/dist-packages/botocore/data/endpoints.json'

I'm having that error when the scheduler runs over 12 tasks with that operator at the same time. If I rerun them manually, they work fine.

I tried increasing the ulimit as suggested by the answer of this question but it didn't work for me: Errno 24: Too many open files. But I am not opening files?

It's odd that error is coming up, as I'm only running 12 tasks at the same time. Is it an issue with the S3 sensor operator?

What is the output of `sudo lsof | grep -c endpoints.json` and `ulimit -a | grep 'open files'` — lllrnr101, Mar 04 '21 at 17:09
@lllrnr101 The output of the first command is 0 and the 2nd command is ```open files (-n) 500240``` — ultraInstinct, Mar 04 '21 at 17:18
Damn! Then I will bookmark the question to see how you finally solved it. I would also keep an eye on `lsof | grep -c ` after starting the program. — lllrnr101, Mar 04 '21 at 17:21
Are you importing the sensor from `airflow.sensors.s3_prefix_sensor.S3PrefixSensor` ? — Elad Kalif, Mar 08 '21 at 14:12

score 3 · Accepted Answer · answered Mar 08 '21 at 17:02

You mentioned in comments that you use airflow.sensors.s3_prefix_sensor.S3PrefixSensor This is an outdated version of the sensor.

The updated version contains PR which caches the hook and prevent creating a new connection every time the sensor poke.

For Airflow<2.0.0 install backport provider:

pip install apache-airflow-backport-providers-amazon

For Airflow>=2.0.0 install providers:

pip install apache-airflow-providers-amazon

Then import the sensor via:

from airflow.providers.amazon.aws.sensors.s3_prefix import S3PrefixSensor

1 Answers1