Android: Too many open files error

Question

I have the following operation which runs every 3 seconds.
Basically it downloads a file from a server and save it into a local file every 3 seconds.
The following code does the job for a while.

public class DownloadTask extends AsyncTask<String, Void, String>{

    @Override
    protected String doInBackground(String... params) {
        downloadCommandFile( eventUrl);
        return null;
    }


}

private void downloadCommandFile(String dlUrl){
    int count;
    try {
        URL url = new URL( dlUrl );
        NetUtils.trustAllHosts();
        HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
        con.setDoInput(true);
        con.setDoOutput(true);
        con.connect();
        int fileSize = con.getContentLength();
        Log.d(TAG, "Download file size = " + fileSize );
        InputStream is = url.openStream();
        String dir = Environment.getExternalStorageDirectory() + Utils.DL_DIRECTORY;
        File file = new File( dir );
        if( !file.exists() ){
            file.mkdir();
        }

        FileOutputStream fos = new FileOutputStream(file + Utils.DL_FILE);
        byte data[] = new byte[1024];
        long total = 0;

        while( (count = is.read(data)) != -1 ){
            total += count;
            fos.write(data, 0, count);
        }

        is.close();
        fos.close();
        con.disconnect(); // close connection


    } catch (Exception e) {
        Log.e(TAG, "DOWNLOAD ERROR = " + e.toString() );
    }

}

Everything works fine, but if I leave it running for 5 to 10 minutes I get the following error.

06-04 19:40:40.872: E/NativeCrypto(6320): AppData::create pipe(2) failed: Too many open files 06-04 19:40:40.892: E/NativeCrypto(6320): AppData::create pipe(2) failed: Too many open files 06-04 19:40:40.892: E/EventService(6320): DOWNLOAD ERROR = javax.net.ssl.SSLException: Unable to create application data

I have been doing some researches for the last 2 days.
There are suggestions that they are many connections open, like this one https://stackoverflow.com/a/13990490/1503155 but still I can not figure out what's the problem.
Any ideas what may cause the problem?
Thanks in advance.

Did you manage to eventually figure out what was the problem? — Bundeeteddee, Aug 05 '14 at 09:00

score 9 · Accepted Answer · answered Jun 05 '13 at 01:20

9

I think you get this error because you have too many files open at the same times, meaning that you have too many async tasks running in the same time (each async task opens a file), which makes sense if you say that you run a new one every 3 seconds.

You should try to limit the number of async task running in the same time using a thread pool executor.

answered Jun 05 '13 at 01:20

lukasz

3,121
1
21
20

2

thanks for the valuable information. Any idea how to limit the number of async task running at the same time? – Lazy Ninja Jun 05 '13 at 01:38
3

If you don't support Android 2.x, you can execute your asynctask by providing your own thread pool executor : `Executor e = Executors.newFixedThreadPool(3)` and then running your async tasks : `executeOnExecutor(e);` – lukasz Jun 05 '13 at 01:47
As an alternative option, I would you to somewhat manage your async tasks progress (by checking tasks that are finished via onPostExecute()) and controlling manually the number currently running asynctasks. – lukasz Jun 05 '13 at 01:50
thanks! Now I am testing without using Async task by adding StrictMode.ThreadPolicy to permit all. If it works, I will try your suggestion. By the way the app supports only 4.xxx – Lazy Ninja Jun 05 '13 at 01:54
Ok! Try to find by making some tests the optimal limit for your thread pool that allows an appropriate number of parallel tasks without crashing. – lukasz Jun 05 '13 at 02:01
Also check out this thread: http://stackoverflow.com/questions/20367647/android-httpurlconnection-is-not-closing-eventually-results-to-socketexceptio Something to do with Keep-Alive connections not closing the connection properly – Bundeeteddee Aug 05 '14 at 10:27
1

AsyncTask already runs on a job queue with a limited number of threads. So it seems not very relevant to cause this error. – stdout Apr 10 '16 at 08:01
@stdout is correct. `AsyncTask` already runs in a threadpool that is common and shared amongst all `AsyncTasks`, unless you specify otherwise. The issue here is that the file descriptors are not being closed properly and in time. – Joshua Pinter Oct 05 '19 at 16:23

score 1 · Answer 2 · answered Oct 05 '19 at 16:27

Try using OkHttp Instead.

Your issue isn't with too many threads, although that's what's causing your issue to surface.

As @stdout mentioned in the comments, AsyncTask already runs in a threadpool that is common and shared amongst all AsyncTasks, unless you specify otherwise. The issue here is that the file descriptors are not being closed properly and in time.

The issue is that your file descriptors are not being closed fast enough.

I struggled with this for hours/days/weeks, doing everything you should like setting small read/connect timeouts and using a finally block to close out connections, input streams, output streams, etc. But we never found a working solution. It seemed like HttpsUrlConnection was flawed in some way.

So we tried OkHttp as a drop-in replacement for HttpsUrlConnection and voila! It worked out of the box.

So, if you're struggling with this and are having a really hard time fixing it, I suggest you try using OkHttp as well.

Here are the basics:

Once you get the Maven dependency added, you can do something like the following to download a file:

OkHttpClient okHttpClient = new OkHttpClient.Builder().build();

OutputStream output = null;

try {
  Request request   = new Request.Builder().url( download_url ).build();
  Response response = okHttpClient.newCall( request ).execute();

  if ( !response.isSuccessful() ) {
    throw new FileNotFoundException();
  }

  output = new FileOutputStream( output_path );

  output.write( response.body().bytes() );
}
finally {
  // Ensure streams are closed, even if there's an exception.
  if ( output != null ) output.flush();
  if ( output != null ) output.close();
}

Switching to OkHttp instantly fixed our leaked file descriptor issue so it's worth trying if you're stuck, even at the expense of adding another library dependency.

That didn't helped. Looks like Android frees pipes and sockets from connections after some time. The only solution worked for me was using a ThreadPool of 8 threads. — Artem Mostyaev, Dec 10 '21 at 12:57

score 0 · Answer 3 · answered Dec 10 '21 at 13:17

I had to download several hunders of files at a time and meet the error.

You may check open descriptors with the following command:

adb shell ps

Find your application PID in the list and use another command:

adb shell run-as YOUR_PACKAGE_NAME ls -l  /proc/YOUR_PID/fd

I see about 150 open descriptors on a usual launch. And there are 700+ when files are downloading. Their number decreases only after some minutes, looks like Android frees them in the backgound, not when you can close on a stream.

The only working solution was use a custom ThreadPool to limit the concurrency. Here is the Kotlin code:

private val downloadDispatcher = Executors.newFixedThreadPool(8).asCoroutineDispatcher()

private suspend fun downloadFile(sourceUrl: String, destPath: String, progressBlock: suspend (Long) -> Unit) = withContext(downloadDispatcher) {
    val url = URL(sourceUrl)
    url.openConnection().apply { connect() }
    url.openStream().use { input ->
        FileOutputStream(File(destPath)).use { output ->
            val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
            var bytesRead = input.read(buffer)
            var bytesCopied = 0L
            while (bytesRead >= 0) {
                if (!coroutineContext.isActive) break
                output.write(buffer, 0, bytesRead)
                bytesCopied += bytesRead
                progressBlock(bytesCopied)
                bytesRead = input.read(buffer)
            }
        }
    }
}

Android: Too many open files error

3 Answers3

Try using OkHttp Instead.

Linked