Java Mail IMAP search taking lot of time when there are lot of messages in mail box

Question

I have written a scheduler to run every one minute and use IMAP (SSL) API to get all unread messages and then mark only 200 messages from it as read and then read content of each message (200) one by one. Issue I am facing is when mail box is having around 400k messages (100GB) the first command to get unread messages itself taking more than 10 mins. I am not sure if this is how imap behaves or is it something slow at mailbox or network level. Eventually my target is to read around 500k emails in 24 hours from mail box where one message will be of around 250kb and then store each HTML message in oracle DB as blob. Currently I am no where near to achieve this target. I am attaching my code below. It is processing only 50 messages in a minute. I would really appreciate if some one can guide me to fix any performance issues in my code. Also if some has experience of extracting HTML email from mail box and persisting in database using any way, It would be really helpful if you can share your knowledge. Thanks!

public void readMails() {



    int arraySize = fetchSize / threadCount;

    try {
            FlagTerm ft = new FlagTerm(new Flags(Flags.Flag.SEEN), false);
            msgs = inbox.search(ft);

            if (msgs.length > fetchSize) {
                batchMsgs = Arrays.copyOfRange(msgs, 0, fetchSize - 1);
            } else {
                batchMsgs = msgs;
            }

            inbox.setFlags(batchMsgs, new Flags(Flags.Flag.SEEN), true);


        if (batchMsgs.length != 0) {
            archiveTaskExecutor.initialize();
            List<Message> tempMsgs = new ArrayList<Message>();
            // Message[] tempMsgs = new Message[arraySize];
            // int i = 0;
            int j = batchMsgs.length;
            for (Message m : batchMsgs) {
                tempMsgs.add(m);
                // i++;
                if (tempMsgs.size() >= arraySize || j <= 1) {
                    archiveTaskExecutor
                            .execute(new ExtractAndPersist(tempMsgs
                                    .toArray(new Message[tempMsgs.size()])));
                    tempMsgs = new ArrayList<Message>();
                    // tempMsgs = new Message[arraySize];
                    // i = 0;
                }
                j--;
            }
            archiveTaskExecutor.shutdown();
            try {
                archiveTaskExecutor.getThreadPoolExecutor()
                        .awaitTermination(15, TimeUnit.MINUTES);
            } catch (InterruptedException e) {

                archiveTaskExecutor.getThreadPoolExecutor().shutdownNow();
            }

        }
    } catch (Exception e) {
        /** revert all messages to UNREAD here **/

    }
}

private class ExtractAndPersist implements Runnable {

    final Logger log = Logger.getLogger(ExtractAndPersist.class);

    private Message[] messages;

    public ExtractAndPersist(Message[] m) {
        this.messages = m;
    }

    @Override
    public void run() {

        try {


            for (Message message : messages) {
                if (message != null) {


                    String mailContent = processMessageBody(message);


                    status = updateMailContent(mailId, mailContent);

                    }
                }
            }

         catch (Exception e) {
            /** set messages as UNREAD **/

        }
        }
    }


}

Answer 1

At a guess, the problem you're facing is that your IMAP server stores flags in the message, so that search means 100GB of disk I/O. Storing the flags there is stupid, but at least one IMAP server does it.

If I'm right, then you can speed it up quite a bit by using a range search. The search you do now is unseen. The one you should be doing is uid 12345:* unseen, where 12345 is one higher than the highest UID you've processed before. That frees the IMAP server from having to look at the first part of the mailbox. In Javamail I think the code will look like new AndTerm(new MessageNumberTerm(...), new FlagTerm(...)).

The way to high performance is to use all of the search result, though. Use it at once or cache it, but don't throw it away. Throwing away the result of a remote IMAP operation doesn't lead to high performance.