0

Implementing an email client (to Yahoo server) in Python using tkinter. Very basic functionality, browse folders, messages in the selected folder, new, forward, reply, delete message. At present it is too slow (takes too much time to see the changes done remotedly). My Yahoo mailbox has ~170 messages.

To approach the problem created scripts fetch_idle.py, fetch_poll.py (below)

Looks like neither Yahoo, nor Gmail supports IDLE command. The fetch_idle.py script:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Fetch unseen messages in a mailbox. Use imap_tools. It turned out
that neither yahoo, nor gmail support IDLE command"""
import os
import ssl
from imap_tools import MailBox, A
import conf


if __name__ == '__main__':
    args = conf.parser.parse_args()
    host, port, env_var = conf.config[args.host]
    if 0 < args.verbose:
        print(host, port, env_var)
    user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
    ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    # ctx.options &= ~ssl.OP_NO_SSLv3
    with MailBox(host=host, port=port, ssl_context=ctx) as mbox:
        # Looks neither Yahoo, nor Gmail supoort IDLE
        data = mbox.idle.wait(timeout=60)
        if data:
            for msg in mbox.fetch(A(seen=False)):
                print(msg.date, msg.subject)
        else:
            print('no updates in 60 sec')

gives the following errors accordingly (for yahoo and gmail):

imap_tools.errors.MailboxTaggedResponseError: Response status "None" expected, but "b'IIDE1 BAD [CLIENTBUG] ID Command arguments invalid'" received. Data: IDLE start

imap_tools.errors.MailboxTaggedResponseError: Response status "None" expected, but "b'GONM1 BAD Unknown command s19mb13629058ljg'" received. Data: IDLE start

Resorted to reading all uids in the mailbox, getting the difference (new - old), and thus getting known what has changed. To learn I use the fetch_poll.py script:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Get all uids in the mailbox. Wait 60 secs. Fetch uids again. Print
the changes. Repeat"""
import os
import ssl
from threading import Thread, Condition, Event
import time
import imaplib
import imap_tools
# from progress.bar import Bar
# from imap_tools import MailBox
import conf
all_uids = deleted_uids = new_uids = set()
POLL_INTERVAL = 10
request_to_terminate = Event()


def fetch_uids(mbox, cv):
    mbox.folder.set(mbox.folder.get())
    try:
        uids = [int(i.uid) for i in mbox.fetch(headers_only=1, bulk=1)]
    except (imaplib.IMAP4.error, imap_tools.errors.MailboxFetchError):
        uids = []
    return uids


def update_uids(mbox, cv):
    global all_uids, deleted_uids, new_uids
    while True:
        if request_to_terminate.is_set():
            break
        with cv:
            start = time.perf_counter()
            uids = set(fetch_uids(mbox, cv))
            print(f'Fetching {len(uids)} uids '
                  f'took {time.perf_counter() - start} secs')
            new_uids = uids - all_uids
            deleted_uids = all_uids - uids
            all_uids = uids
            if deleted_uids or new_uids:
                cv.notify()
        time.sleep(POLL_INTERVAL)


if __name__ == '__main__':
    cv = Condition()
    args = conf.parser.parse_args()
    host, port, env_var = conf.config[args.host]
    user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
    ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    ctx.options &= ~ssl.OP_NO_SSLv3
    with imap_tools.MailBox(host=host, port=port, ssl_context=ctx) as mbox:
        mbox.login(user, pass_, initial_folder='INBOX')
        all_uids = set()
        uids_delta = set()
        update_thread = Thread(target=update_uids, args=(mbox, cv),
                               daemon=True)
        update_thread.start()
        while True:
            try:
                with cv:
                    while not (deleted_uids or new_uids):
                        cv.wait()
                    if deleted_uids:
                        print(f'deleted_uids = {deleted_uids}')
                    if new_uids:
                        print(f'new_uids = {new_uids}')
                    deleted_uids = set()
                    new_uids = set()
            except KeyboardInterrupt:
                ans = input('Add marker/terminate [M/t] ?')
                if ans in ['', 'm', 'M']:  # write marker
                    continue
                else:
                    request_to_terminate.set()
                    update_thread.join()
                    break

The script takes from 10 to 30 seconds to fetch all uids (in fetch_uids function).

Experimented with Debian Evolution (3.38) and macOS High Sierra (10.13.6) Mail (11.6). Evolution sees the changes instantly (I need more time to press File > Send/Receive > Send/Receive F12, than Evolution to get the changes). For macOS I need to Mailbox > Get New Mail, to get the new mail. It is equally fast. I deleted some old messages to see how quick the clients will see the deletion. Again, explicitly doing the mentioned commands give the quick result.

I created/deleted messages using https://mail.yahoo.com

How to speed up my script and see the changes made elsewhere quicker than in 15 (avg) seconds?

Python 3.9.2, Debian GNU/Linux 11 (bullseye)

UPDATE

Credit to @Max who suggested the solution in comments. New much faster version of fetch_uids(), only 2 secs against 15

def fetch_uids(mbox, cv):
    mbox.folder.set(mbox.folder.get())
    try:
        uids = map(int, mbox.uids('ALL'))
    except (imaplib.IMAP4.error, imap_tools.errors.MailboxFetchError):
        uids = []
    return uids
tripleee
  • 175,061
  • 34
  • 275
  • 318
  • 2
    You’re fetch the headers for every message just to get the UIDs? Just use a search for all, to get all UIDs. – Max Jul 20 '22 at 18:42
  • 1
    Tangentially, the code you put inside `if __name__ == ’__main__’:` should be absolutely trivial. The condition is only useful when you `import` this code; if useful functionality is excluded when you `import`, you will never want to do that anyway. See also https://stackoverflow.com/a/69778466/874188 – tripleee Jul 21 '22 at 08:08
  • @tripleee you're right. I'll fix this. – Vladimir Zolotykh Jul 21 '22 at 09:53

0 Answers0