Implementing an email client (to Yahoo server) in Python using tkinter. Very basic functionality, browse folders, messages in the selected folder, new, forward, reply, delete message. At present it is too slow (takes too much time to see the changes done remotedly). My Yahoo mailbox has ~170 messages.
To approach the problem created scripts fetch_idle.py, fetch_poll.py (below)
Looks like neither Yahoo, nor Gmail supports IDLE command. The fetch_idle.py script:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Fetch unseen messages in a mailbox. Use imap_tools. It turned out
that neither yahoo, nor gmail support IDLE command"""
import os
import ssl
from imap_tools import MailBox, A
import conf
if __name__ == '__main__':
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
if 0 < args.verbose:
print(host, port, env_var)
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
# ctx.options &= ~ssl.OP_NO_SSLv3
with MailBox(host=host, port=port, ssl_context=ctx) as mbox:
# Looks neither Yahoo, nor Gmail supoort IDLE
data = mbox.idle.wait(timeout=60)
if data:
for msg in mbox.fetch(A(seen=False)):
print(msg.date, msg.subject)
else:
print('no updates in 60 sec')
gives the following errors accordingly (for yahoo and gmail):
imap_tools.errors.MailboxTaggedResponseError: Response status "None" expected, but "b'IIDE1 BAD [CLIENTBUG] ID Command arguments invalid'" received. Data: IDLE start
imap_tools.errors.MailboxTaggedResponseError: Response status "None" expected, but "b'GONM1 BAD Unknown command s19mb13629058ljg'" received. Data: IDLE start
Resorted to reading all uids in the mailbox, getting the difference (new - old), and thus getting known what has changed. To learn I use the fetch_poll.py script:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Get all uids in the mailbox. Wait 60 secs. Fetch uids again. Print
the changes. Repeat"""
import os
import ssl
from threading import Thread, Condition, Event
import time
import imaplib
import imap_tools
# from progress.bar import Bar
# from imap_tools import MailBox
import conf
all_uids = deleted_uids = new_uids = set()
POLL_INTERVAL = 10
request_to_terminate = Event()
def fetch_uids(mbox, cv):
mbox.folder.set(mbox.folder.get())
try:
uids = [int(i.uid) for i in mbox.fetch(headers_only=1, bulk=1)]
except (imaplib.IMAP4.error, imap_tools.errors.MailboxFetchError):
uids = []
return uids
def update_uids(mbox, cv):
global all_uids, deleted_uids, new_uids
while True:
if request_to_terminate.is_set():
break
with cv:
start = time.perf_counter()
uids = set(fetch_uids(mbox, cv))
print(f'Fetching {len(uids)} uids '
f'took {time.perf_counter() - start} secs')
new_uids = uids - all_uids
deleted_uids = all_uids - uids
all_uids = uids
if deleted_uids or new_uids:
cv.notify()
time.sleep(POLL_INTERVAL)
if __name__ == '__main__':
cv = Condition()
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.options &= ~ssl.OP_NO_SSLv3
with imap_tools.MailBox(host=host, port=port, ssl_context=ctx) as mbox:
mbox.login(user, pass_, initial_folder='INBOX')
all_uids = set()
uids_delta = set()
update_thread = Thread(target=update_uids, args=(mbox, cv),
daemon=True)
update_thread.start()
while True:
try:
with cv:
while not (deleted_uids or new_uids):
cv.wait()
if deleted_uids:
print(f'deleted_uids = {deleted_uids}')
if new_uids:
print(f'new_uids = {new_uids}')
deleted_uids = set()
new_uids = set()
except KeyboardInterrupt:
ans = input('Add marker/terminate [M/t] ?')
if ans in ['', 'm', 'M']: # write marker
continue
else:
request_to_terminate.set()
update_thread.join()
break
The script takes from 10 to 30 seconds to fetch all uids (in fetch_uids function).
Experimented with Debian Evolution (3.38) and macOS High Sierra (10.13.6) Mail (11.6). Evolution sees the changes instantly (I need more time to press File > Send/Receive > Send/Receive F12, than Evolution to get the changes). For macOS I need to Mailbox > Get New Mail, to get the new mail. It is equally fast. I deleted some old messages to see how quick the clients will see the deletion. Again, explicitly doing the mentioned commands give the quick result.
I created/deleted messages using https://mail.yahoo.com
How to speed up my script and see the changes made elsewhere quicker than in 15 (avg) seconds?
Python 3.9.2, Debian GNU/Linux 11 (bullseye)
UPDATE
Credit to @Max who suggested the solution in comments. New much faster version of fetch_uids(), only 2 secs against 15
def fetch_uids(mbox, cv):
mbox.folder.set(mbox.folder.get())
try:
uids = map(int, mbox.uids('ALL'))
except (imaplib.IMAP4.error, imap_tools.errors.MailboxFetchError):
uids = []
return uids