0

Trying to store/re-use cookies between code-executions, similar to this question, but using only python3's urllib.

Thanks to this answer for the process for creating a cookiejar for automatic use across urllib.request.Request calls:

cookie_jar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookie_jar))
urllib.request.install_opener(opener)

Initial searches on cookie storage led to SO questions regarding requests (the module) + cookies, but unfortunately, http.cookiejar objects cannot be pickled (below), as opposed to their requests brethren, which can:

>>> pickle.dumps(cookie_jar)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't pickle _thread.RLock objects

The other queries' results mostly pointed back the python3 docs: https://docs.python.org/3/library/http.cookiejar.html#http.cookiejar.FileCookieJar

Playing around with the FileCookieJar class led to noticing in the description: A CookieJar which can load cookies from, and *perhaps* save cookies to, a file on disk. and the associated error:

>>> FCJ=cookiejar.FileCookieJar("unversioned.cookies")              
>>> FCJ.save()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/http/cookiejar.py", line 1785, in save
    raise NotImplementedError()

Both of which were clarified by this answer - which suggested using LWPCookieJar (among others) instead, but I couldn't find any existing examples on how to use this class in practice.

Adam Smooch
  • 1,167
  • 1
  • 12
  • 27

1 Answers1

0

LWPCookieJar was the answer:

  1. Create the CookieJar + try loading from any existing cookie file:
cookie_filename = "unversioned.cookies"
cookie_jar = cookiejar.LWPCookieJar(cookie_filename)
try:
    cookie_jar.load()
except FileNotFoundError as fnfe:
    # No existing/adjacent cookie file
    pass

opener = request.build_opener(request.HTTPCookieProcessor(cookie_jar))
request.install_opener(opener)
  1. complete any urllib.request.Request call that populates cookies
  2. store the cookies to file (named in the constructor):
cookie_jar.save()

And to confirm, after having saved some cookies in a previous execution-run:

  1. prepare the cookiejar, but do not call .load()
  2. test-request the page -> logged-out version received
  3. call cookiejar.load()
  4. test-request the page -> logged-in version received
Adam Smooch
  • 1,167
  • 1
  • 12
  • 27