YouTube Data API v3 Refresh Token Keeps Expiring on App with Publishing Status Set to Testing

Question

WHAT I'M TRYING TO DO:

I'm trying to build a Python 3.9 program to make daily calls of the YouTube Data API v3 using OAuth2 credentials (set to the "testing" publishing status, as a "web application" type, and "external" user type), by storing the refresh token to get a new access token each time I make a unique call.

I've been using the YouTube Data API v3 official documentation, the Python code examples from the Google API repository on GitHub, along with this OAuth token solution I found from Corey Schafer on YouTube.

WHAT I'VE TRIED SO FAR:

Here's my Python code (I've scrambled the playlist ID for the sake of anonymity, but the code runs fine if you put in your own playlist ID for a channel that you have credentials for):

# YouTube Data API v3
# Pulling data for the brand account

import os
import pickle
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from googleapiclient.discovery import build

credentials = None


# youtube_data_token_brand.pickle stores the user's credentials from previously successful logins
if os.path.exists('youtube_data_token_brand.pickle'):
    print('Loading Credentials From File...')
    with open('youtube_data_token_brand.pickle', 'rb') as token:
        credentials = pickle.load(token)

# If there are no valid credentials available, then either refresh the token or log in.
if not credentials or not credentials.valid:
    if credentials and credentials.expired and credentials.refresh_token:
        print('Refreshing Access Token...')
        credentials.refresh(Request())
    else:
        print('Fetching New Tokens...')
        flow = InstalledAppFlow.from_client_secrets_file(
            'client_secrets_youtube_data_brand.json',
            scopes=[
                'https://www.googleapis.com/auth/youtube.readonly'
            ]
        )

        flow.run_local_server(port=8080, prompt='consent',
                              authorization_prompt_message='')
        credentials = flow.credentials

        # Save the credentials for the next run
        with open('youtube_data_token_brand.pickle', 'wb') as f:
            print('Saving Credentials for Future Use...')
            pickle.dump(credentials, f)

youtube = build('youtube', 'v3', credentials=credentials)

request = youtube.playlistItems().list(
        part="status, contentDetails", playlistId='UUlG34RnfYmCsNFgxmTmYjPA', maxResults=28
    )

response = request.execute()

for item in response["items"]:
    vid_id = (item["contentDetails"]["videoId"])
    yt_link = f"https://youtu.be/{vid_id}"
    print(yt_link)

THE RESULTS I'VE BEEN GETTING:

My program runs for about a week, then I get the following errors (again, I've redacted part of the file path for anonymity):

/Users/…/PycharmProjects/GoogleAPIs/HelloYouTubeDataAPIOAuth.py

Loading Credentials From File...

Refreshing Access Token...

Traceback (most recent call last):
  File "/Users/.../PycharmProjects/GoogleAPIs/HelloYouTubeDataAPIOAuth.py", line 23, in <module>
    credentials.refresh(Request())
  File "/Users/.../PycharmProjects/GoogleAPIs/venv/lib/python3.9/site-packages/google/oauth2/credentials.py", line 200, in refresh
    access_token, refresh_token, expiry, grant_response = _client.refresh_grant(
  File "/Users/.../PycharmProjects/GoogleAPIs/venv/lib/python3.9/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/.../PycharmProjects/GoogleAPIs/venv/lib/python3.9/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/Users/.../PycharmProjects/GoogleAPIs/venv/lib/python3.9/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n  "error": "invalid_grant",\n  "error_description": "Bad Request"\n}')

Process finished with exit code 1

I can circumvent these errors by deleting the 'youtube_data_token_brand.pickle' file from my directory and re-running the program (which then asks me to log in with my google account and manually reauthorizing access via the OAuth2 steps).

This leads me to believe that my refresh token is expiring (I swear I read somewhere in the documentation that it shouldn't expire until access is revoked - which I have not done - but I can't find that note anymore after repeatedly searching for it).

Interestingly, I'm able to run the same program for a different YouTube account I control, and that account hasn't experienced the same refresh token error issues. I've also been able to use the identical token.pickle approach for storing the refresh token to other Google APIs (Google Analytics, YouTube Analytics, etc) and not experienced this issue with any of them.

Thanks in advance for any help you can provide!

Answer 1

According to the official documentation, one case a refresh token gets invalidated is when the app associated with the respective refresh token gets its permission of operation revoked.

Another case of refresh token invalidation is when the respective account has exceeded a maximum number of granted refresh tokens:

There is currently a limit of 50 refresh tokens per Google Account per OAuth 2.0 client ID. If the limit is reached, creating a new refresh token automatically invalidates the oldest refresh token without warning.

You may well check that indeed this is what happened on that account's permission page.

Answer 2

After reading the official documentation that @stvar posted in one of the answers, the problem seems to be that this particular refresh token always has a one-week lifespan. This is only the case BECAUSE my situation is a "perfect storm":

1. The OAuth2 Client ID credentials for the problematic program were created using the Google Cloud Console (while the other was created with the Developers' Console).
2. The OAuth Consent Screen App for the problematic program's API credentials are set to the "external user" type (both OAuth2 Client ID credentials for the two programs actually are).
3. The OAuth Consent Screen App has a publishing status of "testing" (again, this is true of both OAuth2 Client ID credentials - the one tied to the problematic program and the other that is running just fine with the same code but a different refresh token created through the Developers' Console).

The only solution appears to be publishing the OAuth Consent Screen App.

Answer 3

There are several reasons why the refresh token may appear to be expired. The main one being that every time your code runs if the auth server is returning a new refresh token and you are not storing it then after fifty runs the refresh token that you had stored will expire.

Note: the auth server does not return a new refresh token every time, the access token is refreshed. This it seems to be based upon language some how C# does, php does not, I dont think node does either. I have yet to track down why this happens I suspect its something in the libraries, I am not sure if the python library does this or not but either way its best to let it handle things.

Have a look at this code it allows the library to handle all storage of the refresh token. You appear to be doing a lot of this manually. Which may or may not result in contrition of your refresh token.

"""Hello YouTube API ."""

import argparse

from apiclient.discovery import build
import httplib2
from oauth2client import client
from oauth2client import file
from oauth2client import tools

SCOPES = ['https://www.googleapis.com/auth/youtube.readonly']
CLIENT_SECRETS_PATH = 'client_secrets.json' # Path to client_secrets.json file.



def initialize_youtube():
  """Initializes the youtube service object.

  Returns:
    youtube an authorized youtube service object.
  """
  # Parse command-line arguments.
  parser = argparse.ArgumentParser(
      formatter_class=argparse.RawDescriptionHelpFormatter,
      parents=[tools.argparser])
  flags = parser.parse_args([])

  # Set up a Flow object to be used if we need to authenticate.
  flow = client.flow_from_clientsecrets(
      CLIENT_SECRETS_PATH, scope=SCOPES,
      message=tools.message_if_missing(CLIENT_SECRETS_PATH))

  # Prepare credentials, and authorize HTTP object with them.
  # If the credentials don't exist or are invalid run through the native client
  # flow. The Storage object will ensure that if successful the good
  # credentials will get written back to a file.
  storage = file.Storage('youtube.dat')
  credentials = storage.get()
  if credentials is None or credentials.invalid:
    credentials = tools.run_flow(flow, storage, flags)
  http = credentials.authorize(http=httplib2.Http())

  # Build the service object.
  youtube= build('youtube', 'v3', http=http)

  return youtube

You may also want to ensure that the user didn't revoke your access though their google account, however i assume you already checked that.