4

I want to achieve the same effect of the gcloud projects list via an api call (python) specifically.

However, the only thing I can come across by browsing docs is this.

Is this operation tied specifically to the Resource Manager API?

What happens regarding visibility to other projects? If the RM is tied to a project, how can it view (and therefore list) the other ones?

John Hanley
  • 74,467
  • 6
  • 95
  • 159
pkaramol
  • 16,451
  • 43
  • 149
  • 324

3 Answers3

10

You can only list projects for which you have permissions to access. This means that you cannot see all projects unless you have the rights to access them. In my examples below I show which scopes are required. This also means that you can list projects across accounts. This allows you to see which projects you have access to using the credentials specified in the examples. I show how to use Application Default Credentials (ADC) and Service Account Credentials (Json file format).

For more information you can read my article here about projects.

These examples have been tested with Python 3.6 on Windows 10 Professional. These examples will display the project list exactly as the CLI.

Example 1 using the Python Client Library (services discovery method):

from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
from google.oauth2 import service_account

# Example using the Python Client Library

# Documentation
# https://github.com/googleapis/google-api-python-client
# https://developers.google.com/resources/api-libraries/documentation/cloudresourcemanager/v2/python/latest/

# Library Installation
# pip install -U google-api-python-client
# pip install -U oauth2client

# Requires one of the following scopes
# https://www.googleapis.com/auth/cloud-platform
# https://www.googleapis.com/auth/cloud-platform.read-only
# https://www.googleapis.com/auth/cloudplatformprojects
# https://www.googleapis.com/auth/cloudplatformprojects.readonly

print('{:<20} {:<22} {:<21}'.format('PROJECT_ID', 'NAME', 'PROJECT_NUMBER'))

# Uncomment to use Application Default Credentials (ADC)
credentials = GoogleCredentials.get_application_default()

# Uncomment to use Service Account Credentials in Json format
# credentials = service_account.Credentials.from_service_account_file('service-account.json')

service = discovery.build('cloudresourcemanager', 'v1', credentials=credentials)

request = service.projects().list()

while request is not None:
    response = request.execute()

    for project in response.get('projects', []):
        print('{:<20} {:<22} {:<21}'.format(project['projectId'], project['name'], project['projectNumber']))

    request = service.projects().list_next(previous_request=request, previous_response=response)

Example 2 using the Python Google Cloud Resource Manager API Client Library:

from google.cloud import resource_manager

# Example using the Python Google Cloud Resource Manager API Client Library

# Documentation
# https://pypi.org/project/google-cloud-resource-manager/
# https://github.com/googleapis/google-cloud-python
# https://googleapis.github.io/google-cloud-python/latest/resource-manager/index.html
# https://googleapis.github.io/google-cloud-python/latest/resource-manager/client.html
# https://googleapis.github.io/google-cloud-python/latest/resource-manager/project.html

# Library Installation
# pip install -U google-cloud-resource-manager

# Requires one of the following scopes
# https://www.googleapis.com/auth/cloud-platform
# https://www.googleapis.com/auth/cloud-platform.read-only
# https://www.googleapis.com/auth/cloudplatformprojects
# https://www.googleapis.com/auth/cloudplatformprojects.readonly

print('{:<20} {:<22} {:<21}'.format('PROJECT_ID', 'NAME', 'PROJECT_NUMBER'))

# Uncomment to use Application Default Credentials (ADC)
client = resource_manager.Client()

# Uncomment to use Service Account Credentials in Json format
# client = resource_manager.Client.from_service_account_json('service-account.json')

for project in client.list_projects():
    print('{:<20} {:<22} {:<21}'.format(project.project_id, project.name, project.number))
John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • Very nice and elaborated. And what a coincidence, I was just going through this article of yours: https://www.jhanley.com/google-cloud-where-are-my-credentials-stored. Any suggestion about this would be also highly valuable for me: https://stackoverflow.com/questions/58366923/best-auth-strategy-for-gcp-client-tool – pkaramol Oct 13 '19 at 19:12
  • Question: what is this bracket for in: for project in response.get('projects', []): .. I am just learning GCP SDK and not entirely sure on this. Thanks – ColeGulledge Jan 19 '23 at 13:59
  • @ColeGulledge Link for the API docuemntation: https://cloud.google.com/resource-manager/reference/rest/v1/projects/list The `[]` is an empty list used to filter projects. Since my example lists all projects, the filter is empty. Note: At the time I wrote this answer, the API was version 1. Now version 3 is available. https://cloud.google.com/resource-manager/reference/rest/v3/projects/list – John Hanley Jan 19 '23 at 20:42
  • Right, so I assume its best practice to use Version 3? I am looking at documentation and they seem quite different.. – ColeGulledge Jan 19 '23 at 21:23
  • @ColeGulledge 1/2: As with most vendors, libraries are in constant development. Always use the latest GA release version unless you have a specific requirement to use an older library version. For Google Cloud, this can be confusing as there are often two or more library families for the same service, each with multiple versions. – John Hanley Jan 20 '23 at 00:52
  • @ColeGulledge 2/2: I try to locate the GitHub source link for a library and read the notes and release history to understand the details. Sometimes new versions are complete rewrites. Sometimes Google states that a library is now mature with no more development and recommends to use a different library for new features. – John Hanley Jan 20 '23 at 00:52
  • ah okay I see, I mean my company is just now taking "multi-cloud" and this is quite overwhelming for a newcomer.. Thanks for the help! – ColeGulledge Jan 20 '23 at 14:18
  • @JohnHanley Hey Another question for you, sorry, but I am seeing for instance there is a V3 version of CRM API, but there is only a V1 Version of IAM API (v2 is very limited). Is it bad practice to use different versions for each resource.? I am very confused. Thanks – ColeGulledge Jan 20 '23 at 14:56
  • 1
    @ColeGulledge The APIs for most services are developed independently. You can use v1 for one service and v3 for another. In general, use the latest published GA version. For learning, study each API version. The API history can be very interesting. – John Hanley Jan 20 '23 at 19:22
4

In newer versions of the Python client library, ProjectsClient.search_projects has functionality similar to the pre-release Client.list_projects which in turn is similar to gcloud projects list from the command line.

Minimal Sample

from google.cloud.resourcemanager import ProjectsClient

for project in ProjectsClient().search_projects():
    print(project.display_name)
Aaron Nall
  • 41
  • 2
1

The client library has changed since @john-hanley's thoroughgoing answer. Here's a version using v3 of the API and v1.3.3 of the client:

from google.cloud.resourcemanager import ProjectsClient

ORGANIZATION_ID = 'your-org's-numeric-id'

client = ProjectsClient()
project_pager = client.list_projects(parent=f'organizations/{ORGANIZATION_ID}')
pjs = []
for page in project_pager:
    pjs.append(page)

pjs is a list of google.cloud.resourcemanager_v3.types.projects.Project objects.

Note that this assumes you're using Application Default Credentials (ADC), and that the user account in question has the resourcemanager.projects.list permission.

To use ADC, run gcloud auth application-default login or gcloud auth login --update-adc after changing user accounts.

If you can successfully run gcloud projects list, then you have the necessary level of access to use the code above.

chb
  • 1,727
  • 7
  • 25
  • 47