Python: Fetching nested dictionary values GCP Recommendations

Question

I am trying to illicit certain key values out of an API JSON output from GCP Recommendations API using Python, and am newer to using Python. Most of the values that I am trying to illicit I can fetch without an issue, however, when I try to illicit certain values within a more deeply nested code block in the JSON, it fails with the error: TypeError: 'OperationGroup' object is not subscriptable

The full output of the JSON response is here (some of the values are changed to protect company information):

name: "projects/12345678910/locations/us-central1-a/recommenders/google.compute.instance.MachineTypeRecommender/recommendations/abcd-efg-hijk-lmnop-qrstuv-123456"
description: "Save cost by changing machine type from e2-medium to e2-small."
last_refresh_time {
  seconds: 1623222401
}
primary_impact {
  category: COST
  cost_projection {
    cost {
      currency_code: "USD"
      units: -12
      nanos: -98539964
    }
    duration {
      seconds: 2592000
    }
  }
}
content {
  operation_groups {
    operations {
      action: "test"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/xyz/zones/us-central1-a/instances/abcname123"
      path: "/machineType"
      value_matcher {
        matches_pattern: ".*zones/us-central1-a/machineTypes/e2-medium"
      }
    }
    operations {
      action: "replace"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/xyz/zones/us-central1-a/instances/abcname123"
      path: "/machineType"
      value {
        string_value: "zones/us-central1-a/machineTypes/e2-small"
      }
    }
  }
}
state_info {
  state: ACTIVE
}
etag: "\"abc-123-def-456\""
recommender_subtype: "CHANGE_MACHINE_TYPE"

name: "projects/12345678910/locations/us-central1-a/recommenders/google.compute.instance.MachineTypeRecommender/recommendations/abcdefg-hijklmnop-123-456"
description: "Save cost by changing machine type from e2-medium to e2-small."
last_refresh_time {
  seconds: 1623222401
}
primary_impact {
  category: COST
  cost_projection {
    cost {
      currency_code: "USD"
      units: -12
      nanos: -99648292
    }
    duration {
      seconds: 2592000
    }
  }
}
content {
  operation_groups {
    operations {
      action: "test"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/xyz/zones/us-central1-a/instances/instance-example1"
      path: "/machineType"
      value_matcher {
        matches_pattern: ".*zones/us-central1-a/machineTypes/e2-medium"
      }
    }
    operations {
      action: "replace"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/xyz/zones/us-central1-a/instances/instance-example1"
      path: "/machineType"
      value {
        string_value: "zones/us-central1-a/machineTypes/e2-small"
      }
    }
  }
}
state_info {
  state: ACTIVE
}
etag: "\"abcdefg12345\""
recommender_subtype: "CHANGE_MACHINE_TYPE"

name: "projects/12345678910/locations/us-central1-a/recommenders/google.compute.instance.MachineTypeRecommender/recommendations/abcd1234"
description: "Save cost by changing machine type from e2-medium to e2-small."
last_refresh_time {
  seconds: 1623222401
}
primary_impact {
  category: COST
  cost_projection {
    cost {
      currency_code: "USD"
      units: -11
      nanos: -568971875
    }
    duration {
      seconds: 2592000
    }
  }
}
content {
  operation_groups {
    operations {
      action: "test"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/abcd/zones/us-central1-a/instances/instance-example2"
      path: "/machineType"
      value_matcher {
        matches_pattern: ".*zones/us-central1-a/machineTypes/e2-medium"
      }
    }
    operations {
      action: "replace"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/abcd/zones/us-central1-a/instances/instance-example2"
      path: "/machineType"
      value {
        string_value: "zones/us-central1-a/machineTypes/e2-small"
      }
    }
  }
}
state_info {
  state: ACTIVE
}
etag: "\"abcd1234\""
recommender_subtype: "CHANGE_MACHINE_TYPE"

Here is my code in Python:

from google.cloud import recommender
import os

client = recommender.RecommenderClient()

def main():
    name = client.list_recommendations(parent='projects/xyzproject/locations/us-central1-a/recommenders/google.compute.instance.MachineTypeRecommender')
    for element in name:
        # print(element)
        print(element.description)
        print(element.primary_impact.category)
        print(element.primary_impact.cost_projection.cost.currency_code)
        print(element.primary_impact.cost_projection.cost.units)
        print(element.state_info.state)
        print(element.content.operation_groups)
        for item in element.content.operation_groups:
            print(item['resource_type'])
main()

The following portions of the above work:

print(element.description)
print(element.primary_impact.category)
print(element.primary_impact.cost_projection.cost.currency_code)
print(element.primary_impact.cost_projection.cost.units)
print(element.state_info.state)
print(element.content.operation_groups)

but the one that I'm having the error and trouble with is:

for item in element.content.operation_groups:
    print(item['resource_type'])

Whenever I try to use that portion of the python script it fails with the error:

TypeError: 'OperationGroup' object is not subscriptable

So, can someone help me understand how I can properly tap into the JSON response and illicit the information within the below block (e.g. the 'resource_type')?

content {
  operation_groups {
    operations {
      action: "test"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/abcd/zones/us-central1-a/instances/instance-example2"
      path: "/machineType"
      value_matcher {
        matches_pattern: ".*zones/us-central1-a/machineTypes/e2-medium"
      }
    }
    operations {
      action: "replace"
      resource_type: "compute.googleapis.com/Instance"
      resource: "//compute.googleapis.com/projects/abcd/zones/us-central1-a/instances/instance-example2"
      path: "/machineType"
      value {
        string_value: "zones/us-central1-a/machineTypes/e2-small"
      }
    }
  }
}

The response appears to not be JSON or a standard dictionary, as you're using '.' to access it's methods, not keys. Whatever object OperationGroup is, it isn't subscriptable, so you can't iterate over it. You need to read up on the object type, or find a way to convert this all to a `dict` first. — match, Jun 10 '21 at 20:39
Most likely you are dumping an object in Python instead of converting the object to a JSON string for output. In the JSON that Google returns, `operationGroups` is an array of operations which is another array. Pseudo data model: `content.operationGroups[].operations[]`. — John Hanley, Jun 10 '21 at 21:42
@JohnHanley- Are you suggesting to try writing content.operation_groups[].operations[] in the code in order to access the array within operations? — Ben, Jun 11 '21 at 00:40
So after some research the response is a type of JSON called JSON-PATCH. I'm still getting the same error though even when I change things to print this way: `print(element.content.operation_groups[0]['resource_type'])` -- that results in the `TypeError: 'OperationGroup' object is not subscriptable`. Can anyone help? — Ben, Jun 14 '21 at 14:12

John Hanley · Accepted Answer · 2021-06-15T00:01:45.407

I took the code in your question and rewrote it to match the Python Recommender API. The primary issue is that you are accessing internal member and field names. I have changed the code to use public interfaces.

Review the type definitions for the API:

Types for Recommender API Client

GitHub Gist: Python example demonstrating the Google Cloud Compute Recommender list recommendations api

# pip install google-cloud-recommender

from google.cloud import recommender
import os

# Enter values for your Project ID and Zone
PROJECT=
LOCATION=

RECOMMENDER = 'google.compute.instance.MachineTypeRecommender'

def print_recommendations():
    client = recommender.RecommenderClient()

    parent = client.recommender_path(PROJECT, LOCATION, RECOMMENDER)

    recommendations = client.list_recommendations(parent=parent)

    for recommendation in recommendations:
        print('Recommendation:')
        print('  Description:     ', recommendation.description)
        print('  Impact Category: ', recommendation.primary_impact.category)
        print('  Currency Code:   ', recommendation.primary_impact.cost_projection.cost.currency_code)
        print('  Units:           ', recommendation.primary_impact.cost_projection.cost.units)
        print('  State:           ', recommendation.state_info.state)
        print('')

        for op_group in recommendation.content._pb.operation_groups:
            for operation in op_group.operations:
                print('  Operation:')
                print('    Action:            ', operation.action)
                print('    Resource Type:     ', operation.resource_type)
                print('    Path:              ', operation.path)
                print('    Value:             ', operation.value.string_value)
                print('')

print_recommendations()

Example output:

Recommendation:
  Description:      Save cost by changing machine type from e2-standard-4 to e2-highmem-2.
  Impact Category:  Category.COST
  Currency Code:    USD
  Units:            -31
  State:            State.ACTIVE

  Operation:
    Action:             test
    Resource Type:      compute.googleapis.com/Instance
    Path:               /machineType
    Value:

  Operation:
    Action:             replace
    Resource Type:      compute.googleapis.com/Instance
    Path:               /machineType
    Value:              zones/us-east1-b/machineTypes/e2-highmem-2

Hi John, thank you so much, this worked! Can you please explain the use of `._pb` within the `for op_group in recommendation.content._pb.operation_groups:`? I looked at some of the documentation and it is a bit obtuse. — Ben, Jun 17 '21 at 15:05
@Ben pb stands for protobuf which is a grpc item. I did not know the API well enough to explain some of the design decisions. There should be a method to use instead of directly accessing that member. https://grpc.io/ https://developers.google.com/protocol-buffers/docs/proto3 — John Hanley, Jun 17 '21 at 17:13

Python: Fetching nested dictionary values GCP Recommendations

1 Answers1