job failing with ERROR: gcloud crashed (AttributeError): 'bool' object has no attribute 'lower'

Question

We noticed our jobs are failing with the below error on the dataproc cluster.

ERROR: gcloud crashed (AttributeError): 'bool' object has no attribute 'lower'

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

Running diagnostics does not produce valuable insights:

$ gcloud info --run-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

Property diagnostic detects issues that may be caused by properties.
Checking hidden properties...done.
Hidden Property Check passed.
Property diagnostic passed (1/1 checks passed).

Same issue. Our CI/CD started failing with this error couple of minutes ago. Feels like either a breaking gcloud change or API outage. — Gajus, May 02 '23 at 23:16
Appears that gcloud v429 was just released and it has breaking changes. https://cloud.google.com/sdk/docs/release-notes#42900_2023-05-02 — Gajus, May 02 '23 at 23:17
Same issue in Bitbucket Pipelines as well. Trying to deploy to app engine, so not related to any of those break changes.. Same version runs fine locally — CJSewell, May 02 '23 at 23:24
After a bit of investigation, this is not a breaking CLI change. Reverting back to 426.0.0 produces the same error. Appears to be a change in the API. — Gajus, May 02 '23 at 23:28
Are you using a service account key for authorization? I am getting this error specifically when running `gcloud auth login --cred-file=...` — Yetanotherjosh, May 03 '23 at 00:14

sean · Answer 1 · 2023-05-03T00:21:59.570

If your local gcloud is still broken, force a re-fetch of the feature flags configuration as Google has turned the problematic flag off, then re-run your gcloud invocation:

rm ~/.config/gcloud/.feature_flags_config.yaml

This appears to be a bad feature flag value that was recently released, which explains why prior versions of the gcloud SDK are impacted (at least the last few, going back to e.g. 412.0.0 works fine).

If you run gcloud with --verbosity=debug you'll see a stack ending in:

 4281, in _GetBoolProperty
    return property_value.value.lower() in ['1', 'true', 'on', 'yes', 'y']
AttributeError: 'bool' object has no attribute 'lower'

Which occurs when parsing the default value of the auth/service_account_use_self_signed_jwt flag.

Two viable workarounds until Google fixes their flag:

# Not a great idea for CI/CD use cases; this will fail as soon
# as the feature flag is removed.
gcloud config set 'auth/service_account_use_self_signed_jwt' false

# Likely not a great idea either, but it does work via a sledgehammer
# approach by preventing feature flags from being evaluated.
gcloud config set 'core/enable_feature_flags' false

As an update: As of around 2023-05-03T00:00:00Z the new flag disappeared from https://www.gstatic.com/cloudsdk/feature_flag_config_file.yaml so impact should be mitigated outside of people's laptops, which presumably only fetch feature flags periodically.

Our own CI systems have recovered.

It was rolled out as a 10% feature flag (no clue how they bucket, though) which is presumably why impact was relatively small:

# Feature Flag Config File
test/feature_flag:
- value: true
  weight: 0
- value: false
  weight: 100
auth/service_account_use_self_signed_jwt:
- value: true
  weight: 10
- value: false
  weight: 90

Thanks for investigating @sean I thought I was going crazy. My github actions CI was giving these errors last night and altering versions did not fix anything. When I woke up, everything worked again without changes, so this makes a lot of sense. Sometimes its best to just go to sleep and wait for the problem to get fixed by somebody else. — Phil, May 03 '23 at 23:47

score 3 · Answer 2 · answered May 02 '23 at 23:41

3

I had the same error, as a work around I change to version 417.0.1 and worked.

gcloud --version  
Google Cloud SDK 417.0.1

answered May 02 '23 at 23:41

Andres Restrepo

39
2

Confirmed, 417 works! – Michael May 02 '23 at 23:55
same. does anybody have an official google support URL that confirms the issue we can monitor? – Frank Schwichtenberg May 02 '23 at 23:59

score -1 · Answer 3 · answered May 02 '23 at 23:58

-1

To solve it you must rollback on your workflow to use an old version (I tried 417 and worked). Github Workflow example:

- name: "Set up Cloud SDK"
  uses: "google-github-actions/setup-gcloud@v1"
  with:
      version: "= 417.0.1"

answered May 02 '23 at 23:58

Felipe Correa

1

score -1 · Answer 4 · answered May 03 '23 at 00:02

This seemingly fixed it for me.

/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core# diff -u properties.py properties.py.fixed 
--- properties.py       2023-05-02 17:00:46.021340418 -0700
+++ properties.py.fixed 2023-05-02 17:00:18.793438813 -0700
@@ -4279,7 +4279,7 @@
   if property_value is None or property_value.value is None or Stringize(
       property_value.value).lower() == 'none':
     return None
-  return property_value.value.lower() in ['1', 'true', 'on', 'yes', 'y']
+  return Stringize(property_value.value).lower() in ['1', 'true', 'on', 'yes', 'y']
 
 
 def _GetIntProperty(prop, properties_file, required):

job failing with ERROR: gcloud crashed (AttributeError): 'bool' object has no attribute 'lower'

4 Answers4