I was able to successfully upgrade the grpcio library version to 1.31.0. As a first step I would suggest you to upgrade your Cloud Composer instance to the most current version(composer-1.11.3-airflow-1.10.9), since it should be more stable and solved some issues.
To install the newer version of PyPi package in Cloud Composer, you can follow these steps or official documentation:
- Create the
requirements.txt file and pass the package name with version:
grpcio>=1.29.0
- Execute the update command with providing the ENVIRONMENT_NAME, full path to the newly created file and LOCATION of the Composer environment (i.e. europe-west1).
gcloud composer environments update <ENVIRONMENT_NAME> \\
--update-pypi-packages-from-file </PATH/requirements.txt> \\
--location <LOCATION>
The steps that I have taken to confirm that grpcio version has changed from 1.23.0 to 1.31.0 are following:
- Connect to the GKE cluster for your environment by executing following command in Cloud Shell. You can find the
GKE_CLUSTER value within your environment in ENVIRONMENT CONFIGURATION -> GKE cluster variable. GKE_LOCATION replace with the zone name:
gcloud container clusters get-credentials ${GKE_CLUSTER} --zone ${GKE_LOCATION}
- Connect to the worker POD in the GKE cluster. The NAMESPACE name should start with:
composer-<version>-..., and POD_NAME with: airflow-worker-...:
kubectl get pods --all-namespaces
kubectl exec -itn <NAMESPACE_NAME> <POD_NAME> -- /bin/bash
- Run pip freeze to see all packages installed in the environemnt with corresponding version:
pip freeze
- Look for
grpcio==1.31.0 package.
Update:
When you try to install the new packages Composer tries to create a new build, but it fails, this can be seen in the logs for Cloud Build by using the following advanced filter in Stackdriver Logging:
resource.type="build"
The problem is conflicting PyPi dependencies, each update operation triggers a Cloud Build operation that rebuilds images. In the version of Composer that you are using Cloud Build failed the update operation when saw any conflict. In the newest version 1.11.3 you will be able to choose if you want to allow conflicts or not.
Based on this, I would share two recommendations to avoid the issue you are experiencing:
- Create a new environment with a more recent Composer version or upgrading the existing one. In more recent versions (>=1.10.0) Composer use pip in version
19.0.2 instead of 9.0.3. Therefore pip itself may be finding dependencies better.
- Analyse Cloud Build logs and pin the packages to versions that are not conflicting; however, this maybe quite difficult because once you resolve one conflict the other conflict could appear.
