23

My pipelines and schedulers were running smoothly without any problems. After I went out to lunch, I changed the number of epochs a Neural Network would run, save the .yaml file again and leave it in the bucket named "budgetff".

Afterwards, everything stopped working.

No clue

enter image description here

There are the errors and I have 0 clue as to how they are appearing. The code within the components doesn't even seem to start. I've made several different components without any success because they just fail at this step.

If it helps, I installed the kfp --pre and did the imports like this

import kfp.v2.dsl, kfp.v2.compiler
from kfp.v2.dsl import Artifact, Dataset, Input, Metrics, Model, Output

kfp-2.0.0-beta.15 - This is the kfp version running on VertexAi and I'm using Kubeflow with @kfp.v2.dsl.components.

I was trying to just run my pipelines. Forcing a run on the scheduler. When it didn't work, I just tried on the notebook.

filipe
  • 275
  • 1
  • 1
  • 8

9 Answers9

19

The cause is that the latest version of requests does not support urllib3 2.0.0. This is fixed in kfp-2.0.0b16 (see PR with the change), so you can either upgrade to that, or create a new image that downgrades urllib.

Maybe this is triggered by versions of requests-toolbelt and/or urllib3 that were both released in the past few days (May 1 and May 4, 2023, resp).

I have fixed this by building a new container with the following Dockerfile (I use Python 3.9 but use whatever you want):

FROM python:3.9

RUN pip install urllib3==1.26.15 requests-toolbelt==0.10.1

I recommend to build the image using Cloud Build and specify it as the base image for the component.

Paul
  • 1,939
  • 1
  • 18
  • 27
9

I ran into the same issue today and was scratching my head. I found adding appengine-python-standard to the packages_to_install argument in the component decorator solved the issue:

@component(base_image="python:3.7", packages_to_install=["appengine-python-standard",...])

For context, I'm using KFP v1.8.20 through Vertex workbench. Fingers crossed, it works for v2.0.0-beta.15.

mwtmurphy
  • 324
  • 1
  • 8
  • 3
    This is the simplest fix now, works for python 3.7 and 3.9 – pajamas May 05 '23 at 06:52
  • 1
    Thank you so much. This works great if you base a Docker image on python:3.7.16-bullseye, which installs requests-toolbelt-0.9.1 when asking for requests, which in turn produces the appengine dependency error. Just install 'appengine-python-standard' before you install requests and you should be fine. – blurryroots May 06 '23 at 15:34
4

These issues were fixed in kfp==1.8.21 and kfp==2.0.0b16.

Abdel M
  • 51
  • 1
3

Unrelated to OP's specific issue, but outdated versions of twine are also subject to this same error. pip install --upgrade twine won't fix it, but pip install --upgrade twine requests-toolbelt will fix it.

Robin De Schepper
  • 4,942
  • 4
  • 35
  • 56
2

I had the same problem. I updated the pipeline component to use a more recent Python version and now it works:

@component(
    base_image="python:3.11",
Jeremy Caney
  • 7,102
  • 69
  • 48
  • 77
Nelson Filipe Rodrigues Nunes
  • 21
  • 1
2

You should use kfp==1.8.21 ref. https://github.com/kubeflow/pipelines/pull/9323

rayuron
  • 31
  • 1
0

Used a base_image="python:3.11 and all is working now. Thank you so much for the valuable feedback and help! Have a great day

Edit: Was having a lot of compatibility issues, decided to go with the first suggestion of adding appengine-python-standard on the packages to install :)

developerjamiu
  • 590
  • 8
  • 19
filipe
  • 275
  • 1
  • 1
  • 8
  • 1
    Good to hear! Would be great if you could mark my answer as the accepted one :) – mwtmurphy May 06 '23 at 19:19
  • Please mark the answer that helped you out as accepted. Also, please put any edits / updates in the original question, so people can see what changed more easily. – blurryroots May 09 '23 at 22:11
0

This happened to me in my GitHub Action and the problem was that I was using an old version of poetry. Using poetry 1.4.2 fixed it.

Source of an issue with the same root problem: https://github.com/ionrock/cachecontrol/issues/292#issuecomment-1536120527

jorgmo02
  • 41
  • 7
0

In my case it has worked when in my dependencies I've updated requests-toolbelt from version 0.10.1 to version 1.0.0.

westman379
  • 493
  • 1
  • 8
  • 25