use openAI CLIP with torch tensor or numpy array as input

Question

How to use a pytorch tensor or an openCV iamge correctly as input for for OpenAi CLIP?

I tried the following but this didn't work so far :

device = "cuda" if torch.cuda.is_available() else "cpu"
clip_model, clip_preprocess = clip.load("ViT-B/32", device=device)
clip_preprocess(torch.from_numpy(OpenCVImage)).unsqueeze(0).to(device)

the preprocess step fails with message Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
openCVImage is a object that was already processed with cv2.COLOR_BGR2RGB

"opencv images" are numpy arrays. this is not an OpenCV issue. I have removed the tag. -- may I also recommend that you take the [tour]? you don't seem to have accepted any answers on any of your questions so far. — Christoph Rackwitz, Dec 17 '22 at 20:47

score 1 · Accepted Answer · answered Jan 10 '23 at 18:03

If you read the transforms code for CLIP, it shows that you need a PIL Image Object not a Numpy Array or Torch Tensor. These lines

def _transform(n_px):
    return Compose([
        Resize(n_px, interpolation=BICUBIC),
        CenterCrop(n_px),
        _convert_image_to_rgb,
        ToTensor(),
        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
    ])

In PyTorch, mostly Transforms are written for PIL not for Numpy or Torch due to performance and good results. So you have to convert each Image to PIL first via

image = Image.fromarray(np_img)

or you can simply use the last transform manually like

my_transform = Compose([
        ToTensor(),
        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
])
# Use it on np image now

my_transform(torch.from_numpy(OpenCVImage)).unsqueeze(0).to(device)

thanks, that's exactly how I fixed this, although not sure if the transforms are really made only for the PIL images — werber bang, Jan 20 '23 at 13:37

score 0 · Answer 2 · answered Dec 18 '22 at 05:57

0

It seems like a system-related issue (the SIGKILL interrupt).
What system are you using? What OS? What type of device? Is it possible you exceeded CPU/GPU memory limit?

answered Dec 18 '22 at 05:57

Shai

111,146
38
238
371

I am using wsl-2 with cpu, with a PIL image everything works fine, so I am thinking maybe something is wrong with my code – werber bang Dec 18 '22 at 11:56
@werberbang what is the size of the image, in `PIL` and in `openCV`? – Shai Dec 18 '22 at 12:00
with PIL it works fine on any images I have locally, the image I get from OpenCV that I feed to CLIP is 376*297*3 – werber bang Dec 18 '22 at 12:17

use openAI CLIP with torch tensor or numpy array as input

2 Answers2