3

I am using the StableDiffusionPipeline from the Hugging Face Diffusers library in Python 3.10.2, on an M2 Mac (I tagged it because this might be the issue). When I try to generate 1 image from 1 prompt, the output looks fine, but when I try to generate multiple images using the same prompt, the images are all either black squares or a random image (see example below). What could be the issue?

My code is as follows (where I change n_imgs from 1 to more than 1 to break it):

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe = pipe.to("mps")  # for M1/M2 chips
pipe.enable_attention_slicing()

prompt = "a photo of an astronaut driving a car on mars"

# First-time "warmup" pass (because of weird M1 behaviour)
_ = pipe(prompt, num_inference_steps=1)

# generate images
n_imgs = 1
imgs = pipe([prompt] * n_imgs).images

I also tried setting num_images_per_prompt instead of creating a list of repeated prompts in the pipeline call, but this gave the same bad results.

Example output (for multiple images):

white noise image generated by machine learning transformer model

[edit/update]: When I generate the images in a loop surrounding the pipe call instead of passing an iterable to the pipe call, it does work:

# generate images
n_imgs = 3
for i in range(n_imgs):
    img = pipe(prompt).images[0]
    # do something with img

But it is still a mystery to me as to why.

Neele22
  • 373
  • 2
  • 18

3 Answers3

1

Apparently it is indeed an Apple Silicon (M1/M2) issue, of which Hugging Face is not yet sure why this is happening, see this GitHub issue for more details.

Neele22
  • 373
  • 2
  • 18
1

I think it might be a PyTorch issue given that a pure MPS version of the code (in Swift) worked fine last time I tested:

import MetalPerformanceShadersGraph

let graph = MPSGraph()
let x = graph.constant(1, shape: [32, 4096, 4096], dataType: .float32)
let y = graph.constant(1, shape: [32, 4096, 1], dataType: .float32)
let z = graph.matrixMultiplication(primary: x, secondary: y, name: nil)
let device = MTLCreateSystemDefaultDevice()!
let buf = device.makeBuffer(length: 16384)!
let td = MPSGraphTensorData(buf, shape: [64, 64], dataType: .int32)
let cmdBuf = MPSCommandBuffer(from: device.makeCommandQueue()!)
graph.encode(to: cmdBuf, feeds: [:], targetOperations: nil, resultsDictionary: [z:td], executionDescriptor: nil)
cmdBuf.commit()

See this thread for details: https://github.com/pytorch/pytorch/issues/84039

pcuenca
  • 91
  • 3
0

You will encounter this error now if you ran that or anything related to MPS using M1/M2 apple devices.

NotImplementedError: The operator 'aten::index.Tensor' is not current implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1 to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

I'm actually looking for a workaround here too in case you guys know.

vlodia
  • 1
  • 1