loading docker image fails

Question

I am using golang, docker client to load a docker image in .tar format.

func loadImageFromTar(cli *client.Client, tarFilePath string) (string, error) {
    // Read tar file
    tarFile, err := os.Open(tarFilePath)
    if err != nil {
        return "", fmt.Errorf("failed to open tar file: %w", err)
    }
    defer tarFile.Close()

    // Create a pipe to stream data between tar reader and Docker client
    pr, pw := io.Pipe()

    // Set up a WaitGroup for synchronization
    var wg sync.WaitGroup
    wg.Add(2)

    // Load the Docker image in a separate goroutine
    var imageLoadResponse types.ImageLoadResponse
    go func() {
        defer wg.Done()
        imageLoadResponse, err = cli.ImageLoad(context.Background(), pr, false)
        if err != nil {
            err = fmt.Errorf("failed to load Docker image: %w", err)
        }
    }()

    // Read tar file metadata and copy the tar file to the pipe writer in a separate goroutine
    var repoTag string
    go func() {
        defer wg.Done()
        defer pw.Close()

        tarReader := tar.NewReader(tarFile)

        for {
            header, err := tarReader.Next()
            if err == io.EOF {
                break
            }
            if err != nil {
                err = fmt.Errorf("failed to read tar header: %w", err)
                fmt.Printf("Error: %v", err)
                return
            }

            // Extract the repository and tag from the manifest file
            if header.Name == "manifest.json" {
                data, err := io.ReadAll(tarReader)
                if err != nil {
                    err = fmt.Errorf("failed to read manifest file: %w", err)
                    fmt.Printf("Error: %v", err)
                    return
                }

                var manifest []map[string]interface{}
                err = json.Unmarshal(data, &manifest)
                if err != nil {
                    err = fmt.Errorf("failed to unmarshal manifest: %w", err)
                    fmt.Printf("Error: %v", err)
                    return
                }

                repoTag = manifest[0]["RepoTags"].([]interface{})[0].(string)
            }

            // Copy the tar file data to the pipe writer
            _, err = io.Copy(pw, tarReader)
            if err != nil {
                err = fmt.Errorf("failed to copy tar data: %w", err)
                fmt.Printf("Error: %v", err)
                return
            }
        }
    }()

    // Wait for both goroutines to finish
    wg.Wait()

    // Check if any error occurred in the goroutines
    if err != nil {
        return "", err
    }

    // Close the image load response body
    defer imageLoadResponse.Body.Close()

    // Get the image ID
    imageID, err := getImageIDByRepoTag(cli, repoTag)
    if err != nil {
        return "", fmt.Errorf("failed to get image ID: %w", err)
    }

    return imageID, nil
}

// func: getImageIDByRepoTag

func getImageIDByRepoTag(cli *client.Client, repoTag string) (string, error) {
    images, err := cli.ImageList(context.Background(), types.ImageListOptions{})
    if err != nil {
        return "", fmt.Errorf("failed to list images: %w", err)
    }

    for _, image := range images {
        for _, tag := range image.RepoTags {
            if tag == repoTag {
                return image.ID, nil
            }
        }
    }

    return "", fmt.Errorf("image ID not found for repo tag: %s", repoTag)
}

The getImageIDByRepoTag always returns fmt.Errorf("image ID not found for repo tag: %s", repoTag). Also when I run docker images I do not see the image being loaded. Looks like the image load is not completing .

In my other code, The docker images load usually takes time although the docker client cli.ImageLoad returns immediately. I usually add some 30 seconds wait time before checking for getImageIDByRepoTag. Adding a wait time also did not help in this case.

Thanks

Answer 1

There's a couple of issues:

• the two goroutines share err so some error handling may get lost
  • you should use unique error variables here for each goroutine & inspect both errors after the wg.Wait()
• the main issue: you are reading from the tar reader to find the manifest file and extract the tag info - which is fine - but upon finding this, you copy the rest of the byte stream to your pipe. You are thus losing a chunk of the byte stream which never reaches the docker client

---

To avoid reading the tar byte stream twice, you could use io.TeeReader. This allows you to read the tar archive - to scan for the manifest file - but also have this stream written in full elsewhere (i.e. to the docker client).

Create the TeeReader:

tr := io.TeeReader(tarFile, pw)  // reading `tr` will read the tarFile - but simultaneously write to `pw`

the image load will now read from this (instead of the pipe):

//imageLoadResponse, err = cli.ImageLoad(context.Background(), pr, false)
imageLoadResponse, err = cli.ImageLoad(context.Background(), tr, false)

then change your archive/tar reader to read from the pipe:

//tarReader := tar.NewReader(tarFile) // direct from file
tarReader := tar.NewReader(pr) // read from pipe (indirectly from the file)

you can then drop the your io.Copy block:

// no longer needed:
//
// _, err = io.Copy(pw, tarReader)
//

since the tar-inspection code will read the entire stream to EOF.

P.S. you may want to reset the io.EOF to nil to avoid thinking a EOF is a more critical error, when you inspect any potential errors from either of the goroutines:

header, err = tarReader.Next()
if err == io.EOF {
    err = nil  //  EOF is a non-fatal error here
    break
}