How do you decompress a gzip file when using the aws-sdk-go-v2 s3 Downloader without writing the compressed file to disk first?

Question

As per the changelog provided in the aws-sdk-go-v2 module we can see that they have

Disable[d] automatic decompression of getting Amazon S3 objects with the Content-Encoding: gzip metadata header.

They go on to say that you should use the aws/smithy-go's "SetHeaderValue" or "AddHeaderValue":

If you'd like the client to sent the Accept-Encoding: gzip request header, you can add this header to the API operation method call with the SetHeaderValue. middleware helper.

However, using either of those does not seem to cause the downloaded file to decompress the gzip'd file when downloading from S3. The example below shows my code which currently downloads the compressed file despite using the SetHeaderValue method suggested by AWS.

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/smithy-go/middleware"
    "github.com/aws/smithy-go/transport/http"
)

func main() {
    ctx := context.Background()
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        panic(err)
    }

    // Here I attempt to set the header at the client level
    client := s3.NewFromConfig(cfg, s3.WithAPIOptions(http.SetHeaderValue("Accept-Encoding", "gzip")))

    downloader := manager.NewDownloader(client, func(d *manager.Downloader) {
        d.Concurrency = 1
    })

    fdst, err := os.Create("decompressed.txt")
    if err != nil {
        panic(err)
    }

    bucket := "bucket"
    key := "test6.gz"
    n, err := downloader.Download(ctx, fdst,
        &s3.GetObjectInput{Bucket: &bucket, Key: &key},
        // Here I attempt to set the header on a per-call basis
        manager.WithDownloaderClientOptions(
            func(o *s3.Options) {
                o.APIOptions = append(o.APIOptions, []func(*middleware.Stack) error{
                    http.SetHeaderValue("Accept-Encoding", "gzip"),
                }...)
            },
        ),
    )
    if err != nil {
        panic(err)
    }

    fmt.Println(n)
}

So my question is, how do I get this to actually decompress the gzip file when it downloads it? Ideally I want to control the header on a per-call basis, changing the header for the client is less useful.

Answer 1

downloader.Download takes a io.WriterAt. So basically everything from this Question (Buffer implementing io.WriterAt in go) works for getting the download into memory where you can decompress it.