Spring Feign Not Compressing Response

Question

I am using spring feign to compress request and response

On Server Side:

server:
  servlet:
    context-path: /api/v1/
  compression:
    enabled: true
    min-response-size: 1024

When I hit the api from chrome, I see that it adds 'Accept-Encoding': "gzip, deflate, br"

On Client Side:

    server:
      port: 8192
      servlet:
        context-path: /api/demo



feign.compression.response.enabled: true

feign.client.config.default.loggerLevel: HEADERS

logging.level.com.example.feigndemo.ManagementApiService: DEBUG

eureka:
  client:
    enabled: false

management-api:
  ribbon:
    listOfServers: localhost:8080

When I see the request headers passed, feign is passing two headers.

Accept-Encoding: deflate
Accept-Encoding: gzip

gradle file

plugins {
        id 'org.springframework.boot' version '2.1.8.RELEASE'
        id 'io.spring.dependency-management' version '1.0.8.RELEASE'
        id 'java'
    }

    group = 'com.example'
    version = '0.0.1-SNAPSHOT'
    sourceCompatibility = '1.8'

    configurations {
        compileOnly {
            extendsFrom annotationProcessor
        }
    }

    repositories {
        mavenCentral()
    }

    ext {
        set('springCloudVersion', "Greenwich.SR2")
    }

    dependencies {
        implementation 'org.springframework.boot:spring-boot-starter-web'
        compile ('org.springframework.cloud:spring-cloud-starter-netflix-ribbon')
        compile('org.springframework.cloud:spring-cloud-starter-openfeign')
    // https://mvnrepository.com/artifact/io.github.openfeign/feign-httpclient
    // https://mvnrepository.com/artifact/io.github.openfeign/feign-httpclient
        //compile group: 'io.github.openfeign', name: 'feign-httpclient', version: '9.5.0'

        compileOnly 'org.projectlombok:lombok'
        annotationProcessor 'org.projectlombok:lombok'
        testImplementation 'org.springframework.boot:spring-boot-starter-test'
    }

    dependencyManagement {
        imports {
            mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
        }
    }

The response is not compressed. What I have seen is that Spring feign is sending the "Accept-Encoding" as two different values

Let me know if thing is wrong here

Answer 1

I have faced the same issue a couple of weeks back and I came to know that there is no fruitful/straight forward way of doing it. I have also got to know that when @patan reported the issue with the spring community @patan reported issue1 and @patan reported issue2 there was a ticket created for the tomcat side to attempt to fix the issue (issue link). There has been also a ticket (ticket link) present in the Jetty side related to the same. Initially, I planned to use the approach suggested in github but later came to know that the library had been already merged into spring-cloud-openfeign-core jar under org.springframework.cloud.openfeign.encoding package. Nevertheless, we could not achieve compression as expected and faced the following two challenges:

1. When we enable the feign compression by settings the org.springframework.cloud.openfeign.encoding.FeignAcceptGzipEncodingInterceptor (code-link) class adds the Accept-Encoding header with values as gzip and deflate but due to the issue (ticket) the tomcat server could not interpret it as a sign of compression signal. As a solution, we have to add the manual Feign interpreter to override the
   FeignAcceptGzipEncodingInterceptor functionality and concatenate the headers.
2. The default compression settings for Feign perfectly work in the most simple scenarios but when there is a situation when Client calling microservice and that microservice calling another microservice through feign then the feign cannot handle the compressed response because Spring cloud open feign decoder does not decompress response by default (default spring open feign decoder) which eventually ends with the issue (issue link). So we have to write our own decoder to achieve decompression.

I have finally found a solution based on various available resources so just follow the steps for the spring feign compression:

application.yml

spring:
  http:
    encoding:
      enabled: true

#to enable server side compression
server:
  compression:
    enabled: true
    mime-types:
      - application/json
    min-response-size: 2048

#to enable feign side request/response compression
feign:
  httpclient:
    enabled: true
  compression:
    request:
      enabled: true
      mime-types:
        - application/json
      min-request-size: 2048
    response:
      enabled: true

NOTE: The above feign configuration my default enables compression to all feign clients.

CustomFeignDecoder

import feign.Response;
import feign.Util;
import feign.codec.Decoder;
import org.springframework.cloud.openfeign.encoding.HttpEncoding;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.reflect.Type;
import java.nio.charset.StandardCharsets;
import java.util.Collection;
import java.util.Objects;
import java.util.zip.GZIPInputStream;

public class CustomGZIPResponseDecoder implements Decoder {

    final Decoder delegate;

    public CustomGZIPResponseDecoder(Decoder delegate) {
        Objects.requireNonNull(delegate, "Decoder must not be null. ");
        this.delegate = delegate;
    }

    @Override
    public Object decode(Response response, Type type) throws IOException {
        Collection<String> values = response.headers().get(HttpEncoding.CONTENT_ENCODING_HEADER);
        if(Objects.nonNull(values) && !values.isEmpty() && values.contains(HttpEncoding.GZIP_ENCODING)){
            byte[] compressed = Util.toByteArray(response.body().asInputStream());
            if ((compressed == null) || (compressed.length == 0)) {
               return delegate.decode(response, type);
            }
            //decompression part
            //after decompress we are delegating the decompressed response to default 
            //decoder
            if (isCompressed(compressed)) {
                final StringBuilder output = new StringBuilder();
                final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
                final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
                String line;
                while ((line = bufferedReader.readLine()) != null) {
                    output.append(line);
                }
                Response uncompressedResponse = response.toBuilder().body(output.toString().getBytes()).build();
                return delegate.decode(uncompressedResponse, type);
            }else{
                return delegate.decode(response, type);
            }
        }else{
            return delegate.decode(response, type);
        }
    }

    private static boolean isCompressed(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIPInputStream.GZIP_MAGIC)) && (compressed[1] == (byte) (GZIPInputStream.GZIP_MAGIC >> 8));
    }
}

FeignCustomConfiguration

import feign.RequestInterceptor;
import feign.RequestTemplate;
import feign.optionals.OptionalDecoder;
import org.springframework.beans.factory.ObjectFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.http.HttpMessageConverters;
import org.springframework.cloud.openfeign.support.ResponseEntityDecoder;
import org.springframework.cloud.openfeign.support.SpringDecoder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class CustomFeignConfiguration {


    @Autowired
    private ObjectFactory<HttpMessageConverters> messageConverters;

    //concatenating headers because of https://github.com/spring-projects/spring-boot/issues/18176
    @Bean
    public RequestInterceptor gzipInterceptor() {
        return new RequestInterceptor() {
            @Override
            public void apply(RequestTemplate template) {
                template.header("Accept-Encoding", "gzip, deflate");
            }
        };
    }

    @Bean
    public CustomGZIPResponseDecoder customGZIPResponseDecoder() {
        OptionalDecoder feignDecoder = new OptionalDecoder(new ResponseEntityDecoder(new SpringDecoder(this.messageConverters)));
        return new CustomGZIPResponseDecoder(feignDecoder);
    }
}

Additional tips

if you are planning to build the CustomDecoder with just feign-core libraries

import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.type.TypeFactory;
import feign.Response;
import feign.Util;
import feign.codec.DecodeException;
import feign.codec.Decoder;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.util.StringUtils;
import org.springframework.web.client.HttpMessageConverterExtractor;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.lang.reflect.WildcardType;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.LinkedList;
import java.util.Map;
import java.util.Objects;
import java.util.zip.GZIPInputStream;

import static java.util.zip.GZIPInputStream.GZIP_MAGIC;

public class CustomGZIPResponseDecoder implements Decoder {

    private final Decoder delegate;

    public CustomGZIPResponseDecoder(Decoder delegate) {
        Objects.requireNonNull(delegate, "Decoder must not be null. ");
        this.delegate = delegate;
    }

    @Override
    public Object decode(Response response, Type type) throws IOException {
        Collection<String> values = response.headers().get("Content-Encoding");
        if (Objects.nonNull(values) && !values.isEmpty() && values.contains("gzip")) {
            byte[] compressed = Util.toByteArray(response.body().asInputStream());
            if ((compressed == null) || (compressed.length == 0)) {
                return delegate.decode(response, type);
            }
            if (isCompressed(compressed)) {
                Response uncompressedResponse = getDecompressedResponse(response, compressed);
                return getObject(type, uncompressedResponse);
            } else {
                return getObject(type, response);
            }
        } else {
            return getObject(type, response);
        }
    }

    private Object getObject(Type type, Response response) throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        if (response.status() == 404 || response.status() == 204)
            return Util.emptyValueOf(type);
        if (Objects.isNull(response.body()))
            return null;
        if (byte[].class.equals(type))
            return Util.toByteArray(response.body().asInputStream());
        if (isParameterizeHttpEntity(type)) {
            type = ((ParameterizedType) type).getActualTypeArguments()[0];
            if (type instanceof Class || type instanceof ParameterizedType
                    || type instanceof WildcardType) {
                @SuppressWarnings({"unchecked", "rawtypes"})
                HttpMessageConverterExtractor<?> extractor = new HttpMessageConverterExtractor(
                        type, Collections.singletonList(new MappingJackson2HttpMessageConverter(mapper)));
                Object decodedObject = extractor.extractData(new FeignResponseAdapter(response));
                return createResponse(decodedObject, response);
            }
            throw new DecodeException(HttpStatus.INTERNAL_SERVER_ERROR.value(),
                    "type is not an instance of Class or ParameterizedType: " + type);
        } else if (isHttpEntity(type)) {
            return delegate.decode(response, type);
        } else if (String.class.equals(type)) {
            String responseValue = Util.toString(response.body().asReader());
            return StringUtils.isEmpty(responseValue) ? Util.emptyValueOf(type) : responseValue;
        } else {
            String s = Util.toString(response.body().asReader());
            JavaType javaType = TypeFactory.defaultInstance().constructType(type);
            return !StringUtils.isEmpty(s) ? mapper.readValue(s, javaType) : Util.emptyValueOf(type);
        }
    }

    public static boolean isCompressed(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIP_MAGIC)) && (compressed[1] == (byte) (GZIP_MAGIC >> 8));
    }

    public static Response getDecompressedResponse(Response response, byte[] compressed) throws IOException {
        final StringBuilder output = new StringBuilder();
        final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
        final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
        String line;
        while ((line = bufferedReader.readLine()) != null) {
            output.append(line);
        }
        return response.toBuilder().body(output.toString().getBytes()).build();
    }

    public static String getDecompressedResponseAsString(byte[] compressed) throws IOException {
        final StringBuilder output = new StringBuilder();
        final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
        final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
        String line;
        while ((line = bufferedReader.readLine()) != null) {
            output.append(line);
        }
        return output.toString();
    }

    private boolean isParameterizeHttpEntity(Type type) {
        if (type instanceof ParameterizedType) {
            return isHttpEntity(((ParameterizedType) type).getRawType());
        }
        return false;
    }

    private boolean isHttpEntity(Type type) {
        if (type instanceof Class) {
            Class c = (Class) type;
            return HttpEntity.class.isAssignableFrom(c);
        }
        return false;
    }

    private <T> ResponseEntity<T> createResponse(Object instance, Response response) {

        MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
        for (String key : response.headers().keySet()) {
            headers.put(key, new LinkedList<>(response.headers().get(key)));
        }

        return new ResponseEntity<>((T) instance, headers, HttpStatus.valueOf(response
                .status()));
    }

    private class FeignResponseAdapter implements ClientHttpResponse {

        private final Response response;

        private FeignResponseAdapter(Response response) {
            this.response = response;
        }

        @Override
        public HttpStatus getStatusCode() throws IOException {
            return HttpStatus.valueOf(this.response.status());
        }

        @Override
        public int getRawStatusCode() throws IOException {
            return this.response.status();
        }

        @Override
        public String getStatusText() throws IOException {
            return this.response.reason();
        }

        @Override
        public void close() {
            try {
                this.response.body().close();
            } catch (IOException ex) {
                // Ignore exception on close...
            }
        }

        @Override
        public InputStream getBody() throws IOException {
            return this.response.body().asInputStream();
        }

        @Override
        public HttpHeaders getHeaders() {
            return getHttpHeaders(this.response.headers());
        }

        private HttpHeaders getHttpHeaders(Map<String, Collection<String>> headers) {
            HttpHeaders httpHeaders = new HttpHeaders();
            for (Map.Entry<String, Collection<String>> entry : headers.entrySet()) {
                httpHeaders.put(entry.getKey(), new ArrayList<>(entry.getValue()));
            }
            return httpHeaders;
        }
    }

}

and if you are planning to build your own Feign builder then you can configure like below

 Feign.builder().decoder(new CustomGZIPResponseDecoder(new feign.optionals.OptionalDecoder(new feign.codec.StringDecoder())))
                 .target(SomeFeignClient.class, "someurl");

Update to the above answer: If you are planning to update the dependency version of spring-cloud-openfeign-core to 'org.springframework.cloud:spring-cloud-openfeign-core:2.2.5.RELEASE' then aware of the following Change in FeignContentGzipEncodingAutoConfiguration class. In FeignContentGzipEncodingAutoConfiguration class the Signature of the ConditionalOnProperty annotation changed from @ConditionalOnProperty("feign.compression.request.enabled", matchIfMissing = false) to @ConditionalOnProperty(value = "feign.compression.request.enabled"), so by default FeignContentGzipEncodingInterceptor bean will be injected into spring container if you have application property feign.request.compression=true in your environment and compress request body if default/configured size limit exceeds. This results a problem if your server don't have a mechanism to handle the compressed request, in such cases add/modify the property as feign.request.compression=false

Answer 2

This is actually an exception in Tomcat and Jetty - multiple encoding headers as given above is legal and should work, however Tomcat and Jetty have a bug that is preventing them to both be read.

The bug has been reported in the spring boot github here. And in tomcat here for reference.

In Tomcat the issue is fixed in 9.0.25 so if you can update to that, that can solve it. Failing that, here is a workaround you can do to fix it:

You will need to create your own request interceptor to reconcile your gzip, deflate headers into a single header.

This interceptor needs to be added to the FeignClient configuration, and that configuration added to your feign client.

import feign.RequestInterceptor;
import feign.RequestTemplate;
import feign.template.HeaderTemplate;
import java.lang.reflect.Field;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;

/**
 * This is a workaround interceptor based on a known bug in Tomcat and Jetty where
 * the requests are unable to perform gzip compression if the headers are in collection format.
 * This is fixed in tomcat 9.0.25 - once we reach this version we can remove this class
 */
@Slf4j
public class GzipRequestInterceptor implements RequestInterceptor {
    @Override
    public void apply(RequestTemplate template) {
        // don't add encoding to all requests - only to the ones with the incorrect header format
        if (requestHasDualEncodingHeaders(template)) {
            replaceTemplateHeader(template, "Accept-Encoding", Collections.singletonList("gzip,deflate"));
        }
    }

    private boolean requestHasDualEncodingHeaders(RequestTemplate template) {
        return template.headers().get("Accept-Encoding").contains("deflate")
                && template.headers().get("Accept-Encoding").contains("gzip");
    }

    /** Because request template is immutable, we have to do some workarounds to get to the headers */
    private void replaceTemplateHeader(RequestTemplate template, String key, Collection<String> value) {
        try {
            Field headerField = RequestTemplate.class.getDeclaredField("headers");
            headerField.setAccessible(true);
            ((Map)headerField.get(template)).remove(key);
            HeaderTemplate newEncodingHeaderTemplate = HeaderTemplate.create(key, value);
            ((Map)headerField.get(template)).put(key, newEncodingHeaderTemplate);
        } catch (NoSuchFieldException e) {
            LOGGER.error("exception when trying to access the field [headers] via reflection");
        } catch (IllegalAccessException e) {
            LOGGER.error("exception when trying to get properties from the template headers");
        }
    }
}

I know the above looks overkill, but because the template headers are unmodifiable, we just use a bit of reflection to modify them to how we want.

Add the above interceptor to your configuration bean

import feign.RequestInterceptor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FeignGzipEncodingConfiguration {

    @Bean
    public RequestInterceptor gzipRequestInterceptor() {

        return new GzipRequestInterceptor();
    }
}

You can finally add this to your feign client with the configuration annotation parameter

@FeignClient(name = "feign-client", configuration = FeignGzipEncodingConfiguration.class)
public interface FeignClient {
    ...
}

The request interceptor should now be hit when you send a feign-client request for gzipped information. This will wipe the dual header, and write in an acceptable string concatenated one in the form of gzip,deflate

Answer 3

If you are using latest spring boot version then it provides default Gzip decoder so no need to write your custom decoder. Use the below property instead:-

feign:
  compression:
    response:
      enabled: true
      useGzipDecoder: true