DataBufferLimitException: Exceeded limit on max bytes to buffer webflux error

Question

While sending a file I receive an array of bytes. I always have a problem with webflux to receive an array. the error thrown as below :

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
    at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException

Do you now how to resolve that in webflux ?

Answer 1

This worked for me:

1. Create a @Bean in one of your configuration classes or the main SpringBootApplication class:

   @Bean
   public WebClient webClient() {
       final int size = 16 * 1024 * 1024;
       final ExchangeStrategies strategies = ExchangeStrategies.builder()
           .codecs(codecs -> codecs.defaultCodecs().maxInMemorySize(size))
           .build();
       return WebClient.builder()
           .exchangeStrategies(strategies)
           .build();
   }
   

2. Next, go to your desired class where you want to use the WebClient:

   @Service
   public class TestService {
   
       @Autowired
       private WebClient webClient;
   
       public void test() {
           String out = webClient
               .get()
               .uri("/my/api/endpoint")
               .retrieve()
               .bodyToMono(String.class)
               .block();
   
           System.out.println(out);
       }
   }
   

Answer 2

I suppose this issue is about adding a new spring.codec.max-in-memory-size configuration property in Spring Boot. Add it to the application.yml file like:

spring:
  codec:
    max-in-memory-size: 10MB

Answer 3

Set the maximum bytes (in megabytes) in your Spring Boot application.properties configuration file like below:

spring.codec.max-in-memory-size=20MB

Answer 4

worked for me

webTestClient.mutate()
  .codecs(configurer -> configurer
          .defaultCodecs()
          .maxInMemorySize(16 * 1024 * 1024))
  .build().get()
  .uri("/u/r/l")
  .exchange()
  .expectStatus()
  .isOk()

Answer 5

i was getting this error for a simple RestController (i post a large json string).

here is how i successfully changed the maxInMemorySize

import org.springframework.context.annotation.Configuration;
import org.springframework.http.codec.ServerCodecConfigurer;
import org.springframework.web.reactive.config.ResourceHandlerRegistry;
import org.springframework.web.reactive.config.WebFluxConfigurer;

@Configuration
public class WebfluxConfig implements WebFluxConfigurer {

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {

        registry.addResourceHandler("/swagger-ui.html**")
            .addResourceLocations("classpath:/META-INF/resources/");

        registry.addResourceHandler("/webjars/**")
            .addResourceLocations("classpath:/META-INF/resources/webjars/");
    }

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
    }
}

this was surprisingly hard to find

Answer 6

Instead of retrieving data at once, you can stream:

Mono<String> string = webClient.get()
    .uri("end point of an API")
    .retrieve()
    .bodyToFlux(DataBuffer.class)
    .map(buffer -> {
        String string = buffer.toString(Charset.forName("UTF-8"));
        DataBufferUtils.release(buffer);
        return string;
    });

Alternatively convert to stream:

    .map(b -> b.asInputStream(true))
    .reduce(SequenceInputStream::new)
    .map(stream -> {
        // consume stream
        stream.close();
        return string;
    });

In most cases you don't want to really aggregate the stream, rather than processing it directly. The need to load huge amount of data in memory is mostly a sign to change the approach to more reactive one. JSON- and XML-Parsers have streaming interfaces.

Answer 7

This worked for me

val exchangeStrategies = ExchangeStrategies.builder()
                .codecs { configurer: ClientCodecConfigurer -> configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024) }.build()
        return WebClient.builder().exchangeStrategies(exchangeStrategies).build()

Answer 8

Another alternative could be creating a custom CodecCustomizer, which is going to be applied to both WebFlux and WebClient at the same time:

@Configuration
class MyAppConfiguration {

    companion object {
        private const val MAX_MEMORY_SIZE = 50 * 1024 * 1024 // 50 MB
    }

    @Bean
    fun codecCustomizer(): CodecCustomizer {
        return CodecCustomizer {
            it.defaultCodecs()
                .maxInMemorySize(MAX_MEMORY_SIZE)
        }
    }
}

Answer 9

As of Spring Boot 2.3.0, there is now a dedicated configuration property for the Reactive Elasticsearch REST client.

You can use the following configuration property to set a specific memory limit for the client.

spring.data.elasticsearch.client.reactive.max-in-memory-size= The already existing spring.codec.max-in-memory-size property is separate and only affects other WebClient instances in the application.

Answer 10

For those who had no luck with the myriad of beans, customizers, and properties that could be added to solve this problem, check whether you have defined a bean extending WebFluxConfigurationSupport. If you have, it will disable the autoconfiguration version of the same bean (my personal experience, Boot 2.7.2), somewhere under which Spring loads properties such as the suggested spring.codec.max-in-memory-size. For this solution to work you need to have also configured this property correctly.

To test if this is the cause of your problems, remove your WebFluxConfigurationSupport implementation temporarily. The long term fix that worked for me was to use configuration beans to override attributes for the autoconfigured bean. In my case, WebFluxConfigurer had all of the same methods available and was a drop-in replacement for WebFluxConfigurationSupport. Large JSON messages are now decoding for me as configured.

Answer 11

If you dont want to change the default settings for webClient globally, you can use the following approach to manually merge multiple DataBuffers

webClient
        .method(GET)
        .uri("<uri>")
        .exchangeToMono(response -> {
          return response.bodyToFlux(DataBuffer.class)
              .switchOnFirst((firstBufferSignal, responseBody$) -> {
                assert firstBufferSignal.isOnNext();
                return responseBody$
                    .collect(() -> requireNonNull(firstBufferSignal.get()).factory().allocateBuffer(), (accumulator, curr) -> {
                      accumulator.ensureCapacity(curr.readableByteCount());
                      accumulator.write(curr);
                      DataBufferUtils.release(curr);
                    })
                    .map(accumulator -> {
                      final var responseBodyAsStr = accumulator.toString(UTF_8);
                      DataBufferUtils.release(accumulator);
                      return responseBodyAsStr;
                    });
              })
              .single();
        });

The above code aggregates all the DataBuffers into a single DataBuffer & then converts the final DataBuffer into a string. Please note that this answer wont work as DataBuffers received might not have all the bytes to construct a character (incase of UTF-8 characters, each character can take upto 4 bytes). So we cant convert intermediate DataBuffers into String as the bytes towards the end of buffer might have only part of the bytes required to construct a valid character

Note that this loads all the response DataBuffers into memory but unlike changing global settings for the webClient across the whole application. You can use this option to read complete response only where you want i.e you can narrow down & pick this option only where you expect large responses.

Answer 12

As of Spring boot 2.7.x we should use below property to set the memory size to webclient which is used internally in reactive ElasticSearch

spring.elasticsearch.webclient.max-in-memory-size=512MB

Answer 13

Just add below code in your springboot main class.

@Bean
public WebClient getWebClient() {
    return WebClient.builder()
            .baseUrl("Your_SERVICE_URL")
            .codecs(configurer -> configurer
                      .defaultCodecs()
                      .maxInMemorySize(16 * 1024 * 1024))
            .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
            .build();
}

This work for me.

Answer 14

Note that if you're using custom decoder, you need to set the maximum in-memory size in there, for instance:

Integer CODEC_20_MB_SIZE = 20 * 1024 * 1024;
WebClient.builder()
    .codecs(clientCodecConfigurer -> {
        var codec = new Jackson2JsonDecoder();
        codec.setMaxInMemorySize(CODEC_20_MB_SIZE);
        clientCodecConfigurer.customCodecs().register(codec);
        clientCodecConfigurer.customCodecs().register(new Jackson2JsonEncoder());
    });