Zuul timing out in long-ish requests

Question

I am using a front end Spring Cloud application (micro service) acting as a Zuul proxy (@EnableZuulProxy) to route requests from an external source to other internal micro services written using spring cloud (spring boot).
The Zuul server is straight out of the applications in the samples section

@SpringBootApplication
@Controller
@EnableZuulProxy
@EnableDiscoveryClient
public class ZuulServerApplication {
    public static void main(String[] args) {
        new SpringApplicationBuilder(ZuulServerApplication.class).web(true).run(args);
    }
}

I ran this set of services locally and it all seems to work fine but if I run it on a network with some load, or through a VPN, then I start to see Zuul forwarding errors, which I am seeing as client timeouts in the logs.

Is there any way to change the timeout on the Zuul forwards so that I can eliminate this issue from my immediate concerns? What accessible parameter settings are there for this?

score 41 · Answer 1 · answered Sep 02 '15 at 00:06

41

In my case I had to change the following property:

zuul.host.socket-timeout-millis=30000

answered Sep 02 '15 at 00:06

okrunner

3,083
29
22

12

this is the solution when not using ribbon – Meysam Apr 18 '16 at 08:08
4

is it possible to appy this solution per host to avoid applying it for all endpoints? – Alex Dec 08 '17 at 12:08
Can you please explain what this does? Is it a setting to increase the timeout between user logins, or to kill the page after 30 seconds? – cluis92 Sep 02 '22 at 21:43

score 37 · Accepted Answer · answered Mar 06 '15 at 18:01

37

The properties to set are: ribbon.ReadTimeout in general and <service>.ribbon.ReadTimeout for a specific service, in milliseconds. The Ribbon wiki has some examples. This javadoc has the property names.

answered Mar 06 '15 at 18:01

spencergibb

24,471
6
69
75

Thanks that helps. Where would I find the default settings for all these parameters? – EvilJinious1 Mar 06 '15 at 20:20
These are the configs: com.netflix.client.config.DefaultClientConfigImpl? – EvilJinious1 Mar 06 '15 at 20:24
1

is it possible to set this property not only for ribbon if I'm not using it? – Alex Dec 08 '17 at 09:28

score 23 · Answer 3 · answered Dec 28 '15 at 11:41

23

I have experienced the same problem: in long requests, Zuul's hystrix command kept timing out after around a second in spite of setting ribbon.ReadTimeout=10000.

I solved it by disabling timeouts completely:

hystrix:
  command:
    default:
      execution:
        timeout:
          enabled: false

An alternative that also works is change Zuul's Hystrix isolation strategy to THREAD:

hystrix:
  command:
    default:
      execution:
        isolation:
          strategy: THREAD
          thread:
            timeoutInMilliseconds: 10000

answered Dec 28 '15 at 11:41

codependent

23,193
31
166
308

1

thank you very very very much:-) The last one worked for me! Has it some disadvantages? Oh my god.. no timeouts anymore.. thanks – phil May 12 '16 at 06:36
1

I'm not sure but I don't think there's a problem. Actually, we've used in in production without problems so far. You can read more about why SEMAPHORE is the default implementation here. http://stackoverflow.com/questions/29965817/why-is-zuul-forcing-a-semaphore-isolation-to-execute-its-hystrix-commands – codependent May 12 '16 at 07:35
To use this isolated to a specific service: # Disable Hystrix timeout for a single service: `hystrix.command..execution.timeout.enabled: false` # Increase the Hystrix timeout to 60s (per service) `hystrix.command..execution.isolation.thread.timeoutInMilliseconds: 60000` – Tim Schimandle Aug 24 '16 at 14:36

score 22 · Answer 4 · answered Apr 28 '17 at 12:33

22

This worked for me, I had to set connection and socket timeout in the application.yml:

zuul:
  host:
    connect-timeout-millis: 60000 # starting the connection 
    socket-timeout-millis: 60000  # monitor the continuous incoming data flow

answered Apr 28 '17 at 12:33

Eduardo Sanchez-Ros

1,777
2
18
30

This worked for me. Short and simple. Thanks bro. – Chetan Oswal Mar 15 '22 at 14:18

score 17 · Answer 5 · answered Sep 26 '16 at 18:32

17

I had to alter two timeouts to force zuul to stop timing out long-running requests. Even if hystrix timeouts are disabled ribbon will still timeout.

hystrix:
  command:
    default:
      execution:
        timeout:
          enabled: false
ribbon:
  ReadTimeout: 100000
  ConnectTimeout: 100000

answered Sep 26 '16 at 18:32

dseibert

1,319
9
19

1

thanks for this, last time I logged in to stackoverflow was in 2016, I had to login to say thank you and upvote – Babajide Prince Feb 10 '18 at 13:10

score 7 · Answer 6 · answered Apr 09 '18 at 13:32

If Zuul uses service discovery, you need to configure these timeouts with the ribbon.ReadTimeout and ribbon.SocketTimeout Ribbon properties.

If you have configured Zuul routes by specifying URLs, you need to use zuul.host.connect-timeout-millis and zuul.host.socket-timeout-millis

by routes i mean

zuul:
  routes:
    dummy-service:
      path: /dummy/**

Muhammad Waqas Dilawar · Answer 7 · 2019-05-09T11:26:20.933

I had a similar issue and I was trying to set timeout globally, and also sequence of setting timeout for Hystrix and Ribbon matters.

After spending plenty of time, I ended up with this solution. My service was taking upto 50 seconds because of huge volume of data.

Points to consider before changing default value for Timeout:

Hystrix time should be greater than combined time of Ribbon ReadTimeout and ConnectionTimeout.

Use for specific service only, which means don't set globally (which doesn't work).

I mean use this:

command:
   your-service-name:

instead of this:

command:
   default:

Working solution:

hystrix:
 command:
   your-service-name:
  execution:
    isolation:
      strategy: THREAD
      thread:
        timeoutInMilliseconds: 95000

your-service-name:
 ribbon:
  ConnectTimeout: 30000
  ReadTimeout: 60000
  MaxTotalHttpConnections: 500
  MaxConnectionsPerHost: 100

Reference

What about timeout for specific route of service ? – Ivan Feb 13 '20 at 13:19 — Ivan, Feb 13 '20 at 13:19

score 1 · Answer 8 · answered Apr 16 '18 at 17:36

Only these settings on application.yml worked for me:

ribbon:
    ReadTimeout: 90000
    ConnectTimeout: 90000

eureka:
    enabled: true

zuul:
    host:
        max-total-connections: 1000
        max-per-route-connections: 100
    semaphore:
        max-semaphores: 500

hystrix:
    command:
        default:
            execution:
                isolation:
                    thread:
                        timeoutInMilliseconds: 1000000

Hope it helps someone!

Zuul timing out in long-ish requests

8 Answers8