Set Nginx Timeout & Default JSON Response

Question

I have a simple Falcon REST API for a POST request that accepts the JSON formatted body: { "Input": "input string request" } and returns: { "Output: "output string response" }

It may be helpful to note that in production, all the requests will be coming from a single client and this service should be able to handle 50 req/sec with an average response time below 200ms. Ideally, these requests should be done in batches but due to several factors, unfortunately, it has to be done this way.

I've found Bjoern (in comparison to Gunicorn) to be able to serve my app both the fastest and also handle the most load before causing timeouts. On top of all this is running Nginx and my pretty much default configurations are pasted below.

The problem I'm facing is that I see around 2% of requests timeout (determined with the Linux wrk command line tool) and I think this is due to load.

I'm wondering if I can configure Nginx to set a time limit to respond to the client by (lets say 500ms) and if the upstream server reaches that time limit before responding, send a default JSON response with an empty JSON response: { "Output": "" }

Is it possible to have all requests return in a specified max amount of time?

Thanks in advance for your time! :)

/etc/nginx/nginx.conf:

worker_processes    auto;

error_log   /app/logs/nginx_error.log    warn;
pid         /var/run/nginx.pid;

events {
    worker_connections  1024;   # max connections per worker process

    multi_accept        on;     
    use                 epoll;  # efficient connection processing method for linux
}

http {
    include         /etc/nginx/mime.types;
    default_type    application/octet-stream;

    log_format  main  '[$time_local] resp_status=$status req_time=$request_time '
                      'upstream_header_time=$upstream_header_time';
    access_log      /app/logs/nginx_access.log   main    buffer=500k    flush=5m;

    sendfile    on;
    tcp_nopush  on;
    tcp_nodelay on;

    #keepalive_timeout   65;   # how long to keep an idle connection alive
    #keepalive_requests  50;

    # proxy_connect_timeout   400ms;
    # proxy_send_timeout      400ms;
    # proxy_read_timeout      400ms;
    # send_timeout            500ms;

    #gzip    on;

    include     /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/app.conf

upstream app_servers {
    server 0.0.0.0:5000;
}

server {
    listen          8000;
    server_name     localhost;

    location / {
        proxy_pass      http://app_servers;

        # error_page      502  timeout_502;
        # error_page      504  timeout_502;
    }

    # location timeout_502 {
    #     #default_type    application/json;
    #     return          204  '{"Output": ""}';
    # }
}

score 0 · Answer 1 · answered Jun 12 '22 at 21:40

I have not tested this,

but perhaps a combination of custom http error pages and adding headers to specific locations to add the appropriate Content-Type of application/json. and set request timeout nginx

What's the status code returned? 502? something like

 error_page 502 /502.html;
 location = /502.html {

    add_header Content-Type application/json;
    return 502 '{ "Output": "" }'; # or a different status code
    
  }

http{
   ...
   proxy_read_timeout 300;
   proxy_connect_timeout 300;
   proxy_send_timeout 300;
   ...
}

also this may help?

Set Nginx Timeout & Default JSON Response

1 Answers1