1

We have a problem with a Plumber API which is deployed to Posit Connect (v. 2023.03.0) on an Ubuntu 20.04 virtual machine. The API runs processes which could take longer than 60s to complete (it fetches data from other APIs would could exceed 60s).

Whenever the response time is more than 60s, the API returns a "504 Gateway Time-out - The server didn't respond in time." error. However, the API remains active and even though the 504 error is returned, any processes such as writing data to a Connect pin will be completed after the 504 error message has been returned. The API therefore remains active after the 60s timeout period.

As far as I can establish, there are no settings within Posit Connect (other than that mentioned below) which could result in the timeout occurring. My suspicion is that it is either the virtualisation platform on which Posit Connect is installed, or the firewall of the environment which cause the timeout.

The Runtime Settings for the API on Posit Connect are as following:

Initial Timeout: 600s (increased from the default of 60s to ensure this is not the cause)

Connection timeout: 3600s

Idle Timeout: 120s

Read timeout: 3600s

I include a reprex below of a Plumber API which also returns a 504 timeout when the delay parameter of the API is more than 60.

library(plumber)
library(pins)

## Configure API 
#* @apiTitle Testing Plumber Posit Connect Timeout
#* @apiVersion 0.0.1
#* @apiDescription Reprex to replicate Plumber API 60s timeout

## Configure API endpoint
#* @get /TimeoutTest
#* @param Delay The time in seconds which the API should wait before returning a response.
#* @response default Returns a dataframe in JSON format

TimeoutTest <- function(Delay) {
  
  ## Delay API response for the period specified in Delay
  StartTime = Sys.time()
  Sys.sleep(Delay)
  DelayEnd = Sys.time()
  
  ## Configure API payload reponse
  payload <- paste0("Delay: ", Delay, "s, Start Time: ", StartTime, " - End Time: ", DelayEnd)
  
  ## configure and write the payload to a Posit Connect Pin
  board <- board_connect(
    server = "https://connect.server.com",
    auth = "manual",
    key= Sys.getenv("CONNECT_API_KEY")
  )
  
  pin_write(board = board,
            payload,
            name = "API_Timeout_Pin",
            type = "json")
  
  ## return payload as the API GET request response
  return(payload)
}

If "Delay" is less than 60, then the API returns a JSON payload and the payloads is also written to the pin on Connect. If "Delay" is more than 60s, a 504 error is returned. However the response is still written to the Pin, implying that the API process remains active even after the 504 is returned.

Jaco
  • 31
  • 4
  • 1
    @r2evans The connection timeout is set to 3600s (the default). I've tested the connection timeout with a value of more than 3600s, but it did not have any impact. I did not expect that it should, because the timeout occurs after 60s. – Jaco May 21 '23 at 12:35
  • I apologize, I realize I skipped portions of your question. Another thought, do you have a rev-proxy in front of Connect? – r2evans May 21 '23 at 12:38
  • As far as I know, we do not. I'll have to ask the team who configured the network and server. If there is, is there specific settings which they should check? – Jaco May 21 '23 at 13:42
  • I don't think that Connect has any control over the timeout imposed by the rev-proxy or the client browser (I don't know if or which of those are a factor). In general, if you have really long queries, I suggest that a synchronous HTTP GET is the best way to do it; have you considered multi-process, have the first query return a "token" immediately and have a second API endpoint return data for that token? (It will return "empty" or "in progress" until the actual calculation is complete.) – r2evans May 21 '23 at 13:57
  • The API version in production (i.e. not the reprex) has to fetch information for 7500 projects from an API on a remote server. In order keep the connection alive, the API fetches project info in batches of 50 projects. There is therefore a request and response payload every 5 to 10 seconds. Therefore continuous traffic to and from the API. The Plumber Swagger interface returns the 504 error after 60 seconds. The API however do continue to operate until the complete payload has been fetched and then writes the data successfully to the Connect pin (it takes about 3 minutes to complete). – Jaco May 21 '23 at 14:08
  • If I understand what you suggest correctly, then I have done something similar to your suggestion. Writing to the pin is a workaround. Ideally I would just like to have a json payload returned by the API. – Jaco May 21 '23 at 14:08
  • I'm assuming that you either have multi-processing built in to the production version (not shown here), or you have the Connect runtime constraints such that the fact that one process is locked up on one user does not keep other requests from being serviced ("Load Factor" of 0). "My" expectation of an API is that if it cannot immediately return data, it returns either a token or nothing, where the client has to deal with re-querying (after a delay) if nothing is returned. Having two endpoints (query and retrieve) also makes sense. – r2evans May 21 '23 at 14:35
  • @r2evans. I assume you were referring to the async feature of Plumber using the future package? Will try this as per this video. https://cloud.rstudio.com/resources/rstudioglobal-2021/plumber-and-future-async-web-apis/ – Jaco May 21 '23 at 15:12
  • with or without `promises`, https://rstudio.github.io/promises/articles/future_promise.html – r2evans May 21 '23 at 22:42
  • 1
    @r2evans. Our IT team traced the problem at last to a timeout setting on the proxy server. The issue has been resolved. Thanks for your suggestions regarding async. I learnt something new. – Jaco May 22 '23 at 14:09

1 Answers1

1

The problem was solved by correcting the proxy server timeout settings.

Jaco
  • 31
  • 4
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 23 '23 at 04:46