0

I have the following situation/workflow:

  1. The user utilizes Tool A to capture sensor data and save it to a CSV file.
  2. The user takes the CSV and uploads it to a R Shiny Applications (using fileInput) to process it further.

I would like to get rid of the intermediate CSV and directly open the Shiny application with the data already loaded. I.e. I need a way to transfer the contents of the CSV automatically to the Shiny application.

What could work:

  1. Tool A stores the CSV in a special location that is served by a HTTP server on a fixed endpoint.
  2. The Shiny application requests the file from the known location on startup.

However, this still needs the intermediate CSV file, and adds complexity by introducing an additional server (or at least endpoint). Furthermore it needs additional logic if multiple users are active / multiple files are created at the same time. So it is far from ideal.

Is there any way to get the contents of the CSV directly from Tool A to the Shiny Application? E.g. by mimicking the messages the 'fileInput' widget produces?

AEF
  • 5,408
  • 1
  • 16
  • 30
  • Have you tried using pins?? https://pins.rstudio.com/ – Daniel Jul 15 '21 at 08:41
  • @Daniel No, never heard of it. I will look into it, thanks! – AEF Jul 15 '21 at 08:47
  • You're welcome. If you have access to AWS or GCP you could look into data streaming services (like SQS and Pub/Sub). Your tool A would have to send the messages to the topic and then you can read the messages from there. [This](https://rviews.rstudio.com/2017/11/15/shiny-and-scheduled-data-r/) article basically describes your situation. – Daniel Jul 15 '21 at 08:50
  • But, if I understand you correctly, this would still need additional logic when multiple users are involved? Because Shiny does not know which file to read in if there are mutliple. (I think this does not depend on the type of source - if files are not transferred directly from Tool A, but stored intermediately, something or someone must determine which is the correct one.) – AEF Jul 15 '21 at 08:55
  • If you want to get rid of the csv file - Does "Tool A" offer an API? Or do you mean you want to get rid of the user uploading the csv, but still use them under the hood? – ismirsehregal Jul 15 '21 at 09:10
  • If possible, I would like to get rid of the file completely. Using it under the hood would be OK as workaround too. Tool A is also developed in my team, so we can add any feature we need. (It's written in C and not in R though) – AEF Jul 15 '21 at 09:13
  • A simple scenario still keeping the csv's would be to have "tool A" write the files to a shared network drive (or something dropbox-like if not connected via LAN), so that the shiny host can access them by e.g. `reactiveFileReader` or `reactivePoll`. Or you use a REST API / MQTT or some other protocol for direct communication (more effort). – ismirsehregal Jul 15 '21 at 09:19
  • 1
    Why don't you just transmit the content of the file to a database and let the shiny app read from that database? – nicola Jul 15 '21 at 09:32
  • That's exactly how a Streaming service works, nicola. Tool A will post data points to a topic in the service, and then your shiny can use a plumber API to read from it. It takes care of the multiple-users-issue too since any number of instances can write to the topic concurrently. – Daniel Jul 15 '21 at 10:38
  • @ismirsehregal that is basically the same as the variant with the HTTP server I described above, isn't it? Just that its not a HTTP, but maybe a Samba server or similar when I use a network drive. – AEF Jul 15 '21 at 10:56
  • @nicola that would work, but it has the same limitations as the other suggestions: I would need to tell the app somehow which of all files in the database is the correct one for the current user? Or, other way round, who is the current user. – AEF Jul 15 '21 at 10:57
  • Sure - there are numerous ways to get that data from A to B (And we don't know a lot about your setup LAN/Cloud, Windows/Linux, amount of data (sample rate)), but still `reactiveFileReader` or `reactivePoll` provide you with a convenient way to make those external data source reactive in the shiny context. – ismirsehregal Jul 15 '21 at 11:13
  • @ismirsehregal yes, but the problem is not the reactivity. The data is not reactive. It is created once, and then never altered again. So reactive polling has no advantage over a "normal" upload. The problem is just how to get it to Shiny automatically; in such a way that multiple users can do it at the same time and each user gets his own file without further interaction. Sorry if that was not clear from the question. – AEF Jul 15 '21 at 11:24
  • do you use some sort of login for the applications? E.g. if you host your shiny app with shiny proxy, you can use the AD for login and have the user information. You could also integrate login to your Tool A and with this connect the users when you store the read outs in a DB or somewhere else – starja Jul 15 '21 at 11:26
  • Well, then I guess you will need to somehow mark which files belong to which user (folderstructure, filenames), use one of the above mentioned ways to transfer the data and load the needed dataset according to a user-login in your shiny app e.g. using [shinymanager](https://datastorm-open.github.io/shinymanager/) or [shinyproxy](https://www.openanalytics.eu/tags/shinyproxy/) as mentioned by @starja. – ismirsehregal Jul 15 '21 at 11:55
  • Yes, true that will work :) Thank you! However, I'm still interested if there is a direct way to transmit teh data. It seems strange to me, that shiny cannot just eg. receive data via POST request or similar. – AEF Jul 15 '21 at 12:08
  • Of course you can use http verbs: GET(), HEAD(), PATCH(), PUT(), DELETE() and POST() e.g. via [httr](https://github.com/r-lib/httr) from within the shiny server function. Also see [plumber](https://www.rplumber.io/). – ismirsehregal Jul 15 '21 at 12:14
  • Shiny has the full power of R behind, so you can do everything R can. You can trigger a POST request by pressing a button, or schedule the request at your convenient frequency or whatever you need. – nicola Jul 15 '21 at 12:23
  • I know that R can send such requests, but I would of course need to recieve, them i.e. implement the server side. Like it is possible with plumber. However, afaik plumber cannot easily combined with shiny. – AEF Jul 15 '21 at 12:36
  • I'd give it a try. If necessary run plumber in a child process. E.g. via `r_bg()` from [callr](https://github.com/r-lib/callr) also see [IPC](https://cran.r-project.org/web/packages/ipc/vignettes/shinymp.html). – ismirsehregal Jul 15 '21 at 12:59
  • That is a really nice idea! I'll try it! – AEF Jul 15 '21 at 13:15
  • 1
    For future readers: Please check my related answer [here](https://stackoverflow.com/questions/25283736/r-shiny-rest-api-communication/71064046#71064046) which is showing how to handle POST requests directly in `shiny`. – ismirsehregal Feb 10 '22 at 22:21

0 Answers0