1

I use a web interface to upload files to AWS. How can I increase the upload speed?

I'm running R Shiny on a t2.2xlarge instance and uploading ~50 large CSV files totaling ~500 MB. The slowest part is getting files from my local computer to the web interface instance. Subsequent transfers within AWS are quite fast.

It currently takes me 5-30 minutes to upload files. How do I bring that down to ~2 minutes using Shiny, R, and/or Python?

library(shiny)
options(shiny.maxRequestSize = 1000*1024^2)

shinyApp(
    ui = fluidPage(fileInput("test", "Upload here", multiple = TRUE)),
    server = function(input, output, session)
)

Edit: this should work from any computer without having to download AWS CLI, etc.

Jeff Bezos
  • 1,929
  • 13
  • 23
  • 1
    This might not solve your whole problem but there is a default [upload file size limit](https://stackoverflow.com/questions/18037737/how-to-change-maximum-upload-size-exceeded-restriction-in-shiny-and-save-user) of 5 mb which you'll probably want to change. – Dan Adams Feb 16 '22 at 16:35
  • Thanks @Dan, I updated my code – Jeff Bezos Feb 16 '22 at 16:41
  • 1
    Can you use the AWS CLI to do the upload? It can process files in parallel to make more use of your bandwidth. – John Rotenstein Feb 16 '22 at 20:43
  • @JohnRotenstein something like that, I just don't know how to make it so that anyone can upload data through the web interface without setting up AWS CLI first – Jeff Bezos Feb 16 '22 at 21:57

0 Answers0