0

I am making a shinydashboard app in R. I wanted to make an app that withdraws data from a database in AWS Athena every time it is deployed and display a DT table in my shiny app that gives you the option to see the information based on what date ranges you select in dateInputRange.

I needed an R package for this so I used RAthena I also used the packages DBI for the AWS CLI and paws. I downloaded all what I needed for this like boto3 and so forth. I also made a aws profile that contained all important information like secret key, region, path to bucket etc. Anyways I established a connection to the database and locally I was able to withdraw information from the database. My app was running fine locally. However the app when uploaded to shiny app io and deployed a second time it would not show updated dates in table. I thought this might be due to how I set up the scripts in my app. It now contains three scripts only ui.R, server.R and a global.R However after I did this I kept getting the error:

Error: Boto3 is not detected please install boto3 using either: pip install boto3 in terminal or install_boto(). Alternatively reticulate::use_python or reticulate::use_condaenv will have to be used if boto3 is in another environment.

So I did as the error said to and I added:reticulate::use_virtualenv("RAthena") to my global.R script where all data withdrawals and calculations are done. Here is the stack overflow answer. After I did this I got the error:

Error: Unable to find conda binary. Is Anaconda installed?

I decided to try then reticulate::use_condaenv("RAthena") I then simply got the boto3 error again. The annoying part is I have checked in my terminal and Boto3 is successfully installed. I also have Anaconda installed I even updated Boto3.

I saw from the answer that it is possible to use noctau to establish a database connection. I did this and everything worked fine locally. I was able to withdraw data from the database. Of course when I deployed this to shiny app io I got the error

Error in value[3L] : No region provided

I don't know why I have this error. In my aws profile my region for the database is specified. In R studio I see it in the connections tab in the right hand corner. With the code dbGetInfo(con) I see it there as well. I don't know what the solution is.

M--
  • 25,431
  • 8
  • 61
  • 93
Nick
  • 369
  • 1
  • 3
  • 18

1 Answers1

1

For RAthena it appears shinyapp.io is having difficulty setting up your environment. This tutorial may help solve the RAthena problem. Tutorial: using Shiny + reticulate to create apps with R and Python 3

For noctua, it looks like it is having difficulty getting your aws credentials when on shinyapp.io server. When setting up the shinyapp.io how have you set up your credentials for AWS? Are you using environmental variables or are you using .aws/credentails and .aws/config files?

You might want to consider the config package recommended in the shinyapps.io user guide book: https://docs.rstudio.com/shinyapps.io/applications.html#config-package

Here is an example of how to set it up.

Setting up config.yml

local:
  dataconnection:
    profile: 'my_personal_profile'
    aws_access_key_id: 'my_aws_key'
    aws_secret_access_key: 'my_aws_secret_key'
    region: 'eu-west-1'
    database: 'default'
    s3_bucket: 's3://my/s3/bucket/'
    
shinyapps:
  dataconnection:
    profile: 'shiny_profile'
    aws_access_key_id: 'shiny_aws_key'
    aws_secret_access_key: 'shiny_aws_secret_key'
    region: 'eu-west-1'
    schema: 'default'
    s3_bucket: 's3://shinyapp/bucket/'
library(DBI)
conn_args <- config::get("dataconnection")
con <- dbConnect(noctua::athena(),
   profile_name = conn_args$profile,
   aws_access_key_id = conn_args$aws_access_key_id,
   aws_secret_access_key    = conn_args$aws_secret_access_key,
   region_name    = conn_args$region,
   schema_name   = conn_args$schema,
   s3_staging_dir = conn_args$s3_bucket
)

When the environment variable R_CONFIG_ACTIVE=local, then the local credentials will be used. When R_CONFIG_ACTIVE=shinyapps (on shinyapps.io), the production credentials will be used.

Dyfan Jones
  • 229
  • 2
  • 9
  • Hi to be honest it has been already a while since I set up the aws credentials but I remember simply having to type in the terminal all the necessary information like aws_access_key_id, aws_secret_access_key, region_name etc. I set up the AWS CLI like it said to on the [webiste] (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) I then connected to Athena by only putting the s3_staging_dir. I did this because I could of hard coded it like you showed above but I need my information to be secure. – Nick Nov 11 '20 at 10:10
  • But actually im not sure where you are making the shinyapps dataconnection with the profile: 'shiny_profile'. I think as I set it up I didnt make any profile – Nick Nov 11 '20 at 10:11
  • With what you wrote at the beginning of your answer I have already followed the steps in the tutorial and made an Rprofile but Im not sure if I set it up correctly in the tutorial they said basically just to copy their Rprofile. – Nick Nov 11 '20 at 10:14
  • Hi @Nick, the issue with `noctua` is the aws credentials are not available on shinyapp.io. To fix this, the aws credentials are required to be set on the shinyapp.io. This can be done either through `.aws/credentails` and `.aws/config` files or environmental variables on the shinyapp.io – Dyfan Jones Nov 11 '20 at 16:51
  • The `config` package is a method to pass credentials to the shinyapp.io using a yaml file. The `config` package will switch between credentials depending on the environmental variable `R_CONFIG_ACTIVE`. When on the shinyapp.io the environmental variable `R_CONFIG_ACTIVE` is set to `shinyapps` (https://docs.rstudio.com/shinyapps.io/applications.html#config-package). This will use the `shinyapp` credentials from the yaml (example above) for the shinyapp.io. This method is fairly useful as it prevents hard coding credentials into your shinyapp. I hope this helps :) – Dyfan Jones Nov 11 '20 at 16:54
  • Thank you for the explanation I implemented what you said above with a few changes however. I changed local: to default: in my yaml file since i kept getting an error for that. Anyways I was able to get everything working locally again but when I deployed it to shinyapps io I got the following error message `Info: (Data scanned: 2.61 MB) additional arguments ignored in warning() Error in value[[3L]](cond) : AccessDenied (HTTP 403). Access Denied`. I believe I get something similar when I run things locally but its still works whereas on shiny app execution is halted. – Nick Nov 24 '20 at 10:11
  • I got it to work now by adding `noctua_options(cache_size = 1)#getting one table at a time` from your answer [here](https://github.com/DyfanJones/noctua/issues/96) Thanks again! – Nick Nov 24 '20 at 10:57
  • 1
    Ah perfect. I would also recommend increasing your caching size so that your shiny app can benefit from speed improvements when a user does a repeat query. Here is some documentation around `noctua` and AWS Athena Query Caching: https://dyfanjones.github.io/noctua/articles/aws_athena_query_caching.html – Dyfan Jones Nov 24 '20 at 14:15