`sheets_auth` (from `googlesheets4`) doesn't work with a token created with `googleAuthR` package

Question

I want to create an shiny app that analyses some data from a google sheet. It will be deployed on the server. The sheet is only available for users within the company, so I need to implement authentication (api key approach won't work here) and as I read on the internet googleAuthR for now is the only option (see https://github.com/r-lib/gargle/issues/14)

First, here's the app that doesn't use default auth method:

library(googleAuthR)
library(googlesheets4)
library(shinyjs)

ui <- shinyUI(navbarPage(title = 'sample_app',
                         tabPanel(
                                  useShinyjs(),
                                  tableOutput('test')
                         )
))

server <- function(input, output, session) {

  read_my_sheet <- function() {
    sheets_auth(scopes = "https://www.googleapis.com/auth/spreadsheets.readonly",
                cache = FALSE, use_oob = FALSE)
    read_sheet("<MY_SHEET_ID>")
  }

  output$test <- renderTable({
    read_my_sheet()
  })
}

shinyApp(ui, server)

As you can see, the app asks me to authenticate via Tidyverse API packages application just right after launching the app. What I would like to have is to first load the app, start authentication process by clicking the button that uses googleAuth module, fetch a token and pass it to sheets_auth so no Tidyverse authentication is required. My try:

options("googleAuthR.scopes.selected" = c("https://www.googleapis.com/auth/spreadsheets.readonly"))
options("googleAuthR.webapp.client_id" = "<MY_CLIENT_ID>")
options("googleAuthR.webapp.client_secret" = "<MY_CLIENT_SECRET>")
options(shiny.port = 8787)

library(googleAuthR)
library(googlesheets4)
library(shinyjs)


ui <- shinyUI(navbarPage(title = 'sample_app',
                         tabPanel(
                           useShinyjs(),
                           googleAuthUI('gauth_login'),
                           tableOutput('test')
                         )
))

server <- function(input, output, session) {
  access_token <- callModule(googleAuth, "gauth_login")

  read_my_sheet <- function() {
    fil <- access_token()
    sheets_auth(token = fil, scopes = "https://www.googleapis.com/auth/spreadsheets.readonly",
                cache = FALSE, use_oob = FALSE)
    read_sheet("<MY_SHEET_ID>")
  }

  output$test <- renderTable({
    read_my_sheet()
  })
}

shinyApp(ui, server)

Unfortunately, when launching the app the first that appears is again a window that asks for authentication by Tidyverse. How to switch it off? I want to authenticate using only my client.