Play audio by clicking on ggplot/plotly scatter plot

Question

I've created a scatter plot with ggplot2 and would like to link the data points with corresponding sound files. In a shiny app it should then be possible to click on or hover over a data point to play the sound file. While I manage to get the plot into a shiny app, I am struggling with the inclusion of tags$audio, as described here, into the plot.

library(shiny)
library(plotly)

ui <- fluidPage(
  plotlyOutput('scatter')
)
server <- function(input, output) {
  output$scatter <- renderPlotly({
    ggplotly(p)
  })
}
shinyApp(ui = ui, server = server)

The plotly plot displays a tooltip with the info per data point, among others also the URL for the sound file. See the working example here: https://chart-studio.plotly.com/~petergi/1.

The ggplot2 code contains the URL information in the aes 'text'.

p <- ggplot(without_outliers) +
  aes(x = T2, y = T1, label = labels, col = next_label, 
      text = paste0("URL: ", "https://www.yyyy.zz/audio/", tolower(bundle), ".mp3"))

Any hint to how to bring an audio player to the scatter plot's data points in shiny would be highly appreciated.

Peter · Answer 1 · 2020-04-08T16:54:14.503

In the meantime, I found a working solution in using the R package ggiraph, which enhances ggplot2 with tooltip and onclick functionalities, when used in a shiny application. In a nutshell, a geom_point_interative call can have an onClick argument, which in turn then is a call to a Javascript function to play the audio (without opening a new window or a player).

library(ggplot2)
library(ggiraph)
library(shiny)

p <- ggplot(mtcars, aes(wt, mpg))+
  geom_point_interactive(onclick="var audio= new Audio(\"https://file-examples.com/wp-content/uploads/2017/11/file_example_MP3_700KB.mp3\"); audio.play();")
x <- girafe(ggobj = p)

To allow for all different data points in a data frame to be associated with their corresponding audio files, the data frame should be modified to contain a new column with the individual javascript code to play the audio.

An example shiny app can be found here.

score 1 · Answer 2 · edited Apr 24 '21 at 14:17

Thank you Peter, I managed to get this working with help from your pointers. I know this is a late response but I hope it helps if someone else runs into a similar issue. First, I recommend reviewing the examples shared above.

I managed to get something like this working. However, instead of defining the audio and playing it with the onclick argument, I created a separate javascript file and sourced it through the shiny applet. I created a separate file called fileNames.js where I defined each variable. Using the example above with only one audio file, this would be: var audio= new Audio("https://file-examples.com/wp-content/uploads/2017/11/file_example_MP3_700KB.mp3");.

In the R shiny applet, I changed the onclick argument in geom_point_interactive() to the javascript code to play the audio file. In your example, the string would be simply audio.play();. For whatever reason passing the multi-line javascript code to the onclick argument didn't work for me.

Early in the ui call, I included includeScript(path = "fileNames.js") to source the javascript into the shiny app.

Here's another helpful example in case someone stumbles upon this thread as I did: https://www.ardata.fr/en/post/2019/05/24/whats-up-ggiraph/

Play audio by clicking on ggplot/plotly scatter plot

2 Answers2