Order difference between 'shinytest' expected and current json files

Question

I have an ordinary shinytest script to test my Shiny app. When I run it with shinytest::testApp, it is successful: both the JSON files and the images are similar between current and expected.

But when the test is ran via testhat:

  expect_pass(
    testApp(
      system.file("app", package = "shinyCircularDichroism"),
      compareImages = TRUE
    )
  )

then the images are the same but the JSON files differ:

Name         Status      
001.json  != Files differ
001.png      No change   
002.json  != Files differ
002.png      No change   
003.json  != Files differ
003.png      No change   
004.json  != Files differ
004.png      No change   
005.json  != Files differ
005.png      No change

I've inspected the differences between the JSON files. Actually they are the same up to the order (that is, one is [a, b, ... and the other is [b, a, ...]).

Is it a known issue? Why does it occur, only via testthat? Is there a way to compare the JSON files up to the order?

It's easier to help you if you provide a [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) with sample input that can be used to test and verify possible solutions. — MrFlick, Aug 24 '22 at 14:11
@MrFlick If I could, I would have provided it! I can't : the app is huge and confidential. That said, I will try to do a minimal example, but I'm not sure this will reproduce this bug. — Stéphane Laurent, Aug 24 '22 at 14:49
Moreover, this requires to do a package. I'm not sure someone will take the time to build a package to try to help me... I mainly asked this question in case someone already knew this issue (I also posted the question on RStudio Community). — Stéphane Laurent, Aug 24 '22 at 15:02
This [Github issue](https://github.com/rstudio/shinytest/issues/409) seems to be about this: `devtools::check()` fails with JSON but only because of the ordering. I didn't read everything but it seems to be related to the locale. I hope it's useful — bretauv, Aug 26 '22 at 15:32

score 1 · Accepted Answer · answered Aug 27 '22 at 12:45

Solved thanks to the Github issue spotted by @bretauv. The problem is caused by the LC_COLLATE variable. Depending on the value of this variable, the results of sort can be different.

library(shinytest)
collation <- Sys.getlocale("LC_COLLATE")
Sys.setenv(LC_COLLATE = "en_US.UTF-8")
Sys.setlocale("LC_COLLATE", "en_US.UTF-8")
shinytest::testApp("../")
Sys.setlocale("LC_COLLATE", collation)

and:

test_that("Shiny test", {
  skip_if_not_installed("shinytest")
  library(shinytest)
  collation <- Sys.getlocale("LC_COLLATE")
  Sys.setenv(LC_COLLATE = "en_US.UTF-8")
  Sys.setlocale("LC_COLLATE", "en_US.UTF-8")
  expect_pass(
    testApp(
      system.file("app", package = "shinyCircularDichroism"),
      quiet = TRUE,
      compareImages = TRUE
    )
  )
  Sys.setlocale("LC_COLLATE", collation)
})

Order difference between 'shinytest' expected and current json files

1 Answers1