1

For HTTP performance testing, I'm searching for a way to generate data as efficiently as possible for a repeatable Java InputStream which is incompressible by HTTP gzip compression (or as close as possible). I found this blog post on Randomly Generated Input Stream which already does a very good job of providing a fast random number generator for an input stream and which is nearly incompressible, but it is not repeatable.

The requirements are:

  • use minimal CPU resources
  • be repeatable (e.g. always return the same results when read from beginning)
  • produce an output stream which is as incompressible as possible
  • produce output as fast as possible (e.g. not limited by disk I/O)

I'd be glad to get some tips or pointers how these requirements could be met.

DwB
  • 37,124
  • 11
  • 56
  • 82
Florian Feldhaus
  • 5,567
  • 2
  • 38
  • 46
  • 2
    Why isn't it repeatable? Just seed the `Random` with a known seed. – Louis Wasserman Nov 08 '17 at 21:21
  • Just change the class so you can seed the RNG. Alternatively (and much simpler) just use pre-generated data - say, a bunch of stuff you've gzipped before testing. – pvg Nov 08 '17 at 21:23
  • Or, since compression typically does frequency analysis and requires repeat characters to be compressed, just start at some arbitrary character and increment by one for each next character. Presto, can't compress it and can easily repeat it. – markspace Nov 08 '17 at 21:23
  • @markspace that is not going to produce 'incompressible' data. – pvg Nov 08 '17 at 21:25
  • The digits of PI do not repeat. Why not use those? – DwB Nov 08 '17 at 21:26
  • I think I found a good example here: https://github.com/h2database/h2database/blob/master/h2/src/test/org/h2/test/store/TestStreamStore.java#L197 – Florian Feldhaus Nov 08 '17 at 21:26
  • @DwB the digits of pi compress really nicely. – pvg Nov 08 '17 at 21:28
  • This question has some more details why `Random` is not good and what some alternatives could be https://stackoverflow.com/questions/29193371/fast-real-valued-random-generator-in-java – Florian Feldhaus Nov 08 '17 at 21:38

0 Answers0