5

Would it be possible to make Shake reactive, using inotify (or whatever git-annex and Yesod use) so that if ever the filesystem changes in such a way to imply that rule should execute, it does so at the earliest opportunity?

Petr
  • 62,528
  • 13
  • 153
  • 317
John Wiegley
  • 6,972
  • 1
  • 16
  • 20
  • 4
    This question appears to be off-topic because it a feature request for Shake, not a programming question. – Petr Aug 01 '13 at 20:55
  • 2
    @PetrPudlák I'd interpret this question as "how can I use the Shake library to build a reactive build system", since at least one of the answers requires no new features. – Neil Mitchell Aug 02 '13 at 05:56

1 Answers1

8

The author of Shake, Neil Mitchell, answered this by saying:

There are a few ways to approach this:

  1. You could just rerun Shake every time it detects something has changed. Shake is highly optimised for fast rebuilds, if the change requires doing a compile, then the time for Shake to figure out what to rebuild is probably minimal. Requires no changes to Shake.

  2. There are certain things Shake does on startup, like reading the Shake database. If there is demand, and that turns out to be noticeable in time, I would happily provide a rerun Shake cheaply API of some sort - it's not that difficult to do.

  3. When Shake does do a rebuild-check, the most expensive thing it does is checking file modification times. If the inotify layer gave a list of the files that had changed I could only recheck things that had actually changed. For a massive project you're likely to see ~1s checking modification times, so it probably buys you a little, and isn't too hard to implement.

  4. If Shake is actively building, and then something changes, you could throw an exception, kill whatever is being built, and restart Shake. Shake has been thoroughly tested with exceptions being thrown at it, and does the right thing. I know at least one person uses Shake in this way.

  5. Finally, if Shake is actively building, you could dynamically terminate just those rules whose inputs have changed and go again. Shake could support this model, but it would be a reasonable amount of work, and require re-engineering some pieces. That would be the full reactive model, but I suspect it only starts to be a benefit when you have a massive number of files and a few files are changing almost continuously but most files aren't.

We also determined that combining Shake with a utility like Hobbes (also on Hackage) can make it possible to do reactive builds.

Chris Martin
  • 30,334
  • 10
  • 78
  • 137
John Wiegley
  • 6,972
  • 1
  • 16
  • 20
  • 1
    I use Shake as a build manager and step 2 takes up about 300ms for me, which is too slow for any reasonable automatic polling approach. – nominolo Aug 02 '13 at 11:57
  • @nominolo If it matters to you, let me know, and I'll redo the API to avoid the overhead with rerunning. If you have a Haskell profile for your use case, I'd love to see it, as it might identify something else at fault in the 300ms. – Neil Mitchell Nov 06 '13 at 19:38