Shake: how do I depend on dynamically generated source files?

Question

Given a directory structure like this:

.
├── frontend
│   ├── _build/          -- build dir, all files produced by shake, except for Frontend.elm, go here
│   ├── Build.hs         -- the build script
│   ├── build.sh         -- wrap Build.hs with `stack exec build -- $@`
│   ├── other files ...
│   ├── Frontend.elm     -- generated by a rule in Build.hs, `protoc  -I../proto --elm_out=. ../proto/frontend.proto`
│   ├── Index.elm        -- hand written source file
│   └── other elms ...   -- hand written source files
└── proto
    └── frontend.proto   -- protocol buffer message defination, hand written

Target _build/index.js depends on all .elm files, including Frontend.elm, but Frontend.elm is generated by a rule in Build.hs, if I blindly do:

want ["_build/index.js"]
"_build/index.js" %> \out -> do
    elms <- filter (not . elmStuff)
            <$> (liftIO $ getDirectoryFilesIO "" ["//*.elm"])
    need elms
    blah blah

want ["Frontend.elm"]
"Frontend.elm" %> \_out -> do
    cmd ["protoc", "blah", "blah"]

build.sh clean would give me:

Lint checking error - value has changed since being depended upon:
  Key:  Frontend.elm
  Old:  File {mod=0x608CAAF7,size=0x53D,digest=NEQ}
  New:  File {mod=0x608CAB5B,size=0x53D,digest=NEQ}

Is there a way to tell shake to watch out for the dynamically generated Frontend.elm, maybe build it first so it doesn't change during the rest of the build, I tried priority 100 ("Frontend.elm" %> ...), doesn't work.

Can you generate Frontend.elm not in the current directory but in the _build? And can you move all source files to directory src (as example)? — freestyle, Jul 28 '17 at 19:14
@freestyle 1. no, I cannot move it to _build, because 1.1 Elm uses directory structure as module naming space and 1.2 it need to be version controlled. 2 you mean move all hand written source files to a `src`, but generated source files to somewhere else? no I cannot, because 1.1 — Incömplete, Jul 29 '17 at 00:27
1.1 Usually, there is a way to specify the search path for imports. For example, the GHC has an option -i. 1.2 Also usually there is no sense in keeping the generated file in the VCS, it is enough to keep the original data for generation and the build script itself. 2 I mean restructure like this: frontend\\_build -- contains all generated files; frontend\src -- contains all source files (handwritten). — freestyle, Jul 29 '17 at 08:42

Daniel Wagner · Answer 1 · 2017-07-28T18:35:50.687

2

You should probably:

Switch from getDirectoryFilesIO, which does not track changes to the file system, to getDirectoryFiles, which does.
Declare your dependence on Frontend.elm, which you know you need even if it does not exist in the filesystem yet (hence might not be visible to getDirectoryFiles).
(Optional) Don't bother wanting Frontend.elm, since you only wanted it as a hack to enable _build/index.js.

With these changes, it would look like this:

want ["_build/index.js"]
"_build/index.js" %> \out -> do
    need ["Frontend.elm"]
    elms <- filter (not . elmStuff)
            <$> getDirectoryFiles "" ["//*.elm"]
    need elms
    blah blah

"Frontend.elm" %> \_out -> do
    cmd ["protoc", "blah", "blah"]

Caveat lector: I have not tested this solution.

edited Jul 28 '17 at 18:35

answered Jul 28 '17 at 17:41

Daniel Wagner

145,880
9
220
380

Here is problem. `getDirectoryFiles "" ["//*.elm"]` depends on changes in CD and Frontend.elm is generated in CD. – freestyle Jul 28 '17 at 18:16
It is better to keep the source files (not generated) separately and separately the directory for the files that are being builded. – freestyle Jul 28 '17 at 18:21
@freestyle Ah, right, probably I should move the `need` a bit higher to deal with that infelicity. Modification incoming. – Daniel Wagner Jul 28 '17 at 18:35
This can still lead to a problem. Let us imagine that the Frontend.elm depends (or will be in the future) `getDirectoryFiles "" ["//*.somthingelse"]`. The shake doc talks about getDirectoryFiles: "You should only call this function returning source files". – freestyle Jul 28 '17 at 18:59
seems I can get away with `orderOnly` – Incömplete Jul 29 '17 at 02:46
The above `need ["Frontend.elm"]` situation is exactly what `orderOnly` was designed for. That said, if you `orderOnly` something then know you will later always `need` it then it's really no benefit (or harm) in using `orderOnly` the first time. – Neil Mitchell Jul 30 '17 at 15:21
It is safe to do `getDirectoryFiles` after `orderOnly` or `need`ing everything that might be added to the directory. I've slightly weakened the docs to say "You should normally only call..." - your case is safe. – Neil Mitchell Jul 30 '17 at 15:59
I'm not sure if I understand you right, but if I `orderOnly [frontendElm]` then `need everyElmsIncludingFrontendElm`, the build works, if I just `need everyElmsIncludingFrontendElm`, then it will complain. – Incömplete Jul 31 '17 at 07:30

Shake: how do I depend on dynamically generated source files?

1 Answers1