2

Our testing methodology has test documentation as a first class object in the documentation output. Specifically, it defines the specification that is tested by each behavioural test.

Running cargo doc on a project with suitably documented tests doesn't yield much in the way of documentation derived from the test doc strings and I can't see any obvious way of making it include the test doc strings in the output.

An example module would be as follows:

/// This function does some important stuff
pub fn working_fn() -> bool {
    true
}

#[cfg(test)]
mod tests {
    //! This is some important set of tests
    //!

    use super::*;

    /// The function should work
    #[test]
    fn it_works() {
        assert!(working_fn());
    }
}

I get documentation output for the public working_fn, but nothing for the tests module. I appreciate that an additional complication is that tests are private and ideally I'd be able to document private tests without also documenting other private objects.

Henry Gomersall
  • 8,434
  • 3
  • 31
  • 54
  • @Shepmaster Right, bear with me, I'm learning fast but not necessarily fast enough! Edit made – Henry Gomersall Sep 24 '19 at 14:44
  • @Shepmaster yes, I'd like that documentation to show up somewhere - ideally in a section titled `Specification`, but I'm realistic enough to accept that might be a few steps away. – Henry Gomersall Sep 24 '19 at 14:46

1 Answers1

5

You can introduce a new feature flag that can be used to treat tests specially for the purposes of documentation.

Add the feature to your Cargo.toml:

[features]
dox = []

Use the feature flag in your code.

  1. Compile the tests modules if tests are running or the feature flag is provided.
  2. Only mark #[test] functions if the feature flag is not provided. The #[test] attribute automatically implies #[cfg(test)], so we have to skip that to allow the function to exist.
/// This function does some important stuff
pub fn working_fn() -> bool {
    true
}

#[cfg(any(test, feature = "dox"))]
mod tests {
    //! This is some important set of tests
    //!
    use super::*;

    /// The function should work
    #[cfg_attr(not(feature = "dox"), test)]
    fn it_works() {
        assert!(working_fn());
    }
}

Build the documentation

cargo doc --document-private-items --features=dox

Keep an eye on #[cfg(rustdoc)], which would allow you to remove the need for your own feature flag but is currently unstable.

See also:

ideally I'd be able to document private tests without also documenting other private objects

You could make your tests pub or pub(crate).

If that wasn't an option, I think this will be more annoying than it is valuable. The direct solution I know of would be to follow How do I change a function's qualifiers via conditional compilation? to conditionally make the test pub or not.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
  • This looks to me to be a workaround to the basic problem, but it's frustratingly a bit limited (not your fault!). Ideally I'd like to get hold of an intermediate representation of the docs from inside `rustdoc`, so it looks like I'll need to keep thinking... – Henry Gomersall Sep 24 '19 at 15:14
  • @HenryGomersall As I understand it, since you are using conditional compilation (`#[cfg(...)]` and `#[test]`), there *isn't* an internal representation of the docs — the functions don't even exist by the time rustdoc looks at the code. – Shepmaster Sep 24 '19 at 15:16
  • Interesting, where does `rustdoc` get its input from? – Henry Gomersall Sep 24 '19 at 15:22
  • @HenryGomersall *Rustdoc actually uses the rustc internals directly. [...] It runs the compiler up to the point where we have an internal representation of a crate (HIR) and the ability to run some queries about the types of items. [...] librustdoc performs two major steps [...] 1. "Clean" the AST into a form that's more suited to creating documentation [...] 2. Use this cleaned AST to render a crate's documentation, one page at a time.* [rustc guide](https://rust-lang.github.io/rustc-guide/rustdoc.html) – Shepmaster Sep 24 '19 at 15:26
  • that link is very interesting. Taking the `rustc` build command from `cargo test --verbose`, and appending `-Zunpretty=hir-tree` provides us with what might well be a useful HIR from which I can probably glean the relevant bits and pieces. – Henry Gomersall Sep 24 '19 at 16:02
  • @HenryGomersall if you are interested in going that deep, it's possible you could work with the rustdoc team to add a `--document-test-items` – Shepmaster Sep 24 '19 at 16:04