fatal error: 'stddef.h' file not found when using clang-llvm ASTMatcher

Question

I am new to using ASTMatcher and following a code from a tutorial - https://github.com/peter-can-talk/cppnow-2017. Here, the tool clang-variables can be run using the following command:

cd code/clang-variables 
docker run -it -v $PWD:/home clang 
root@5196c095092d:/home# ./clang-variables test.cpp -- -std=c++14 

Instead of test.cpp file, if I use another source file then I am getting the following error:

fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~

I understand that my source file has these header files that needs to be included. I tried to include them in the Makefile as follows but the error is still present:

clang-variables: $(TARGET).cpp
    $(CXX) $(HEADERS) $(LDFLAGS) $(CXXFLAGS) $(TARGET).cpp $(LIBS) -o $(TARGET) -I$(START_DIR)/source -I$(HOME_ROOT)/extern/include

There was no error while compilation. So I am wondering, is it possible to mention the include files as arguments to ASTMatcher? Please find the code below:

// Clang includes
#include "clang/AST/ASTConsumer.h"
#include "clang/AST/ASTContext.h"
#include "clang/AST/Decl.h"
#include "clang/AST/Type.h"
#include "clang/ASTMatchers/ASTMatchFinder.h"
#include "clang/ASTMatchers/ASTMatchers.h"
#include "clang/Basic/Diagnostic.h"
#include "clang/Frontend/CompilerInstance.h"    
#include "clang/Frontend/FrontendAction.h"
#include "clang/Tooling/CommonOptionsParser.h"
#include "clang/Tooling/Tooling.h"
// LLVM includes
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/raw_ostream.h"

// Standard includes
#include <memory>
#include <string>
#include <vector>

namespace Mutator {

/// Callback class for clang-variable matches.
class MatchHandler : public clang::ast_matchers::MatchFinder::MatchCallback {
 public:
  using MatchResult = clang::ast_matchers::MatchFinder::MatchResult;

  /// Handles the matched variable.
  ///
  /// Checks if the name of the matched variable is either empty or prefixed
  /// with `clang_` else emits a diagnostic and FixItHint.
  void run(const MatchResult& Result) {
    const clang::VarDecl* Variable =
        Result.Nodes.getNodeAs<clang::VarDecl>("clang");
    const llvm::StringRef Name = Variable->getName();

    if (Name.empty() || Name.startswith("clang_")) return;

    clang::DiagnosticsEngine& Engine = Result.Context->getDiagnostics();
    const unsigned ID =
        Engine.getCustomDiagID(clang::DiagnosticsEngine::Warning,
                               "found mutating variable");

    /// Hint to the user to prefix the variable with 'clang_'.
    const clang::FixItHint FixIt =
        clang::FixItHint::CreateInsertion(Variable->getLocation(), "precision & accuracy mutation");

    Engine.Report(Variable->getLocation(), ID).AddFixItHint(FixIt);
  }
};  // namespace Mutator

/// Dispatches the ASTMatcher.
class Consumer : public clang::ASTConsumer {
 public:
  /// Creates the matcher for clang variables and dispatches it on the TU.
  void HandleTranslationUnit(clang::ASTContext& Context) override {
    using namespace clang::ast_matchers;  // NOLINT(build/namespaces)


    const auto Matcher = declaratorDecl(
    isExpansionInMainFile(), 
    hasType(asString("int"))
    ).bind("clang");

     /*
    // clang-format off
    const auto Matcher = varDecl(
      isExpansionInMainFile(),
      hasType(isConstQualified()),                              // const
      hasInitializer(
        hasType(cxxRecordDecl(
          isLambda(),                                           // lambda
          has(functionTemplateDecl(                             // auto
            has(cxxMethodDecl(
              isNoThrow(),                                      // noexcept
              hasBody(compoundStmt(hasDescendant(gotoStmt())))  // goto
      )))))))).bind("clang");
    // clang-format on
    */
    MatchHandler Handler;
    MatchFinder MatchFinder;
    MatchFinder.addMatcher(Matcher, &Handler);
    MatchFinder.matchAST(Context);
  }
};

/// Creates an `ASTConsumer` and logs begin and end of file processing.
class Action : public clang::ASTFrontendAction {
 public:
  using ASTConsumerPointer = std::unique_ptr<clang::ASTConsumer>;

  ASTConsumerPointer CreateASTConsumer(clang::CompilerInstance& Compiler,
                                       llvm::StringRef) override {
    return std::make_unique<Consumer>();
  }

  bool BeginSourceFileAction(clang::CompilerInstance& Compiler,
                             llvm::StringRef Filename) override {
    llvm::errs() << "Processing " << Filename << "\n\n";
    return true;
  }

  void EndSourceFileAction() override {
    llvm::errs() << "\nFinished processing file ...\n";
  }
};
}  // namespace Mutator

namespace {
llvm::cl::OptionCategory ToolCategory("clang-variables options");
llvm::cl::extrahelp MoreHelp(R"(
  Finds all Const Lambdas, that take an Auto parameter, are declared Noexcept
  and have a Goto statement inside, e.g.:

  const auto lambda = [] (auto) noexcept {
    bool done = true;
    flip: done = !done;
    if (!done) goto flip;
  }
)");

llvm::cl::extrahelp
    CommonHelp(clang::tooling::CommonOptionsParser::HelpMessage);
}  // namespace

auto main(int argc, const char* argv[]) -> int {
  using namespace clang::tooling;

  CommonOptionsParser OptionsParser(argc, argv, ToolCategory);
  ClangTool Tool(OptionsParser.getCompilations(),
                 OptionsParser.getSourcePathList());

  const auto Action = newFrontendActionFactory<Mutator::Action>();
  return Tool.run(Action.get());
}

I appreciate any help.

Answer 1

Clang tools instantiate a compiler object in order to produce the AST. Unlike the compiler that gets installed from a distribution (that gets called when building your project), this compiler object is not configured with header path information.

There are (at least) two ways to add that information: configure the compiler with standard header paths, or add the paths to the compilation database.

First, you can add the paths programmatically with the ClangTool::appendArgumentsAdjuster() method. Here's an example from apps/FuncLister.cc in CoARCT (https://github.com/lanl/CoARCT):

ClangTool tool(OptionsParser.getCompilations(),
               OptionsParser.getSourcePathList());
// add header search paths to compiler
ArgumentsAdjuster ardj1 =
    getInsertArgumentAdjuster(corct::clang_inc_dir1.c_str());
ArgumentsAdjuster ardj2 =
    getInsertArgumentAdjuster(corct::clang_inc_dir2.c_str());
tool.appendArgumentsAdjuster(ardj1);
tool.appendArgumentsAdjuster(ardj2);
if(verbose_compiler){
  ArgumentsAdjuster ardj3 = getInsertArgumentAdjuster("-v");
  tool.appendArgumentsAdjuster(ardj3);
}

CoARCT defines the include directories in several steps: first, the top level CMakeLists.txt guesses where the directories are and puts that information into macros; second, lib/utilities.h puts the macros into clang_inc_dir1/2 strings in the form of compiler flags (i.e. clang_inc_dir1="-I/path1/to/headers"); then clients like FuncLister.cc use those as arguments to appendArgumentsAdjuster().

A second way to add compiler search paths is to change the compiler command in the compilation database. If you are using CMake, add set(CMAKE_EXPORT_COMPILE_COMMANDS TRUE) to the top level CMakeLists.txt. This should produce a compilation database file called compile_commands.json in the build directory. Each source file will have an entry; the entry will include

"command":"compiler command line here"

You can add -I/path/to/headers to the compile command for any file you want to run your tool on. You would then invoke the tool with something like

clang-variables test.cpp -p /path/to/compile_commands.json

If your project is not using CMake, the structure of the database file is described here: https://clang.llvm.org/docs/JSONCompilationDatabase.html.