PRE_TARGETDEPS fails when using parallel builds

Question

I'm using a custom ressource management (in replacement to qrc) and I'm trying to integrate it to QtCreaor.

I have a Python script that generates a source file to be compiled. I use QMAKE_EXTRA_TARGETS/PRE_TARGETDEPS to tell QMake that this script must be executed before files are compiled. So I do, in my pro file:

CONFIG += ordered

generated_file.target   = my_custom_target
generated_file.commands = echo "Generating..." && d:/dev/vobs_ext_2015/tools_ext/python/Python34_light/python.exe $$PWD/pyc_res_generator.py -o $$PWD/generated/generated.cpp && echo "Generated!"
generated_file.depends  = FORCE

QMAKE_EXTRA_TARGETS            += generated_file

PRE_TARGETDEPS                 += my_custom_target

SOURCES                        += generated/generated.cpp

#DEPENDPATH = ./generated

With pyc_res_generator.py simply being:

#! /usr/bin/env python
# -*- coding: utf8 *-*

import argparse

parser = argparse.ArgumentParser(description="", formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument("-o", type=str, help="Output file name")

args = parser.parse_args()

with open(args.o, 'w') as output_file:
    output_file.write( "// This is valid C++!" )

To test this, I write some invalid C++ in generated.cpp (like fjkfkfk). When I compile (target is Android), I see in the log:

echo Generating... && python.exe pyc_res_generator.py -o generated/generated.cpp && echo Generated!
Generating... 
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o main.obj ..\TestQt\main.cpp
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o dialog.obj ..\TestQt\dialog.cpp
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o generated.obj ..\TestQt\generated\generated.cpp
B:\QtCreator5_6_1\5.6\android_armv7\bin\moc.exe -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -IB:/QtCreator5_6_1/5.6/android_armv7/mkspecs/android-g++ -IC:/Users/jp225611/Documents/TestQt -IB:/QtCreator5_6_1/5.6/android_armv7/include -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtWidgets -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtGui -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtCore -I. -IB:\Android\android-ndk-r11b/sources/cxx-stl/gnu-libstdc++/4.9/include -IB:\Android\android-ndk-r11b/sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/include -IB:\Android\android-ndk-r11b/platforms/android-9/arch-arm//usr/include ..\TestQt\dialog.h -o moc_dialog.cpp
..\TestQt\generated\generated.cpp:1:1: error: 'fjkfkfk' does not name a type
   // This is valid C++!
   ^
makefile:670: recipe for target 'generated.obj' failed
mingw32-make: *** [generated.obj] Error 1
mingw32-make: *** Waiting for unfinished jobs....
Generated!
13:59:08: Le processus "B:\QtCreator5_6_1\Tools\mingw492_32\bin\mingw32-make.exe" s'est terminé avec le code 2.
Erreur lors de la compilation/déploiement du projet TestQt (kit : android_armeabi-v7a)
When executing step "Make"
13:59:08: Temps écoulé : 00:03.

And I see that generated\generated.cpp was correctly generated by pyc_res_generator.py (it now contains // This is valid C++!)...but apparently too lately as the compiler saw fjkfkfk... As you can see, the output reports:

Generating...
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ .... ..\TestQt\generated\generated.cpp
Generated...

While one would expect:

Generating...
Generated...
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ .... ..\TestQt\generated\generated.cpp

This only occurs when parallel builds are enabled (I set MAKEFLAGS environment variable to '-j8`)

It looks like my custom command generated_file is started before generated\generated.cpp is compiled, but the system does not wait for it to end before actually compiling generated\generated.cpp.

Am I doing something wrong?

Answer 1

PRE_TARGETDEPS (custom targets) is not the good approach here as generated_file.target must match exactly the Makefile target name which may be vary on different options:

• shadow build enabled or not
• host platform (using slashs or backslash as a folder separator)
• probably targetted compiler/platform

However, custom compiler apparently works much better. That's how moccing files is done. A custom compiler can generate a new file to be compiled (and then the system waits for the file to be generated before compiling it, even when performing parallel builds).

# Set fuiles to be generated in a variable
TO_GENERATE = $$PWD/generated/generated.cpp

# Specify custom command output file:
custom_generator.output  = $$PWD/generated/generated.cpp
# Specify custom command:
custom_generator.commands = 'echo "Generating..." && python $$PWD/pyc_res_generator.py -o $$PWD/generated/generated.cpp && echo "Generated..."'
# dependency:
custom_generator.depends = FORCE
# link to input file variable
custom_generator.input = TO_GENERATE
# link to variable to store generated file to
custom_generator.variable_out = SOURCES
# add to qmake:
QMAKE_EXTRA_COMPILERS += custom_generator

Then, you don't even need to specify SOURCES += $$PWD/generated/generated.cpp. And also, $$PWD/generated/generated.cpp gets deleted upon clean!

This works perfectly as expected!

Answer 2

You need to set the name of the target to the name of the file that it generates. Here's a full project file that works for me (qmake 3.1, Qt 5.10.1):

# Filename relative to the *build* directory.
generated_file.target   = generated/generated.cpp
generated_file.commands = 'echo "Generating..." && mkdir -p generated && echo "int main() {}" > generated/generated.cpp'
generated_file.depends  = FORCE

QMAKE_EXTRA_TARGETS     += generated_file

# Filename relative to the *source* directory.
SOURCES                 += $$OUT_PWD/generated/generated.cpp

The reason is that generated_file.target becomes the name of the target in the Makefile, so when generated.cpp is needed, your custom rule will be executed to create the file.

You can also remove it from PRE_TARGETDEPS, because you're already adding the generated file to SOURCES instead, which creates the proper dependency.

Note that this approach generates the file in the build directory, not the source directory. This is why you need to specify $$OUT_PWD when adding it to SOURCES: by default, qmake seems to assume that SOURCES are relative to the source directory.

If you want to generate it in the source directory instead, you might think that you just prefix $$PWD everywhere, but then qmake sometimes uses absolute paths and sometimes relative, and everything fails. You can probably work around that with $$absolute_path($$PWD) but I haven't tried.

Relevant sections from the generated Makefile (comments are mine):

# How to create generated/generated.cpp.
# Depending on FORCE to make it always stale (fails if you create a file named FORCE :)).
generated/generated.cpp: FORCE
    echo "Generating..." && mkdir -p generated && echo "int main() {}" > generated/generated.cpp

# generated.o depends on generated/generated.cpp.
# Note that it's not in the same directory; we don't care.
generated.o: generated/generated.cpp 
    $(CXX) -c $(CXXFLAGS) $(INCPATH) -o generated.o generated/generated.cpp

# The final executable depends on the compiled generated.o.
OBJECTS = generated.o
$(TARGET):  $(OBJECTS)  
    $(LINK) $(LFLAGS) -o $(TARGET) $(OBJECTS) $(OBJCOMP) $(LIBS)

The qmake documentation is pretty vague about this, only stating that target should be "The name of the custom build target.". However, the example shows that they're setting it to the name of the output file. (As a bonus, it also shows how to not repeat that name in the commands.)

Answer 3

An alternative relies on the TEMPLATE=subdirs, to honour libname.depends in a parallel build environment. You would need two .pro files in each application folder. And one in the shared library folder. Any other folders in your development area that use TEMPLATE=subdirs need to be made compatible.

eg TestApp using LibLarry.

Folder TestApp has bld.pro and TestApp.pro.

TestApp.pro:

#This is a TEMPLATE=subdirs, which gmake will interpret;  
# making sure the build dependencies are respected
TEMPLATE = subdirs
SUBDIRS += bld  mylib
bld.file = bld.pro
bld.depends = mylib # force library to be built first.
mylib.file = $${PATH_TO_LIB_LARRY}/liblarry.pro

bld.pro:

TARGET=TestApp
SOURCES+=TestApp.cpp
PRE_TARGETDEPS+=$${PATH_TO_LIB_LARRY}/liblarry.a

#boilerplate...
QT -= gui
CONFIG = ...
include ($${PATH_TO_LIB_LARRY}/liblarry.pri) # contains include file info
CONFIG += link_prl # allow lib larry to specify its position in the linker input command.

Folder LibLarry has LibLarry.pro containing

LibLarry.pro:

TARGET=larry
TEMPLATE=lib
CONFIG+=staticlib 
CONFIG+=create_prl # export information, so apps can link correctly
SOURCES+=my_shared_function.cpp
include (liblarry.pri)

liblarry.pri:

HEADERS += $$PWD/my_shared_function.h
INCLUDEPATH += $$PWD