Xcode build failing randomly

Question

We have an iOS automated build system that is randomly failing the codesign for certain projects.

The build system uses xcodebuild (followed by xcrun) to build and package the application. We use this process across multiple apps. The build machines we run this on have multiple distribution signing certificates, and are running Xcode 5.1.1 (5B1008) and 10.9.4. We've been using the system for about two years now, and builds work fine. When they don't work, there's usually a clear xcodebuild error message. But right now we are getting random failures of the codesign call on xcodebuild for two of our apps.

The xcodebuild command is as follows:

xcodebuild -verbose -project ./src_dir/TechApp.xcodeproj ONLY_ACTIVE_ARCH=NO ARCHS="armv7 armv7s arm64" -scheme ${SCHEME} QA PROVISIONING_PROFILE=${PROFILE} CODE_SIGN_IDENTITY=${IDENTITY} CONFIGURATION_BUILD_DIR=./bld_dir/builds/QA build

And, after lots and lots of xcodebuild compile statements, it fails with the following:

CodeSign /Users/bldUser/buildScripts/archive_dir/1404933111/bld/bld_dir/builds/QA/Tech\ App.app
    cd /Users/bldUser/buildScripts/archive_dir/1404933111/bld/src_dir
    export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate
    export PATH="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Applications/Xcode.app/Contents/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin"
    Using code signing identity "iPhone Distribution: Build Certificate" and provisioning profile "BuildWildcard" (C137C14D-630F-4D67-BD2B-2FD4AB4F2BA4)
    /usr/bin/codesign --force --sign 637514541007B62BCEA94ACC41A98A0E1391C3D4 --resource-rules=/Users/bldUser/buildScripts/archive_dir/1404933111/bld/bld_dir/builds/QA/Tech\ App.app/ResourceRules.plist --entitlements /Users/bldUser/buildScripts/archive_dir/1404933111/bld/build/TechApp.build/Debug-iphoneos/TechApp\ QA.build/Tech\ App.xcent /Users/bldUser/buildScripts/archive_dir/1404933111/bld/bld_dir/builds/QA/Tech\ App.app


** BUILD FAILED **

The projects in which this are failing are big apps... and they have multiple dependent projects attached to them (and I think that it might be the dependent projects that are contributing to the random failure of the codesign, but I'm not sure).

The thing is that this randomly fails at this statement. If I run it again, it will work fine. So it's the randomness that confuses me. And there really is not any error message. It just says "BUILD FAILED". Some things I've tried to get this to work:

1. I've passed in additional parameters into xcodebuild OTHER_CODE_SIGN_FLAGS=--verbose=4 --continue --deep in order to see if any combination of these parameters "help." This was based on the problem described in this link, as it seemed to match the failure conditions for our particular projects that are failing. This post had me the most interested, as it seems like this issue may be related to the OS + build environment + the fact that the projects this is failing on have lots of sub-projects attached.
2. We currently have all our certificates in the login.keychain... but, as per this link (amongst others) there's a suggestion to run certificates off of a build-specific keychain. This is great, and a much cleaner way to do things-- but, after trying this, this had no impact for the issue above.
3. I thought maybe this might have to do with confusion on certificates-- so I've removed all iPhone Distribution certificates except the the one I need for these apps, and it still comes back with this error.
4. I've gone in to each of the targets for the projects, including the sub-projects, and put in code signing identity just to make sure there wasn't a possibility that there was some project or sub-project setting that would pick the wrong certificate. In addition, on the xcodebuild command above, I've put in the explicit scheme name and provisioning profile id-- so this is not an issue where it can't find the provisioning profile/certificate combination (or at least I don't think it is-- there's no indicator that it can't find the provisioning profile or certificate)

Given that it's random, I suspect it's either (1) a case where there is some unset code signing value, and it therefore randomly picks one; or (2) some issue with xcode or the os that was only recently introduced.

Any thoughts or ideas on what to try would be much appreciated.

One update: This works just fine when run through Xcode. So this is something specific to xcodebuild for these projects. When compiled through Xcode: it works 100% of the time. When compiled through xcodebuild: it works like 25-50% of the time. So I'm really hoping there's something I"m missing on the xcodebuild command and this does not require an actual project change (since it works when compiled directly through Xcode).

Update on trojanfoe's question: There may very well be an issue with spaces. This is the actual xcodebuild call on one of the projects:

xcodebuild -verbose -project /Users/bldUser/buildScripts/archive_dir/1404904225/bld/src_dir/MyTechApp.xcodeproj ONLY_ACTIVE_ARCH="NO" ARCHS="armv7 armv7s arm64" -scheme "TechApp Dev" PROVISIONING_PROFILE="" CODE_SIGN_IDENTITY="iPhone Distribution: Build Certificate" CONFIGURATION_BUILD_DIR="/Users/bldUser/buildScripts/archive_dir/1404897462/bld/bld_dir/builds/Dev" build

After running the command above, there is (lots) of build commands (there are lots and lots of warnings, but no errors). It finally stops at the following:

CodeSign /Users/bldUser/buildScripts/archive_dir/1404897462/bld/bld_dir/builds/Dev/Tech\ App.app
    cd /Users/bldUser/buildScripts/archive_dir/1404904225/bld/src_dir
    export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate
    export PATH="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Applications/Xcode.app/Contents/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin"
    Using code signing identity "iPhone Distribution:  Build Certificate" and provisioning profile "BuildWildcard" (C137C14D-630F-4D67-BD2B-2FD4AB4F2BA4)
    /usr/bin/codesign --force --sign 637514541007B62BCEA94ACC41A98A0E1391C3D4 --resource-rules=/Users/bldUser/buildScripts/archive_dir/1404897462/bld/bld_dir/builds/Dev/Tech\ App.app/ResourceRules.plist --entitlements /Users/bldUser/buildScripts/archive_dir/1404904225/bld/build/MyTechApp.build/Debug-iphoneos/TechApp\ Dev.build/Tech\ App.xcent /Users/bldUser/buildScripts/archive_dir/1404897462/bld/bld_dir/builds/Dev/Tech\ App.app

** BUILD FAILED **

All of the "cd" commands and the "export" commands are run as part of the codesign command. In terms of the xcodebuild statement, we're leaving the provisioning profile blank so it picks up a default-- although even if I specify the specific provisioning profile, it will still fail (and from the code sign above, it IS picking up the correct provisioning profile). The above also specifies the scheme. This is normally run from a bash script where it will loop through an array of schemes and generate builds via xcodebuild for each scheme. As you can see above, there are spaces in the scheme and the project and the product name, so I would not be at all surprised if that were an issue in some way. I'm not sure why it would work one time and not another, but I'm going to try out removing spaces. The src_dir and the funky archive_dir numeric is a directory created by our build system, so the files are all checked out to the src_dir location.