4

Has anyone been able to successfully pass the KW_SPEC variable to xctool. I'm trying to run a single KIWI spec by using https://github.com/kiwi-bdd/Kiwi/wiki/Kiwi-FAQ#q-how-do-i-run-a-single-spec-describecontextit.

I can run all the tests successfully with xctool but it doesn't seem to pick up the KW_SPEC value. I've tried it in a lot of different places with the command line but no luck.

e.g.:

xctool -destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
    -sdk iphonesimulator -workspace SampleProject.xcworkspace 
    -scheme SampleProject KW_SPEC=NewAssessmentTests.m:12 test 
    -only SampleProject_Acceptance_Tests

Using Kiwi v2.3.1 and xctool 0.2.3

Cheers, Mo

TimoStaudinger
  • 41,396
  • 16
  • 88
  • 94
Mo Jaimangal
  • 63
  • 6
  • Have you ever succeeded in this? I'm trying to do the same with no luck. From reading the Kiwi code I gather: (1) `KW_SPEC` is retrieved using `[[[NSProcessInfo processInfo]environment]` which means build-time settings like you used above won't work - I believe you need to set it as an environment variable in the scheme (see http://nshipster.com/launch-arguments-and-environment-variables/) (2) The filename and line are compared against an `atos` result which appears consistent with the format you used, but when I tried it myself nothing ran (0 tests executed) – Ohad Schneider Oct 27 '15 at 19:15
  • Related source files: `https://github.com/kiwi-bdd/Kiwi/blob/ae9f6d83faf774754ea60a9dfc6556eb23800f47/Classes/Core/KWExampleSuiteBuilder.m`, `https://github.com/kiwi-bdd/Kiwi/blob/ae9f6d83faf774754ea60a9dfc6556eb23800f47/Classes/Core/KWSymbolicator.m`. Related GitHub issue: https://github.com/kiwi-bdd/Kiwi/issues/614. – Ohad Schneider Oct 27 '15 at 19:18

1 Answers1

2

Update 11/03/2015

@OhadSchneider's comment made me realise that KW_SPEC worked for me because I had set it in the test configuration for my scheme (Edit scheme->Test->Arguments->Environment variables). Setting the variable from shell doesn't work as that variable applies only to the actual build, and not when executing the unit test target.
But there's a workaround to this, by modifying the test phase of your scheme and adding a KW_SPEC environment variable with the value $KW_SPEC, this will expand when running xcodebuild to the value passed to the xcodebuild command (as in my original answer). After this, xcode will will gracefully use the passed KW_SPEC variable, xctool still has the skipped teste marked as failure issue. KW_SPEC scheme setting

Original answer

If you want KW_SPEC as an environment variable to xctool (or to any *nix tool), then you have to place it before the command, otherwise it will be considered a build setting:

KW_SPEC=NewAssessmentTests.m:12 xctool 
    -destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
    -sdk iphonesimulator -workspace SampleProject.xcworkspace 
    -scheme SampleProject test 
    -only SampleProject_Acceptance_Tests

This will however lead to another problem: xctool will report as errored the tests that don't run, and will report the test as failed, even if no tests have failed. xcodebuild doesn't have this problem as it either doesn't do unit tests discovery, or ignore tests that didn't run, a thing that xctool fails to do.

Community
  • 1
  • 1
Cristik
  • 30,989
  • 25
  • 91
  • 127
  • It doesn't work even with `xcodebuild` (actually even when I define it inside XCode, the spec simply doesn't run), but the bounty is about to expire so I might as well award it for effort :) – Ohad Schneider Nov 02 '15 at 22:47
  • @OhadSchneider thanks for checking it, I rerun on my machine and it still works (cliche doesn't it :) The problem was the fact that I altered my scheme from Xcode to pass that environment variable, making the command line one useless. I will update my answer with this, as I found the workaround for xcodebuild. – Cristik Nov 03 '15 at 06:28
  • Thanks for following up, but I was actually aware of that. My problem is that the spec simply doesn't run (I know it reads the flag becuase when it's defined nothing runs, when not everything runs). I'm using the line number where `SPEC_BEGIN` is defined. Anyway relying on line numbers is probably too fragile anyway, so I'll just use a different target for each spec. Too bad using the name of the spec doesn't work. – Ohad Schneider Nov 03 '15 at 09:34
  • Hmm.. I used the line where the `it` I wanted to run began, and I got running only that test only, I don't think using the line where with `SPEC_BEGIN` would help... – Cristik Nov 03 '15 at 10:07
  • In the FAQ, they claim `spec`/`describe`/`context`/`it` are all supported: https://github.com/kiwi-bdd/Kiwi/wiki/Kiwi-FAQ. I think `it` failed for me too, but if you get the chance it would be interesting to know whether the `SPEC_BEGIN` line works for you (may be something in our build that makes `atos` fail). – Ohad Schneider Nov 03 '15 at 12:32
  • Yeah, it looks like it works for describe/context/it, but it doesn't work for SPEC_BEGIN. Wrapping all tests of a spec in a describe should work. – Cristik Nov 03 '15 at 14:05
  • Thanks for confirming. Like I said I've deemed this as too fragile for my taste, but it's good to know. – Ohad Schneider Nov 03 '15 at 14:14
  • Hi Ohad and Cristik, thanks for following up with these answers. I still didn't have any luck with the command line and xctool – Mo Jaimangal Dec 21 '15 at 14:15
  • @MoJaimangal are you still not able to run the tests that you want, or did you run into the other problem of `xctool` - reporting as failed test that didn't run? – Cristik Dec 21 '15 at 15:55
  • Works for Xcode if one sets the environment variable to `filename.m:line` as described in the Kiwi FAQ. Too bad one has to manually edit the scheme to achieve that. – user3099609 Jan 26 '16 at 10:30