Test types and their configurations#
In order to run iOS tests you have to define configurations for every test bundle in emceeplan. Each configuration must be populated at least with artifact bundles and devices.
There are different build artifacts depending on type of test you want to run. In any case, xcodebuild build-for-testing
will produce these artifacts for you. You may use local file paths in emceeplan, thus Emcee will be responsible for serving build artifacts for workers. Or you may upload them on some server and use direct URLs in emceeplan so Emcee workers would be able to download and use them to run tests.
Different test types require different set of build artifacts to be provided to Emcee in order to successfully execute tests.
UI Tests#
Also known as XC UI native tests. In this test scenario, main app is launched and then controlled by another app (XCTRunner.app
). The latter also runs your tests. This allows to achieve black box-like test mode.
The required artifacts are:
-
appBundle
, app under test,.app
bundle of the app being tested -
runnerBundle
, Xcode will addXCTRunner.app
into DerivedData folder, typically named after your UI tests target, e.g.UITests-Runner.app
-
xcTestBundle
,.xctest
bundle - usually it is located insideUITests-Runner.app/PlugIns/
.
tests:
configurations:
- platform: ios
xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app/PlugIns/EmceeSampleUITests.xctest
appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
runnerBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app
device:
#...
Application Tests#
In this test scenario, the app under test starts and runs your test bundle, thus, the app itself acts like a test hosting app. Tests have access to all objects and memory of the app under test. This allows to write gray or white box tests.
The required artifacts are:
-
appBundle
,.app
bundle to host tests -
xcTestBundle
,.xctest
bundle - usually it is located insideYourApp.app/PlugIns/
.
tests:
configurations:
- platform: ios
xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app/PlugIns/EmceeSampleHostedTests.xctest
appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
device:
#...
Unit Tests#
This test environment does not require a hosting app. Tests are loaded and executed by xctest
process. Usually things like CocoaPods dev pods can be tested in logic test mode.
The required artifacts are:
xcTestBundle
,xctest
bundle - usually it is located inside your Derived Data folder.
tests:
configurations:
- platform: ios
xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleTestsWithoutHost.xctest
device:
#...
Xcode Test Plan#
A Xcode test plan is a document in your Xcode project that describes which tests should run whenever a test invocation gets placed and the configurations Xcode uses to run the tests.
If you use organized Test Plans to keep test configurations and target settings, you can reference xctestrun
file from emceeplan. xctestrun
is in turn result of "compiling" Test Plan into low level format that is readable by both xcodebuild and Emcee. xctestrun
format has its own versioning (look for __xctestrun_metadata__
). Version 2 is specific for building Test Plans. Version 1 is for schemes that don't use Test Plans.
Info
Emcee supports only xctestrun
of version 2 (i.e. derived from Test Plan). If you need to describe other tests, stick to manually specifying configurations.
Normally xctestrun
resides in a Products
directory in derived data and has a name comprised of project name, scheme name, target platform/version name and architecture. There will be one xctestrun
per Test Plan in a scheme that is built. All relative file references in xctestrun
files will be provided by build system and are guaranteed to be available on file system.
Info
You cannot explicitly set xctestrun filename when building project. Since it depends on toolchain, you have to adjust xctestrun
reference when you update sdk.
tests:
configurations:
- platform: ios
xcTestRun: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample_AllTests_iphonesimulator17.4-arm64-x86_64.xctestrun
device:
#...
With Test Plan enabled in a scheme you build project as usual:
xcodebuild build-for-testing\
-project EmceeSample.xcodeproj\
-destination "generic/platform=iOS Simulator"\
-scheme EmceeSample\ #Scheme should have test plans enabled
-derivedDataPath derivedData
Test Plan and emceeplan configuration parameters conflict resolution#
With xctestrun
reference you can still use emceeplan configuration parameters (like testTimeout
) which may conflict. Common rule is that emceeplan parameters have precedence, if set, over parameters from xctestrun
. Except for environment
and filter
that are merged. You can find all possible parameters here.
tests:
configurations:
- platform: ios
xcTestRun: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample_AllTests_iphonesimulator17.4-arm64-x86_64.xctestrun
testTimeout: 30 # effective test timeout will be 30 regardless value in xctestrun
filters: [SomeAdditionalTestClass] # `SomeAdditionalTestClass` also will be tested as well as tests specified in xctestrun
device:
#...