Skip to content

Test types and their configurations#

In order to run iOS tests you have to define configurations for every test bundle in emceeplan. Each configuration must be populated at least with artifact bundles and devices.

There are different build artifacts depending on type of test you want to run. In any case, xcodebuild build-for-testing will produce these artifacts for you. You may use local file paths in emceeplan, thus Emcee will be responsible for serving build artifacts for workers. Or you may upload them on some server and use direct URLs in emceeplan so Emcee workers would be able to download and use them to run tests.

Different test types require different set of build artifacts to be provided to Emcee in order to successfully execute tests.

UI Tests#

Also known as XC UI native tests. In this test scenario, main app is launched and then controlled by another app (XCTRunner.app). The latter also runs your tests. This allows to achieve black box-like test mode.

The required artifacts are:

  • appBundle, app under test, .app bundle of the app being tested

  • runnerBundle, Xcode will add XCTRunner.app into DerivedData folder, typically named after your UI tests target, e.g. UITests-Runner.app

  • xcTestBundle, .xctest bundle - usually it is located inside UITests-Runner.app/PlugIns/.

tests:
  configurations:
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app/PlugIns/EmceeSampleUITests.xctest
      appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
      runnerBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app
      device:
        #...

Application Tests#

In this test scenario, the app under test starts and runs your test bundle, thus, the app itself acts like a test hosting app. Tests have access to all objects and memory of the app under test. This allows to write gray or white box tests.

The required artifacts are:

  • appBundle, .app bundle to host tests

  • xcTestBundle, .xctest bundle - usually it is located inside YourApp.app/PlugIns/.

tests:
  configurations:
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app/PlugIns/EmceeSampleHostedTests.xctest
      appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
      device:
        #...

Unit Tests#

This test environment does not require a hosting app. Tests are loaded and executed by xctest process. Usually things like CocoaPods dev pods can be tested in logic test mode.

The required artifacts are:

  • xcTestBundle, xctest bundle - usually it is located inside your Derived Data folder.
tests:
  configurations:
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleTestsWithoutHost.xctest
      device:
        #...

Xcode Test Plan#

A Xcode test plan is a document in your Xcode project that describes which tests should run whenever a test invocation gets placed and the configurations Xcode uses to run the tests.

If you use organized Test Plans to keep test configurations and target settings, you can reference xctestrun file from emceeplan. xctestrun is in turn result of "compiling" Test Plan into low level format that is readable by both xcodebuild and Emcee. xctestrun format has its own versioning (look for __xctestrun_metadata__). Version 2 is specific for building Test Plans. Version 1 is for schemes that don't use Test Plans.

Info

Emcee supports only xctestrun of version 2 (i.e. derived from Test Plan). If you need to describe other tests, stick to manually specifying configurations.

Normally xctestrun resides in a Products directory in derived data and has a name comprised of project name, scheme name, target platform/version name and architecture. There will be one xctestrun per Test Plan in a scheme that is built. All relative file references in xctestrun files will be provided by build system and are guaranteed to be available on file system.

Info

You cannot explicitly set xctestrun filename when building project. Since it depends on toolchain, you have to adjust xctestrun reference when you update sdk.

tests:
  configurations:
    - platform: ios
      xcTestRun: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample_AllTests_iphonesimulator17.4-arm64-x86_64.xctestrun
      device:
        #...

With Test Plan enabled in a scheme you build project as usual:

xcodebuild build-for-testing\
  -project EmceeSample.xcodeproj\
  -destination "generic/platform=iOS Simulator"\
  -scheme EmceeSample\  #Scheme should have test plans enabled
  -derivedDataPath derivedData

Test Plan and emceeplan configuration parameters conflict resolution#

With xctestrun reference you can still use emceeplan configuration parameters (like testTimeout) which may conflict. Common rule is that emceeplan parameters have precedence, if set, over parameters from xctestrun. Except for environment and filter that are merged. You can find all possible parameters here.

tests:
  configurations:
    - platform: ios
      xcTestRun: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample_AllTests_iphonesimulator17.4-arm64-x86_64.xctestrun

      testTimeout: 30  # effective test timeout will be 30 regardless value in xctestrun
      filters: [SomeAdditionalTestClass]  # `SomeAdditionalTestClass` also will be tested as well as tests specified in xctestrun

      device:
        #...