Skip to content

for iOS

This guide will demonstrate you how to use Emcee for iOS testing. We will use two MacOS machines to run unit and UI tests from a sample project. You can also use a single machine to try out Emcee to see if it works for your project. In this case, a single machine will act as a queue and a worker simultaneously.

If you encounter any issues while proceeding through the guide, please reach out via https://t.me/emcee_ios.

Table of contents#

  1. Prerequisites
  2. Build the sample project
  3. Create emceeplan
  4. Run tests
  5. Further steps

Prerequisites #

Install Emcee#

Install Emcee using Homebrew.

Toolchain#

We will use Xcode 15.2 and the iOS 17.2 simulator runtime (installed with xcodebuild -downloadPlatform iOS). If you want to use a specific version of simulator runtime, proceed to Xcode -> Preferences... -> Components -> Simulators and install the runtime on all the worker machines, where you want the tests to execute with the needed runtime version.

SSH access#

In this tutorial we use technique when Emcee uses ssh to deploy itself to the machines specified as queue and workers. You will need to grant SSH access to your machines. Enable SSH in your System Preferences -> Sharing -> Remote Login.

We will use SSH key-based authentication. So, you have to prepare SSH keys beforehand. See manuals about SSH from Atlassian or Github.

Also you need to add your SSH key to all hosts in the Emcee system. For local Emcee setup it will be the local machine.

You can do this by executing command:

ssh-copy-id <YOUR_USER>@<SERVER_IP>

If you don't have ssh-copy-id utility then you can install it via brew install ssh-copy-id or copy your key manually.

SSH key requirement

For now, to operate correctly, your SSH key must be without passphrase.

Build the sample project #

To run Emcee you have to build tests for iOS simulator first. If you've got compiled test artifacts you can skip this step.

In this step, we will build a sample project that features different types of tests. Xcode and xcodebuild will produce build artifacts in derived data.

Clone the sample project:

git clone https://github.com/avito-tech/Emcee.git
cd Emcee/Samples/EmceeSample

and buid:

xcodebuild build-for-testing\
  -project EmceeSample.xcodeproj\
  -destination "generic/platform=iOS Simulator"\
  -scheme EmceeSample\
  -derivedDataPath derivedData

Create emceeplan #

To run tests you have to define emceeplan file. Take a look on full_regress.yml configurations we prepared for this sample project:

tests:
  configurations:
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleTestsWithoutHost.xctest
      device:
        simDeviceType: iPhone-15
        simRuntime: iOS-17-2 # select installed runtime
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app/PlugIns/EmceeSampleHostedTests.xctest
      appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
      device:
        simDeviceType: iPhone-15
        simRuntime: iOS-17-2
    - platform: ios
      xcTestBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app/PlugIns/EmceeSampleUITests.xctest
      appBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app
      runnerBundle: derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app
      device:
        simDeviceType: iPhone-15
        simRuntime: iOS-17-2
  outputs:
  	- xcresult
queue:
  type: deploy
  host: localhost
  ssh:
    username: <YOUR_USER> # for simplicity you can select your current user
    identity:
      type: publicKey
      keyPath: /Users/<YOUR_USER>/.ssh/id_ed25519 # select your private ssh key
  workers:
    - id: w1
      host: localhost

Before proceeding you have to set ssh username and keyPath to match those you created on prerequisites step.

Configurations#

In this minimalistic emceeplan we've created 3 configurations:

  • for unit testing, to run lightweight tests that do not depend on host application
  • for integration testing, to run tests that require UI instantiation
  • for end to end testing with full device access.

Learn more about Xcode Test Plans and build artifacts requirements for different iOS tests here.

All tests will be run on iPhone-15 with iOS-17-2:

      device:
        simDeviceType: iPhone-15
        simRuntime: iOS-17-2

As a result we want to get only Xcode compatible xcresult:

  outputs:
  	- xcresult

Queue#

Emcee requires queue and at least one worker in the system to operate. In emceeplan above we specified that Emcee should deploy itself using ssh with public key identity:

queue:
  type: deploy
  ssh:
    username: <YOUR_USER>
    identity:
      type: publicKey
      keyPath: /Users/<YOUR_USER>/.ssh/id_ed25519

Where queue and worker will be deployed to localhost:

queue:
  #...
  host: localhost
  workers:
    - id: w1
      host: localhost

Run tests #

To run all tests described in emceeplan use CLI in following way:

emcee run full_regress.yml --outdir=results

After all tests finish executing you'll get short info in shell:

[INFO] 9 tests completed successfully
[INFO] 2 tests were skipped
[INFO] 3 tests completed with errors
[INFO] Test <EmceeSampleTestsWithoutHost/test___from_tests_without_host___that_always_fails> failed after 1 runs
[INFO]    executed on localhost at 2024-05-11 14:33:07.632 using AC785F64-5B52-4953-B5D7-31512A748E51
[INFO]    caught 1 exceptions
[INFO]        EmceeSampleTestsWithoutHost/test___from_tests_without_host___that_always_fails /Users/vvishutin/ios-test-runner/Samples/EmceeSample/EmceeSampleTestsWithoutHost/EmceeSampleTestsWithoutHost.swift:17: failed - Failure from tests without host
[INFO] Test <EmceeSampleTestsWithHost/test___from_tests_with_host___that_always_fails> failed after 1 runs
[INFO]    executed on localhost at 2024-05-11 14:33:32.994 using AC785F64-5B52-4953-B5D7-31512A748E51
[INFO]    caught 1 exceptions
[INFO]        EmceeSampleTestsWithHost/test___from_tests_with_host___that_always_fails /Users/vvishutin/ios-test-runner/Samples/EmceeSample/EmceeSample/ViewController.swift:13: EmceeSample.ViewController test exception (NSGenericException)
[INFO] Test <EmceeSampleUITests/test___from_xcui_tests___that_always_fails> failed after 1 runs
[INFO]    executed on localhost at 2024-05-11 14:33:29.177 using EFAE58CF-24D2-4CA6-BF71-572533997BA6
[INFO]    caught 1 exceptions
[INFO]        EmceeSampleUITests/test___from_xcui_tests___that_always_fails /Users/vvishutin/ios-test-runner/Samples/EmceeSample/EmceeSampleUITests/EmceeSampleUITests.swift:37: failed - Failure from xcui tests

and merged xcresult in result directory.

This is how the test run might look:

running_tests

Emcee queue and workers keeps running after tests run's finished. If you want to terminate Emcee run:

pkill "Emcee*"

Where to go from here #

In this simplified guide we utilized only a single worker that unlikely will boost your pipeline significantly. To get better performance you should scale this guide up to as many machines as you have. It's recommented that you control the deployment process. Learn more on possible deployment options in production environment.

Find all possible test parameters you may use to configure in emceeplan to reflect your tests correctly.

You also may want to know all possible reports Emcee may yield during tests run.

Look here how you can configure simulator before running tests.