IOS app tests with Appium, Github Actions and AWS Device Farm

Background

When we started our journey with Alluo mobile app, we knew that our goal was to automate tests before every release is merged on Github. While looking for a testing platform, we took into consideration the following features:

• It needs to be reliably connected to our existing infrastructure
• is competitive in terms of cost
• support both iOS and Android systems

Based on those, we came to the conclusion that AWS Device Farm is the best option. It was mainly due to our existing AWS-hosted back-end infrastructure. As a testing framework, we chose Appium as the most mature technology, and everything was supposed to be orchestrated by Github Actions with the help of Expo CLI. Everything went rather smoothly for Android, but iOS is a different story.

The first problem with Expo CLI is that builds created by it were incompatible with Device Farm. The only builds that run on Device Farm can be created with Xcode build-for-tests option, therefore we had to opt for building it ourselves on Github actions.

Solution Breakdown

The solution we came up with is a single workflow .yml file placed in the .github/workflows/ folder that consists of 5 different jobs:

1. Building and uploading the iOS app based on the mobile app repository
2. Creation and upload of the Appium package and custom test settings based on the test repository
3. Running the tests on Device Farm
4. Creating unit tests
5. Displaying the results in the console

We decided to go with a single workflow file instead of any extra processors in .js or .py files to ensure faster processing time and transparency on the concurrent steps.

Prerequisites

Before creating the pipeline we need to ensure that we have 5 elements in place:

1. Existing project on AWS Device Farm with device pool for iOS https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-create-project.html https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-create-device-pool.html
2. Mobile App Github Repository
3. Appium test code Github repository
4. Sharing git tokens between repositories to be able to checkout within one workflow to another repository. More on this https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token
5. AWS Credential obtained by following the instructions https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html and saved as DEV_AWS_ACCESS_KEY_ID and DEV_AWS_SECRET_ACCESS_KEY in Github secret storage

Code breakdown

In the first snippet, it is important to run the first job on macOS Action environment, because of the pre-installed xCode that is necessary to run the iOS build. The checkout function serves as a way to add files from the current repository and the QA repository to the running macOS machine. At the end of the script, we are logging in to AWS with credentials stored in GitHub Secrets. The job only starts when the source branch is the release branch and the pull request-target branch is the main.

name: E2E Device Farm Flow - iOS
on:
  pull_request:
    types: [opened, labeled, reopened]
    branches:    
      - 'main' 
env:
  DEV_DEVICES_POOL: "arn:aws:devicefarm:us-west-2:xxx:devicepool:xxx/yyy"
  DEV_PROJECT_ARN: "arn:aws:devicefarm:us-west-2:xxx:project:xxx"
      
jobs:
  job_1:
    if:  startsWith(github.head_ref, 'release/')
    name: Create Test App iOS
    runs-on: macos-latest
    strategy:
     matrix:
       include:
         - xcode: "13.2"
           ios: "15.2"
    steps:
    - uses: actions/checkout@v2
    - name: Checkout actions
      uses: actions/checkout@v2
      with:
         repository: Test/qa-automation
         path: qa-automation
         ref: ET-400-iOS-allFeatures
         token: ${{ secrets.ORG_TOKEN_GIT }}
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
         aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
         aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}
         aws-region: us-west-2

In the second snippet, we are installing pods as well as building the app with the XCodeBuild CLI tool. It is VERY important to include 2 elements in that command that are very vaguely described on the internet so:

1. build-for-testing as a way to generate a prod build that doesn’t contain Expo client and
2. CODE_SIGNING_ALLOWED=NO that prevents errors concerning lack of provisioning profiles.

For us, the solution described on the official Github actions page didn’t work https://docs.github.com/en/actions/deployment/deploying-xcode-applications/installing-an-apple-certificate-on-macos-runners-for-xcode-development and because our version of the app is never publishable in the AppStore we can skip signing as such.

After the .ipa file is generated we need to repack it into .ipa file zipping files inside the original folder. As a second last step, we need to upload the file labelled as IOS_APP into the device farm. Once this is done we need to pass the upload URL into the artefact store so it can be reused by the next jobs that will run the Appium tests themselves.

- name: CocoaPod Install
      run: |
        yarn
        cd ios
        ls
        pod install
    - name: Build the .app
      run: |
        cd ios
        echo "make dir"
        ls
        xcodebuild build-for-testing -workspace Alluo.xcworkspace -scheme Alluo -destination generic/platform=iOS -allowProvisioningUpdates CODE_SIGNING_ALLOWED=NO
        echo "listing after app creation"
        ls -R     
    - name: Generate IPA
      run: |
       ls
       echo "listing before ipa generation"
       cd
       echo "listing root dir"
       ls
       find ./Library/Developer/Xcode/DerivedData -name Alluo.app > filepath.txt
       app_file_path=`cat filepath.txt`
       echo $app_file_path
       mkdir -p iphoneos/Payload
       mv $app_file_path iphoneos/Payload
       ls -R
       echo "before zip"
       cd iphoneos
       zip -r Alluo.ipa Payload
       rm -rf Payload
       echo "list of dir"
       ls
       response_upload_app=$(aws devicefarm create-upload --project-arn $DEV_PROJECT_ARN --name AlluoTestIPA.ipa --type IOS_APP)
       echo $response_upload_app
       url_upload_app=$(echo $response_upload_app | jq '.upload.url' | sed -e 's/^"//' -e 's/"$//')
       echo $url_upload_app
       url_upload_arn=$(echo $response_upload_app | jq '.upload.arn' | sed -e 's/^"//' -e 's/"$//')
       echo $url_upload_arn
       curl -T Alluo.ipa $url_upload_app
       echo $url_upload_arn > /Users/runner/work/mobile-app-account-holders/mobile-app-account-holders/apk_upload_url.txt
       sleep 180
       echo "uploaded"
    - name: Upload ipa arn
      uses: actions/upload-artifact@v3
      with:
       name: apk_upload_url
       path: /Users/runner/work/mobile-app-account-holders/mobile-app-account-holders/apk_upload_url.txt

The second job starts similar to the first one with checking out repositories as well as providing AWS credentials from the storage. Bear in mind that we use ORG git token to checkout to test repository. The checkout process has to be repeated in each job because each job runs in a separate environment that shares only the artefact space. Moreover, the second job runs on an Ubuntu machine that is 10x cheaper than the macOS instance (USD 0.08 vs USD 0.008 at the time of writing this article), because we don’t need XCode at this step.

Afterwards, the job moves into installing Java and creating a test package with Maven. More details on how to do that will be published in another article in the upcoming weeks. Once this is done the job uploads the .zip test file as well as the test specification and passes the upload URL to the artefact store for the next job.

job_2:
    if:  startsWith(github.head_ref, 'release/')
    name: Create TestSet
    runs-on: ubuntu-latest
    steps:
      - name: Checkout actions
        uses: actions/checkout@v2
      - name: Checkout actions
        uses: actions/checkout@v2
        with:
          repository: GetAlluo/qa-automation
          path: qa-automation
          ref: ET-400-iOS-allFeatures
          token: ${{ secrets.ORG_TOKEN_GIT }}
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-2
      - name: Create test package
        run: |
          cd qa-automation
          wget -O- <https://apt.corretto.aws/corretto.key> | sudo apt-key add -
          echo 'deb <https://apt.corretto.aws> stable main' | sudo tee /etc/apt/sources.list.d/corretto.list
          sudo apt-get update;
          sudo apt-get install -y java-1.8.0-amazon-corretto-jdk
          sudo update-alternatives --config java
          sudo update-java-alternatives -s java-1.8.0-amazon-corretto   
          export JAVA_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto/     
          export PATH=$PATH:$JAVA_HOME      
          mvn clean package -DskipTests=true
          ls ${{ github.workspace }}
      - name: Upload test package
        run: |
          response_upload_test=$(aws devicefarm create-upload --project-arn $DEV_PROJECT_ARN --name test-action-github.zip --type APPIUM_JAVA_TESTNG_TEST_PACKAGE)
          echo $response_upload_test
          url_upload_test=$(echo $response_upload_test | jq '.upload.url' | sed -e 's/^"//' -e 's/"$//')
          echo $url_upload_test
          echo $response_upload_test | jq '.upload.arn' | sed -e 's/^"//' -e 's/"$//' > ./test_upload_url.txt
          ls ./qa-automation/target
          ls 
          curl -T ./qa-automation/target/zip-with-dependencies.zip $url_upload_test
          sleep 100
          
      - name: Upload test spec yml
        run: |
         response_testSpec_upload=$(aws devicefarm create-upload --name testSpec_ios.yml --type APPIUM_JAVA_TESTNG_TEST_SPEC --project-arn $DEV_PROJECT_ARN)
          echo $response_testSpec_upload
          url_upload_testSpec=$(echo $response_testSpec_upload | jq '.upload.url' | sed -e 's/^"//' -e 's/"$//')
          arn_upload_testSpec=$(echo $response_testSpec_upload | jq '.upload.arn' | sed -e 's/^"//' -e 's/"$//')
          echo $arn_upload_testSpec > upload_testspec_arn.txt
          ls ./qa-automation
          curl -T ./qa-automation/testSpec_ios.yml $url_upload_testSpec
          sleep 20
          echo "upload completed"
          response_getupload_testSpec=$(aws devicefarm get-upload --arn $arn_upload_testSpec)
          echo $response_getupload_testSpec
      - name: Upload url upload
        uses: actions/upload-artifact@v3
        with:
          name: test_upload_url
          path: ./test_upload_url.txt
          
      - name: Upload test spec arn
        uses: actions/upload-artifact@v3
        with:
          name: upload_testspec_arn
          path: ./upload_testspec_arn.txt

In the last step, we are retrieving the upload results from job 1 and job 2 in order to schedule the test itself with the command schedule run. While the job runs we are checking every 30 seconds if the job has already been completed or not. There is no existing webhook for Device Farm at the moment that could loop back the result to Github actions, but it is crucial to display the results and allow pull requests to be merged based on it.

job_3:
     name: Run and Publish Test Results
     needs: [job_1, job_2]
     runs-on: ubuntu-latest
     steps:
       - name: Checkout actions
         uses: actions/checkout@v2
       - name: Configure AWS Credentials
         uses: aws-actions/configure-aws-credentials@v1
         with:
           aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}
           aws-region: us-west-2
       - name: Download latest IPA
         uses: actions/download-artifact@v3
         with:
           name: apk_upload_url
       - name: Download testspec yml
         uses: actions/download-artifact@v3
         with:
           name: upload_testspec_arn
       - name: Download test suite
         uses: actions/download-artifact@v3
         with:
           name: test_upload_url
       - name: Schedule run
         run: |
           arn_upload_app=`cat apk_upload_url.txt`
           echo $arn_upload_app
           arn_upload_test=`cat test_upload_url.txt`
           echo $arn_upload_test
           arn_upload_testspec=`cat upload_testspec_arn.txt`
           echo $arn_upload_testspec
           sleep 30
           response_upload_zip=$(aws devicefarm get-upload --arn $arn_upload_test)
           echo $response_upload_zip
           response_upload_apk=$(aws devicefarm get-upload --arn $arn_upload_app)
           echo $response_upload_apk
           sleep 10
           response_run_tests=$(aws devicefarm schedule-run --project-arn $DEV_PROJECT_ARN \\
           --app-arn $arn_upload_app \\
           --device-pool-arn $DEV_DEVICES_POOL \\
           --name 'first_cli_run' \\
           --test testSpecArn=$arn_upload_testspec,type=APPIUM_JAVA_TESTNG,testPackageArn=$arn_upload_test)
           echo $response_run_tests
           arn_test_run=$(echo $response_run_tests | jq '.run.arn' | sed -e 's/^"//' -e 's/"$//')
           echo $arn_test_run > runtime_arn.txt
           
           aws_resp=$(echo $response_run_tests | jq '.run.status')
           while [ "$aws_resp" != *"COMPLETED"* ]
           do
            echo "entering while..."
            sleep 30
            aws_resp=$(aws devicefarm get-run --arn $arn_test_run | jq '.run.status')
            if [[ "$aws_resp" == *"COMPLETED"* ]]; then 
            echo "COMPLETED" 
            break
            fi
           done
           echo "$aws_resp"
           
           aws_resp=$(aws devicefarm get-run --arn $arn_test_run | jq '.run.counters')
           echo $aws_resp > aws_response_result.json

The outcome of the pipeline can be used to block merging new releases as PRs visible in the GUI:

Conclusions

The presented solution is scalable and doesn’t depend on any macOS virtual instances on AWS that can cost up to USD 700 per month or physical MacOS running onsite as a server that creates a maintenance problem. Each run of the GitHub Actions costs around USD 6 and takes around an hour to complete, which is much more cost-efficient than the other solutions. It can be further amended with conditional app deployment to Appstore with Expo CLI for Testflight beta testing.