Snappy Shrimp - New Snapshot Testing Experience

Snappy Shrimp - is a library for snapshot testing. It is based on FBSnapshotTestCase library and actually, it does the same thing - it records a reference image of your view and verifies it every time you launch tests. But how much time does it takes to make tests on all possible devices and in all possible orientations? This library allows you to save your time by decreasing the number of necessary devices up to three - iPhone, iPhone Plus and iPad and you will see why.

Snappy Shrimp - New Snapshot Testing Experience

1. Features
   1. iPhone, iPhone SE and iPhone Plus examples
   2. iPad with multitasking support examples
   3. iPhone X Example
2. Requirements
3. Install
   1. Using Carthage
   2. Using Cocoapods
4. Configuring
5. How to write tests
6. Snapshot Testing example
7. Automation example
8. Why you need three devices instead one
9. Specific notes about snapshot testing
10. Contributions

Features

Things that we've improved to make snapshot testing experience much better:

• All devices are presented programmatically, including actual sizes and trait collections;
• We don't use Host Application, so it makes our tests run faster;
• Improved images naming, by adding to the image name: device, orientation and os version;
• Implemented iPhone X safe areas and masks to capture snapshots exactly as it looks like on a real device;
• You're able to see how your app looks like in all orientations and in all multitasking modes for iPads;

iPhone, iPhone SE and iPhone Plus examples

iPad with multitasking support examples

iPhone X examples

Requirements

• iOS 9.0+
• Xcode 9.0+
• Swift 3.2+

Install

We use Carthage to install our framework. Follow the link if you don't know how to install Carthage.

Using Carthage

1. Create Cartfile in your project;
2. Add this repo to the file:

github "AndriiDoroshko/SnappyShrimp"

3. Use to get our library:

carthage update --platform iOS

4. Add both frameworks to Build Settings/Link Binary With Libraries
5. In your test target add the Run Script

Shell: /bin/sh

/usr/local/bin/carthage copy-frameworks

Input Files:

$(SRCROOT)/Carthage/Build/iOS/FBSnapshotTestCase.framework
$(SRCROOT)/Carthage/Build/iOS/SnappyShrimp.framework

Using Cocoapods

1. Create Podfile
2. Add the following line to it:

pod 'SnappyShrimp'

3. Install this and other dependencies in your podfile using:

pod install

Configuring

1. Create two schemes with your test target - first for testing, second for recording.
2. Add evnironmental variables to your test schemes:

• FB_REFERENCE_IMAGE_DIR - folder for reference images - $(SOURCE_ROOT)/Tests/ReferenceImages e.g;
• IMAGE_DIFF_DIR - folder for failed and difference images - $(SOURCE_ROOT)/Tests/FailureDiffs e.g..
• RECORD_MODE - variable to decide if tests should run in record mode. In Record scheme, set this value as TRUE, to record new references.

3. In case you want to have different snapshots on screens with different display gamut (P3 and sRGB), you should set isGamutSupportEnabled value to true.
4. If you want your custom implementation of record mode using , just setup the recordMode variable.

Example of the setup method:

override open func setup() {
	super.setup()
	recordMode = #?@!&
	isGamutSupportEnabled = #?@!&
}

Note: this is our implementation of Snapshot tests implementation. After install, you're free to configure it as you want.

How to write tests

1. Add new Unit test target;
2. Create new class inherited from SnapshotTest;
3. Inside the class, create test...() method;
4. Create and setup your ViewController for testing;
5. Use method verify, that needs your controller and a Presentation object, that describes a device.

Presentations of all devices are specified in Device enum. Here's an example

verify(controller, for: Device.iPadPro9.portrait.oneThird)

That's how the method looks like

verify(controller: UIViewController, 
	for presentation: Presentation)

Where:

• controller - your view controller that you want to test;
• presentation - contains information about device, including size, trait colections, masks, safe areas and name;

To create snapshot with custom traits and size, just create a Presentation object on your own.

Example:

public let smallViewController = Presentation(
            name: "Custom small window",
            size: CGSize(width: 250, height: 375),
            traitCollection: UITraitCollection(
                traitsFrom: [Display.InterfaceIdiom.phone,
                             Display.SizeClass.Vertical.compact,
                             Display.SizeClass.Horizontal.compact,
                             Display.Scale.x2]))

Snapshot Testing example

Example of using Snappy Shrimp framework

Our example of snapshot testing has two schemes - one for recording, another one is for testing. We've picked this way to make things faster and easier.

To run our test example, we're using Fastlane. It allows you to run testing or recording with a simple command

fastlane test

or

fastlane record

Inside the Fastfile everything is simple. A scan action with required scheme and devices.

  lane :test do
    scan(
      scheme: 'SnappyShrimpTests',
      devices: ['iPhone 8', 'iPhone 8 Plus', 'iPad Pro (12.9-inch)']
    )
  end
  lane :record do
    scan(
      scheme: 'SnappyShrimpRecord',
      devices: ['iPhone 8', 'iPhone 8 Plus', 'iPad Pro (12.9-inch)']
    )
  end

This is an example of the class for snapshot testing. All you need is to setup your controller and to call verify method with Presentation that you want to test.

import SnappyShrimp

class SnappyShrimpTests: SnapshotTest {
    
    func testExample() {
        let vc = ViewController()
        
        verify(vc, for: Device.iPhone8.landscape)
        verify(vc, for: Device.iPhone8.portrait)
        verify(vc, for: Device.iPhoneSE.portrait)
        
        verify(vc, for: Device.iPhone8Plus.landscape)
        verify(vc, for: Device.iPhone8Plus.portrait)
        
        verify(vc, for: Device.iPadPro12.portrait.fullScreen)
        verify(vc, for: Device.iPadPro12.portrait.oneThird)
        verify(vc, for: Device.iPadPro12.portrait.twoThirds)
        
        verify(vc, for: Device.iPhoneX.portrait)
        verify(vc, for: Device.iPhoneX.landscapeLeft)
        verify(vc, for: Device.iPhoneX.landscapeRight)
}

Automation example

We've used snapshot testing to verify that our app looks exactly as we expect in all cases. But obviously, no one wants to run tests before pushing, or simply forgets, so we run these tests on the CI. But if the tests fail on the CI, you just can't take a look what exactly has happened there and you have to run tests on your own machine.

So we found a way for that, We upload those images out of Travis in cloud storage, and then, if that's a pull request, they're posted in PR comments, if no, they're comming directly to our slack channel.

Things that we're using for that:

• Fastlane - if you still don't use fastlane, you better start. It allows you to run tests much easier and has a lot of options for build and testing.

• AWS S3 - best storage for your files, but you can use any you want.

• Danger - provides you with an ability to post comments, warnings, and failures to GitHub PR. It supports both Swift and Ruby, so you can write your own script, that will insert failed images into PR comments. With Danger, you will always have only one comment, that will be updated. If there will be no failurs, warnings or messages, it will remove comment.

Why you need three devices instead one

The main idea was to use only one device with iOS SDK for snapshot testing, when the controller will have all needed trait collections, including size classes, idiom, scale and so on. But in fact, you can't override device specific trait collections, like scale, gamut, idiom. So when you launch tests for iPhone Plus on an iPad, you may have wrong behavior, because it will have scale @2 instead of @3, same with the idiom. That is why you need at least three. If you want to test @1 scale (old iPads or iPhone 3GS) you should add more, obviously.

Specific notes about snapshot testing

There're some things that you should note from our own experience.

1. Always use the same devices for testing.

First of all, because we didn't add differences for devices with different gamut - SRGB and P3. So, when you run snapshot tests on iPhone 6 Plus and iPhone 7 Plus, you will have tests failed, but you will have empty difference images and both reference and failed images will look absolutely the same. But for the machine, there is a difference. So, if you see an empty diff in your folder, first of all, make sure that you're running on the same device, that was used when you were writing these refs.

2. Issues with empty diffs after layout updates

If for the previous note there is an explanation, for the next one I haven't found one. We've had some issues when we've had empty diffs after we've changed view hierarchy. We've simply added an empty view, that served as a container and some of the tests have failed. The only way here - record new refs. The issue can be in the original framework's issue, or in the iOS SDK. Had no time to research that issue. Feel free to share your thoughts about that.

Contributions

Feel free to contribute to make that framework even better. Here's small check list that you should read:

• Read Contributions guide first;
• Check already opened issues and requirements file if your issue is already mentioned, or in progress;
• Open an issue;
• Create a PR with your fixes or improvements, if you have some.