Menu

UI Testing with Xamarin Test Cloud

on xamarin, mobile, testing

One thing I wanted to touch on early is UI Testing because it's something you should be doing from the start of creating your app. It helps to make sure that things are still working as you make changes and also because nobody wants to go back after the fact and write a bunch of tests at the end. If you add them as you finish a feature you can go to bed safe in the knowledge that what you just did three views down didn't just cause that welcome screen for your users not to work any longer.

Why Do I Need UI Testing?

Are you a mobile developer? Then I don't really need to mention device fragmentation do I? Let's just quickly gloss over the numbers and move on. There are 5 different iOS versions and 9 different Android OS versions. There are about 20 different Apple devices, and over 24,000 Android devices. There are close to thirty different Android screen sizes and Apple is starting to play catch up now with their 6 different sizes.

Beyond just fragmentation, there is also the complexity of modern apps to take into account. Just think about the number of different screens you have in your app. Now think about how many screens you have that your users might only ever see once or twice. How confident are you that the latest Android OS upgrade won't cause havoc on that screen that you might not even think to test? What about the expectations your users have? If your app isn't flawless they are just going to go grab one of the 90 other apps (if you're lucky there are only 90 of them) that perform a similar function to yours. You want your app to be the best and the only way to do that is to make sure it's constantly being tested.

Xamarin UITest to the Rescue!

Xamarin has a great tool called UITest that you have at your disposal. You can use this to write simple scripts that will let you navigate through your app and test it the same way your users might. You have access to many interactions, such as; Tap, Scroll, Swipe, Pinch, Text Entry, Rotation, GPS. They are also constantly improving it with new features like testing under particular network conditions coming up in the future. Also, it's free to use! You can use it all you want in the simulator/emulator and make sure everything is rock solid. Beyond that you can use Test Cloud to test all the other devices you don't have, like one of the 24,000 different Android devices we mentioned earlier. As a licensed Xamarin developer you also have access to 60 free minutes per month.

Ok, now enough with the sales pitch, let's get to work.

Adding Tests to an Existing App

Just like adding a project, just right click on your solution, add new project, under tests create a new UITest project. This will create your new project, just make sure you update your nuget packages. For iOS, you will need to add in the Xamarin.UITest package from nuget or grab it from the Xamarin Component Store.

If you're creating a new Xamarin.Forms project, it will create a UITest project for you so you won't need to worry about the setup portion of things. All that said, you would probably get more out of following along with James Montemagno on his Motz Codes Live channel since he walks you through all of this.

Local Testing

So once you've got your project set up you should see a sample Tests.cs file that has your first test in it. If you've created a brand new Xamarin.Forms app you can just run this test and see it pass. If you've added it to your existing app or once you've got your new app with more than just the boiler plate code you will need to have a way to figure out what's on the screen. The first thing I do is add a test for opening the REPL Command Window. 

[Test]
public void OpenRepl()
{
    app.Repl();
}

Once that is in place you can run the test and you'll get a REPL window that you can work through your various tests. Some examples:

// wait for something with the word submit on it to be on the screen
app.WaitForElement(c => c.Marked("Submit")); 

// take a screenshot and tag it with this text, only works in test cloud
app.Screenshot("The app started");

// enter text into a text field with an id of "text_conference"
app.EnterText("text_conference", "text to enter");

// get a list of everything marked "search"
app.Query(c => c.Marked("search"));

First Run Testing

So one of the great new features in Xamarin.UITest 1.0 that was just released is the ability to recreate that new install experience. Most of your users will decide if they want to continue using your app based on this initial experience so making sure it is thoroughly tested is essential to retaining users. This is done through a new enum called AppDataMode which has a default value of Clear, and has the option of DoNotClear. While I know that the default is to Clear, as a TDD practitioner I want my Unit Tests to be explicit so I feel like my UI Tests should be the same. What I like to do is just update that AppInitializer to take in the AppDataMode enum and then I know that my tests are doing what I expect them to do.

public static IApp StartApp (Platform platform, AppDataMode appDataMode)
{
    if (platform == Platform.Android) {
        return ConfigureApp.Android.StartApp (appDataMode);
    }
    return ConfigureApp.iOS.StartApp (appDataMode);
}

After that you just have to update your [Setup] function and now you know that the test is doing what you intended for it to do.

[Setup]
public void BeforeEachTest()
{
    app = AppInitializer.StartApp (platform, AppDataMode.Clear);
}

Test Cloud

Once you're done, it's pretty easy to move on to test cloud from Xamarin Studio. Just right click on the tests and instead of selecting "Run Test", you just select "Run in Test Cloud". Thanks for making it that easy. Ok, well maybe not that easy, you then just have to select the devices and schedule the run. So it's that easy.

Sample Code

Of course everything is uploaded to GitHub, and this sample app is something I'm working on. It is a work in progress, but that's the best time to put in the tests. So for this sample I'm testing out the first screen a user will see when they start the app for the first time.

Sample Code: github