How we test our apps to make them ready for a new OS.
NB: all links are correct at the time of writing but as some point to live development resources, it’s likely that these will be retired or updated to reflect newer OS versions over time.
What is beta software?
“Beta” refers to the final stage of development and testing software, broadly when software in development is opened out to a wider community (generally outside the company that is developing it) for testing. Companies do this to get a much broader range of (free!) testing and feedback about the software from developers and early adopters, and it’s used across the software industry.
When you’re talking about mobile device operating systems (OS), then there is the added need to allow 3rd party developers to get their own software applications (apps) updated and ready to work with the new OS. Over the last few years in particular, the timing of these OS betas has become reasonably predictable, with the new versions announced and available for developers to test in the summer, followed by the release to the public in the winter.
Why beta test?
As soon as the final versions of iOS or Android are launched to the general public, we want to make sure that Perfect Ward continues to work brilliantly, with no interruptions for our users. This is generally referred to as “Day Zero compatibility”, meaning that you know that on the very launch day your app will work fine with the new OS. The is very important for Apple iOS, as Apple have very high adoption rates of the new versions of iOS and we will get a reasonable number of users who try out Perfect Ward on the new iOS version on the day it gets released (hi Julia!). Oh, and of course some of the geekier members of our own team.
Because of the way Android is developed (Google build the core OS, but each manufacturer needs to add any customisation and release themselves), adoption rates for the latest version are far lower for Android, though they are getting better.
Testing the beta versions also helps us get a good feel for the new features and styles of the OS, and also get a feel for how other applications are adapting. Understanding the general look and feel of the operating system is very important to us, we put a lot of effort into making the app feel a natural fit for users of iOS or Android, and whilst Apple and Google publish extensive guides to how apps should look and feel on their OS, it really helps to be able to experience that first-hand. That all said, we generally choose to adopt to the newer OS styles more gently over time, as we know not all of our users will be familiar with them immediately.
We can test Perfect Ward on the simulators that come with software development kits (SDKs – these are released alongside each beta for developers to build apps) and this helps us identify any major bugs, but there’s no substitute for trying it on real devices. This gives you a much better feel for using it, but also there are some features (like scanning QR codes) that aren’t truly possible to simulate in the SDK.
When we beta test
It’s a fine balance. If you test to early, it can be a poor use of time. Apple and Google release multiple changes to their beta OSs, and you may end up repeating a lot of the work you’ve done for little benefit (at the time of latest editing, Apple are up to their 9th beta version of iOS 12). It can also be a risky business – some early beta versions can cause issues, and if you’re always working with the latest version you risk things like this.
Even if it’s not as bad as totally ruining your phone (!), the software and features can undergo fairly major changes during beta tests. However, if you leave it too late, you might not be able to adapt to the changes in the OS ahead of Day Zero – particularly since Apple and Google don’t give us much notice when they’re going to be releasing the final version to customers. So we try to reach a compromise, by doing a quick test on one of the early beta releases. This allows us to understand if there are any major issues that we need to be working on. If this goes smoothly then we will not carry out extensive testing until much later in the beta release cycle. If it doesn’t go so well and we find any issues, then we obviously get to work on fixing them!
Apple have a very well established pattern now of presenting the new OS early in the summer, complete with some headline new features. Shortly after they will release the beta to developers, ahead of a full consumer launch generally early in the autumn.
This year, iOS doesn’t look to have many dramatic changes for us. That’s not to say there aren’t exciting things in the new version, both for users and certain types of developers (e.g. for games), but there aren’t the core OS changes that we have seen in previous versions of iOS. This has made it a relatively straightforward version for us to test.
How We Tested
Apple make this a very easy process for developers, and after a while even the more keen members of the general public too. We can enrol any compatible (recent) iOS device into the beta scheme and then it will automatically have the latest version of the beta available for testing. It’s straightforward to do, but because the beta isn’t final software it’s a good idea to do this on a dedicated device rather your day-to-day iPhone or iPad. We’ve tested on an iPod touch and then later on an iPad, both specifically set aside for beta testing.
What we found
So far so good! iOS 12 doesn’t introduce any dramatic new styles or compatibility issues, and this is not that surprising – it seems a lot of what Apple has focused on in this release are “under the hood” improvements rather than significant changes for developers. We haven’t found any major changes that we need to adjust for so far. We will consider some of the new features available to developers over time, but as these will only be available to users with iOS 12 or higher it will be a while until we can offer them as part of the core Perfect Ward experience.
Android 9 ("Pie')
Android has a convention of naming each new version of Android after something sweet, starting with the next letter of the alphabet. We’re currently on “Oreo” (v8), before that was “Nougat” (v7), then Marshmallow (v6) and so on. Therefore it is no surprise that the new beta is codenamed "P".
EDIT hot off the press, the new version is called… “Pie”. I’m not sure this translates across the Atlantic particularly well, as here in the UK most people think of pies as more savoury than sweet, but in the USA they generally use the term refer to dessert pies – think more “apple pie” than “steak and kidney pie”.
How we tested
Beta testing on Android is a much more limited affair, with only a handful of devices available able to accept the beta version, and with varying levels of technical knowledge required. This is largely due to the fact that each manufacturer is fundamentally in control of the versions of Android (whereas Apple are in charge of everything iOS), but it has improved recently with the introduction of Android One, and maybe this will be even better as Project Treble becomes more widespread)
That all said, Android make it quite clear which devices are ready for Beta testing: https://developer.android.com/preview/devices/
Attempt 1 - Xiaomi Mi Mix 2s
Our original plan was to try a slightly different device to those you’d normally see in the UK, the Xiaomi Mi Mix 2s. Xiaomi are a massive electronics manufacturer; you might not hear much of them in the UK, but they’re huge in China and elsewhere in the world, and that helps make them the 4thbiggest manufacturer of smartphones globally.
The Mi Mix line are great phones, with a really clean design and a very big screen for the size of the phone. That leads to a slight quirk of having the selfie camera at the bottom, but that’s not much of an issue for Perfect Ward! On top of that, they are about the cheapest way to beta test Android 9, because despite a high-end design they’re very low cost for the specification. This is important, as the instability that you can get with beta software means that you really don’t want to be running it on anyone’s main phone. They are a bit tricky to get hold of as they are only sold in China, but we managed to track one down (thanks Isabella!). Great…
…in theory. Unfortunately, Xiaomi have put a number of security steps in place that made it impossible for us to load the beta, and we’re not quite sure why. Various attempts at loading different Xiaomi desktop applications and re-flashing the ROM through boot mode got us absolutely nowhere. There may be help out there in the Xiaomi forums, but it’s all in Chinese.
The good news is, we’ve still managed to get it running Perfect Ward, so it makes a nice demo phone. Just not the beta testing device we wanted.
Attempt 2 - Nokia 7
So then we tried the Nokia 7, available on the high street for a fairly reasonable £350, not bad for a mid-spec phone like this. And it looks pretty classy too:
This, in comparison, was a breeze to beta test on. You sign up to the Nokia developer portal, register the device and then the Android 9 beta is available to download directly on the phone.
What we found
Well, almost everything worked well. In fact the only adjustment that we’ve had to make so far is to include a Perfect Ward icon matching the new Android 9 format; by default it boxes the Android 8 icon into a circle, which doesn’t look great.
Some quick adjustments from one of our development team (thanks Alex!) and we have the icon back in full Android 9 glory:
Note that several other quite major apps haven’t managed to update theirs yet - examples above. This either suggests they haven’t been testing it, or that they have but haven't updated their app store versions yet.
We also took the time to test out the new Android gesture system, to make sure that Perfect Ward is still comfortably usable with the layout that Android have implemented (it's the new lozenge shape at the bottom, and the removal of the square):
We had to do similar testing when the iPhone X came out and introduced new gestures to the iOS interface, and indeed we made some adjustments to the layout. In this case the gestures didn’t seem to impact usability, and overall we’re happy with how the testing has gone.
We’re as confident as we can be at this stage that Perfect Ward is ready for the next releases of Android and iOS.
We’ll be vigilant in particular as we come closer to the final beta of iOS 12 and through the public launch itself, and we’ll look to correct any minor bugs that we do find as we move into using iOS 12 on a daily basis.
Android is an incredibly diverse ecosystem. That’s both its strength (you get a wide choice of devices, and price competition as a result) and its weakness (it’s hard to standardise software for such a wide range of devices). This means that we’re expecting manufacturers to make their own adaptations for Android P when it’s released, and these always run the risk of having specific bits of code that we need to adapt Perfect Ward to work with – for example, a different way of handling photos.
If you do spot any bugs or even minor visual glitches when you update to newer versions of iOS or Android please let us know immediately, by contacting your Perfect Ward lead or just firstname.lastname@example.org.
Despite all this work to get test beta software, we’re aware that not all our users want to, or indeed are able to be at the very bleeding edge – many will not update to the latest OS for a long time, and some may not do so at all. Therefore we aim to maintain at least one mobile device running on the previous two versions of iOS and Android, so that we can still carry out hands-on testing of Perfect Ward using those older versions of the OS
Oh… and we can now start guessing what sweet treat Google can think of beginning with the letter “Q” to name the next version of Android after.
Written by the team at Perfect Ward.
Follow us on Twitter @Perfect_Ward