Any web-app or app intent to provide a truly universal experience, must be seamlessly functional on whatever device/ OS/ browser combinations their customer prefers. Having access to the latest hardware, operating systems, and platforms allows the testing process to be more streamlined, less expensive, and more innovative. In order to meet this objective, device availability becomes imperative and organisations spend several hundreds to thousands of dollars in procuring and maintaining these physical devices which are used for testing. This necessitates the need for a cloud-based device testing environment such as AWS Device Farm.
There are many cloud based solutions such as SauceLabs, BrowserStack, Perfecto mobile etc. AWS is another cloud testing platform which allows testing of apps over real android and iOS devices. AWS Device farm is a cloud-based application testing environment that enables remote testing of websites and mobile apps on a significant number of real mobile devices and an extensive range of desktop browsers, without the hassles of managing of building any testing infrastructure. It offers access to both modern and legacy versions of devices from multiple manufacturers.
Pricing
AWS device farm has three pricing options for mobile app testing.
Pay as you go There won’t be any charge for the first 1000 minutes and you will be charged $0.17per device per minute.
Unlimited test & access This is a slot based plan. You can buy a slot for $250/per month. Let’s say you buy an Android slot. Now, you can access any real Android device using the slot for unlimited hours. If you want to run in parallel, you need to buy more slots. I.e. one slot, you can access one device at a time.
Private devices In this option, you can choose devices of specific hardware and software configurations that you wish to run your tests, those devices will be reserved for you till you unsubscribe from the plan. This plan starts from $200 per month.
Remote Access
An interesting feature is the possibility to connect remotely to a specific device model via a remote screen and install our app. As of today, AWS device farm has 162 device combinations. You can access devices which are eligible for remote access. If you want a real device for manual testing, then open Device Farm Page and start a session. You can hold a session for a maximum of 150 minutes.
Private Devices
You may ask a question: Why should I go for private devices as I already have a slot for manual testing. What difference does it make? Private devices are dedicated to you. In other options, the devices which you access on remote sessions are actually used by someone a while ago. However, the private devices will be allocated dedicatedly for your account. No other users can access it.
Automated Testing
Appium is a widely used tool for automating your mobile application, you can run Appium scripts using TestNG, Junit, Node.JS, Python and Ruby. Many cloud device services allow you to upload test scripts, however in AWS device farm, you have to upload a test package to execute on multiple devices simultaneously. Uploading the script has upside and downside. Upside – we don’t need a highly configured machine to run the test scripts. Downside – sometimes it is hard to figure out scripting issues.
Usually, on some cloud testing provider such as SauceLabs, the test code will run on your computer while instructing cloud devices to do something, but in Device Farm you need to package all your tests to be run on the cloud. When executing tests, ADF uses an approach which we call as server-side execution. This means that the test code runs entirely on their infrastructure. The alternative to this is what providers like Browserstack or Saucelabs use where the test code runs on the user’s machine, called client-side execution, and interacts with the cloud service via a web service.
In general, client-side execution is easier to implement; if you already know how to execute tests locally, the configuration changes you need to make are minimal. Amazon Device Farm approach is a bit trickier as it requires a few extra steps, namely packaging and uploading everything necessary to run your tests to their servers.
Plugins
Automated tests can be scheduled in Jenkins to perform continuous testing using the Amazon Device Farm for each new build of the application. There is an Amazon Device farm plugin available for Jenkins. Here are the steps on how to configure your AWS account with Jenkins.
Steps to use this plugin:
1. Log into your Jenkins web UI.
2. Click on the job you wish to edit.
3. On the left-hand side of the screen, click “Configure”.
4. Scroll down to the “Post-build Actions” header.
5. Click “Add post-build action” and select “Run Tests on AWS Device Farm”.
6. Select the project you would like to use.
7. Select the device pool you would like to use.
8. Select if you’d like to have the test artifacts (such as the logs and screenshots) archived locally.
9. In “Application”, fill in the path to your compiled application for testing native or hybrid app. Check “It is a web application.” for testing web app.
10. Choose the test framework, provide the path to your test package location and other relevant details.
11. Configure device state parameters like radio details, extra data and device locations.
12. Configure the maximum execution timeout. The default execution timeout is 60 minutes.
13. Set the execution configuration parameters: video recording and app performance monitoring.
14. Click “Save”.
If you don’t want to use Jenkins UI and are looking for a code based plugin, then you can try aws-device-farm-gradle-plugin. Using this plugin, you can configure your application file and test files in the Gradle build system.
Security
AWS allows third-party security auditors to assess the security and compliance of AWS Device Farm. AWS device farm has data retention policies. The data entered or saved in the device will be wiped out once the session is closed. However, Uploaded applications & test packages, logs, and video recordings will be retained certain days.
Limits
Almost most of the devices are public and shared ones, you cannot hold the devices for more than 150 minutes per session for both manual and automation testing. There is no information about the access limit for private devices.
Conclusion
If you have robust automated test scripts and you have performed multiple dry runs locally, then AWS device farm is a suitable platform for your cloud devices testing. However, don’t forget to use the 1000 minutes free trial before you go for any plans. We, as a software testing company, use different cloud testing platforms to test mobile applications on real devices. We would recommend choosing the platforms based on your testing needs. Before choosing a cloud testing tool, check how quickly the new devices are added and whether the remote access provides the selected device without any delay.
One of the best advantages of AWS device farm is uploading the test package once and getting the result for multiple devices. Whereas in other cloud testing platforms, you need to execute the same test package using your machine. With device farm we get detailed logs, performance graphs and screenshots are generated during each run to provide general and device-specific feedback. The service offers a lot of flexibility by allowing the state and configuration of each device to be altered in order to reproduce very specific test scenarios. Ensure the test cases selection for automation test execution is done via build file arguments. So that you can use a single build file with the same test package.
We hope you have enjoyed reading this blog article and found it insightful.
Comments(0)