by admin | Feb 27, 2020 | Performance Testing, Fixed, Blog |
This blog is mainly going to cover performance testing and its strategy. Performance Testing helps the business to understand the stability and reliability of the application upon stringent circumstances.
What is performance Testing?
To define in laymen terms, a Performance test is conducted to mainly observe the behavior of application which is hosted on a server upon various loads. The goal is not to validate the functional flow and identify the bugs in that manner, rather test a maturity attained application under critical load, volume, stress conditions to verify the set benchmark parameters are satisfied before the application goes live.
Types of Performance Testing
In levels of testing, the performance test is classified under the non-functional testing , given the nature of the test. This is again categorized into various types of testing which are listed below
Load testing
This test determines the application behavior under a certain load. When we say load we can consider a couple of illustrations as the number of user hits per second or the number of transactions. The goal is to verify how the application performance when there is a load on the server.
Stress Testing
Unlike the above test, this is done by injecting the load higher than the threshold to see the behavior of the application; this can tell us the breakeven point of an application also the frequency of getting down.
Volume Testing
This test is also called as flood testing as we are going to inject too much of data into the application to see, how the queue or any other interface is withstanding the inflow
Endurance Testing
This testing is done by subjecting the application to a nominal load but to make it operate for quite longer than its usage in the field. This test gives the reliability of the application as we are considering the time that the application lasts before it sees failure.
Spike testing
This test is performed by applying a significant steep of raise load on the application to see the withstanding nature of an application.
Scalability testing
This testing is done to verify how the application behaving when the load is kept increasing by adding a lot of user data volume.
Why is performance testing needed?
Given the capabilities of functional testing, a performance test is more useful to understand the reliability parameters. The intention of the test is not just to find the defects but more about examining the application behavior under load. Spec is defined in such a way that it mainly denotes following
How many concurrent users can log in to the application?
How many transactions should the application be processing upon peak load?
How much minimum the response time must be of?
There are a few other parameters which are gathered as part of Non-Functional Requirements (NFR) gathering and tested accordingly.
Has there been no performance testing we wouldn’t be able to understand the capacity, flexibility and the critical parameters of an application under load and there are great chances that we fail in field due to the improper understanding of the benchmark parameters.
Parameters that are validated during test
When any test is conducted, we look for certain parameters as validation elements. In performance testing, these parameters are again considered as two parameters
Server-side parameters:-
CPU utilization
Memory utilization
Disc utilization
Garbage collector
Heap dump/ thread pooling
Client-side parameters:-
Load
Hits per second
Transactions per second
Response times
Entry & Exit criteria
Like every other type of test, performance test also defines its own entry and exit criteria.
Entry criteria
This imperatively defines, when the application is eligible to undergo a performance test. Following are the key pre-requisites to define entry criteria
Functional testing must have been thoroughly conducted
Application must be stable
No major showstoppers must have been in open state
When a dedicated NFT environment is made available
Needed test data should be ready
Exit criteria
This defines how and when to tear down the test, the following activities must be completed to give a proper sign-off from a performance test perspective
An application must have been tested with all the NFR requirements
No major defects or customer use cases are to be in failing state
All the test results should be properly analyzed
Reports and results must be published to all stakeholders for sign-off
After discussing about various parameters and basics about Performance Testing, we will now discuss in detail about Performance Testing Strategy
Performance Testing Strategy
Below are the key pointers to consider in defining the Performance Testing strategy
Understanding Organization Requirement
Assess the Application nature and category
Take stock of last year’s performance of existing live applications
Defining Scope
Tool identification
Environment readiness
Risk assessment
Results analysis and report preparation
Understanding Organization Requirement
Every Organization provides certain service to Customer and based on the services and the nature of assessing requirement to its Customer, Performance Testing requirement gain importance. For example, an e-commerce company is relying 100% on Customer hit rates and successful transactions. 1000+ customers login at a time to buy products. During Offer sale, this can turn to the peak of million customers login at a time. This type of Organization needs strong performance testing before hosting it live. Their spending on Performance tool is inevitable.
Assess the Application nature and category
Identify the # of Applications that are in life on that year and assess the application based on its nature, whether it is Customer facing or assessed by the back end-user or assessed by operations team.
This brings in the criticality of application for performance testing. Also, understand the technology in which it is built with and the number of releases planned with a criticality of release. Prepare the list based on the above pointers.
Take stock of last year’s performance of existing live applications
Few applications would be in live for the past few years. Understand the challenges with respect to performance it faced and do the impact assessment based on past history. This exercise would give us the concrete requirement for Performance Testing for that application.
Defining Scope
After assessing the application nature and taking stock of past history, it gives us a requirement on what to test. Prepare NFR Requirements. This is nothing but the process of coming up with a well-documented test plan for that application. This phase essentially includes the identification of the non-functional requirements such as stated below
Whether the application is a multiuser purpose one or a limited user purpose one.,
Identification of key business transactions that are needed to test
Identify key blockers in the past from performance perspective and target to include in the testing scope
Forecasting the future load on certain services and conducting testing as needed
Tool identification
This is essential in the business; the right choice of tool will help in managing the cost as well as the hassle-free test conduction. The proper analysis must be done before we go for a selection that best suits the assignment.
Whether or not it’s an open-source tool or licensed
The tool that is selected should support any test activity with minimal configuration
The tool should help us in generating the best reporting mechanism
The tool must have got a decent community to get any solution for the problem being faced
To insist on the first point, the tool need not be freeware all the time, it’s completely a business call considering the high-end features that a licensed tool offers, no possibility for any breaches. If at all the test engineer identifies that the same can be accomplished by going with any other alternative tool that is available for free and supports the operation equally with a licensed one, then the choice of open-source one is desired as that way we can save the cost that is incurred. Below are some tools list
Popular Performance Testing tools are:
Jmeter (open source)
Load runner (commercial)
Silk performer (commercial)
Neo load (Commercial)
Application performance management tools:
App Dynamics
Wily-introscope
Perf mon
N-mon
Environment Readiness
This is one of the major considerations as far as the performance test is concerned. Unless and otherwise there is an isolated environment for NFT testing we can deem the test results and the application critical parameters (response time, think time, throughput..etc). So, a dedicated environment is highly recommended as the tasks more or related to
Insert a huge number of transactions in DB through API hit
Perform concurrent login scenario
Observe the response time while navigating between the pages
The environment must be maintained properly and should be highly stable
Risk assessment
It is quite obvious that on certain tasks, there are one or multiple hurdles seen down the line. We should foresee them during the initial discussions about the planning and proactively calling them out to all stakeholders will help come up with a stringent plan, that way we can reduce the negative impact on the business.
This concept is to learn the impact on a business if at all
Analyzing the dependency with the development and environmental team to get the support required during application being broke/ downtime instances
Going live with the presence of a defect that may occur at customer sight
Can’t test a particular scenario due to the unavailability of the infrastructure in the lower environments
Testing a feature by taking some deviation
Results analysis and report preparation
For any execution we make, test results are the key as they are the outcome of the effort that is being spent.
Post execution, results generated from the performance testing tool have to be analyzed and various parameters are assessed and a decent report in an easily understandable way has to be shared to all stakeholders on the outcome. We have tools which capture results of a test run, we can fairly extract from them.
Few instances, performance engineering activity also done by going one step further deep to understand where the blockers are and what could be the solution. This activity requires architectural understanding and finding the error-prone areas.
by admin | Dec 8, 2019 | Performance Testing, Fixed, Blog |
Performance testing falls within the wide and interesting gamut of software testing, and the aim of this form of testing is to ascertain how the system functions under defined load conditions. This form of testing focuses on the performance of the system based on standards and benchmarks, and is not about uncovering defects / bugs. As a leading Performance Testing Services Company we understand the importance of this type of testing in providing problem solving information to help with removing system blocks. Test execution and result analysis are the two sub-phases within the performance test plan, which encompasses comprehensive information of all the tests that would be executed (as mentioned below) within performance testing.
Defining Test Execution within Performance Testing
Test execution within performance testing denotes testing with the help of performance testing tools, and includes tests such as load test, soak test, stress test, spike test and others. There are several activities that are performed within test execution. These include – implementation of the pre-determined performance test, evaluation of the test result, verification of the result against the predefined non-functional requirements (NFRs), preparation of a provisional performance test report, and finally a decision to end or repeat the test cycle basis the provisional report.
Our team has years of experience in performance testing services, and includes experts who consistently execute tests within the timelines specified for test completion. A structured approach is quintessential to the success of test execution within performance testing. Once the tests are underway, the testers must consistently study the stats and graphs that figure on the live monitors of the testing tool being used.
It is important that the testers pay close attention to the performance metrics such as the number of active users, transactions and hits per second. Metrics such as throughput, error count and type, and more also need to be closely monitored, while also examining the ‘behavior’ of users against a pre-determined workload. Finally, the test must halt properly and the test results should be appropriately collated for the location. Post the completion of test execution, it is important that testers gather the result and commence result analysis – a post execution task. It is necessary for testers to understand the importance of a structured analysis of the test result – the second sub-phase of the performance test plan.
Methodology for Result Analysis
Result analysis is an important and more technical portion of performance testing. This is the task of experts who would need to assess the blocks and the best options for alleviating the problems. The solutions would need to be applied to the appropriate level of the software to optimize the effect. It is also essential for testers to understand certain points prior to starting the test result analysis. Some of these include – ensuring the tests run for a pre-determined period, remove ramp up and down duration, assess that no errors specific to the tool are present (load generator, memory issues, and others), ensure that there are no network related issues, and evaluate that the testing tool collects the results from all load generators and a consolidated test report is prepared. Further, adequate granularity must be there to identify the right peaks and lows, and the testers must make a note of the utilization percentage of the CPU and Memory before, during, and post the test. Filter options must be used and any unrequired transactions must be eliminated.
Testers must use some basic metrics to begin the result analysis – number of users, response time, transactions per second, throughput, error count, and passed count transactions (of first and last transactions). Testers must also analyze the graphs and other reports in order to ensure accurate result analysis.
As leaders in performance testing within the realm of software testing services, we believe there are some tips and practices that when implemented, would enhance the quality of test execution and result analysis.
- Generate individual test reports for each test
- Use a pre-determined and set template for the test report and for report generation
- Accurate observations (including root cause) and listing of the defects (description and ID) must be part of the test report
- All relevant reports (AWR, heap dump analysis and others) must be attached with the provisional test report
- Conclude the result with a Pass or Fail status
A successful digital strategy is one where there is speedy and reliable software that is released fast into the market. Being able to create such software with the help of a structured performance testing process is the edge that a business would gain. We at Codoid are leaders in this realm and more, since our testing methods are designed by experts who understand the importance of consistently superior load and performance testing. Connect with us to accelerate the cause of your business through highest quality software – we ensure it.
by admin | Oct 8, 2019 | Performance Testing, Fixed, Blog |
When customers walk into a brick-and-mortar store/shop they expect to be greeted warmly and enter clean and well-stocked premises. Most businesses also have an online presence and the websites serve as the store in a digital world – it is the face of a business. A good website is one that allows speedy and easy navigation and ensures that customers remain happy with their experience. Customer happiness is directly proportionate to the ROI of any business – there is a proven connection between the loading speed of a website and gaining new customers and retaining existing ones. Fast loading pages will keep visitors and customers coming back, and they would be more likely to conduct business on the site. However, as a company excelling at Performance Testing Services we know the converse of this truth as well. A slow loading site is one of the top reasons for customers/visitors to skip a website and abandon shopping carts midway. Customers expect websites to load within a few seconds and if a business expects them to remain engaged, the customer wishes must be granted.
We at Codoid consistently help clients to keep a strict watch on how their website functions and perform – and ensure that all functionalities and features run optimally, even under load conditions. There are times when every business experiences peak loads such as during holidays, major announcements or events, and other such times. It is necessary for a business to know the peak volume/load their website can withstand else it may shut down leaving customers furious. When existing and new customers are unable to access services and products, a slump in revenue and reputation are expected outcomes. Keep your business safe from these risks – speak with our experts.
The Importance and Best Practices of Load Testing
Load testing is a critical method that helps to alleviate risk. A fast-loading site is a great way to ensure customer happiness, bring peace of mind for the stakeholders, and add significantly to the bottom line of a business. We, therefore, put forth some of the best practices that we as a leading performance testing company follow, and apply to the projects entrusted to us.
Understanding Business GoalsWe help businesses understand the correlation between their goals and the testing environment, such that the right aspects of the application are tested to ensure performance under load. Businesses must pay attention to the user experiences that would be the drivers for engagement, revenue, advertising, and other such metrics, and know what expectations users have that would accelerate these metrics.
Understanding Web and Application Key Performance IndicatorsKnowing business objectives would not suffice without establishing some indicators of performance – real-time performance compared to expected performance. Some of the KPIs would be response time, the number of requests the system processes per second, utilization of resources (memory and CPU consumption and other resources)
Selecting the Right Tool for Load Testing Since the success of load testing depends on the kind of tool and methodology, your business would need the best-in-class partner to deliver proactive service. It is important to understand the possible issues even before they affect the performance – this will ensure an optimally functioning website and ultimately happy customers consistently doing business and spreading the word of mouth to others, which would create more customers.
Putting Together the Best Test CaseA test case that simulates the user behavior in the real world will be best equipped to predict the performance of the website under a similar load. Experts create these test cases with virtual users on emulators or even physical devices.
Ascertaining the Load EnvironmentThe premise of load testing is the similar replication of the production environment – we understand that a test environment can never exactly duplicate the production environment, but it is important to gain as much accuracy as possible. It is necessary for a business to understand the limits of the hardware and through that ascertain the possible defects before they occur.
Load Tests to be Run GraduallyIt is never wise or feasible to test everything at once – beginning to test a smaller number of scattered users is a better way to identify any problems and the breakpoint of the system. Post running the first set of tests, it is necessary to check the results and remedy any defects/obstacles, before moving to the next set of testing scenarios.
Load Testing is Best Done from the End User PerspectiveThe entire objective of load testing is to ensure that the end-user/customer is happy with their experience when using the website and or application of a company. Load testing must revolve around the aim of elevating these experiences, such that customers remain engaged and continue to do business with the company thereby maintaining a good bottom line.
In Conclusion
As experts we know that there are several things that are strictly unacceptable to run load tests – for instance running tests in a real environment. There are several factors and network traffic that affect a website, and running tests in such a scenario is sure to skew results. Another thing that must not be done is trying to break down the tested server – this is not the objective of load testing – but rather to check the performance of a website under varying load conditions. The best practices mentioned do not constitute an exhaustive list but do provide a clear indication of how to ensure meticulous load testing. Connect with our load testing champions and leave the worrying to them.
by admin | Nov 3, 2019 | Performance Testing, Fixed, Blog |
Performance Testing checks the ‘performance’ of an application or software under workload conditions, in order to determine stability and responsiveness. The objective of performance testing is also to discover and eliminate any hindrances that may lower performance. As an expert Performance Testing Services Company we conduct this type of testing using contemporary methods and tools to ensure that software meets all the expected requirements such as speed (response time of the application), stability (steady under varying loads), and scalability (load capacity of the software). Performance testing falls within the larger realm of performance engineering.
Why is Performance Testing Required?
Most importantly performance testing uncovers areas that may need to be improved and enhanced before launch, and ensure that the business is aware of the speed, stability, and scalability of their software/application. As an experienced software testing company, we understand that without this type of testing, software could contain several issues such as slow performance when being used by several users, inconsistent ‘behavior’ across varying operating systems and devices, and other such issues.
If a software/application were to be released with issues to the market not only would it not meet the sales goals, the company would also gain a bad reputation and customer ire. Performance testing is especially necessary for applications that would be critical to the mission of companies – applications need to run interrupted and without variations and divergent behavior.
Types of Performance Testing
There are different types of performance tests within the realm of software testing and fall under non-functional testing (determining the readiness of the software) and functional testing (focus on individual functions of the software). It is important for companies to partner with a professional and experienced service provider to undertake this type of testing, amongst others, for enhanced ROI in business.
Load TestingAs a highly proficient load testing services company we have helped several businesses to measure the performance of their software under increasing workload – a number of users or transactions simultaneously on the system. Load testing is about measuring the response time and stability of the system with the increase in workload – which is within the bounds of normal working conditions.
Stress TestingThis is also referred to as ‘fatigue testing’ and measures the performance of software beyond the bounds of normal working conditions. The software is loaded with users /transactions that would be beyond the capacity of the software, and the aim of this testing is to check the stability of the software under such ‘stress’ conditions. It helps to determine the point that software would crash and how soon it would be able to ‘recover’ from the crash.
Spike Testing This is a type of testing within the realm of stress testing and tests the performance of software when there are quick, repeated, and substantial increases in workloads. The workloads are beyond the normally expected loads in shorter timeframes.
Endurance TestingThis is also referred to as soak testing and evaluates the performance of software under normal work conditions but over an extended timeframe. The aim of endurance testing is to determine problems such as memory leaks in the system (a memory leak happens when the system is unable to release discarded memory, and this can significantly damage the performance of the system)
Scalability TestingThis testing determines whether or not the software is able to successfully handle increases in workloads, and is done by gradually enhancing data volume or user load while monitoring the performance of the system. In addition, factors such as the CPU and memory could be changed while workload remains constant – the system should perform well.
Volume TestingThis is also referred to as flood testing and is conducted to determine the efficiency of the software with projected large amounts of data – ‘flooding’ of the system with data.
Common Problems Occurring During Performance Testing
Apart from slow responses and load times, testers usually encounter several other problems while conducting performance testing.
Blockages: Data flow is interrupted or stopped when the system does not have the capacity to handle the workload.
Reduced Scalability: When the software is unable to handle the required number of simultaneous tasks, the results may not be as expected, there could be an increase in the errors and possible other unexpected results.
Issues with Software Configuration: The settings may not be sufficient to handle workload.
Inadequate Hardware: There could be poorly performing CPUs or even lowered physical memory of the system.
Steps within Performance Testing
Experienced testers use what is known as a ‘test bed’ or the testing environment, wherein the software, hardware, and networks are put up to run performance tests. The following are the steps to conduct performance testing within the testbed.
Ascertain the Testing Environment By ascertaining and putting together the required tools, hardware, software, network configurations, testers would be better equipped to put design the tests and also assess the possible testing challenges early on in the process.
Determine Performance Metrics Testers must determine performance metrics such as response time, throughput and constraints, while also assessing the success criteria for performance testing.
Meticulous Planning and Designing of Tests Experienced testers would take into account target metrics, test data, and variability of users, which would constitute test scenarios. In addition, testers would need to put together the elements of the test environment and monitor resources.
Put Together Test Design and Execute Tests The next step would be to develop the tests based on the above, followed by the structured execution of the tests. This would include monitoring and capturing all the generated data.
Lastly – Analysis, Reporting, and Retesting For successful completion of the testing, it is necessary to analyze and report the findings. Post analysis and reporting, a rerun of the tests using varying parameters – some same and some different, is essential.
Performance Testing Metrics Measured
Every round of performance testing would not be able to use all the metrics to measure speed, scalability, and stability. Some of the metrics for performance testing are:
Response Time – The total time spent to send and receive a response from the system
Wait Time – Developers are able to determine the time elapsed before they receive the first byte after sending a request – this is also known as average latency.
Average Load Time – From the user perspective one of the top indicators of quality is the average time the system takes to deliver their request.
Peak Response Time – This measures the longest time taken to complete a request, and a peak response time that goes above the average, could be an indication of a variance that could likely pose a problem later.
Error Rate – When compared to all requests, this would be a percentage of requests that end in errors, and usually occurs when the load goes beyond the capacity of the system.
Concurrent Users – Also referred to as load size or the number of active users at any point of time.
Requests per Second and Number of Passed / Failed Transactions – The number of requests handled and the measurement of the total successful and unsuccessful requests.
Throughput, CPU and Memory Utilization – Throughput displays the bandwidth utilized during testing and CPU Utilization is the time that the CPU need to process requests, and Memory utilization is the amount of memory required to process each request.
Right time to Conduct Performance Testing
There are two phases in the lifecycle of an application – the development and deployment phases – and this is true for both mobile and web applications. The performance testing during development focuses on components, and the sooner these components are tested the earlier would anomalies be detected, lowering the amount of time and costs required to rectify the errors. As the application is developed, performance tests must be more comprehensive and some should be carried out in the deployment phase.
In Conclusion
Performance Testing is an absolute necessity in the realm of Performance Engineering – and must be conducted prior to the release of the software. By ensuring that the software/application is working well, a business would be able to gain customer satisfaction and long-term engagement. Costs of poor performance of an application can be extremely high and would include damaged reputation, customer ire, bad publicity, and huge losses in sales. To protect your company from flawed products, connect with our experts and leave the worrying to us.
by admin | Sep 19, 2019 | Performance Testing, Fixed, Blog |
Soak Testing is part of performance testing and is a kind of software testing for Systems under Load (SUL). This testing is carried out to determine and verify that the software is able to withstand high volumes over an extended period. Soak testing can uncover issues such as memory allocation, log file handles, database resource utilization, and others. A performance testing company uses soak testing to measure the stability of the system over an extended period and as part of load testing services.
A soak test is expected to execute more transactions in the entire day – those that would be typical on a busy day – to determine any issues with performance that could crop up due to a large number of transactions. This kind of test is also referred to as a stability test. Software testing experts recommend soak testing since the analysis from it forms a critical part of app development.
It is important for businesses to understand that soak testing can only be carried out by an expert company proficient in the realm of software testing. This is so since soak testing uses a whole lot of data, and only experts would be able to remove any incidental issues without wasting time and other precious resources. Such a company would be able to swiftly diagnose and rectify any performance issues, before they become bottlenecks using the most contemporary techniques, tools, and methodology.
Characteristics of Soak Testing Soak Testing is also called Endurance Testing – as the name suggests, it helps to determine whether the software being tested can ‘endure’ continuous high loads and estimate the reaction parameters of the system. There are several occasions when a business would experience peak usage over a longer duration, of their software and unless the system is able to endure such loads, it would crash and result in mayhem. For example: a popular e-commerce site declared massive discounts of a range of items over a period of 3 days. However, within a few hours the site crashed leaving the company embarrassed and the consumers extremely irate and feeling cheated. Such incidents bring a bad reputation to a company and in some cases can turn litigious.
A soak test should have the following ‘qualities’:
- The duration of the peak soak test would be regularly controlled by the available time
- The app should continue functioning without any issues even in a broader timeframe
- It must cover the testing parameters as determined by the business and testing partner
Every system has a typical support window time period. The time between such ‘windows’ helps to determine soak test coverage. A soak test is indispensable to identify a memory leak, depletion of system resources, and lessening of resources with a database management system.
In Conclusion:
Given the importance and sensitivities involved in a perfect software/system, it is necessary for specialists to conduct soak testing. Companies may not be equipped to understand and determine their soak testing requirements, and need to partner with a leading software testing company. Connect with us today to help determine your soak testing requirements, and know more about the wide range of services we offer.
by admin | Sep 28, 2019 | Performance Testing, Fixed, Blog |
Continuous performance testing gives your development teams a fighting chance against hard-to-diagnose performance and load-handling bugs, as well as quickly identifying major functional bugs. One of the key tenets of continuous integration is to reduce the time between a change being made and the discovery of defects within that change. “Fail fast” is the mantra we often use to communicate this tenet. This approach provides us with the benefit of allowing our development teams to quickly pinpoint the source of an issue compared to the old method of waiting weeks or months between a development phase and a test phase.
For this approach to work, however, development and QA teams have to be able to run a consistent suite of automated tests regularly, and these tests must have sufficient coverage to ensure a high likelihood of catching the most critical bugs. If a test suite is too limited in scope, then it misses many important issues; a test suite that takes too long to run will increase the time between the introduction of a defect and the tester raising the issue. This is why we introduce and continue to drive automated testing in our agile environments.
We have observed a recurring set of three major factors that, when present, significantly increase the effectiveness of our tests in a continuous integration environment:
- Flexibility: Our tests must be able to be executed on demand at any time.
- Coverage: Our test coverage must be maximized with respect to the time available.
- Effectiveness: We must be able to catch the hard-to-pinpoint defects immediately.
When the concept of continuous integration and continuous testing was introduced to me some years ago, the discussion centered primarily on the unit and functional testing. Our teams implemented unit tests into their code, and the test team wrote a small set of automated functional tests that could be run on demand. Performance tests, however, were still largely relegated to the back of the room until the project was nearly completed. We thought it necessary to wait until functional testing was almost over in order to get the level of quality “high enough” so that performance testing would run without (functional) issues.
Flexibility: Performance Tests Are Automated
Performance tests are, by their very nature, almost always automated. They have to be because it is very difficult to drive large levels of load or volume using manual testing methods. Pressing the “submit” button on your mouse ten thousand times in succession is far more difficult and far less repeatable than submitting the same transaction via an automated test. Because of this high degree of automation inherent in performance tests, they can be executed any time as needed, including off days and weekends. This flexibility allows our teams to run tests overnight on changes that are made late in the day before the testers and developers arrive the next morning.
Coverage: Performance Tests Quickly Cover Broad Areas of Functionality
Performance tests generally provide “good enough” coverage of major functions without going too deep into the functionality. They cover a broad swath of commonly used functions within a quick turnaround time. If a functional bug exists in a major feature, it very often gets caught in the net of a performance test. Your performance tester is likely to be one of the first to begin screaming about a major bug in a functional feature. This is not to say that continuous performance testing can or should take the place of automated functional testing, but performance tests do, inherently, add a strong measure of functional validation.
You ought to be cautious not to allow your performance tests to become the de facto functional tests, as doing so can cause the team to lose focus on finding performance issues. When used together, however, functional and performance tests become effective partners in finding those bugs that otherwise bring your testing to a grinding halt.
Effectiveness: Performance Tests Catch Hard-to-Pinpoint Defects Immediately
Another important lesson we have learned managing performance test teams is that it’s rare for a performance issue to be caused by a code change that was made to intentionally impact performance. In other words, the majority of performance-related bugs occur in otherwise innocuous code. Quite often, we find that a defective change has a very minor performance impact when the lines of code are executed once, but when executed thousands or millions of times, they have a major cumulative slowing effect.
Consider the otherwise harmless line of code that, when changed, creates a performance delay of, say, only ten milliseconds per iteration. Now assume that the code iterates through that loop ten times per transaction. That ten-millisecond delay per loop is now compounded into a hundred-millisecond delay per transaction. If we multiply that one-tenth of a second delay by hundreds or even thousands of transactions per second, this tiny performance delay is now causing a major decrease in the number of transactions our system can process per second.
The key here is that functional changes are generally prescriptive. By this, I mean that a functional code change makes the system behave differently by design and by intention. Performance changes, however, especially negative performance changes, are less likely to be prescriptive and more likely to be an unintentional side effect of an otherwise well-intended change.
Identifying and eliminating these unintentional side effects and figuring out why a system is slowing down becomes increasingly difficult as more time passes between the introduction of the issue and when our tester catches it. If the next performance test doesn’t occur for weeks or even months later, performing root cause analysis on the issue can become next to impossible. Catching these types of performance issues quickly is key to giving your developers the best chance of pinpointing the source of the bug and fixing it. Developers and testers alike will be able to spend less time searching for the proverbial needle in the haystack and more time focusing on getting the product ready for a quality release.
Scaling Up Your Performance Testing
If you don’t do performance testing, start now! Even basic performance tests can provide major benefits when run in a continuous fashion. Start with a single transaction, parameterize the test to accept a list of test inputs/data, and scale that transaction up using a free tool such as JMeter or The Grinder. Add additional transactions one at a time until you’ve got a good sampling of the most important transactions in your system. Today’s performance test tools are much easier to use than previous generations, and most basic tools today support features that were once considered advanced, such as parameterization, assertions (validation of system responses), distributed load generation, and basic reporting.
If you do performance testing, but only occasionally or at the end of a project, select a subsection of those tests and run them every day. Or, if constraints dictate otherwise (such as test environment availability), run them as often as you possibly can, even if that means running them weekly or less often. The key here is to pick up the repetitions and reduce the amount of time between repetitions, failing as fast as possible. Remember, the word “continuous” doesn’t have to mean “constant.”
Report the results of your continuous performance tests in a way that makes them accessible to everyone who needs them. We recommend a dashboard that provides an at-a-glance overview of the current state of your performance tests with the ability to drill down into more detailed results.
Conclusion
Troubleshooting and fixing performance issues is difficult enough without having to wade through weeks or months of code changes to find the source of an issue. By closing the gap between the time a performance issue is introduced and the time we find it, we simplify the process of troubleshooting, eliminate a major source of frustration, and give our teams more time to work on the overall quality of our products. Because they contain a compelling mix of flexibility, coverage, and effectiveness, performance tests are very often powerful candidates for continuous testing.
