Performance Testing and its Strategy - Codoid
Select Page
Performance Testing

Performance Testing and its Strategy

Every Organization provides certain service to Customer and based on the services and the nature of assessing requirement to its Customer, Performance Testing requirement gain importance.

Performance Testing and its Strategy

This blog is mainly going to cover performance testing and its strategy. Performance Testing helps the business to understand the stability and reliability of the application upon stringent circumstances.

What is performance Testing?

To define in laymen terms, a Performance test is conducted to mainly observe the behavior of application which is hosted on a server upon various loads. The goal is not to validate the functional flow and identify the bugs in that manner, rather test a maturity attained application under critical load, volume, stress conditions to verify the set benchmark parameters are satisfied before the application goes live.

Types of Performance Testing

In levels of testing, the performance test is classified under the non-functional testing , given the nature of the test. This is again categorized into various types of testing which are listed below

Load testing

This test determines the application behavior under a certain load. When we say load we can consider a couple of illustrations as the number of user hits per second or the number of transactions. The goal is to verify how the application performance when there is a load on the server.

Stress Testing

Unlike the above test, this is done by injecting the load higher than the threshold to see the behavior of the application; this can tell us the breakeven point of an application also the frequency of getting down.

Volume Testing

This test is also called as flood testing as we are going to inject too much of data into the application to see, how the queue or any other interface is withstanding the inflow

Endurance Testing

This testing is done by subjecting the application to a nominal load but to make it operate for quite longer than its usage in the field. This test gives the reliability of the application as we are considering the time that the application lasts before it sees failure.

Spike testing

This test is performed by applying a significant steep of raise load on the application to see the withstanding nature of an application.

Scalability testing

This testing is done to verify how the application behaving when the load is kept increasing by adding a lot of user data volume.

Why is performance testing needed?

Given the capabilities of functional testing, a performance test is more useful to understand the reliability parameters. The intention of the test is not just to find the defects but more about examining the application behavior under load. Spec is defined in such a way that it mainly denotes following

How many concurrent users can log in to the application?

How many transactions should the application be processing upon peak load?

How much minimum the response time must be of?

There are a few other parameters which are gathered as part of Non-Functional Requirements (NFR) gathering and tested accordingly.

Has there been no performance testing we wouldn’t be able to understand the capacity, flexibility and the critical parameters of an application under load and there are great chances that we fail in field due to the improper understanding of the benchmark parameters.

Parameters that are validated during test

When any test is conducted, we look for certain parameters as validation elements. In performance testing, these parameters are again considered as two parameters

Server-side parameters:-

CPU utilization

Memory utilization

Disc utilization

Garbage collector

Heap dump/ thread pooling

Client-side parameters:-

Load

Hits per second

Transactions per second

Response times

Entry & Exit criteria

Like every other type of test, performance test also defines its own entry and exit criteria.

Entry criteria

This imperatively defines, when the application is eligible to undergo a performance test. Following are the key pre-requisites to define entry criteria

Functional testing must have been thoroughly conducted

Application must be stable

No major showstoppers must have been in open state

When a dedicated NFT environment is made available

Needed test data should be ready

Exit criteria

This defines how and when to tear down the test, the following activities must be completed to give a proper sign-off from a performance test perspective

An application must have been tested with all the NFR requirements

No major defects or customer use cases are to be in failing state

All the test results should be properly analyzed

Reports and results must be published to all stakeholders for sign-off

After discussing about various parameters and basics about Performance Testing, we will now discuss in detail about Performance Testing Strategy

Performance Testing Strategy

Below are the key pointers to consider in defining the Performance Testing strategy

Understanding Organization Requirement

Assess the Application nature and category

Take stock of last year’s performance of existing live applications

Defining Scope

Tool identification

Environment readiness

Risk assessment

Results analysis and report preparation

Understanding Organization Requirement

Every Organization provides certain service to Customer and based on the services and the nature of assessing requirement to its Customer, Performance Testing requirement gain importance. For example, an e-commerce company is relying 100% on Customer hit rates and successful transactions. 1000+ customers login at a time to buy products. During Offer sale, this can turn to the peak of million customers login at a time. This type of Organization needs strong performance testing before hosting it live. Their spending on Performance tool is inevitable.

Assess the Application nature and category

Identify the # of Applications that are in life on that year and assess the application based on its nature, whether it is Customer facing or assessed by the back end-user or assessed by operations team.

This brings in the criticality of application for performance testing. Also, understand the technology in which it is built with and the number of releases planned with a criticality of release. Prepare the list based on the above pointers.

Take stock of last year’s performance of existing live applications

Few applications would be in live for the past few years. Understand the challenges with respect to performance it faced and do the impact assessment based on past history. This exercise would give us the concrete requirement for Performance Testing for that application.

Defining Scope

After assessing the application nature and taking stock of past history, it gives us a requirement on what to test. Prepare NFR Requirements. This is nothing but the process of coming up with a well-documented test plan for that application. This phase essentially includes the identification of the non-functional requirements such as stated below

Whether the application is a multiuser purpose one or a limited user purpose one.,

Identification of key business transactions that are needed to test

Identify key blockers in the past from performance perspective and target to include in the testing scope

Forecasting the future load on certain services and conducting testing as needed

Tool identification

This is essential in the business; the right choice of tool will help in managing the cost as well as the hassle-free test conduction. The proper analysis must be done before we go for a selection that best suits the assignment.

Whether or not it’s an open-source tool or licensed

The tool that is selected should support any test activity with minimal configuration

The tool should help us in generating the best reporting mechanism

The tool must have got a decent community to get any solution for the problem being faced

To insist on the first point, the tool need not be freeware all the time, it’s completely a business call considering the high-end features that a licensed tool offers, no possibility for any breaches. If at all the test engineer identifies that the same can be accomplished by going with any other alternative tool that is available for free and supports the operation equally with a licensed one, then the choice of open-source one is desired as that way we can save the cost that is incurred. Below are some tools list

Popular Performance Testing tools are:

Jmeter (open source)

Load runner (commercial)

Silk performer (commercial)

Neo load (Commercial)

Application performance management tools:

App Dynamics

Wily-introscope

Perf mon

N-mon

Environment Readiness

This is one of the major considerations as far as the performance test is concerned. Unless and otherwise there is an isolated environment for NFT testing we can deem the test results and the application critical parameters (response time, think time, throughput..etc). So, a dedicated environment is highly recommended as the tasks more or related to

Insert a huge number of transactions in DB through API hit

Perform concurrent login scenario

Observe the response time while navigating between the pages

The environment must be maintained properly and should be highly stable

Risk assessment

It is quite obvious that on certain tasks, there are one or multiple hurdles seen down the line. We should foresee them during the initial discussions about the planning and proactively calling them out to all stakeholders will help come up with a stringent plan, that way we can reduce the negative impact on the business.

This concept is to learn the impact on a business if at all

Analyzing the dependency with the development and environmental team to get the support required during application being broke/ downtime instances

Going live with the presence of a defect that may occur at customer sight

Can’t test a particular scenario due to the unavailability of the infrastructure in the lower environments

Testing a feature by taking some deviation

Results analysis and report preparation

For any execution we make, test results are the key as they are the outcome of the effort that is being spent.

Post execution, results generated from the performance testing tool have to be analyzed and various parameters are assessed and a decent report in an easily understandable way has to be shared to all stakeholders on the outcome. We have tools which capture results of a test run, we can fairly extract from them.

Few instances, performance engineering activity also done by going one step further deep to understand where the blockers are and what could be the solution. This activity requires architectural understanding and finding the error-prone areas.

Written By

Submit a Comment

Your email address will not be published. Required fields are marked *


This blog is mainly going to cover performance testing and its strategy. Performance Testing helps the business to understand the stability and reliability of the application upon stringent circumstances.

What is performance Testing?

To define in laymen terms, a Performance test is conducted to mainly observe the behavior of application which is hosted on a server upon various loads. The goal is not to validate the functional flow and identify the bugs in that manner, rather test a maturity attained application under critical load, volume, stress conditions to verify the set benchmark parameters are satisfied before the application goes live.

Types of Performance Testing

In levels of testing, the performance test is classified under the non-functional testing , given the nature of the test. This is again categorized into various types of testing which are listed below

Load testing

This test determines the application behavior under a certain load. When we say load we can consider a couple of illustrations as the number of user hits per second or the number of transactions. The goal is to verify how the application performance when there is a load on the server.

Stress Testing

Unlike the above test, this is done by injecting the load higher than the threshold to see the behavior of the application; this can tell us the breakeven point of an application also the frequency of getting down.

Volume Testing

This test is also called as flood testing as we are going to inject too much of data into the application to see, how the queue or any other interface is withstanding the inflow

Endurance Testing

This testing is done by subjecting the application to a nominal load but to make it operate for quite longer than its usage in the field. This test gives the reliability of the application as we are considering the time that the application lasts before it sees failure.

Spike testing

This test is performed by applying a significant steep of raise load on the application to see the withstanding nature of an application.

Scalability testing

This testing is done to verify how the application behaving when the load is kept increasing by adding a lot of user data volume.

Why is performance testing needed?

Given the capabilities of functional testing, a performance test is more useful to understand the reliability parameters. The intention of the test is not just to find the defects but more about examining the application behavior under load. Spec is defined in such a way that it mainly denotes following

How many concurrent users can log in to the application?

How many transactions should the application be processing upon peak load?

How much minimum the response time must be of?

There are a few other parameters which are gathered as part of Non-Functional Requirements (NFR) gathering and tested accordingly.

Has there been no performance testing we wouldn’t be able to understand the capacity, flexibility and the critical parameters of an application under load and there are great chances that we fail in field due to the improper understanding of the benchmark parameters.

Parameters that are validated during test

When any test is conducted, we look for certain parameters as validation elements. In performance testing, these parameters are again considered as two parameters

Server-side parameters:-

CPU utilization

Memory utilization

Disc utilization

Garbage collector

Heap dump/ thread pooling

Client-side parameters:-

Load

Hits per second

Transactions per second

Response times

Entry & Exit criteria

Like every other type of test, performance test also defines its own entry and exit criteria.

Entry criteria

This imperatively defines, when the application is eligible to undergo a performance test. Following are the key pre-requisites to define entry criteria

Functional testing must have been thoroughly conducted

Application must be stable

No major showstoppers must have been in open state

When a dedicated NFT environment is made available

Needed test data should be ready

Exit criteria

This defines how and when to tear down the test, the following activities must be completed to give a proper sign-off from a performance test perspective

An application must have been tested with all the NFR requirements

No major defects or customer use cases are to be in failing state

All the test results should be properly analyzed

Reports and results must be published to all stakeholders for sign-off

After discussing about various parameters and basics about Performance Testing, we will now discuss in detail about Performance Testing Strategy

Performance Testing Strategy

Below are the key pointers to consider in defining the Performance Testing strategy

Understanding Organization Requirement

Assess the Application nature and category

Take stock of last year’s performance of existing live applications

Defining Scope

Tool identification

Environment readiness

Risk assessment

Results analysis and report preparation

Understanding Organization Requirement

Every Organization provides certain service to Customer and based on the services and the nature of assessing requirement to its Customer, Performance Testing requirement gain importance. For example, an e-commerce company is relying 100% on Customer hit rates and successful transactions. 1000+ customers login at a time to buy products. During Offer sale, this can turn to the peak of million customers login at a time. This type of Organization needs strong performance testing before hosting it live. Their spending on Performance tool is inevitable.

Assess the Application nature and category

Identify the # of Applications that are in life on that year and assess the application based on its nature, whether it is Customer facing or assessed by the back end-user or assessed by operations team.

This brings in the criticality of application for performance testing. Also, understand the technology in which it is built with and the number of releases planned with a criticality of release. Prepare the list based on the above pointers.

Take stock of last year’s performance of existing live applications

Few applications would be in live for the past few years. Understand the challenges with respect to performance it faced and do the impact assessment based on past history. This exercise would give us the concrete requirement for Performance Testing for that application.

Defining Scope

After assessing the application nature and taking stock of past history, it gives us a requirement on what to test. Prepare NFR Requirements. This is nothing but the process of coming up with a well-documented test plan for that application. This phase essentially includes the identification of the non-functional requirements such as stated below

Whether the application is a multiuser purpose one or a limited user purpose one.,

Identification of key business transactions that are needed to test

Identify key blockers in the past from performance perspective and target to include in the testing scope

Forecasting the future load on certain services and conducting testing as needed

Tool identification

This is essential in the business; the right choice of tool will help in managing the cost as well as the hassle-free test conduction. The proper analysis must be done before we go for a selection that best suits the assignment.

Whether or not it’s an open-source tool or licensed

The tool that is selected should support any test activity with minimal configuration

The tool should help us in generating the best reporting mechanism

The tool must have got a decent community to get any solution for the problem being faced

To insist on the first point, the tool need not be freeware all the time, it’s completely a business call considering the high-end features that a licensed tool offers, no possibility for any breaches. If at all the test engineer identifies that the same can be accomplished by going with any other alternative tool that is available for free and supports the operation equally with a licensed one, then the choice of open-source one is desired as that way we can save the cost that is incurred. Below are some tools list

Popular Performance Testing tools are:

Jmeter (open source)

Load runner (commercial)

Silk performer (commercial)

Neo load (Commercial)

Application performance management tools:

App Dynamics

Wily-introscope

Perf mon

N-mon

Environment Readiness

This is one of the major considerations as far as the performance test is concerned. Unless and otherwise there is an isolated environment for NFT testing we can deem the test results and the application critical parameters (response time, think time, throughput..etc). So, a dedicated environment is highly recommended as the tasks more or related to

Insert a huge number of transactions in DB through API hit

Perform concurrent login scenario

Observe the response time while navigating between the pages

The environment must be maintained properly and should be highly stable

Risk assessment

It is quite obvious that on certain tasks, there are one or multiple hurdles seen down the line. We should foresee them during the initial discussions about the planning and proactively calling them out to all stakeholders will help come up with a stringent plan, that way we can reduce the negative impact on the business.

This concept is to learn the impact on a business if at all

Analyzing the dependency with the development and environmental team to get the support required during application being broke/ downtime instances

Going live with the presence of a defect that may occur at customer sight

Can’t test a particular scenario due to the unavailability of the infrastructure in the lower environments

Testing a feature by taking some deviation

Results analysis and report preparation

For any execution we make, test results are the key as they are the outcome of the effort that is being spent.

Post execution, results generated from the performance testing tool have to be analyzed and various parameters are assessed and a decent report in an easily understandable way has to be shared to all stakeholders on the outcome. We have tools which capture results of a test run, we can fairly extract from them.

Few instances, performance engineering activity also done by going one step further deep to understand where the blockers are and what could be the solution. This activity requires architectural understanding and finding the error-prone areas.