The testing process, which was once confined to pre-production environments, has shifted to testing in production. The evolution of test in production has provided organizations with numerous benefits, including faster time-to-market and improved quality. This article aims to explore the evolution of testing in production, starting from the initial stages to the current stage.


Testing only in Production

In the early days of software development, testing was mostly done in production. There were no test environments or dedicated QA team. The software was developed on physical servers, and monolithic applications were the norm. Testing was done yearly, following the waterfall testing approach. The lack of a dedicated testing environment and the use of physical servers made testing challenging. Issues discovered in production were costly to fix, and the time taken to fix them was often long.


Testing in Production with Limited Users

As software development evolved, virtualization became the norm, and independent QA teams emerged. Applications were developed using the N-Tier architecture, and dedicated test and staging environments were created. Monthly releases replaced yearly releases, and Agile Testing methodologies became popular.


Testing in production still occurred, but the testing was done with limited or no users to minimize the impact of issues discovered. In the retail industry, a point of sale (POS) system is critical for completing transactions and conducting sales. Any issues or errors with the POS system can result in a significant loss of revenue for the organization. To ensure that the POS system is functioning correctly, retail organizations typically perform testing before opening the store for the day.


In this scenario, the retail organization is using testers to validate the features and workflows of a new POS release in the production environment. The testing is performed before the store opens, so there are no real production users during the testing period. If the testers identify any issues or errors during their testing, the team will roll back the build to the previous stable version. This approach provides confidence to the users that the system will function correctly and reduces the risk of failure.


Test in Production in the Digital Era

Today, software development has shifted to microservices and serverless architecture. Organizations have integrated QA into the software development process, and cloud infrastructure has become the norm. Daily releases have replaced monthly releases, and DevOps practices have emerged. Testing in production is now done in real-time, with no limitations around application configurations, interfaces, or infrastructure.


For example, consider a ride-sharing application. When a user requests a ride, the application performs various actions, such as calculating the fare, identifying the nearest driver, and updating the driver’s location. These actions must be performed in real-time, and any delays or issues can result in a poor user experience. Testing in production ensures that the application performs as expected in a real-world scenario.


Benefits of Testing in Production

Drawing from the various stages of evolution in test in production , the practice offers organizations distinct advantages in today’s complex software landscape. It is also important to know about staging environment vs production environment  and its limitations while defining your testing startegy. One of the most impactful benefits is undoubtedly the ability for quicker time-to-market. In my experience, testing in a live environment allows for more accurate data, which helps development and QA teams make faster and more informed decisions, thereby accelerating the release cycle. This aligns well with today’s Agile and DevOps methodologies that prioritize continuous delivery.


Another major benefit lies in quality improvement. In the past, I’ve observed that testing in isolated environments sometimes fails to expose certain issues that only manifest in real-world conditions. When testing occurs in production, the uncovered insights are immediately actionable and relevant, reducing the risk of post-release defects.


Lastly, real-time data collection during production testing offers a goldmine of information, right from system performance to user behavior. This production feeback data is invaluable for not just troubleshooting, but also for strategizing future development cycles.


Key Inferences on Testing In Production

The evolution of testing in production has provided organizations with numerous benefits, including faster time-to-market, improved quality, and real-time data. Testing in production is now the norm and organizations that fail to adopt this approach risk falling behind their competitors. As software development continues to evolve, testing in production will continue to play a crucial role in ensuring that software performs as expected in a real-world scenario.


Leave a Reply