Introduction to High Risk Software Automated Testing

Brett Castano
Note: The views expressed in this article are those of the author and do not necessarily represent those of his employer, GxP Lifeline, its editor or MasterControl Inc.

In today's world, high risk software applications are managing a majority of clinical trial data collected in the field. In the early 2000s, the clinical trial industry saw a switch from managing the data collection process through manual means to doctors utilizing secure software applications that allow the doctor to enter the data electronically through the use of a computer or mobile device. MasterControl itself is an example of a high risk software system that is utilized within the clinical trial industry, as it is focused on creating and regulating the process by which clinical trial documentation is managed. In this article, applications created by vendors (such as MasterControl) that will be utilized by doctors or other managers in the field to aide and assist in the management of clinical trials will be referred to as eClinical software, while the collective group of software providers comprise the eClinical industry (

As one may imagine, there are strict regulations as to how eClinical software is created, tested, and delivered to pharmaceutical companies. Should one of these high risk eClinical software systems malfunction, an entire clinical trial's data could be called into question by the FDA.

The change from the antiquated manual process of clinical trial data collection has come with many benefits for pharmaceutical companies, but primarily the biggest benefit is a decrease in time for study data collection, which leads to a decrease in time of trial data analysis and subsequent earlier submission of the clinical trial to the FDA for approval of a new drug, etc. In the pharmaceutical industry, time really is money, since drug patents only last a finite amount of time before generics of the same medication or process are made. After a clinical trial is approved to be conducted on human subjects by the appropriate governing body, it usually takes between 12 and 16 weeks for a supporting eClinical company to create and validate a supporting clinical trial software system - and often, there are more than three of these eClinical software systems, serving different functions, for each clinical trial. This means that the pharmaceutical company must wait those 12 to 16 weeks before being able to register the first patient in their newly created software. However, what if there was a technology solution that could shorten the twelve to sixteen weeks for software creation to four to six weeks, or even less?

As one may imagine, there are strict regulations as to how eClinical software is created, tested, and delivered to pharmaceutical companies ( Should one of these high risk eClinical software systems malfunction, an entire clinical trial's data could be called into question by the FDA. Therefore, development and testing processes, methodologies, and technology solutions utilized in the creation of eClinical software remain at the forefront of most conversations.

The scope of this article will focus on the software testing process since the time spent validating applications plays an integral role in any software provider's decision-making process. The software testing process plays a major role in any software development life cycle (SDLC), accounting for approximately 60% of the software creation and process time. Continuously refining and improving testing methodologies is critical for eClinical businesses wanting to increase both customer satisfaction and profitability. Sponsor requests for more system functionality have caused eClinical software to become increasingly complex, with more frequent release cycles. As a result, the probability of software defects has increased. Software defects in the eClinical industry are a constant source of anxiety for software providers since defects in critical areas of the software could cause an entire trial to be nullified by the FDA. Currently, there is a problem that exists for most software vendors in the eClinical space: pharmaceutical companies are demanding additional and complex functionality to be delivered in the same or reduced timelines without sacrificing software quality.

Software vendors can look at this impasse three ways: by managing client satisfaction and expectations at the risk of losing the client to competitors; increasing spending on software testing resources that are typically outsourced - leading to questionable results; or adapting their current processes and SDLC via a technology solution in order to find efficiencies that will produce the desired result. To this point, the eClinical software industry has mainly relied on manual functional software testing practices which entail a human tester following handwritten instructions to navigate a software application and database. While the manual testing process may find many defects in software applications, it has proven to be a laborious, costly, frustrating, inconsistent, and time-consuming process that does not guarantee results or the ability to fully regression test a software system.

By implementing automated functional testing, an eClinical software provider can dramatically increase both the speed and accuracy of software testing, providing a higher return on investment (ROI) from software projects while dramatically cutting the risk of defects. This article will introduce the reader to the industry of eClinical software and its pressure points, and concludes with an overview of the advantages and challenges of implementing automated functional testing, in addition to an assessment aimed at identifying if automated software testing is right for you and your company.

Automated Functional Testing Advantages and Challenges

There is no question that rigorous functional testing, manual or automated, is critical to successful application development. The challenge for developers, QA teams, and management alike is how to speed up testing processes and increase accuracy and completeness—without breaking budgets. By automating key elements of functional testing, a company can meet aggressive release schedules; test more thoroughly and reliably; regression test parts of or the entire application to achieve ultimate quality control; verify that business processes function correctly; and ultimately generate higher revenue and customer satisfaction from its online operations by condensing its Software Development Life Cycle (see Figure A).

Figure A

Functional Test Automation Concerns

The concerns of functional test automation are mostly rooted in adopting a new application, including:

  • What are the costs of automating the testing processes and what is the return on investment (ROI)?
  • Which applications/processes are candidates for automated testing and which are not?
  • Will new training be required and how will that impact current development project schedules?
  • What is the proper methodology or best practice for automating a testing effort?
  • What is involved in maintaining automated tests?
  • How does this fit in with current Standard Operating Procedures (SOP(s)) and will current operational business processes have to be updated?

  • When comparing test automation products, which features are most important for my business and software application?

First, let's step back and explain the concept of automated testing. Automated software testing is already being used in every industry where software helps perform a function. When you purchase products online from, you are using a hosted software solution provided by Amazon. In order to ensure that the purchasing process goes smoothly for its users, Amazon deploys automated software testing to test their software before you, the consumer, utilize their application. Automated software testing is the process of creating software testing instructions, known as automation test scripts. Once completed, automation test scripts will be able to programmatically perform the software testing instructions as they interact with a desired application or program. In contrast, the manual process utilizes the same written testing instruction method; however, a human must then follow the written testing instructions rather than having the testing accomplished programmatically - double the effort of automated testing. Once automation test scripts have been created, they can be rerun effortlessly, repeatedly and most important, cost effectively for software products that have a long maintenance life (see Figure B).

Figure B


Why automate functional testing processes? The pressure on today's biotechs' and clinical research organizations' IT departments continues to build. Management is looking to new technologies to deliver new capabilities, unlock new business opportunities, and deliver competitive advantages through software. This means the addition of more business application development projects with tighter deadlines—but not always with more budget or resources. At the same time, management is becoming progressively more aware of the critical links between software and revenue. Multiply that awareness by the increasing demand that preexisting applications may need to work on different operating systems, use several different front-end interfaces, involve numerous business processes, and interact with many separate data sets. Trying all possible permutations of these components creates a highly complex testing situation with hundreds or thousands of testing scenarios. When software fails, the expense can be extremely high—from lost sales to unproductive employees to unhappy customers, at-risk patients and demoralized development and quality assurance (QA) teams. The later in the development cycle defects are found, the more expensive they are. A defect identified in a production environment can be greater than 100 times more expensive to fix than the same defect identified earlier in the design phase.

Clearly, automated software testing is a key to improving the speed, accuracy, and flexibility of the software testing process, enabling your company to find and fix more defects earlier and increase efficiency.

Functional Test Automation Benefits

Speed, accuracy and flexibility are only three benefits of functional test automation. Other advantages include:

  • Reduced risk. By providing more comprehensive testing coverage, automated functional testing reduces the risk of production failures and increases ROI.
  • Enable faster execution. Computers are orders of magnitude faster than humans at executing functional test scripts, so more tests can be executed in less time, more applications can be tested in a given period, and more projects can be delivered on schedule. And unlike humans, computers work 24 hours a day, including nights, weekends, and holidays; they don't get bored or tired; and they don't make assumptions about what works and what doesn't. Here, ROI can be measured in terms of time and man-days.
  • Facilitate greater test coverage. Automated functional testing products support the execution of test scripts across all popular browsers, operating systems, etc. Regression testing of ever-changing applications and environments with automated tools is easier than with manual processes. And with integrated data-driven spreadsheet capabilities, test automation products allow your developers and QA teams to perform calculations, manipulate data sets, and quickly create multiple test iterations to expand test case coverage. With automated testing tools, any mix of transactions and any user workload can be quickly emulated.
  • Deliver higher test accuracy and find more defects earlier — even on the most dynamic of software applications. Test automation gives your developers an easier way to replicate and document software defects, helping expedite development processes while verifying correct functionality across all environments, data sets, and business processes.
  • Provide formalized processes. The introduction of automated testing encourages your test teams to formalize their processes, resulting in higher test consistency and better documentation records.
  • Facilitate the re-use of tests. Once tests are scripted, your developers can use, re-use, and add to the test suite as they make changes to their applications. There is no need to recreate scripts to test the same functionality for each application modification.

  • Support ERP/CRM implementations. As more companies are implementing ERP/CRM solutions, the multiple builds/patches/fixes need to be tested on numerous hardware or software configurations. In addition, if repetitive tasks such as data loading and system configuration are involved, or if the application needs to meet a specific service-level agreement (SLA), automation will certainly make economic sense.

Identifying Candidacy

In general, it makes sense to focus automation efforts on critical business processes, complex applications, and the use-cases that comprise them (as opposed to lower-level tasks, such as system-level verifications). If an enterprise is dedicating resources to testing software each day and is still finding quality and functional problems in production, the move to automated testing should be high on the priority list. The decision of whether to automate the software testing for a business is usually driven by ROI considerations. In general, there will be a positive return if the application requires any regression testing during its lifetime. When it comes to automating functional testing processes, the costs are tangible but the net value also includes many intangible factors. The best approach is to determine with as much precision as possible what the hard-dollar costs are and then compare them to the hard- and soft-dollar benefits of automating the test efforts.

Brett Castano is the Clinical Operations Manager at RealityCorp He has extensive experience in interactive web and voice response technologies, electronic document management systems and electronic data capture systems. The goal of RealityCorp can be found in its slogan - innovative eClinical solutions. ClinTest™, RealityCorp's proprietary automation platform, provides a targeted, next-generation solution for all testing, validation, reporting, and quality assurance needs and regulations. By adopting ClinTest™ as a business solution, clients expedite the task of eClinical software validation, thereby shortening the software development life cycle as a whole — resulting in achieving 'Go-Live' faster. The ClinTest™ Automation Suite has been utilized on a wide variety of eClinical software including EDC, IWR, IVR, EDM and CTM Systems; in addition to applications from the insurance, government, and financial industries. RealityCorp is a MasterControl Partner Company. For more information about automated software testing, call Brett Castano at (302) 995-2770 or visit the company's website at