Executing Qualio Validation

How to prepare for and execute validation steps

Written by Lydia Olu-Harding
Updated over a week ago

Customers who have evaluated Qualio’s validation approach and choose to accept our Computer Software Assurance model can use this article for step-by-step guidance. Alternatively, customers wishing to create their own validation documentation and testing can still leverage the resources and links provided in this article to assist their efforts.

Jump ahead to:

Validation Planning

Begin validation with a Plan; identify what features will be evaluated, who is doing what, any business risks associated, and what to do with the results.

1 - Some customers prefer to validate the Organization Settings, Documents, and Training first, and create an Interim Validation Summary Report for those features. They then continue to configure and validate the other features (Events, Suppliers, etc.) USING the validated Qualio Documents feature. They anticipate this and include details in the Validation Plan. Other customers deem these extra steps to be unnecessary.

2 - QMS Validation is different from medical and pharmaceutical process and systems validation because your eQMS software does not directly impact patient safety or product quality. Qualio does not make or market drugs or medical devices, so the guidelines followed by Qualio to produce eQMS software differ from your requirements. Items such as Change Management and Risk Assessment would fall under guidelines such as ISO 9001 and 27001, which focus on data integrity and security, as opposed to a regulation like ISO 13485 or 21CFR Part 210/211, which specifically apply to the manufacture and distribution of medical devices or drugs.

If you do not currently have a unique validation policy/process for a QMS, you can use Qualio’s in part or in full at your discretion.

3 - Qualio Validation Plan - Example
Qualio has developed a Validation Plan example for customers to use in part or full based on their review of the document and its alignment with their organization's policies and procedures. Also, carefully review the sections describing the Business Processes and Risk Assessment. Evaluate for consistency with your intended use of Qualio.

Document Business Processes & Risks

Computerized System Assurance guidelines heavily emphasize the importance of documenting QMS processes and associated risks. A risk assessment of your business process is recommended to help you scope any additional testing activities you may undertake. As we’ve seen, system activity like document control, training, CAPAs, and supplier management do not generally have any direct impact to product quality or patient safety. With that in mind, it’s your responsibility to pinpoint and work on any system processes you deem of higher risk.

The Validation Plan and Summary Report provided by Qualio address the baseline business processes and associated risks to product and patient safety from the QMS perspective.

4 - Qualio uses a single validation template encompassing the Validation Plan, Feature Validation, and Validation Summary Report related to the implementation and maintenance of Qualio. This template, named Qualio Validation Document (QVAL-), has a single section. The text within each QVAL will reflect the information in the baseline templates found in the Supporting Documents Dropbox Folder.

5 - Some customers may choose to use their existing document control process (outside Qualio) to approve the Validation Plan because they don’t consider Qualio “operational” until it has been fully validated. Other customers agree that CSA principles are sufficiently satisfied due to multi-tenancy and proceed with creating and approving the Validation Plan within Qualio. This is the customer’s discretion and responsibility. Notify your COM of your decision and work with them to clarify who is doing what.

Test Evidence, Documentation & Configuration (if applicable)

Qualio performs documented testing as part of our software release process. We make this available to customers to use in order to streamline their CSA since the majority of customers use Qualio as we intend for the software to be used.

Test evidence and documentation is organized by feature and is stored in a Dropbox folder accessible to customers. During this validation step, we recommend customers view the latest version of test evidence and documentation for each feature they are implementing.

Below is an example of the automated test video evidence. On the left, you can see the test script being executed. And on the right, you can see the app view of Qualio where the test is being performed. Screenshots of the final result are also included in the Test Evidence folder. More information on understanding test evidence is below.

In addition to test evidence like videos and screenshots, we provide the following documentation for each release:

  • User Needs Document PDF
    Lists the feature’s requirements specific to that release

  • User Story Document PDF
    Lists the software design specifications of the feature specific to that release

  • Test Plan PDF
    The Test Plan summarizes all test scripts as written, and describes the automated testing tools used to capture the "passing" results, along with the date the testing was performed. The test plan is approved under our quality system umbrella, and the output is attached here for your review.

  • Requirements Traceability Matrix (RTM) XLSX
    The RTM illustrates the traceability between User Needs (product requirements) to User Stories (design specification to address the user needs) to the tests confirming the design meets the needs. Any additional user needs are the customer’s responsibility to document, assess risk and test.

  • Qualio Features Validation document
    Details described below in list item #8.

  • Change History PDF

    The Change History file is a record of new, changed, and deleted User Needs, User Stories, and Tests from the previous Launch Train. Learn more about the Change History file below along with suggestions on how to identify substantial changes.

To learn more about test evidence folder structure, tips on viewing test evidence, and included documentation, view our Validation Resources article.

6 - Feature Val-Packs (test evidence & documents)
See Understanding Test Evidence below for more details.

This document describes which features are being implemented in your Production account. Links to the Dropbox where test evidence and documentation is located is described in this document along with a flowchart of how this evidence is generated by Qualio’s Design Control tool. View an example in our Support Documentation Dropbox folder.

8 - QMS Platform Configuration - Example
Because Qualio is a GAMP 5 Category 4 software, there are some components of Qualio which customers will configure to match their business processes. Examples include the Suppliers Policy and Design Control integrations. These configurations should be documented for change control purposes.

The Configuration file contains

  • Organization Configurations

  • Document Configurations

  • Training Configurations

  • Event Configurations (if applicable)

  • Supplier Configurations (if applicable)

  • Design Control Configuration (if applicable)

Understanding Test Evidence

Qualio manages the tools used to support validation efforts as controlled suppliers, and have qualified the tools for use. The relevant tools here are JIRA and Circle CI. We are happy to make these qualifications available to customers for review during the supplier qualification process.

Our requirements and testing are written using software industry standard methodology, and this methodology aligns nicely with validation principles. We utilize "Epics", "stories", and "user needs" to capture the requirements to which the software is built. We write testing of these functions of the software in tools that then perform the testing in an automated way. The testing language is known as "gherkin", which sets the test up in a standard way. See the example below.



**Scenario**: Change issue owner

**Given** I log in as a Normal user

**Given** I open an issue details

**And** I click change owner

**When** I assign issue to different user

**Then** 'Owner changed successfully.' notification is displayed

This example is following simple rules:

  • Sentence starting with keyword Given describes the initial point of the test

  • Sentence starting with keyword When describes the key action taken in this test (normally we would like to test only one action in test, however in E2E tests we tend to test the process with multiple actions). This sentence should contain "Test Procedure" - or the actions to be performed to test the requirement.

  • Sentence starting with keyword Then describes the expected behavior. This sentence should contain (if possible) the "Expected Results" , which maps to what we used to call “as expected/as found” in old validation methodology.

  • Keyword followed by "I" means that the user is taking an action (clicks, types, etc)

  • If the keyword is not followed by "I", it means that the action is taken by the system.

The testing software executes all the tests in a fraction of the time it takes a person to run, and if a test fails, it is clearly flagged and the run will stop. Successful passing of the tests is required before the changes are incorporated into the application on staging. If all is successful on staging, the code is deployed to production, and the testing will run again as part of that code merge.

The recordings of the automated test running are located in the referenced dropbox file as the evidence of test execution. They are named by the requirement at test, the number that corresponds can be found in the record itself, it will have the "[feature abbreviation]-XXX" identifier.

Additionally, screenshots of final outcomes have been added to the folder for further verification where the video may have ended too quickly to verify. Please keep in mind that not every single requirement will have an individual test, as workflows are grouped together to ensure appropriate testing of the system functions are captured. In other words, related items may be grouped for testing. Coverage across these elements is shown in the "RTM" document.

Lastly, it's important to note that the numbers are not sequential coming out of Jira. Every ticket created in the system is numbered, but not every ticket is a requirement. Bug fixes are also tickets in the same project, and all tickets, regardless of type, are numbered sequentially.

Understanding the Change History file

Qualio releases new feature and feature updates on scheduled, cadenced releases called Launch Trains. To support the newly released features, a Val-pack is generated for each Launch Train and represents a complete validation of all Qualio features and functionality. This simplifies validation for new customers because they will simply use the latest Launch Train Val-Pack documents and evidence. But existing customers want to know what changed. That’s where the Change History file comes into play.

Change History files are another auto-generated output of Qualio’s Design Controls feature. It’s a record of new, changed, and deleted User Needs, User Stories, and Tests. Here’s a couple things to keep in mind as you view a Change History file.

  • Non-updated features may have documentation only updates that may not be called out in the release notes. You will, however, see this noted in the Change History document.

  • You will see multiple “DCC’s” in those Change History documents. Many of these are “documentation” only updates, meaning we are pulling from Jira the updates to the requirements for typos, grammar errors, or other “non functional” changes.

  • You will always see ALL tests linked to the Change History and flagged by the system as ‘CHANGED’. This is due to the most recent date test execution being flagged by Jira as a “change”, which it then pushes to the Design Control tool. This does not mean that every test has changed. You can simplify your review of the updates by focusing on the User Needs and User Story updates. We also occasionally consolidate tests, and we will call that out in the Change History file, which will display a test as “Deleted” (see image below).

Change History Document - EXAMPLE

How should I save test evidence for my validation records?

Because Dropbox folders with test evidence will remain available to customers in perpetuity, we actually don’t recommend customers waste time with recreating the test evidence. That is of course an option and up to the discretion of each customer. However, we recommend simply including the link to the latest dropbox folder (feature and version specific) in the [feature] Validation Document. Those links are readily accessible for an auditors requirements.

Validation Summary Report

This final report should summarize validation activities and results to confirm that the Validation Plan was properly executed. We recommend creating a Validation Summary Report (example available in the Support Documentation Dropbox folder) utilizing the Qualio Validation Document template (QVAL-) to generate this Qualio document and include the following items:

  • Smartlink to the Qualio Feature Validation document

  • Attached QMS Configuration document

  • Smartlinks to your organization’s policies and procedures relevant to the validation process, such as the e-signature policy, Change Management, and Supplier Qualification policies and procedures.

  • Link to Dropbox folder containing Qualio’s policies and certifications, which Qualio maintains with updated versions for the support of our customers

The Validation Summary Report document can then be reviewed, approved, and stored in the customer's instance of Qualio. Discuss with your COM how you would like to document your validation summary.

Revalidation Process

Other than initial validation, some feature updates will require revalidation. Because the Qualio Validation Support pack is divided by feature, when revalidation becomes suggested/necessary, customers can access and complete revalidation of those features exclusively. It is impossible to prescribe how often revalidation will be required. However, the Qualio Product and Quality teams are cognizant of the resources required to revalidate features and attempt to minimize and condense whenever possible.

Additionally, our Quality Director will review each Launch Train for which features may require revalidation, and this will be annotated in the Release Notes of that Launch Train.

Recommendations posted in the Release Notes represent the standard use of Qualio and the typical customer setup. We strongly recommend that each customer review the validation note of each item listed in the Release Notes and make a final judgment if revalidation is necessary.

When revalidation becomes necessary, Quality Users will be notified and provided with Dropbox links to the updated version of validation test evidence and documentation. The new feature/functionality will first be released only to sandboxes on the date of the notification for a 30-day validation period. At the end of the 30 days, the feature will be released/pushed to production accounts.

Customers should take the following steps during the 30-day validation period:

  • Review new test evidence and documentation

  • Observe new functionality in their sandbox environment and complete any additional testing

  • Update¹ your internal process documentation/training related to the feature

  • Update¹ the Configuration document if necessary

  • Update¹ the Validation Summary Report to document what changes were made if necessary

1. Any validation document updates will require the Qualio document to be reverted to draft, updated, then sent for review and approval.

Assistance & Next Steps

This article helped you to prepare for and execute Qualio Validation, but you may still have questions about validation or concerns over the new method. Review our FAQ article for common questions. And if you still cannot find the answer to your question, reach out to Qualio Support.

Did this answer your question?