Customers who have evaluated Qualio’s validation approach and choose to accept our Computer Software Assurance model can use this article for step-by-step guidance. Alternatively, customers wishing to create their own validation documentation and testing can still leverage the resources and links provided in this article to assist their efforts.

Jump ahead to:

Begin validation with a Plan; identify what features will be evaluated, who is doing what, any business risks associated, and what to do with the results.

1 - Some customers prefer to validate the Organization Settings, Documents, and Training first, and create an Interim Validation Summary Report for those features. They then continue to configure and validate the other features (Events, Suppliers, etc.) USING the validated Qualio Documents feature. They anticipate this and include details in the Validation Plan. Other customers deem these extra steps to be unnecessary.

2 - QMS Validation is different from medical and pharmaceutical process and systems validation because your eQMS software does not directly impact patient safety or product quality. Qualio does not make or market drugs or medical devices, so the guidelines followed by Qualio to produce eQMS software differ from your requirements. Items such as Change Management and Risk Assessment would fall under guidelines such as ISO 9001 and 27001, which focus on data integrity and security, as opposed to a regulation like ISO 13485 or 21CFR Part 210/211, which specifically apply to the manufacture and distribution of medical devices or drugs.

If you do not currently have a unique validation policy/process for a QMS, you can use Qualio’s in part or in full at your discretion.

3 - Qualio Validation Plan - Example
​Qualio has developed a Validation Plan example for customers to use in part or full based on their review of the document and its alignment with their organization's policies and procedures. Also, carefully review the sections describing the Business Processes and Risk Assessment. Evaluate for consistency with your intended use of Qualio.

Document Business Processes & Risks

Computerized System Assurance guidelines heavily emphasize the importance of documenting QMS processes and associated risks. A risk assessment of your business process is recommended to help you scope any additional testing activities you may undertake. As we’ve seen, system activity like document control, training, CAPAs, and supplier management do not generally have any direct impact to product quality or patient safety. With that in mind, it’s your responsibility to pinpoint and work on any system processes you deem of higher risk.

The Validation Plan and Summary Report provided by Qualio address the baseline business processes and associated risks to product and patient safety from the QMS perspective.

4 - Qualio uses a single validation template encompassing the Validation Plan, Feature Validation, and Validation Summary Report related to the implementation and maintenance of Qualio. This template, named Qualio Validation Document (QVAL-), has a single section. The text within each QVAL will reflect the information in the baseline templates found in the Supporting Documents Dropbox Folder.

5 - Some customers may choose to use their existing document control process (outside Qualio) to approve the Validation Plan because they don’t consider Qualio “operational” until it has been fully validated. Other customers agree that CSA principles are sufficiently satisfied due to multi-tenancy and proceed with creating and approving the Validation Plan within Qualio. This is the customer’s discretion and responsibility. Notify your COM of your decision and work with them to clarify who is doing what.

Qualio performs documented testing as part of our software release process. We make this available to customers to use in order to streamline their CSA since the majority of customers use Qualio as we intend for the software to be used.

Test evidence and documentation is organized by feature and is stored in a Dropbox folder accessible to customers. During this validation step, we recommend customers view the latest version of test evidence and documentation for each feature they are implementing.

Below is an example of the automated test video evidence. On the left, you can see the test script being executed. And on the right, you can see the app view of Qualio where the test is being performed. Screenshots of the final result are also included in the Test Evidence folder. More information on understanding test evidence is below.

In addition to test evidence like videos and screenshots, we provide the following documentation for each release:

To learn more about test evidence folder structure, tips on viewing test evidence, and included documentation, view our Validation Resources article.

6 - Feature Val-Packs (test evidence & documents)
See Understanding Test Evidence below for more details.

7 - Qualio Features Validation - Example

This document describes which features are being implemented in your Production account. Links to the Dropbox where test evidence and documentation is located is described in this document along with a flowchart of how this evidence is generated by Qualio’s Design Control tool. View an example in our Support Documentation Dropbox folder.

8 - QMS Platform Configuration - Example
​Because Qualio is a GAMP 5 Category 4 software, there are some components of Qualio which customers will configure to match their business processes. Examples include the Suppliers Policy and Design Control integrations. These configurations should be documented for change control purposes.

The Configuration file contains

Qualio manages the tools used to support validation efforts as controlled suppliers, and have qualified the tools for use. The relevant tools here are JIRA and Circle CI. We are happy to make these qualifications available to customers for review during the supplier qualification process.

Our requirements and testing are written using software industry standard methodology, and this methodology aligns nicely with validation principles. We utilize "Epics", "stories", and "user needs" to capture the requirements to which the software is built. We write testing of these functions of the software in tools that then perform the testing in an automated way. The testing language is known as "gherkin", which sets the test up in a standard way. See the example below.

This example is following simple rules:

The testing software executes all the tests in a fraction of the time it takes a person to run, and if a test fails, it is clearly flagged and the run will stop. Successful passing of the tests is required before the changes are incorporated into the application on staging. If all is successful on staging, the code is deployed to production, and the testing will run again as part of that code merge.

The recordings of the automated test running are located in the referenced dropbox file as the evidence of test execution. They are named by the requirement at test, the number that corresponds can be found in the record itself, it will have the "[feature abbreviation]-XXX" identifier.

Additionally, screenshots of final outcomes have been added to the folder for further verification where the video may have ended too quickly to verify. Please keep in mind that not every single requirement will have an individual test, as workflows are grouped together to ensure appropriate testing of the system functions are captured. In other words, related items may be grouped for testing. Coverage across these elements is shown in the "RTM" document.

Lastly, it's important to note that the numbers are not sequential coming out of Jira. Every ticket created in the system is numbered, but not every ticket is a requirement. Bug fixes are also tickets in the same project, and all tickets, regardless of type, are numbered sequentially.

Qualio releases new feature and feature updates on scheduled, cadenced releases called Launch Trains. To support the newly released features, a Val-pack is generated for each Launch Train and represents a complete validation of all Qualio features and functionality. This simplifies validation for new customers because they will simply use the latest Launch Train Val-Pack documents and evidence. But existing customers want to know what changed. That’s where the Change History file comes into play.

Change History files are another auto-generated output of Qualio’s Design Controls feature. It’s a record of new, changed, and deleted User Needs, User Stories, and Tests. Here’s a couple things to keep in mind as you view a Change History file.

Change History Document - EXAMPLE

How should I save test evidence for my validation records?

Because Dropbox folders with test evidence will remain available to customers in perpetuity, we actually don’t recommend customers waste time with recreating the test evidence. That is of course an option and up to the discretion of each customer. However, we recommend simply including the link to the latest dropbox folder (feature and version specific) in the [feature] Validation Document. Those links are readily accessible for an auditors requirements.

This final report should summarize validation activities and results to confirm that the Validation Plan was properly executed. We recommend creating a Validation Summary Report (example available in the Support Documentation Dropbox folder) utilizing the Qualio Validation Document template (QVAL-) to generate this Qualio document and include the following items:

The Validation Summary Report document can then be reviewed, approved, and stored in the customer's instance of Qualio. Discuss with your COM how you would like to document your validation summary.

Other than initial validation, some feature updates will require revalidation. Because the Qualio Validation Support pack is divided by feature, when revalidation becomes suggested/necessary, customers can access and complete revalidation of those features exclusively. It is impossible to prescribe how often revalidation will be required. However, the Qualio Product and Quality teams are cognizant of the resources required to revalidate features and attempt to minimize and condense whenever possible.

Additionally, our Quality Director will review each Launch Train for which features may require revalidation, and this will be annotated in the Release Notes of that Launch Train.

Recommendations posted in the Release Notes represent the standard use of Qualio and the typical customer setup. We strongly recommend that each customer review the validation note of each item listed in the Release Notes and make a final judgment if revalidation is necessary.

When revalidation becomes necessary, Quality Users will be notified and provided with Dropbox links to the updated version of validation test evidence and documentation. The new feature/functionality will first be released only to sandboxes on the date of the notification for a 30-day validation period. At the end of the 30 days, the feature will be released/pushed to production accounts.

Customers should take the following steps during the 30-day validation period:

1. Any validation document updates will require the Qualio document to be reverted to draft, updated, then sent for review and approval.

This article helped you to prepare for and execute Qualio Validation, but you may still have questions about validation or concerns over the new method. Review our FAQ article for common questions. And if you still cannot find the answer to your question, reach out to Qualio Support.