Customers who have evaluated Qualio’s validation approach and choose to accept our Computer Software Assurance model can use this article for step-by-step guidance. Alternatively, customers wishing to create their own validation documentation and testing can still leverage the resources and links provided in this article to assist their efforts.
Jump ahead to:
Begin validation with a Plan; identify what features will be evaluated, who is doing what, and what to do with the results.
1 - Some customers prefer to validate the Organization Settings, Documents and Training first, and create an Interim Validation Summary Report for those features. They then continue to configure and validate the other features (Events, Suppliers, etc.) using the validated Documents feature. They anticipate this and include details in the Validation Plan. Other customers deem these extra steps to be unnecessary.
2 - QMS Validation is different from medical and pharmaceutical process and systems validation because your eQMS software does not have a direct impact on the patient safety or product quality. Qualio does not make or market drugs or medical devices, so the guidelines followed by Qualio to produce eQMS software are different than your requirements. Items such as Change Management and Risk Assessment would fall under guidelines such as ISO 9001 and 27001 which focus on data integrity and security, as opposed to a regulation like ISO 13485 or 21CFR Part 210/211, which specifically apply to the manufacture and distribution of medical devices or drugs.
If you do not currently have a unique validation policy/process for a QMS, you can use Qualio’s in part or full at your discretion.
3 - Master Validation Plan Template
Qualio has developed a Validation Plan document for customers to use in part or full based on their review of the document and its alignment with their organization policies and procedures.
4 - Discuss with your Customer Onboarding Manager (COM) what document templates will be created in Qualio to support the validation process. Some customers may already have a Validation template/defined layout, while other customers are new to validation and looking for template guidance and best practices.
Qualio recommends creating three validation templates which can also be used to validate other systems, not just Qualio. This recommendation allows for more specific document formatting. However, customers may decide that a single, generalized validation template is sufficient.
Validation Master Plan
Validation Summary Report
5 - Some customers may choose to use their existing document control process (outside Qualio) to approve the Validation Plan because they don’t consider Qualio “operational” until it has been fully validated. Other customers agree that CSA principles are sufficiently satisfied due to multi-tenancy and proceed with creating and approving the Validation Plan within Qualio. This is the customer’s discretion and responsibility. Notify your COM of your decision and work with them to clarify who is doing what.
Document Business Processes & Risks
Computerized System Assurance guidelines heavily emphasize the importance of having QMS processes and associated risks documented. Risk assessment of your business process is recommended to help you scope any additional testing activities you may undertake. As we’ve seen, system activity like document control, training, CAPAs and supplier management do not generally have any direct impact to product quality or patient safety. With that in mind, it’s your responsibility to pinpoint and work on any system processes you deem of higher risk.
6 - Qualio Risk Assessment Template
Qualio has developed a template for customers to use in part or full based on their QMS processes. Customers should review this template and modify as needed for any additional risks unique to their organization. This sample image below displays the Documents feature, but you can view the whole template here.
Because Qualio is a GAMP 5 Category 4 software, there are components of Qualio which customers can configure to match their business processes. Examples include the Suppliers Policy and different event workflows. These configurations should be documented for change control purposes.
During onboarding projects, this step of validation is performed for each feature implemented as well as the next step described for Test Evidence. That means these two steps are not a single event at the beginning of the onboarding process since you won’t know what configuration to document until you dive into that feature with your COM.
COMs are responsible for documenting each customer’s baseline configuration of Qualio. Once Onboarding is complete, customers will need to update this Qualio document whenever configuration changes occur. Discuss with your COM what document template should be used to create this document. We recommend the Validation Record document template.
Test Evidence & Documentation
Qualio performs documented testing as part of our software release process. We make this available to customers to use in order to streamline their CSA since the majority of customers use Qualio as we intend for the software to be used.
Test evidence and documentation is organized by feature and is stored in a Dropbox accessible to customers. During this validation step, we recommend customers view the latest version of test evidence and documentation for each feature they are implementing.
Below is an example of the automated test video evidence. On the left, you can see the test script being executed. And on the right, you can see the app view of Qualio where the test is being performed. Screenshots of the final result are also included in the Test Evidence folder. More information on understanding test evidence is below.
In addition to test evidence like videos and screenshots, we provide the following documentation for each release:
User Needs Document PDF
Lists the feature’s requirements specific to that release
User Story Document PDF
Lists the software design specifications of the feature specific to that release
Test Plan PDF
The Test Plan summarizes all test scripts as written, and describes the automated testing tools used to capture the "passing" results, along with the date the testing was performed. The test plan is approved under our quality system umbrella, and the output is attached here for your review.
Requirements Traceability Matrix (RTM) XLSX
The RTM illustrates the traceability between User Needs (product requirements) to User Stories (design specification to address the user needs) to the tests confirming the design meets the needs. Any additional user needs are the customer’s responsibility to document, assess risk and test.
[feature] Validation Document
Details described below in list item #9.
To learn more about test evidence folder structure, tips on viewing test evidence, and included documentation, view our Validation Resources article.
8 - [Feature] Val-Pack (test evidence & documents)
9 - [Feature] Validation document
This document describes the validation of that specific feature. This document is an output of Qualio’s validation process, but we recommend customers create and mirror this document in their own instance of Qualio as an output of their validation process.
Understanding Test Evidence
Qualio manages the tools used to support validation efforts as controlled suppliers, and have qualified the tools for use. The relevant tools here are JIRA and Circle CI. We are happy to make these qualifications available to customers for review during the supplier qualification process.
Our requirements and testing are written using software industry standard methodology, and this methodology aligns nicely with validation principles. We utilize "Epics", "stories", and "user needs" to capture the requirements to which the software is built. We write testing of these functions of the software in tools that then perform the testing in an automated way. The testing language is known as "gherkin", which sets the test up in a standard way. See the example below.
**Scenario**: Change issue owner
**Given** I log in as a Normal user
**Given** I open an issue details
**And** I click change owner
**When** I assign issue to different user
**Then** 'Owner changed successfully.' notification is displayed
This example is following simple rules:
Sentence starting with keyword Given describes the initial point of the test
Sentence starting with keyword When describes the key action taken in this test (normally we would like to test only one action in test, however in E2E tests we tend to test the process with multiple actions). This sentence should contain "Test Procedure" - or the actions to be performed to test the requirement.
Sentence starting with keyword Then describes the expected behavior. This sentence should contain (if possible) the "Expected Results" , which maps to what we used to call “as expected/as found” in old validation methodology.
Keyword followed by "I" means that the user is taking an action (clicks, types, etc)
If the keyword is not followed by "I", it means that the action is taken by the system.
The testing software executes all the tests in a fraction of the time it takes a person to run, and if a test fails, it is clearly flagged and the run will stop. Successful passing of the tests is required before the changes are incorporated into the application on staging. If all is successful on staging, the code is deployed to production, and the testing will run again as part of that code merge.
The recordings of the automated test running are located in the referenced dropbox file as the evidence of test execution. They are named by the requirement at test, the number that corresponds can be found in the record itself, it will have the "[feature abbreviation]-XXX" identifier.
Additionally, screenshots of final outcomes have been added to the folder for further verification where the video may have ended too quickly to verify. Please keep in mind that not every single requirement will have an individual test, as workflows are grouped together to ensure appropriate testing of the system functions are captured. In other words, related items may be grouped for testing. Coverage across these elements is shown in the "RTM" document.
Lastly, it's important to note that the numbers are not sequential coming out of Jira. Every ticket created in the system is numbered, but not every ticket is a requirement. Bug fixes are also tickets in the same project, and all tickets, regardless of type, are numbered sequentially.
How should I save test evidence for my validation records?
Because Dropbox folders with test evidence will remain available to customers in perpetuity, we actually don’t recommend customers waste time with recreating the test evidence. That is of course an option and up to the discretion of each customer. However, we recommend simply including the link to the latest dropbox folder (feature and version specific) in the [feature] Validation Document. Those links are readily accessible for an auditors requirements.
Validation Summary Report
This final report should summarize validation activities and results to confirm that the Validation Plan was properly executed. We recommend creating a Validation Summary Report template to generate this Qualio document and include the following items:
Smartlinks to all the [feature] Validation Documents
Smartlink to the QMS Business Process & Risk Assessment document
Smartlink to the QMS Configuration document
Smartlinks to your organization’s policies and procedures relevant to the validation process such as the e-signature policy, Change Management and Supplier Qualification policies and procedures.
Link to Dropbox folder containing Qualio’s policies and certifications which Qualio maintains with updated versions for the support of our customers
The Validation Summary Report document can then be reviewed, approved and stored in the customer's instance of Qualio. Discuss with your COM how you would like to document your validation summary.
Other than initial validation, some feature updates will require revalidation. Because the Qualio Validation Support pack is divided by feature, when revalidation becomes suggested/necessary, customers can access and complete revalidation of those features exclusively. It is impossible to prescribe how often revalidation will be required. However, the Qualio Product and Quality teams are cognizant of the resources required to revalidate features and attempt to minimize and condense whenever possible.
When revalidation becomes necessary, customers Quality Users will be notified and provided with Dropbox links to the updated version of validation text evidence and documentation. The new feature/functionality will first be released only to sandboxes on the date of the notification for a 30-day validation period. At the end of the 30 days, the feature will be released/pushed to production accounts.
Customers should take the following steps during the 30-day validation period:
Review new test evidence and documentation
Observe new functionality in their sandbox environment and complete any additional testing
Update¹ process documentation related to the new feature and risk assessment if necessary
Update¹ the Configuration document if necessary
Update¹ the relevant [feature] Validation Document with the latest Dropbox link and any new information, process changes, etc.
1. Any document updates will require the Qualio document to be reverted to draft, updated, then sent for review and approval.
Assistance & Next Steps
This article helped you to prepare for and execute Qualio Validation, but you may still have questions about validation or concerns over the new method. Review our FAQ article for common questions. And if you still cannot find the answer to your question, reach out to Qualio Support.