eCOA FAQs
eCOA (electronic clinical outcome assessment)is the use of an electronic system or platform(technology) to capture Clinical Outcome Assessments in clinical trials. Just like COAs, eCOA includes outcomes reported by patients (Patient Reported Outcomes, PRO), clinician Reported Outcomes (ClinROs) and observer Reported Outcomes (ObsROs) , as well as performance Outcomes (PerfOs) , and can be captured using various methods, such as smartphones and sensor-based digital health technologies. Daily or event driven diaries can also be collected electronically and are collectively referred to as eDiaries.
eDiary: Our eCOA Lexicon defines an eDiary (electronic diary) as a digital tool that allows clinical trial participants to record responses to patient reported outcome (PRO) measures, and directly provide other study-related data such as symptom severity and frequency, adverse events, or medication usage in real-time, at specified interval or when triggered by a clinical event. eDiaries are often “home grown” (i.e., developed by the sponsor), and are typically used to collect patient information away from the clinical site.
ePRO (electronic patient-reported outcome): In basic terms, ePRO is PRO data (Information reported directly by the patient about their health status without interpretation of their response by a health-care professional) collected electronically. Types of PROs include quality of life measures, functional status measures and health behaviors. Patients complete these assessments or questionnaires on a device, at the site or in the field.
ClinRO (clinician-reported outcome): These are assessments completed by healthcare professionals after observation of a patient’s health condition. In most cases, completion of these measures requires medical knowledge/experience with the trial disease indication.
PerfO (performance outcome):Observable tasks that the patient is asked to perform according to instructions provided by a healthcare professional. A common example of a PerfO measure would be the 6-Minute Walk Test, where participants are asked to walk and are timed on the distance they cover.
ObsRO (observer-reported outcome): A questionnaire or assessment completed by someone other than the patient enrolled in a clinical trial (not including site users). This user is typically a caregiver/care partner, such as a spouse or parent, and provides knowledge from the observer’s perspective, e.g., a person living with Alzheimer’s or a young child. NOTE: ObsROs requires a separate log in to the patient to ensure the data is attributable to the observer.
Electronic collection of COA data presents multiple advantages in comparison to paper collection, largely due to the controls that can be implemented in an electronic system.
Primarily, electronic methods ensure adherence with ALCOA+ principles. Electronic systems can increase data accuracy and completeness by imposing specific time windows, prompting the correct set of questionnaires based on the progress in the trial, and using branching logic so that relevant measures are displayed according to responses or protocol design. Branching logic may also reduce patient burden, by only presenting the relevant questions and prevent completion of those which are not applicable. Audit trails in electronic systems ensure data contemporaneousness and support review for irregularities. With paper sources, the authenticity of date and time cannot be verified, and can easily result in cases of back- or forward-filling of entries. Additionally, electronic systems can control user activity by granting specific access according to trial roles and providing all users with individual credentials to ensure the attributability of data entry and review. This also supports improved compliance, since notifications of non-compliance can be issued when participants fail to complete entries as expected.
Finally, the approach may save money as printing, shipping and storing paper versions can be costly, as well as the cost of manual data entry by site staff or data managers and the potential source data verification. eCOA offers data security and encryption vs paper which can be lost, damaged or sensitive data exposed.
These advantages have been recognized by regulatory authorities, who recommend electronic use for these reasons in addition to the usefulness of digital tools and decentralized procedures for reducing trial patient fatigue and burden.
All capable eCOA providers should be compliant with regulations and laws governing their involvement in clinical trials in the countries they operate in (e.g., 21-CFR Part 11, ICH GCP E6 R3). When engaging with a new provider, conversations around that provider’s familiarity with such regulations are advised before being contracted, and due diligence should be performed by the sponsor organization which typically takes the form of a capabilities assessment and audit. A capable provider will be able to share insights into internal SOPs and work instructions as well as training provided to staff. Often, pharmaceutical companies will require the provider project teams to undergo yearly refresher training or sign agreements that attest to them working in a compliant manner. In practical terms, having an in-depth conversation at the time of choosing a provider can be telling. Reputable providers will provide these insights on demand.
eCOA providers who are members of the eCOA Consortium help define best practices and recommended approaches, each being involved in the development of the eCOA Consortium’s output.
/Each region may have specific regulatory requirements or guidelines related to electronic data capture in clinical trials, and it is essential that these are carefully considered while implementing eCOA, and electronic data capture more broadly. The International Conference on Harmonization (ICH) guidelines, particularly E6(R3) for Good Clinical Practice, provide a framework that influences how eCOA should be implemented across different jurisdictions. Depending on the region, there may be additional local regulations or guidelines that impact eCOA implementation.
Device availability, shipment and import
There are regions where smartphones are not available to a proportion of the population – this has been observed in some African, Latin American and Asian countries. It is advisable to plan the trial’s eCOA strategy based on patient demographics – for example, it may not be wise to plan a Bring Your Own Device (BYOD) approach in countries where smartphones are not easily accessible. Similarly, it is essential to consider countries where there known challenges with shipment and import, such as:
- China: shipping devices into China requires several layers of approval which can impact start up timelines.
- Mexico: only allows for temporary import of devices up to a limited total cost per device
Additionally, shipments to some countries are facilitated by a broker, which requires a contract between broker and CRO/Sponsor. The contracting process can take time and should be factored into trial timelines. Countries requiring proforma invoices for shipment also require review and approval by the importer of record prior to shipping, which can add to timelines.
Low- or Middle-Income Countries (LMIC)
LMICs do not always have access to WIFI and suitably modern devices. It is essential to plan trials around this if LMIC locations are involved. Consider if patients need to be reimbursed for mobile data usage, mitigations around power outages and availability of mobile Wi-Fi device (MIFI).
Language text length
Translations into certain languages and the resulting extended text length may impact how a COA item displays on a screen. Languages of notable challenge are Tamil and Malayalam. Thai is also challenging due to continuous script.
Any electronic system for collecting PRO data should verify the user’s identity by requiring login credentials such as a user/role name and self-selected PIN. This can be further enhanced through the use of multi-factor authentication and biometrics (where available and permissible) as a means to ensure that the correct individual is the only one able to access the assessment.
Sponsors will often want “ease of use” to prevail, believing that additional layers to authenticate participants can create additional burden. It is important, however, to recognize that these same approaches are things we deal with in our day-to-day lives now more than ever, and as such, the perceived burden may actually be an expectation of the participant.
Below are the best practices as per the eCOA Consortium membership:
On-Device Training: Site users (and CRAs) should have an opportunity to train “hands on” with the system. The training can be done during investigator meetings or site initiation visits.
Comprehensive Training Materials: Provide detailed, role-appropriate training materials, including quick reference guides, FAQs, and video/web training.
Interactive Training Sessions: Conduct interactive training sessions where site staff can ask questions and receive real-time feedback. Additional video training should be made available online for refresher purposes.
Dummy participant creation: Where supported, allow sites to set up dummy participants in the eCOA system to allow for a deeper understanding of the workflow.
Live Support: Offer continuous support through helpdesks or dedicated support teams. This ensures that site staff can get assistance whenever they encounter issues with the eCOA modality regardless of the time or country locale. 24/7 support is the gold standard, but may not always be necessary if sites are triaging all issues.
Assessment and Certification: Provide a quiz or practical assessment to certify that site staff and participants adequately understand. Provide Refresher Training: Provide regular updates and refresher training sessions to keep site staff informed about any changes or new features in the eCOA system. Participants should have training on the modality or available online if they need a refresher.
Given the complexity of many clinical trials, and the volume of information both site staff and participants are required to be aware of, it makes sense for helpdesks to have the ability to serve both site staff and trial participants. Each group will have different needs and encounter different challenges along the way, and users will often need to speak to an individual to help with their issue. If the helpdesk is participant-facing, it is crucial to ensure the helpdesk agents are trained on how to handle any call that might relate to the participant’s health status, and appropriate processes are in place to protect identifiable data. Availability of support in required languages and locales is a key consideration for participant support.
In the case of BYOD studies, some challenges faced by participants will differ from those of site staff; for example, if a participant losing access to their relevant app store login details. While helpdesks can steer patients toward how this can be rectified, they cannot reset those credentials.
fAn article generated by C-Path entitled “Best Practice Recommendations: User Acceptance Testing for Systems Designed to Collect Clinical Outcome Assessment Data Electronically” provides direction. Key highlights include:
Collaboration: Emphasizes the importance of close collaboration between clinical trial sponsors and eCOA providers to ensure that trial-specific requirements are met.
Detailed UAT Plan: Recommends creating a clear UAT plan that includes necessary documentation and outlines the scope, objectives, responsibilities and timeline.
Thorough testing: Suggests testing all components of the eCOA system, such as the trial-specific application, customization features, trial portal, and custom data transfers/integrations. We strongly recommend that the eCOA provider does not create or test the scripts.
Regulatory compliance: Highlights the need for documentation that may be requested by regulatory authorities, depending on the country. Support: eCOA provider should provide support throughout the UAT process and manage the findings, if applicable.
In addition to those recommendations, we would also encourage the following:
– A discussion around how defects are documented and tracked to resolution.
– Employing testers who will actually use the system in the trial.
Please note that User Acceptance Testing (UAT) is distinct from Usability Testing (UT).
The eCOA Lexicon was created in collaboration with the PRO Consortium to provide a common lexicon for eCOA-related terminology across the eCOA ecosystem. The Lexicon is a live resource that is updated regularly.
Combining paper and electronic data capture in the same clinical trial can risk the data’s accuracy, attributability, contemporaneousness and integrity. While adding a paper back-up option to an eCOA solution can seem helpful, this approach requires manual data entry, which increases the risk of data error, and demands source data verification, which is largely impossible for subjective PROs. It also increases data management activities and their associated time and costs. Having disparate systems for data collection can also increase the risk for data duplication, make it more difficult to identify missing data, and require additional resources for reconciliation. With today’s eCOA systems offering a range of electronic back-up solutions, from web-based collection to interview methods, it introduces greater risk to add a paper option than keeping a fully closed electronic system.
From the site perspective, adding a paper option requires additional instruction and training; and remote monitoring activities cannot be performed with paper data entry which could require additional travel for the clinical teams or the participant.
Costs: Implementing eCOA can be perceived as expensive upfront, especially if migrating from paper to electronic. This cost includes any necessary process to validate new measures in an electronic format.
Trial timeline concerns: When developing eCOA solutions, especially for the first time, delays to the trial startup timeline might occur if you do not plan for eCOA early in the process.
Knowledge and experience: Many sponsors do not have internal expertise to manage the implementation of electronic clinical outcome measures, which might impact a successful deployment.
Prior negative experience: If a sponsor has had any previous negative experiences with eCOA, this may impact a sponsor’s decision to implement eCOA in the future.
Training Concerns: Proper training for both site staff and trial participants is essential to ensure they can effectively use the system.
Appropriateness of the COA for digitization: Some COAs may require consideration for appropriateness for an eCOA application. While the overwhelming majority of COAs are suitable for electronic use, some, such as those incorporating free text, or require audio visual recording, require careful planning.
qYes, clinical trials that are collecting PRO data electronically can use a strategy where participants use their own personal device to download the app or access the web to complete the PRO measures, and regulatory bodies are accepting of data captured in this way. In 2023, a C-Path-led eCOA Forum that included FDA representation concluded that there are no specific additional regulatory requirements for BYOD1. It is known that a regulatory approved labeling claim based on ePRO data collected using BYOD has occurred; the Pfizer and BioNTech mRNA Vaccine trial for the treatment of COVID-19 collected PRO safety data from participants worldwide, with 79% using BYOD for ePRO completion2, and Members of the eCOA Consortium have anecdotal evidence of other pivotal trials employing the BYOD model successfully. The below references provide useful commentary on this topic:
- Regulatory Acceptance of Patient-Reported Outcome (PRO) Data from Bring-Your-Own-Device (BYOD) Solutions to Support Medical Product Labeling Claims : Let’s Share the Success Stories to Move the Industry Forward (Mowlem et al., 2022)
- Measurement Equivalence of Patient-Reported Outcome Measures Migrated to Electronic Formats: A Review of Evidence and Recommendations for Clinical Trials and Bring Your Own Device (Byrom et al., 2018)
- Progressing BYOD Adoption (eCOA Consortium, 2022)
References
Yes – a BYOD approach to data collection can be used for the collection of both ePRO and data. The important consideration in these trials is if the same BYOD device is being used for the completion of the ePRO and the measures; if the same device is being used (for example, in instances where the participant is a child who does not have their own phone), then it is important to have a clear plan for attributability. If multiple observers are involved, the recorded data should be attributable to each observer.
This will vary by participant population, trial protocol, eCOA design, and the countries in which the trial is held. The eCOA platform being used may be a factor also. Some eCOA designs are suitable for a wide range of device sizes and operating systems, and others have important limitations. Your provider will be able to advise on per-country BYOD uptake rates based on previous trials.
BYOD studies should always include a provisioned device option for participants, which means that if they don’t own a suitable device, or prefer not to use their own device, they should be provided with one. Typically, these are locked-down devices for completing trial activities only. If a participant needs to change approach mid-trial, then they can be issued with a provisioned device at this point, or vice versa.
It is likely that a participant will replace a lost device quickly due to the personal inconvenience factor. To reduce the impact of potential loss of data in the interim, they can be provided with a provisioned device to use, download the app on another device, or potentially employ an electronic backup option. For example, a web backup solution would enable the participant to continue to complete tasks on another internet-enabled device (such as a laptop or tablet) in instances where an app-based solution was the principal mode deployed.
Depending on the type of ClinRO, or more specifically what the ClinRO is measuring, ClinROs may be administered via a televisit, though consideration must be given to the indication or concept of interest.
There are several examples of ClinROs being performed remotely and with consistent and accurate results. Much will depend on the trial design, the trial population, therapeutic area, and in some cases the measure itself. There is no regulatory guidance that specifically prohibits the use of ClinROs in a virtual setting. It is more important that we choose the most suitable measures for the given context of use, and that probe the right concepts of interest. The 2024 FDA guidance on conducting clinical trials with DCT elements lists Telehealth as an appropriate method if an assessment made in that setting does not pose significant risk to trial participants and if there are plans in place to ensure that adverse events identified during such visits will be properly assessed, managed, and documented.
No – rater training is not required for all ClinROs. The need for rater training is typically a function of the study phase, complexity of the measure, the concepts and domains being covered, and the instructions for use as laid out by the measure author or copyright holder. A sponsor may also decide to include rater training for some scales depending on the measures’ position in the endpoint hierarchy, and/or desire to reduce variability and interpretability between administrations of the scale in their trial. Rater training is a recognized approach to reduce inaccuracy and variability and may produce more reliable and consistent data.
Most children are familiar with mobile technologies and use them regularly, but it is about considering the age at which a child has the capacity to reliably and validly comprehend the measure and self-report on their health status. There is no set answer to this, and it will differ by developmental age, not necessarily chronological age. Thus, an individual approach is required that considers the level of understanding and comprehension the child has of the questions being asked, as well as their reading ability.
A good place to start is the ISPOR recommendations paper (Matza et al, 2013) which identifies that while it is not possible to provide age cutoffs that will fit every situation, the following four age groups are a helpful start for further consideration:
- < 5 years old: No clear evidence of reliability/validity for self-report
- 5-7 years old: Self-report may be possible, but reliability and validity are often questionable
- 8-11 years old: Improved reliability and validity of self-report
- 12-18 years old: Self-report is typically preferred.
For Phase 3 studies, sponsors would benefit from engaging with regulatory bodies on what they will accept as evidence of a primary endpoint being a valid measure for use in a labeling claim in the age groups included in their trial population. Sponsors should also look to specific instrument guidance from the relevant copyright holder/author.
Further relevant literature:
This is an extremely context-dependent matter. Historically, it has been the case that changes to eCOA data should be avoided, with the possible exception of contextual metadata (e.g., dates of completion, where supported by other source records). Additionally, some sponsor representatives had created workflows that required sponsor “approval” of data change requests from sites which in some cases were denied. Today, there is alignment between stakeholders that investigative sites must exert control over the source data that eCOA providers host on behalf of the sites, and if an Investigator requests a change to eCOA data because the originator of that data (whether participant, caregiver, observer, or clinician) has noted that the original entry is incorrect, preventing the change is inappropriate.
To help inform a balanced position on eCOA data changes, we strongly recommend reviewing Best Practice Recommendations for Electronic Clinical Outcome Assessment Data Changes, co-written by the eCOA Consortium and published in the Journal for the Society of Clinical Data Management in 2023. This key paper analyses all considerations for eCOA data change strategies and provides clear guidance on how sponsors, sites, and eCOA providers can collaborate to ensure data is maintained in a compliant and defensible manner.
The eCOA Consortium collaborated with the PRO Consortium, CDISC, analytics experts and the US FDA to develop Best Practice Recommendations for Electronic Patient-Reported Outcome (ePRO) Dataset Structure and Standardization to Support Drug Development, which was published in Value in Health in 2023. This paper details a series of recommendations that reinforce the critical need to establish and adhere to standards for ePRO and eCOA data from the point of capture through to regulatory submission.
There are two key regulatory guidances available that address DHTs:
1) European Medicines Agency published its Guideline on computerised systems and electronic data in clinical trials in 2023.
2) US FDA published Digital health Technologies for Remote Data Acquisition in Clinical Investigations in December 2023.
Sensor-based DHTs can provide objective evidence on health status, which can be used to complement subjective PRO data. Evidence exists of trials incorporating DHTs demonstrating treatment effects quicker than those reported using PRO responses alone. PROs seek to obtain the participant’s perspective on how they feel or function, as opposed to measuring their feelings and functioning directly. This is particularly challenging for sleep and cognitive conditions.
In our collected experience, where device use has been optional and/or used as part of a sub-study, recruitment or uptake of device use has been a challenge. Generally, we would not recommend device use to be optional with a trial.
Using a device as part of a sub-study may be a good option for later phase trials where making device use mandatory for all participants could be cost-prohibitive. Alternatively, device use could be mandatory for certain cohorts of a study where statistical significance might be achievable via the use of the DHT with fewer patients than the full trial population.
We strongly recommend that this be part of the trial design, but given the nascence of sensor-based DHTs, we recommend that sponsors consider bringing sensors in at the early stage of the clinical development program as a whole. Use in Phases 1 and 2 can provide vital information and provide assurances that the technology can be used and will provide valuable data in later phases.
However, a controlling factor is the intended use of the DHT and the trial endpoint the data will support. Safety monitoring with sensor-based DHTs can be used in any clinical phase, whereas efficacy measurement typically occurs in phases 2-4. Natural History studies are an excellent way to understand baselines prior to interventional trials.
The FDA Guidance Digital Health Technologies for Remote Data Acquisition in Clinical Investigations specifically states that the guidance does not address if a DHT meets the definition of a device under section 201(h) of the FD&C Act. However, FDA outlines that sponsors must confirm, through verification and validation activities, that their proposed DHTs are fit for purpose. However, the terms “verification” and “validation” as used in this guidance are not the same as in 21 CFR 820.3(aa) and 820.3(z) under the QSR for devices (21 CFR part 820) or the terms “device software function verification” and “validation” as described in FDA’s own guidance General Principles of Software Validation. Sponsors must assess both the context and the intended use of the DHT to determine the applicability of medical device regulations.
Sponsors should be aware that while it is possible to include DHTs in clinical trials that do not have regulatory certifications, there will be an increased level of scrutiny from the IRB and ECs. In our experience, additional discussion and documented confirmations will need to be provided by the sponsor and manufacturer to satisfy IRB and EC concerns prior to the approval of protocols.
While consumer-grade devices can collect health-related data, sponsors should evaluate the devices’ suitability for the context of use and compliance with regulations. As outlined in the FDA’s DHT guidance, sponsors are responsible for reviewing verification and validation documentation to ensure the device’s appropriateness.
Consumer-grade devices may lack proper documentation supporting verification and validation. It is critical that sponsors verify that the DHT complies with applicable regulations (potentially: 21 CFR 812.3(m), EU MDR 2017/745, and EU IVDR 2017/746), as well as data privacy laws (e.g., HIPAA and EU GDPR 2016/679). Sponsors should also be cognizant of end-user license agreements that may allow wellness companies to collect usage data, and ensure these agreements are reviewed by an ethics committee and consented to by study participants. Furthermore, sponsors should also document if the consumer-grade device undergoes firmware updates, the battery lifespan, retirement, and succession strategy during the trial conduct, and if such updates impact the computation of outcome measures, as well as the availability of the raw data from the devices.
Some commercial devices have elements of their hardware, firmware, or software approved for use in clinical trials by regulators, but this does not mean all elements are. For example, the Apple Watch’s Heart monitoring tool has been approved for use in clinical trials for atrial fibrillation, but other capabilities of the watch have not.
The highest potential for commercial-grade devices is in long-term, observational monitoring using a patient’s preferred device that may support baselining for both clinical trial enrollment and understanding of disease progression and/or clinical outcomes over time.
BYOD approaches for wearables (sometimes termed Bring your own Wearable (BYOW)) or sensor-based DHTs is challenging. Late phase, natural history and observational studies that have less stringent regulations may lend themselves more readily to BYOD. At this time, the eCOA Consortium is not aware of any registrational trials that have incorporated a BYOD approach for wearables or sensor-based DHTs. In any case, sponsors are responsible for ensuring that DHTs are fit-for-purpose and compliant with relevant regulations.
Sharing data with the participant can be considered when this supports the clinical trial’s objectives; at a minimum, data could be shared with participants at the end of the trial. Certain types of data (not used to support study endpoints) might be shared during the trial.
Sharing of data with participants can help with retention/compliance if participants can see some meaningful data. This has to be balanced with not influencing participant behavior had they not seen the data.
For safety or condition management purposes (e.g., continuous glucose monitoring [CGM] data for diabetes), we would advise that participants have access to their data to manage their condition rather than requiring them to have a study CGM and their own CGM.
Understanding the value of the data and its intended use is pivotal to ensuring compliance.
Site understanding will ensure they are motivated to encourage participants to take part in the study with the device. Sites can be overloaded with technology, so it is important for them to understand the value the collected data could bring. Sites need to understand what the data is used for so they can answer participant questions and concerns.
For participants, learning how the data will be used and for what purpose should be detailed as part of the informed consent process. The value that it could bring to the trial as a whole or to future trials/treatments should be outlined, as this will help to motivate participants to take part in the trial and drive compliance.
There is no established answer to this. The number of potential failure points (screens, batteries, motherboard failures), varying device manufacturers, and operating systems make it impossible to quantify failure rates universally. A capable DHT partner will ensure devices are thoroughly tested and failures recorded accordingly. Trials must have contingency plans for device failure, loss, and damage throughout the trial study and a plan in place to re-supply the participant with a new device. This plan should also include mitigations to minimize data loss, such as statistical analysis plans documenting how to account for missing data.
Training should be a must to ensure the correct usage of the device. Devices (and companion apps) can and do provide instructions on use, and ensure users view the training before triggering data collection on the device. This is very much device-dependent, though. Devices/apps can also provide coaching and feedback to participants to aid them in taking good quality measures (remote spirometers are a good example of this).
The eCOA Consortium has co-developed a series of publicly available tools that serve as the basis for a sponsor-level eCOA strategy and include considerations around typical workflow and stakeholder roles and responsibilities. Those tools are:
These tools were developed in collaboration with sponsor representatives within the Patient-Reported Outcome Consortium and so provide a balanced perspective of both eCOA providers and sponsors on how best to achieve the successful implementation of eCOA systems in clinical trials.
The prevailing industry norm involves displaying a single item per screen. However, this practice may pose accessibility challenges on smaller screens, such as reduced readability due to small font sizes. Splitting response options across multiple screens can be a solution in certain circumstances but can risk the integrity of the measure due to not presenting an item in a self-contained format. In contrast, leveraging scrolling functionality can effectively address these concerns.
Despite legacy reservations regarding scrolling’s impact on measurement comparability, the eCOA Consortium advocates for enabling vertical scrolling over reducing font sizes to maintain readability as a best practice in Mowlem et al. (2024). This is supported by evidence that has shown that vertical scrolling is very unlikely to compromise measurement properties (Shahraz et al., 2021). When disabling scrolling, if this compromises readability, careful device selection may facilitate good usability. Where appropriate, it’s advisable to engage in discussions with instrument developers or copyright holders to find a suitable implementation that minimizes adverse effects on usability.
Mowlem et al. (2024) provides detailed examples of how to implement scrolling functionality in a scientifically robust manner.
Accumulating evidence has demonstrated that, for common response scale types (VRS, VAS, NRS), respondents answer consistently across data collection modes. A recent ISPOR Task Force report (O’Donohoe et al, 2023) summarized this literature base, concluding that where sufficient evidence already exists to demonstrate comparability and best practices for faithful migration are followed, additional comparability testing is unnecessary.
Where a response scale that is not common (e.g., a body map) is being migrated from paper to electronic, and there is not yet sufficient evidence to support comparability, then a measurement comparability assessment may be needed. The latest ISPOR guidance states that “a decision regarding the need for generation of additional evidence is dependent on the amount of available supporting evidence demonstrating whether the change [in formatting between paper and electronic] has affected the PROM’s measurement properties in regard to content and meaning”. Key considerations include how significant a formatting change was made in the migration process and what type of response option is being employed. Other articles such as Byrom et al. (2019) & Mowlem et al. (2024) provide important additional guidance on this topic.
No – they are not the same. Usability testing within a representative population is extremely important to conduct – especially when considering special populations such as pediatric populations, populations that may have dexterity challenges, visual challenges, etc. Regulatory agencies are keen to see evidence of both comparability and usability when a novel electronic administration of an instrument is used in a trial.
There are numerous changes that are known to have no meaningful effect on measurement properties when migrating from paper to electronic modes, such as changing instructional text to ‘select’ instead of circle’, and reformatting to display as one item per screen. Mowlem et al. (2024) synthesizes the current, published industry best practice guidance for the migration of a paper COA to electronic. It also dives deeper to provide practical applications, including the rationale behind the practice, the regulatory considerations, a classification, and mock screens to illustrate each practice, leading users through the best approaches to eCOA implementation while maintaining data comparability and optimizing usability. For response scale types outside of common COA attributes (such as body maps, activities included in some cognitive batteries), there is not yet enough evidence on their migration and comparability. For those scale types, implementation guidance provided by the instrument owner or licensor should be referred to. Additional information on considerations specifically for ClinRO migration is included in Romero et al. (2022).
Avoiding over-design in eCOA systems is crucial to ensure that the system is efficient, user-friendly, and cost-effective. Here are several strategies to achieve this:
- When planning a trial with eCOA, the first point of reference should be the protocol with emphasis on primary/secondary objectives, schedule of assessments and COAs being used in the trial. It is imperative to prioritize essential functions and focus on core functionalities that directly support data collection, validation, and compliance with regulatory requirements. Avoid adding unnecessary features that could complicate the system.
- Ensure that all stakeholders are available to make key decisions throughout the design/planning process; i.e., Sponsor, eCOA Provider, CRO, Clinicians (if applicable). Discuss needs, workflows, and preferences with the team to tailor the system effectively. This ensures that features are not excessive and relevant to the protocol. Discuss trial/system-specific recommendations/lessons learnt with the eCOA provider based on previous studies conducted with similar COAs.
- Ensure patient convenience by focusing on the trial population (e.g., age, location) and ensure that the chosen technical solution aligns with the capabilities of the target user group and the intended environment of use. This is especially important when deciding on device type, translations, shipment implications and data transmission. Challenge over complex design requests if they are not a mandatory requirement of the protocol.
- Follow regulatory guidelines and industry standards relevant to eCOA systems (e.g., FDA guidance). This ensures that the system meets essential requirements without unnecessary complexity.
- Conduct reviews/usability testing at various stages of development to identify potential usability issues and refine the design accordingly. This is important to keep in mind during eCOA specification sign off, demos and UAT, optimizing the system for ease of use and efficiency.
- Provide adequate training and support materials for users to maximize the system’s effectiveness, this includes trial specific site and patient guides, investigator meeting presentations and demos, device training and trial specific eLearning.
It’s important to also acknowledge that it is virtually impossible to account for every possible scenario that may emerge with the system – this is a mistake sponsors can and do make. By following these strategies, eCOA system designers can avoid over-designing and create systems that are robust, user-friendly, and aligned with the specific needs of clinical trials and research studies.
For eCOAs, having a backup strategy defined during trial set-up phase during trial set-up phase is important. In general, the preferred backup method would be another electronic system to collect data (e.g., web backup for trials with primarily a device-based strategy), ideally one that mimics the exact way a participant would be collecting their data (i.e., their eCOA device has a certain look and feel, and the back-up looks, acts and feels as similar as possible). Mixing paper and electronic data collection methods would not be the recommended approach, as discussed in the ISPOR Updated Recommendations on Evidence Needed to Support Measurement Comparability Among Modes of Data Collection for Patient-Reported Outcome Measures (O’Donohoe et al. 2023).
Back-up options are more challenging for ePerfOs collected using sensor-based digital health technologies. If there is a need to change technologies mid-trial, it will be important to have data demonstrating that any differences in usability or technology performance are within acceptable ranges”.
Each IRB/EC may have different requirements, and it is becoming more common for each entity to have published guidance on their expectations for what each submission should include. In general, Sponsors and investigators should consider including an overview of the technical specification; a summary of the use specification; a description of potential safety issues associated with the technology with appropriate risk mitigation; and a summary of privacy and security protections, including data ownership.
As with all IRB/EC submissions, participant-facing material should be included. For digital health technologies, this may also include the end user license agreement, privacy policy, or terms of use. The ICF should address technology-specific considerations such as privacy and security; data ownership; whether any diagnostic tests or other health data will be made available to them or their healthcare providers (including atypical readings or concerning results); and arrangements for any costs that may be incurred (such as use of cellular data). The NIH Office of Science Policy has developed a document including sample ICF language for studies including digital health technologies.
When implementing sensor-based technologies such as wearables, Sponsors and investigators should consider all measures generated by the selected tool. For example, the study may require a particular wearable for evaluating sleep, but the tool may also collect heart rate or other metrics that may give additional health insights (such as evidence of arrhythmia). A plan for whether physiological data will be reviewed clinically should be developed, and if not, participants should be made aware that their data will not be actively monitored. If health data will be visible to participants during the study, or provided after study completion, a description of available clinical support should be included where applicable. If the technology is a regulated medical device, product labeling should be included and any applicable regulations regarding investigational devices should be followed.
This common and persistent issue has prevailed for many years. The eCOA Consortium created a self-published white paper entitled eCOA Systems and CE Certification in 2023. This resource provides a detailed analysis of governing regulations in the context of eCOA systems and maps out the requirements for CE certification, as well as providing a decision tree to help sponsors arrive at an informed conclusion as to whether their study eCOA systems are in scope for CE certification or otherwise.
