Surprises You Don’t Want When Adopting eCOAs for Use in Clinical Trials: Cautions for Decision Making and Planning

SPRING 2020, THE EVIDENCE FORUM, WHITE PAPER


Huda Shalhoub, PhD
Former Research Scientist
Patient-Centered Research
Evidera
Mona L. Martin, RN, MPA
Senior Research Leader
Patient-Centered Research
Evidera

Introduction

The surge in the digitalization of communications over the past ten years is not only shaping our day-to-day lives but is also seeping through to scientific methodology such as clinical outcome assessment (COA) data collection methods in clinical trial research. COA tools are used to measure symptoms, health status, or impacts of a disease or condition on functioning.1 A COA can be a standardized measure with multiple items or domains, or an individual item. COAs include patient-reported outcomes (PROs), clinician-reported outcomes (ClinROs), observer-reported outcomes (ObsROs), and performance-based outcomes (PerfOs) measures.

Adoption of electronic COAs (eCOAs) for clinical trial data collection is happening at a faster rate than ever before and many pharmaceutical and healthcare organizations (hospitals, clinics, etc.) are now switching efforts to move away from paper data collection methods.2,3 The types of COAs available for use is getting more varied with electronic modalities such as smartphones, tablets, wearables, interactive voice response systems (IVRS), web-based software, and device apps like Bring Your Own Device (BYOD). Currently, the most popular modalities are smartphone, BYOD, web, and tablet, while IVRS use is on the decline.4,5 It is projected that in the next five years alone eCOA revenue will grow by almost 20% – with the market reaching $160 billion by 2027.6

    eCOA Advantages

    There are numerous advantages to using eCOAs in clinical research. Evidence demonstrates that the use of eCOAs improves data accuracy and site and user compliance. They have the added benefit of significantly reducing missing data.7,8 Moreover, eCOAs allow users to receive reminders to complete their assessments and provide the flexibility of data completion from anywhere (e.g., home, clinic, or hospital). Regulatory bodies like the US Food and Drug Administration (FDA), the European Medicines Agency (EMA), and the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) appear to support the use of eCOAs as electronic data can be easily tracked, time stamped, are confidential, and allow for centralized data monitoring.9-11 Since 2012, many medicinal products approved by the FDA and EMA include an electronic format of a PRO, and especially daily patient diaries.12

    Beyond their overall acceptance by the industry and regulatory bodies to date, eCOAs allow for implementation of branching logic of questions, reducing the length of questionnaires, and patient burden. Skip patterns can be more effectively used to give patients a better personalized and user-friendly experience.13 Finally, their use has been shown to improve patient willingness to answer sensitive questions that they otherwise may not be comfortable answering.14

      eCOA Logistical Considerations

      Although there are advantages to using eCOAs, there are also planning steps, hurdles, and detailed technical requirements to think through (summarized in Figure 1). It is not reasonable to expect to buy the product off the shelf from an eCOA vendor and have it work well in the trial. It is crucial to assure the goal of your study is going to be well supported by the eCOA you select, and be sure you have appropriate buy-in from your internal stakeholders. There are a number of logistical, decision making, and tailoring considerations that must be arranged before, during, and after you decide to use eCOAs.

      Implementing electronic modalities usually requires added time and funding in the early stages. The initial search for a suitable eCOA vendor should include more than investigating budget for services; it should scrutinize the potential vendor’s ability to meet the study specifications and study goals with their services and not present limitations in programming, platforms, or other services that would hinder the successful accomplishment of study goals. The eCOA vendor costs can vary dramatically depending on required logistics and selected devices (e.g., BYOD versus tablets; leasing versus buying devices; global versus country-specific) and complexities (e.g., length of study assessment, COA length and branching logic, type of modality). eCOA vendors know these territories well and will offer options beneficial to their own efficiencies, but it is also important to involve a scientific expert well versed in eCOA to guide this planning phase to be sure the scientific requirements are being properly considered. Some eCOA vendors offer this kind of scientific expertise to assist project design while others do not and are more focused on operationalization of the devices and programming. You will want to consider the scientific input as a key step so that the study design is not compromised later due to implementation, technical, or user issues.

      Once it is determined that implementing an eCOA is the right fit for a study and the project specifications are finalized, conducting due diligence in vendor selection is critical. The level of experience and resources that a vendor has with clinical trials is critical, including how experienced they are with the study intricacies, including CFR Part 11 compliance, which is a regulatory requirement in many clinical trials.1 The juxtaposition with Title 21 CFR Part 11 Compliance requirements is that, to date, it is not as flexible as it appears to be, as it requires a certain infrastructure in place (guidance on establishing security controls, backups, system maintenance, and data integrity).

      Figure 1. Cautions about Using eCOA

      If an eCOA is being used to support an endpoint for a labeling claim in a clinical trial and in order to adhere to regulatory requirements, the best option is usually to use a provisioned device (devices provisioned by the sponsor or site that are specific device models that have undergone study customization). Drawbacks with provisioned devices, however, is that they lack the flexibility or user-friendliness that a BYOD app may provide. Provisioned devices also require device shipping, training, and setup by the user; will have limited device functionalities; and, may add the burden of carrying another device in the respondent’s pocket or purse.

      eCOA training is important especially for target populations that are not as familiar with technology or older populations who did not grow up using electronic devices or apps. Although research suggests that older populations are able to successfully use eCOAs, there is often a learning curve that requires initial device training. Another example of how an eCOA implementation may falter is if the login or setup process is too complex or time consuming for the patient or site. This can deter user engagement dramatically and lead to a loss of critical data for trial success.

      Additionally, an important consideration to make during the planning phases is knowing your target population. If for a neurodegenerative disease, for example, patients have upper extremity difficulties (e.g., difficulty with writing and typing), an eCOA solution that includes the need to type or sign might be a major design flaw. The application of eCOAs can be used across a broad range of therapeutic areas and indications, including oncology, rheumatology, dermatology, gastroenterology, rare diseases and beyond, but with caution.

        In short, if the implementation phases and programming are not well executed, they may add significant burden to those taking part, including sites, the study team, patients, and caregivers, and can fail, especially in the case of multi-national, longitudinal trials that require more than one time point for data entry.

          Completion of eCOAs should also be weighed carefully when combined with other data collection case report forms or electronic modalities. It is considered good practice to match as much as possible with other electronic modalities during that trial to provide the end user with a seamless experience.15 Reducing mixed modes in a trial and using only one device, for example, to complete all data collection would be the ideal scenario, but this depends on the study design, the status and type of instruments being used, the phase of drug development in which the study occurs, and the overall available funding. Often, eCOA implementation is not well thought through, and the patient, for instance, is expected to complete her/his case report forms on a tablet, while their daily diary is on a smartphone. This can be jarring and confusing, leading to lower compliance rates and engagement with the trial. Such issues should be considered and resolved with the expert research scientist, eCOA vendor, and study investigator during project planning phases.

            Cross-Cultural and Geographic Issues

            When it comes to international trials and deployment of eCOAs across countries and languages, special additional considerations must be made regarding the migration of COA instruments to electronic formats. Such issues include the number of words per question across languages, which often differs and can impact the of the size of the screen that must be used. The programming structure of response options across languages will also require detailed scrutiny as languages are conjugated differently and some can be substantially longer than the source document’s character count. Screen size and programmable character count should be considered early to assure the electronic PRO (ePRO) vendor can support the language and COA needs. Another example is the use of English keyboards in international trials; they can create confusion, data quality issues, and decreased patient engagement.

            More problems may arise in translation programming. Some vendors do not have the capability to implement Zulu, for instance, a South African language, or Cantonese characters in Chinese.

            Cross-cultural issues are relevant not only within global trials, but also within one country where more than one language is spoken (e.g., English and Spanish in the US). Such considerations can have budget implications and need to be thought through well in advance.

            Geographic locations of the study population must also be considered. For example, if the study population is mainly in rural areas and patients do not have access to WIFI, yet the device relies on a WIFI connection, the data for that group is at risk. This can result in major study catastrophes on data generalizability and may decrease power from the statistical analyses and result in significant cost burden and delays.

              Additional Scientific Issues

              Other considerations are scientific in nature. Text placement on the screen can make a big difference to user engagement. Text that is centered versus crowded in the top left corner is preferred as it has been shown in cognitive interview work to be easier for patients to read. Word wrapping is another consideration. Where words break can totally change the meaning in some languages. Text should not extend out to the same margins as the item numbers, and response options should have adequate space between their start and the stem item. Both issues have been shown to cause patients to skip reading the items fully because the screen area is visually congested. If users begin to skip reading parts of the message and assume an incorrect message, this can have an impact on the data, the validity of an instrument, and the findings.

              It is important to note that the majority of standardized COAs were originally developed on paper and require migration to an electronic format before use. In those cases, working closely with the COA developer or license holder and the study investigators will be important. Many of the license holders do not yet have clear guidelines for eCOA implementation, so it is important to allow ample time for these tasks to be accomplished as they require longer periods of coordination.

                Validation research must also be conducted before using an eCOA in a study if the instrument is to be an endpoint to support a medical label claim.16 Although equivalence of paper and electronic has been shown to be highly correlated, it is still recommended to consider validation.17,18 This type of research can be divided into three categories and will depend on the level of modification from paper to electronic as shown in Table 1.19 The type of research includes usability testing for minor modifications, cognitive testing for moderate changes and equivalence/full psychometric testing for major change or de novo instruments.

                Table 1. eCOA Validation and Equivalence (Adapted from Fuller et al., 2016)

                  Conclusion

                  We are in a time where technology is rapidly advancing, making new options interesting and attractive. While collecting data by varied electronic platforms opens up the potential for research that increases accuracy, timeliness of reporting, and the types of variables that can be captured, the electronic modality itself must be approached with a variety of cautions in order for the marriage of eCOA data collection and scientific research to be effective.

                  As different electronic modalities become available, more needs to be known about their individual validity and comparative difference between modalities. Pre-planning logistics, costs, and limitations in programming should be researched in great detail in order to be clear what will be required of an ePRO vendor before making the selection.

                  New technology must not be used simply because it can be; it needs to be chosen with an eye to providing the best fit for the study needs with the data objectives, the required logistics, the population characteristics, impacts of the therapeutic area, and overall patient burden well thought out in advance of the final decision. Careful pre-planning and extra awareness will go a long way towards avoiding surprises that nobody wants.

                    References

                    1. US Food and Drug Administration (FDA). Patient-Focused Drug Development Guidance Public Workshop. Methods to Identify What is Important to Patients & Select, Develop or Modify Fit-for-Purpose Clinical Outcomes Assessments. 2018. Available at: https://www.fda.gov/downloads/Drugs/NewsEvents/UCM620708.pdf. Accessed March 7, 2020.
                    2. Yeomans A. The Future of ePRO Platforms. Applied Clinical Trials. Jan 2014. Available at: http://www.appliedclinicaltrialsonline.com/future-epro-platforms. Accessed March 5, 2020.
                    3. Wyllie (King) R. The Rise of Patient-centric eCOA. Journal for Clinical Studies. 7(4):58-59. Available at: https://www.jforcs.com/7/wp-content/uploads/2015/08/The-rise%E2%80%A6pdf.pdf. Accessed March 9, 2020.
                    4. Gwaltney C, Coons SJ, O’Donohue P, et al. “Bring Your Own Device” (BYOD): The Future of Field-Based Patient-Reported Outcome Data Collection in Clinical Trials? Ther Innov Regul Sci. 2015:49(6):783-791. doi: 10.1177/2168479015609104.
                    5. Byrom B, Watson C, Doll H, et al. Selection of and Evidentiary Considerations for Wearable Devices and Their Measurements for Use in Regulatory Decision Making: Recommendations from the ePRO Consortium. Value Health. 2018;21(6):631-639. doi: 10.1016/j.jval.2017.09.012.
                    6. Zion Market Research. Electronic Clinical Outcome Assessment (eCOA) Market Worth Over USD 1,822 Million By 2024, Globally: Zion Market Research. 2018. Available at: https://globenewswire.com/news-release/2018/12/18/1668552/0/en/Electronic-Clinical-Outcome-Assessment-eCOA-Market-Worth-Over-USD-1-822-Million-By-2024-Globally-Zion-Market-Research.html. Accessed March 5, 2020.
                    7. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient Non-Compliance with Paper Diaries. BMJ. 2002;324(7347):1193-1194.
                    8. Zbrozek A, Hebert J, Gogates G, et al. Validation of Electronic Systems to Collect Patient-Reported Outcome (PRO) Data-Recommendations for Clinical Trial Teams: Report of the ISPOR ePRO Systems Validation Good Research Practices Task Force. Value Health. 2013;16(4):480-489.
                    9. European Medicines Agency (EMA). Reflection Paper on Expectations for Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials. 2010. Available at: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/reflection-paper-expectations-electronic-source-data-data-transcribed-electronic-data-collection_en.pdf. Accessed March 9, 2020.
                    10. US Food and Drug Administration (FDA). Oversight of Clinical Investigations A Risk-Based Approach to Monitoring. Guidance for Industry. 2013. Available at: https://www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf. Accessed March 9, 2020.
                    11. US Food and Drug Administration (FDA). Electronic Source Data in Clinical Investigations. Guidance for Industry. 2013. Available at: https://www.fda.gov/downloads/drugs/guidances/ucm328691.pdf. Accessed March 9, 2020.
                    12. Coons SJ, Eremenco S, Lundy JJ, et al. Capturing Patient-Reported Outcome (PRO) Data Electronically: The Past, Present, and Promise of ePRO Measurement in Clinical Trials. Patient. 2015;8(4):301–309. doi:10.1007/s40271-014-0090-z. (Published correction appears in Patient. 2015;8(6):571.)
                    13. Shalhoub H, Reaney M. PROMIS® Tools as Endpoints in Clinical Trials: What Should You Know? A Review of PROMIS Capabilities and the Current Regulatory Status. Int J Clin Trials. 2016;3(4):174-179.
                    14. Tourangeau R, Yan T. Sensitive Questions in Surveys. Psychol Bull. 2007;133(5):859-883. doi: 10.1037/0033-2909.133.5.859.
                    15. Eremenco S, Coons SJ, Paty J, et al. Pro Data Collection in Clinical Trials Using Mixed Modes: Report of the Ispor Pro Mixed Modes Good Research Practices Task Force. Value Health. 2014;17(5):501-516.
                    16. US Food and Drug Administration (FDA). Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Guidance for Industry. 2009. Available at: https://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf. Accessed March 10, 2020.
                    17. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review. Value Health. 2008;11(2):322-333.
                    18. Muehlhausen W, Byrom B, Skerritt B, et al. Standards for Instrument Migration When Implementing Paper Patient-Reported Outcome Instruments Electronically: Recommendations from a Qualitative Synthesis of Cognitive Interview and Usability Studies. Value Health. 2018;21(1):41-48.
                    19. Fuller RLM, McNamara CW, Lenderking WR, et al. Establishing Equivalence of Electronic Clinician-Reported Outcome Measures. Ther Innov Regul Sci. 2016;50(1):30-36.

                      For more information, please contact us.

                      Download White Paper   Return to The Evidence Forum

                      Previous ArticleEvidence Generation Using Innovative, Technology-Driven Data Collection 
                      Next ArticleDigital Therapeutics: Past Trends and Future Prospects