EPI-SET Measures and Outcomes Working Group Meeting Minutes
October 11, 2019
In attendance: Janet, Mary, Melanie, Don, Avra, Gord, Sandy, and Andrea
ACTION ITEMS
Clarify how long post-trainer calls will continue after in-person training - Sandy
Follow-up on how widely available recordings of the November training will be - Sandy
Follow-up on feedback about inter-rater reliability for Fidelity assessments- Mary
Meet with site leads/privacy officers re: logistics of ACCESS database - Sandy/Mary
Follow-up on meeting frequency - Janet
MINUTES
SITE TRAINING UPDATE
First Wave of Training
The 3 original EPI-SET sites followed a specific training process for the first wave of training: 7-8 pre-training calls with NAVIGATE trainers that covered 7 modules of NAV content followed by 2 days of in-person training.
Post-training calls specific to each role led by specialized trainers. The calls go on for a year and are set to finish in March 2020.
ECHO calls started about 5 months after the in-person training and will finish in about 10 months. All sites join the ECHO calls to discuss case studies and support learning.
Second-Wave/Current Training
Sarah Bromley (Slaight) has developed a Training Handbook/Toolkit that is available on the EPI-SET Website. It outlines all the current required training.
There is in-person training on November 18th in Toronto, for all new sites and new members from current sites.
The in-person training covers NAVIGATE content, implementation, delivery and strategies
Before attending the in-person training, staff need to: - Complete the readings in the training handbook, which include an overview of EPI-SET, NAVIGATE and role-specific content. The readings replace the pre-training phone calls. - Watch an animation – still in development, will show how the NAV model works. - Attend 2 one-hour webinars – one is general for all roles and the second is role-specific.
The upcoming trainings require less travel and aim to improve accessibility in anticipation of potential spread of the NAVIGATE model.
November training will be recorded this time as part of the sustainability plan
NAVIGATE manuals are publicly available on the NAVIGATE website.
REVIEW OF EPI FIDELITY METHODOLOGY AROUND RELIABILITY
The current model is set-up so that there are 2 raters for all data that comes in.
The raters get together in-person and rate all material together.
Afterwards there is a consensus meeting with Fidelity experts to weigh in on the ratings and discuss any tricky ratings.
Current question: would inter-rater reliability be better if the two assessors rate entirely separately and then come together at the consensus rating meetings?
Feedback from this meeting: - Our current process aligns with approach demonstrated to be reliable in the U.S. - The current benefit of rating together is to include different perspectives. - The current process seems fine for now, but may want to demonstrate inter-rater reliability in the future by including different groups of raters
Two assessments have been completed; nearing the end of data collection for the 3rd assessment.
Will be booking the consensus rating meetings soon.
Additional assessments will occur as new sites come on and there will be another full round later in the project.
MEASURING NAVIGATE DELIVERY – UPDATE AND DISCUSSION
We are currently in the process of identifying how to use an ACCESS database to collect info about NAVIGATE delivery.
Sites would fill in the database to measure fidelity, delivery, and penetration.
Steering Committee strongly supports the need for the database to be in ACCESS.
Current Challenges:
Privacy issues – current sites cannot have ACCESS as a part of their EMR, would need to be external to EMR - Currently in discussion with the Forensic Unit at CAMH. They use an ACCESS database in a similar way we intend to. Their unit uses an EMR and the database captures service delivery measures that the EMR cannot. They find the database clinically useful because they can use graphs produced by the database to present at team meetings. The database contains a patient’s MRN but no other identifying information.
Who will build the database? Due to time constraints/capacity, hoping to get someone from within CAMH, but will consider developing if we can’t find anyone.
Double-documentation: sites have expressed concern about having to document module in the EMR and ACCESS. We want to ensure the database meets everyone’s needs –clinically and for research. - An advantage of ACCESS is that it can generate reports or use copy/paste to upload data to the EMR to help reduce duplication.
Differences in EMRs across sites are another logistical issue, so building the database likely won’t be one-size-fits-all.
MEETING FREQUENCY
Feedback from the group indicated that the current frequency is okay, but can also be adjusted depending on need.