Why Competency Tracking Breaks Down
A program director stares at a resident’s file, seven rotations in. The decision: ready for unsupervised call, or not yet. The file has grades, a few narrative comments, and a gut feeling. None of it says exactly when readiness arrived.
That is not a paperwork problem. It is a measurement problem. Norman, Norcini, and Bordage argue that pinpointing the exact moment a trainee becomes competent requires an unrealistic sample of observations (Norman et al., 2014). Their own data illustrate this. A progress test given every four months to medical students produced a very gradual growth curve. Defined at a modest 50% threshold, competency could be reached anywhere from 60 to 120 months into training. The gap depended partly on which test happened to land on a harder day.
Tracking systems that promise a precise readiness date are promising something the data cannot support. The realistic goal is not precision. It is enough evidence, gathered consistently, to make a defensible call.
What Entrustable Professional Activities Actually Track
Entrustable Professional Activities exist to answer one question. Can this trainee be trusted to do this specific task without supervision? Ten Cate introduced the concept in 2005 to close the gap between abstract competencies and actual clinical work (ten Cate, 2005). An EPA is a real unit of professional work, like performing a venipuncture or chairing a multidisciplinary meeting. A supervisor can observe it, measure it, and eventually hand it off. Much of that observation happens in structured simulation before it ever reaches a live patient. That is exactly the terrain our guide to medical simulation scenarios covers.
That shift changes what tracking is for. Iobst and colleagues laid out the contrast plainly. Traditional postgraduate training centered on content and a single subjective measure at the end of a rotation (Iobst et al., 2010). Competency-based training centers on outcomes instead, using criterion-referenced evaluation and a portfolio of multiple objective measures. The goal is no longer proving a resident sat through the right rotations. It is proving they can actually do the work.
The Eight-Attribute Test for a Real EPA
Ten Cate’s original eight attributes still work as a test for what belongs in a tracking system. A genuine EPA is part of essential professional work, leads to a recognized output, and is restricted to qualified personnel (ten Cate, 2005). It must also be independently executable within a defined time frame. Tasks that fail this test, like general professionalism or being a team player, are not EPAs. They are qualities a trainee brings to many EPAs at once. Tracking them as if they were discrete tasks turns a system into paperwork for its own sake.
That is the trap. A program that tracks everything ends up tracking nothing well. The fix is not a longer checklist. It is a shorter list of real units of work, each one worth an actual entrustment decision. Getting that list right is largely what the challenges of implementing EPAs come down to.
The Real Cost of Getting It Wrong
Done well, competency tracking catches problems early. Done poorly, it becomes a box-ticking exercise that misses the residents who actually need help.
Holmboe and colleagues documented both sides from the ACGME’s own implementation experience. On one campus, a clinical competency committee’s first review of internal medicine milestones surfaced several curricular gaps (Holmboe et al., 2016). Those gaps had gone unnoticed for years. On another, the same process caught struggling residents earlier than the old system ever had. That gave them time to actually remediate. Programs reported milestones gave faculty a shared language for feedback that had previously been vague.
The same paper is honest about the cost. Faculty needed new assessment skills that their own training never covered. Milestone data entry competed directly with clinical productivity targets. As one program bluntly put it in the paper’s own table of findings, “RVUs always win.” Some programs adopted the forms without adopting the underlying judgment those forms exist to support.
Schumacher and Turner’s review of a decade of EPA implementation found a more troubling pattern still. Multiple studies describe trainees finishing a stage of training without meeting the expected EPA level (Schumacher & Turner, 2021). They were not flagged, not remediated, simply passed through. A tracking system that cannot catch this is not simplifying anything. It is just producing paperwork that looks like oversight.
Why Adding More Forms Isn’t the Fix
More tracking data does not automatically mean better tracking. Sometimes it means less honest data.
Heeneman and Driessen studied portfolios, the most common tool for collecting this evidence (Heeneman & Driessen, 2017). They work only under specific conditions: mentoring, some flexibility for the learner, and a genuinely supportive environment. Strip those away and portfolios become exactly what critics fear. Acceptance of portfolios as a reflection tool ranged from 72% among surgical residents down to 42% among general practice trainees. The gap tracked closely with how obligatory and assessed the reflection felt. Residents who already saw themselves as reflective practitioners felt differently. The requirement got in the way of their own approach to learning.
Self-assessment alone does not fill the gap either. Iobst and colleagues cite a body of research showing physicians in training are simply inaccurate at judging their own performance (Iobst et al., 2010). The fix they point to is not more self-rating forms. It is what Iobst and colleagues describe, citing Eva and Regehr’s term, as self-directed assessment seeking. Trainees actively pull in feedback from multiple outside observers, rather than programs asking them to grade themselves.
The pattern repeats across every tool. Volume without structure produces noise. What actually simplifies tracking is fewer, better-defined data points, not more of the same kind.
How to Simplify Clinical Competency Tracking
Three principles make tracking data usable instead of overwhelming: saturation, triangulation, and aggregation. Mash and colleagues built South Africa’s national family medicine EPA framework directly around these three ideas (Mash et al., 2024). The framework is worth borrowing.
That matters beyond South Africa itself. A scoping review of 80 EPA studies found the evidence base is overwhelmingly American and Dutch (Shorey et al., 2019). Africa and Asia are barely represented.
Saturation, Triangulation, and Aggregation Defined
Saturation asks a simple question. Are there enough entries for this specific EPA to support a decision? One observation is an anecdote. A handful, collected over weeks, becomes evidence. Triangulation asks whether those entries come from different sources. Different assessors, different settings, different assessment types, not the same supervisor’s opinion recorded five different ways. Aggregation asks whether every entry actually links back to the specific EPA it supports. Only then can the evidence pull together into one decision, instead of scattering across unconnected forms.
Ten Cate and Taylor’s guide to writing a usable EPA adds the structural half of the fix (ten Cate & Taylor, 2021). A well-built EPA description specifies exactly what counts as evidence for it. It also states how many data points are expected, and when that entrustment expires if the skill goes unpracticed. Without that structure, saturation and triangulation have nothing to organize around.
South Africa’s National EPA Rollout
South Africa’s rollout shows what this looks like at scale. The country’s family medicine programs settled on 22 EPAs nationally (Mash et al., 2024). Each follows a standard eight-part template covering specification, risk, required knowledge, and expected entrustment level. The programs then built a shared electronic portfolio specifically to aggregate the evidence for each one. Registrars earn one of five entrustment levels, from can observe only up to supervising others. Those levels themselves become the tracking system’s actual output.
None of this requires more assessment. Schumacher and Turner point to programs that shifted collection to the trainee directly (Schumacher & Turner, 2021). Programs gave trainees primary responsibility for logging their own EPA-based observations instead. That single change increased both the volume and the variety of assessments captured. In effect, triangulation and saturation solved themselves through who did the recording. No one had to add another form for faculty to fill out.
What Good Evidence for an Entrustment Decision Looks Like
Ten Cate and Taylor group the useful evidence sources into three approaches: watching, talking, and reviewing results (ten Cate & Taylor, 2021). Not every source carries the same weight, and pretending otherwise is its own kind of noise. Watching covers brief focused observation, like a mini clinical evaluation exercise, plus longitudinal tools like multisource feedback. Talking covers structured conversations such as case-based discussion or an entrustment-based discussion. Both are built around one blunt question. What would this trainee do if something went wrong? Reviewing results covers the paper trail itself, discharge letters, electronic health record entries, quality improvement reports.
A program does not need all three for every EPA. It needs enough from more than one category. That way, a competency committee is not relying on a single supervisor’s memory of a single shift. That same fragility is what made remote OSCE stations worth standardizing in the first place.
Entrustment also has an expiration date, and this is the detail most tracking systems skip entirely. Ten Cate and Taylor compare it to selling a car (ten Cate & Taylor, 2021). The guarantee holds for a certain number of years or a certain amount of use, not forever. A resident entrusted with a complex procedure may not perform it again for two years. No one should assume that resident is still competent by default. The tracking system’s job does not end at certification. It continues for as long as the entrustment does.
How Captured Video Becomes an Aggregated Data Point
A resident’s file, seven rotations in, should not still be a stack of disconnected forms and a supervisor’s memory. It should be a timeline anyone on the committee can actually read.
Video recordings fit directly into the watching category ten Cate and Taylor describe. They solve the fragility problem the other watching tools share. A single mini-CEX captures one supervisor’s impression of one encounter, gone the moment the form is filed. A recorded OSCE station or workplace encounter stays intact. A second assessor can review it for triangulation. It links to the specific EPA it demonstrates, feeding aggregation. Months later, a committee can reference it again to check whether entrustment should still hold.
That is what actually simplifies tracking. Not fewer observations, but observations that survive long enough to feed a real decision, instead of evaporating after the shift ends. Our piece on video in competency-based medical education covers how programs build this into their EPA portfolios directly. It is not an add-on. It is the evidence layer the tracking system was always missing.
Frequently Asked Questions
What’s the difference between competency-based and traditional postgraduate training?
Traditional training centers on content and time served, typically judged by a single subjective measure at the end of a rotation and compared against peers. Competency-based training centers on outcomes instead, using multiple objective measures gathered into a portfolio and judged against a fixed standard rather than the norm. The practical difference is what the program is actually trying to prove: that a resident sat through the right rotations, or that they can do the work.
What are the 5 levels of entrustment supervision?
At level 1, the trainee may only observe. Level 2 adds direct supervision, with the supervisor present in the room. By level 3, supervision becomes indirect: the supervisor stays nearby but is not present. Level 4 means distant supervision, effectively unsupervised practice. The trainee reaches level 5 when ready to supervise others themselves. Most EPAs require level 4 by the end of training.
How many assessments are needed before an entrustment decision is valid?
There is no single number that applies to every EPA. The right count depends on how many contexts the task covers, whether observations come from more than one assessor, and how high the stakes are. Colonoscopy training illustrates the range: one commonly cited rule sets the bar at 140 supervised procedures, but a study following real fellows found some needed close to 500 before reaching a 90% success rate.