Sampling Strategy: How Much Evidence Is Enough?

One of the most common questions training providers ask us is: "How much evidence does my IQA actually need to sample?"

Is it 10%? 20%? One learner from every course? Three portfolios from every cohort?

The honest answer is: it depends on risk. And anyone who gives you a flat percentage without first asking questions about your assessors, qualifications, learners, delivery methods and previous quality assurance history is effectively guessing.

Internal Quality Assurance isn't supposed to be a mathematical exercise. The purpose of IQA sampling is to give your training centre sufficient confidence that assessment decisions are valid, consistent, fair and compliant. That means your sampling strategy should be designed around risk, not an arbitrary percentage.

Highfield describes a sampling strategy as a document setting out the frequency of sampling for qualifications being delivered and says this would normally be determined by the risk rating of tutors, assessors and qualifications, with higher risk leading to more frequent sampling.

QNUK takes a similarly clear position. Its qualification documentation states that the amount of sampling should be outlined within the centre's sampling strategy and should be risk based, with lower-risk assessors potentially being sampled less frequently than higher-risk assessors.

So let's get rid of one of the biggest myths in IQA.

There Is No Magic Sampling Percentage

You'll frequently hear statements such as: "We sample 10% of every course."

That may sound reassuring. But 10% of what? Ten per cent of learners? Ten per cent of assessments? Ten per cent of units? Ten per cent of assessors? And, more importantly: why 10%?

Imagine a training provider has two assessors. Assessor A has ten years' experience, consistently strong assessment decisions, excellent previous IQA reports, current CPD and no previous EQA concerns. Assessor B joined the centre last week, has recently qualified as an assessor, is delivering the qualification for the first time and has never previously been observed by your IQA.

Would it make sense to sample exactly the same percentage from both? Probably not.

That's why good IQA is based on professional judgement supported by documented risk assessment.

What Is an IQA Sampling Strategy?

Your sampling strategy explains how your organisation approaches internal quality assurance. It should establish the principles that determine:

• Who gets sampled
• What gets sampled
• When sampling takes place
• How much gets sampled
• Which assessment methods are sampled
• How assessors are risk-rated
• When sampling levels increase
• When sampling levels may decrease
• How findings are recorded
• How actions are followed up

Highfield describes the sampling strategy as setting the frequency of sampling and notes that risk may sometimes justify sampling every portfolio or set of course paperwork. Alternatively, observation of an assessor may be used to help determine whether their risk rating can subsequently be reduced.

The strategy therefore provides the rules. Your sampling plan then applies those rules to the actual qualifications, assessors and learners you're currently working with.

Sampling Strategy Versus Sampling Plan

The two terms are often mixed up. Think of it this way: your sampling strategy explains HOW your organisation samples. Your sampling plan shows WHAT you are going to sample.

For example, your strategy might state that newly appointed assessors are automatically classified as higher risk until they have successfully completed a defined period of monitoring.

Your sampling plan would then identify:

• The assessor
• Their risk rating
• The qualification
• Their learners
• The assessments taking place
• The units or components being sampled
• The sampling method
• Planned sampling dates

Highfield describes a sampling plan in similar terms, identifying the qualification, tutor or assessor, risk rating, relevant learners or courses, and the units or components that will be sampled. One without the other leaves an incomplete picture.

So, How Much Evidence Should an IQA Sample?

Enough to give the IQA reasonable confidence that assessment is being conducted correctly and consistently. That doesn't mean checking everything. But it doesn't necessarily mean checking only 10% either. Your decision should be influenced by the level of risk.

A good IQA asks: "What evidence do I need to see to have confidence in this assessor's decisions?" Not: "What percentage can I get away with?"

That distinction is important.

What Makes an Assessor Higher Risk?

Risk doesn't automatically mean somebody is a poor assessor. It simply means there are factors that justify greater quality assurance oversight.

A higher-risk assessor might include someone who:

• Is newly qualified
• Is new to your centre
• Is delivering a qualification for the first time
• Has limited occupational experience
• Has recently returned after a long break
• Has received previous IQA actions
• Has inconsistent assessment decisions
• Has had documentation problems
• Has missed assessment requirements
• Has generated learner complaints
• Has had concerns raised during EQA
• Has limited current CPD
• Is delivering a newly introduced qualification

A new assessor could therefore be excellent at their job but still reasonably be considered higher risk until the centre has gathered enough evidence to demonstrate otherwise. That isn't punishment. It's quality assurance.

What Might Make an Assessor Lower Risk?

Conversely, an experienced assessor might eventually justify a lower sampling frequency. Factors could include:

• Significant assessment experience
• Strong previous IQA outcomes
• Consistent assessment decisions
• Good EQA feedback
• Current occupational competence
• Regular CPD
• Attendance at standardisation
• Accurate documentation
• No outstanding actions
• Good knowledge of the qualification
• A history of reliable assessment practice

But there's an important distinction here. Lower risk doesn't mean no risk. Experienced assessors should still be sampled. People change. Qualifications change. Assessment requirements change. Awarding organisation requirements change. Complacency can also creep into organisations that have been delivering the same qualification for years.

Your sampling strategy therefore needs to maintain oversight while recognising genuine differences in risk.

Risk Isn't Just About the Assessor

This is another area providers sometimes miss. Risk-based sampling shouldn't look only at the person carrying out the assessment. You should also consider the qualification.

For example, risk might increase where:

• A qualification is newly approved
• Assessment requirements have recently changed
• Practical assessment forms a significant component
• Assessment involves safety-critical skills
• The qualification contains complex assessment criteria
• Multiple assessors are delivering the programme
• Courses are delivered across several locations
• Different delivery models are being used
• A subcontractor is involved
• Previous EQA actions relate to the qualification

A highly experienced assessor delivering a brand-new qualification could therefore still require increased sampling.

Your Learners Can Also Affect Sampling

A robust sampling strategy should ensure that your IQA gets a representative view of assessment. You don't want to repeatedly sample only the easiest portfolios. Think about the range of learners being assessed.

Your sample could include:

• Different cohorts
• Different assessment outcomes
• Different locations
• Different assessors
• Different units
• Different assessment methods
• Different delivery models
• Learners receiving reasonable adjustments
• Referred assessments where appropriate
• Borderline assessment decisions
• Different stages of the learner journey

The purpose isn't to find problems. It's to make sure the sample genuinely represents what is happening within the centre.

Don't Just Sample Learner Portfolios

Sampling is sometimes treated as: "Give me three portfolios and I'll sign them off." That is far too narrow. An effective IQA should be interested in the assessment process, not simply the final paperwork.

Depending on the qualification and assessment requirements, IQA activity could include:

• Sampling learner work
• Reviewing assessment decisions
• Reviewing assessor feedback
• Observing practical assessments
• Checking assessment records
• Reviewing questioning
• Monitoring assessment practice
• Interviewing learners
• Reviewing reasonable adjustments
• Checking assessment documentation
• Monitoring assessor performance

Highfield's external quality assurance process itself can include sampling learner work and assessment decisions, observing assessments, attending standardisation meetings and monitoring a centre's IQA activity. If external quality assurance looks beyond portfolios, your internal quality assurance should too.

Think CAMERA – Not Just Percentage

One useful approach when developing a sampling strategy is to think about the CAMERA principle.

• Candidates: are you sampling across different learners, cohorts and outcomes?
• Assessors: are all assessors appropriately represented within the sampling programme?
• Methods: are different assessment methods being sampled? Written work shouldn't be the only thing reviewed if practical observation, professional discussion or questioning are also being used.
• Evidence: are different forms of learner evidence being reviewed?
• Records: are assessment records, feedback, tracking documents and audit trails accurate?
• Assessment Sites: if your centre delivers across multiple locations, are those locations represented within your quality assurance activity?

CAMERA can be a useful starting point because it stops IQA sampling from becoming: "I've checked three folders, so we're done."

Vertical Sampling

Vertical sampling follows assessment activity through a particular learner, assessor or qualification. For example, an IQA might select one learner and examine their journey across several units or assessments.

This can help establish whether:

• Assessment decisions are consistent
• Feedback is appropriate
• Documentation is complete
• Assessment criteria are being interpreted correctly
• Actions are being followed up

It gives you depth.

Horizontal Sampling

Horizontal sampling looks across the same assessment activity across several assessors or learners. For example, an IQA might examine how three different assessors have assessed the same practical component.

This can identify inconsistencies between assessors. Assessor A may be applying the assessment requirements correctly. Assessor B may be applying a much higher standard than required. Assessor C may be accepting evidence that isn't sufficient.

Individually, the learner files might look perfectly reasonable. Only by comparing assessment decisions do you identify the inconsistency. That's why horizontal sampling is so important for standardisation.

Interim Sampling Matters

Don't wait until everything has finished before carrying out IQA. Interim sampling allows the IQA to identify problems while the learner journey is still taking place. That means problems can potentially be corrected before they affect an entire cohort.

Highfield also recognises interim sampling within its external quality processes, describing it as reviewing assessment decisions and evidence while a learner has not yet completed the qualification.

Consider the difference. You discover an assessor has misunderstood an assessment requirement after two learners. That's manageable. You discover it after 120 learners. That's potentially a very different conversation.

Summative Sampling Still Has a Place

Summative sampling takes place after assessment has been completed. It allows the IQA to check the completed assessment journey and determine whether assessment decisions are sound.

A balanced IQA strategy will often use both interim sampling, to identify problems early, and summative sampling, to confirm final assessment decisions. The exact balance should again be determined by qualification requirements and risk.

What Happens When an IQA Finds a Problem?

Sampling levels should not be permanently fixed. Your strategy needs to respond to what the IQA discovers.

Imagine an assessor is currently classified as low risk. The IQA samples their assessment activity and discovers inconsistent assessment decisions, missing evidence, weak learner feedback and incomplete assessment records.

The correct response isn't: "They're only on 10% sampling, so we'll check them again next quarter." The risk profile has changed. Sampling may now need to increase.

Actions might include: Identify Concern → Increase Sampling → Support Assessor → Standardise → Re-sample → Review Risk Rating.

That's a functioning quality assurance system. Sampling should be dynamic.

What About 100% Sampling?

There will be situations where 100% sampling may be appropriate, or where specific qualification, awarding organisation or assessment requirements dictate particular levels of oversight.

For example, increased or complete sampling might be justified for:

• A new assessor
• A serious assessment concern
• A newly introduced qualification
• Repeated IQA actions
• Significant EQA concerns
• Assessment malpractice concerns
• An assessor undergoing additional support

Always check the current qualification specification, assessment strategy and awarding organisation requirements before deciding sampling levels. Where an awarding organisation specifically requires a particular level or method of IQA, that requirement takes priority over your centre's general approach.

Your Sampling Strategy Needs to Be Defensible

Imagine your EQA asks: "Why did you only sample two learners from this assessor?"

A weak answer is: "Because we always sample 10%."

A much stronger answer is: "This assessor is currently rated low risk based on previous IQA outcomes, two satisfactory assessment observations, current occupational competence, attendance at standardisation, no outstanding actions and positive previous EQA sampling. Our sampling plan therefore selected learners across two cohorts and different assessment methods. We will review their risk rating again at the next planned monitoring point."

Now you've demonstrated professional judgement. You haven't simply produced a percentage. You've produced a rationale. And that's what good quality assurance looks like.

Document Why You Made the Decision

If your IQA decides to increase sampling, record why. If your IQA decides sampling can safely decrease, record why. If a particular learner wasn't sampled, your sampling plan should help explain how that decision fits within your strategy.

Highfield specifically notes that where a learner, course or unit has not been sampled because of the centre's sampling strategy and plan, those documents can be supplied to demonstrate why IQA sampling wasn't completed on that particular piece of work.

That tells us something important. Not sampling something isn't necessarily the problem. Being unable to explain why you didn't sample it can be.

The Percentage Is Only the Beginning

Let's say two providers both sample 20%. Provider A selects the first two learner portfolios alphabetically from every ten learners. Provider B deliberately samples across different assessors, different cohorts, practical and theory assessment, different learner outcomes, interim and summative stages, different locations and different risk profiles.

Both can truthfully say: "We sampled 20%." But they're not necessarily carrying out the same standard of quality assurance.

That's why percentages alone tell you very little. Coverage matters. Risk matters. Timing matters. Professional judgement matters. And above everything else: the quality of the sampling matters.

How The IQA HUB Can Help

This is one of the reasons we're developing The IQA HUB. Many small and independent training providers deliver excellent training but don't necessarily employ a full-time quality assurance manager. Others have qualified IQAs but need additional specialist support for certain qualifications.

The IQA HUB is designed to connect training providers with appropriately qualified and experienced IQA professionals who can support their quality assurance requirements.

Build a Risk-Based Sampling Strategy

Through The IQA HUB, providers can access IQAs who can help review or develop:

• IQA strategies
• Sampling strategies
• Sampling plans
• Assessor risk-rating systems
• Qualification risk ratings
• Interim sampling programmes
• Summative sampling
• Horizontal sampling
• Vertical sampling
• Assessor observation plans
• Standardisation programmes
• IQA action plans

The objective isn't to give every provider the same generic spreadsheet. It's to help create a system appropriate to the centre.

Find an IQA Who Understands Your Qualification

Holding an IQA qualification doesn't automatically make somebody the right IQA for every qualification. Occupational competence matters. Qualification requirements matter. Sector knowledge matters.

The IQA HUB is designed so IQAs can provide information about their:

• Teaching qualifications
• Assessor qualifications
• IQA qualifications
• Occupational qualifications
• Specialist areas
• Awarding organisation experience
• CPD
• Sector experience

Training providers can therefore look for quality assurance professionals whose competence matches the work they actually need.

Independent Sampling Reviews

Sometimes the most valuable thing a training provider can have is a fresh pair of eyes. You've looked at the same documents for months. Your assessor has looked at them. Your regular IQA has looked at them. Everyone becomes familiar with the system.

An independent IQA review can ask: "Why are you doing it this way?" That's often where weaknesses are discovered.

The IQA HUB can help centres access independent professionals to review areas including:

• Existing sampling plans
• Assessor risk ratings
• IQA reports
• Assessment decisions
• Sampling coverage
• Standardisation
• Previous EQA actions

That can be particularly valuable when preparing for an upcoming EQA review.

Move Away From "How Little Can We Sample?"

The better question isn't: "What's the minimum amount of sampling we can get away with?" It should be: "What level of sampling gives us confidence that our assessment decisions are safe?"

That's a completely different mindset. A good IQA sampling system shouldn't create unnecessary paperwork. But neither should it reduce quality assurance to a meaningless percentage. The aim is proportionate quality assurance.

Higher risk? Increase your oversight. Evidence demonstrates risk has reduced? Review your sampling level. New problem identified? Increase sampling again. That's what risk-based sampling actually means.

If You Can't Explain Your Sampling Decision, Review It

Here's a simple test. Pick any learner who has been sampled. Ask your IQA: "Why did you choose this learner?" Then pick one who wasn't sampled. Ask: "Why wasn't this learner selected?" Then ask: "Why is this assessor classified as low, medium or high risk?"

If the answers are clear, evidenced and consistent with your sampling strategy, you're probably heading in the right direction. If the answer is "Because we've always sampled 10%", it may be time to review your approach.

How Much Evidence Is Enough?

So let's return to our original question. How much evidence does an IQA need to sample?

Enough to provide reasonable confidence that assessment decisions across your centre are consistent, valid, fair and compliant. Sometimes that might mean a relatively small sample. Sometimes it might mean significantly more. And occasionally it could mean everything.

The correct level should be determined by: Risk + Coverage + Evidence + Professional Judgement + Awarding Organisation Requirements. Not an arbitrary percentage.

Because effective internal quality assurance isn't about checking as little as possible. It's about checking enough of the right things to know that your assessment system works. And being able to prove why you made that decision.

Need Help With Your IQA Sampling Strategy?

The IQA HUB is being developed to connect training providers with experienced IQA professionals who can help build, review and manage robust internal quality assurance systems.

Whether you need assistance developing a sampling strategy, risk-rating your assessors, reviewing learner evidence, completing independent IQA sampling or preparing for an EQA visit, our aim is simple: give training providers access to the right quality assurance expertise when they need it.

The IQA HUB. Connecting Training Providers with Quality Assurance Professionals. IQA Support • Sampling • Compliance • Quality Assurance • EQA Readiness