5 Steps to Selecting Risk-Based Sampling Plans for Optimal Process Validation
Most engineers choose the smallest sample size possible without considering whether their process AQLs can actually meet the requirements of the sampling plan. Many times, this is the root cause of failing to meet acceptance criteria in validation studies.
Let’s say you’re preparing to conduct a process validation study of your medical device manufacturing process. After performing Installation Qualification and Operational Qualification, you’re ready to take on Performance Qualification, which consists of two parts:
- Process Performance Qualification – demonstrating that each manufacturing process (e.g., welding, bonding, and coating) consistently produces a subassembly or component that meets the process specification (e.g., tensile force, lubricity, etc.).
- Product Performance Qualification – demonstrating that the entire manufacturing process for a given finished device consistently produces a device that meets the finished product specification.
Step 1: Begin by deciding if the data you’ll be gathering will be attributes data or variables data. Attributes data are pass/fail, no/no-go, qualitative data. Variables data are continuous numeric values such as 1.1, 1.2, 1.3, and so on. Beware that although attributes lend themselves to simple assessment using gauges and jigs, the sample sizes associated with attributes are about one magnitude larger than the comparable variables sample sizes.
Step 2: Consider patient safety next. There should be a risk-based, documented LTPD (Lot Tolerance Percent Defective) or reliability level[i] for each characteristic in the product specification. For example, the failure of an angioplasty balloon to meet its minimum pressure specification is associated with a high risk of patient harm. Therefore, balloon burst pressure should be tied to a high reliability (a low LTPD).
Step 3: Review the available sampling plans for the selected LTPD. Refer to tables of attribute or variables sampling plans commensurate with the LTPD for each characteristic in the product specification. It is essential to prioritize consumer risk by ensuring that you only compare sampling plans with equivalent LTPDs. There will be many sampling plans for each LTPD, and they vary by number of samples required, accept number, and AQL (the number of defective units your process produces per hundred units produced). This is where most mistakes occur; we tend to go for the sampling plan with the smallest sample size, without regard for the AQL. Big mistake! Consider the following sampling plans, which share the same LTPD:
All three plans have the same consumer risk (LTPD≤3%), but the AQLs differ significantly. Recall that AQL is simply the number of defective units per hundred units manufactured.
#1: The smallest sample size, n=100, a=0, requires you to select 100 samples in a representative way from the batch manufactured. Measure each unit and count the number of defective units out of 100 units measured. If there are zero defective units, accept the batch. Otherwise, this batch fails to pass the validation. Remember, you must repeat this step for each of the batches making up your validation study. This sampling plan will routinely reject a manufacturing process unless it’s producing at most 0.05 defective units per hundred units, or 1 defective unit for every 2000 units produced.
#2: The largest sample size plan shown, n=210, a=2, requires you to select 210 samples in a representative way from the batch manufactured. Measure each unit and count the number of defective units out of 210 units measured. If there are two or fewer defective units, accept the batch. Otherwise, this batch fails to pass the validation. This sampling plan will routinely reject a manufacturing process unless it’s producing at most 0.39 defective units per hundred units, or 1 defective unit for every 256 units produced. However, it will cost you an extra 110 samples to get that benefit!
#3: Now let’s take note of the double sampling plan: this plan requires you to select 110 samples in a representative way from the batch manufactured. Measure each unit and count the number of defective units out of 110 units measured. If there are no defectives, accept the batch. You do not need to continue and sample any additional units for this lot. If there are two or more defectives, this batch fails the validation. Likewise, you do not need to continue and sample any additional units for this lot. However, if there is one defective unit in the sample, we have a built-in insurance policy! We do not automatically fail! If there’s one defective unit out of 110, then we take an additional sample of 120 units in a representative way, measure each one and count the number of defective units out of 120 units measured. Next, we add that number of defective units to the number of defective units from round one. If the combined number of defectives is less than or equal to two, accept the lot and move on. Otherwise, reject it—this batch of our validation study has failed. This sampling plan will routinely reject a manufacturing process unless it’s producing at most 0.28 defective units per hundred, or 1 defective unit for every 357 units produced. Round one of this sampling plan is comparable to the smallest plan, but comes with a built-in insurance policy if you encounter a defective unit or two[ii].
Step 4: Review the historic capability (AQL values) of your manufacturing process. If no historic data are available, run a pilot lot or choose a similar process for comparison.
Step 5: Choose a sampling plan that balances Consumer Risk (LTPD) with the defective rate of the process (AQL). For example, let’s say we have a legacy manufacturing process that uses similar equipment and materials to the current process we’re trying to validate. It’s running at an AQL of 0.25% (or 1 defective per 400 produced). Assuming a 3% LTPD is appropriate for consumers for this product attribute, you’ll want to maximize the likelihood that a good process will pass the validation. If you choose the plan with an AQL closest to the historic AQL of .25, you’ll have the best chance of passing this lot in the validation study. Let’s choose the double sampling plan, because its AQL of 0.28% is the closest to the historic AQL of 0.25%, and it gives a built-in insurance policy in case you need it.
If you use this method, the probability of accepting one lot is 95% if your manufacturing process has an AQL equal to that of the sampling plan. If your manufacturing process’ AQL is smaller than that of the sampling plan, your odds of passing go up; if it’s larger, then your odds go down. Now all you have to do is run enough production lots to capture all the variations the process may encompass in regular production (multiple operators, equipment lines, supplier lots, etc.). Moreover, remember probabilities are multiplicative, so the more lots you have, the smaller the likelihood of passing them all gets. From our example above, we have a process AQL of 0.25% and an LTPD of 3%. The process produces defective units at a ratio of 1:400. The sampling plan we chose has an AQL of 0.28% and an LTPD of 3%. This means if our process were producing defective units at a ratio of 1:357, we’d have a 95% probability for each individual lot to pass. Our odds are slightly better than 95% since our defective ratio is better. Let’s say we have a 96.5% probability of passing each individual lot using our chosen sampling plan. If we need to run three total lots in order to encompass all the variation we’re likely to see in manufacturing in the future, then our probability of passing all three lots in the validation study is: (.965) x (.965) x (.965) = 89.9% probability to pass the validation
In the next blog, we’ll explain how to use variables sampling plans to ensure good processes have the best chance of passing the validation. Stay tuned!
[i] Reliability = 1 – LTPD. For example, an LTPD of 0.03 has a reliability of 0.97, or 97%. [ii] Accepting a defective unit or two in this sampling plan makes some people uncomfortable, as if we’re saying we don’t mind putting defective units into the field. It’s important to remember that every sampling plan has an associated LTPD or consumer risk; no plan provides 100% protection to patients. Therefore, it’s not only ethical but also smart to select a plan that provides a level of protection commensurate with the risk to the patient if that characteristic or specification were not met clinically.