method validation research paper

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Method Validation Approaches for Analysis of Constituents in ENDS

Samantha m reilly , phd, tianrong cheng , phd, jenna dumond , phd.

• Author information
• Copyright and License information

Correspondence Dr Reilly; [email protected]

We assessed how many peer-reviewed publications reporting chemical quantities and/or yields from electronic nicotine delivery systems (ENDS) have included adequate method validation characteristics in the publication for appropriate interpretation of data quality for informing tobacco regulatory science.

We searched 5 databases (Web of Knowledge, PubMed, SciFinder, Embase, EBSCOhost) for ENDS publications between January 2007 and September 2018. Of the 283 publications screened, 173 publications were relevant for analysis. We identified the publications that report a certain degree of control in data quality, ie, the publications that report marginally validated methods (MVMs). MVMs refer to the methods that: (1) report 3 or more International Conference on Harmonisation (ICH) method validation characteristics, (2) state the method was validated, (3) cite their own previous publication(s) that report MVMs, or (4) use a method within the accreditation scope of an accredited laboratory.

Overall, 97 publications (56%) report MVMs in their studies. This percentage also reflects the publication distribution for the majority of the 28 chemicals measured by MVMs.

Conclusions:

This study highlights the need for reporting sufficient validation characteristics following appropriate guidance to ensure the accuracy and reliability of the published analytical data for proper data interpretations that may support policy.

Keywords: ENDS, constituents, analytical method validation, International Conference on Harmonisation (ICH) guidelines

Electronic nicotine delivery systems (ENDS) are battery-powered devices that are used to generate an aerosol. ENDS generate aerosols by heating a liquid that usually contains nicotine, often called an e-liquid. The e-liquids typically contain propylene glycol, glycerol, water, nicotine, and various flavor additives. In 2016, the United States Food and Drug Administration (FDA) finalized a rule to regulate all other tobacco products not covered in the 2009 rule, including ENDS. 1 Through this regulation, ENDS are subject to the premarket requirements of the Federal Food, Drug, and Cosmetic Act. For ENDS premarket applications, the guidance published by FDA recommends the analysis of harmful and potentially harmful constituents (HPHCs) in e-liquids and aerosols. 2 In addition, FDA has proposed adding 19 additional chemicals to the HPHC list, many of which apply to ENDS products and aerosols. 3

To understand the potential risks in using ENDS products, it is helpful for both the product manufacturers and researchers to collect scientific data for chemicals that expected to be in ENDS e-liquids and aerosols, their respective quantities and/ or yields, and the factors that contribute to their quantities and/or yields. Since 2007, researchers have published thousands of papers about ENDS, of which hundreds report the detection of harmful chemicals. Many of these publications report the quantities of chemicals observed in the e-liquid or the yields in the aerosols as well as those from leaching or extraction from the metal coils in the ENDs device. However, not all publications report sufficient information regarding test method validity. If a publication lacks adequate method validity information, the accuracy of the data reported and the reliability of using the data in scientific interpretations is unknown. Thus, it is unclear if studies without adequate method validation details in the publications can accurately and reliably inform tobacco regulatory science.

The critical method validation characteristics that are included for this study are based on the International Conference on Harmonisation (ICH) guidelines for analytical method validation. 4 The ICH was established in 1990 and is a tripartite body sponsored by regulators and pharmaceutical industry from the 3 major pharmaceutical markets: The United States (US), European Union (EU), and Japan, as well as the 6 representatives from regulatory authorities and industry in these 3 regions. The ICH is supported by a secretariat provided by the International Federation of Pharmaceutical Manufacturers Association (IFPMA). 5 Therefore, these guidelines are well-established and globally recognized for pharmaceutical industries. The ICH guidelines are set to harmonize, among others (eg, laboratory practice, manufacturing practice, or clinical practice for quality, safety, and efficacy) the analytical method validation requirements globally so that both industry and regulators worldwide are reading the same guidelines for data quality controls. 5 – 7 ICH validation guidelines list 7 validation characteristics for intended purposes: accuracy, precision, specificity, limit of detection (LOD), limit of quantification (LOQ), linearity, and range. A test method should ideally include all 7 of these characteristics to be interpreted as validated. Although the ICH guidelines are a standard for the pharmaceutical field and not ENDS, the assessment of which ENDS publications could potentially contain fully or partially quality controlled data (referred to as ‘quality data’ for the rest of publication) is also useful to determine how far the current ENDS literature is from meeting ICH guidelines. Therefore, in this study, we tally the publications that report fully validated analytical methods by ICH standards examined at a high-level (eg, not examining every detail of ICH standards) to determine whether the provided method information indicates that data quality is controlled. We also report the publications that include partial or “marginal validation criteria” (ie, those including at least 3 validation characteristics, or meet other specified criteria; further details are in Section 2.2 of this publication) for the test methods. This “marginal” validation classification suggests that the publication has the potential to contain quality data. It does not make any statement on whether a publication contains quality data. The publications that have the “marginal” validation classification would need to be examined further to determine if they do contain quality data that may support scientific interpretations of the data for policy implications. This further examination may need more validation data included in the publications from the authors. The classification of “marginal validation” and any evaluation criteria set in this study are limited to this paper and should not be confused with complete method validation to ICH standards and do not reflect the FDA position in data quality requirements.

The objective of this study is to determine the number of publications that report fully validated or “marginally” validated analytical methods by comparing ENDS publications to the ICH guidelines at a high-level to determine whether the provided method information indicates that data quality is controlled. First, we tally the number of publications that measure the chemical quantities in e-liquids, yields in ENDS aerosols, quantities from ENDS leachables and extractables, or any combination of these sources. To assess the areas where ENDS validation procedures may need more research, we also evaluate the most and least common validation characteristics across publications. The reported validation characteristics in ENDS publications are then compared to the most relevant recommendations from the ICH guidelines. This comparison will identify how reporting of validation characteristics in ENDS research could improve. Altogether, this information will determine the percentage of the current ENDS literature that has the potential to provide quality data to inform tobacco regulatory science reliably. The information also will determine the areas of method validation that publications could address to support future scientific studies.

Systematic Literature Selection

We conducted systematic literature searches for articles published between January 2007 and September 2018 to identify research related to ENDS and the chemicals of public interest. We conducted these literature searches weekly between 2013 and 2018. The 28 chemicals of public interest included in this publication are chemicals highly studied in ENDS (eg, propylene glycol, glycerol, and nicotine) and/or of specific toxicological concern (eg, 2,3-pentanedione, acetic acid, acetaldehyde, acetoin, acrolein, acrylonitrile, benzene, benzyl acetate, butyraldehyde, cadmium, chromium, crotonaldehyde, diacetyl, diethylene glycol, ethyl acetate, ethylene glycol, formaldehyde, isobutyl acetate, lead, n-butanol, nickel, nicotine, N-nitrosonornicotine (NNN), 4-[Methyl(nitroso)amino]-1-(3-pyridinyl)-1-butanone (NNK), propionic acid, and toluene). 2 , 3 We searched 5 reference databases (Web of Knowledge, PubMed, SciFinder, Embase and EBSCOhost) using a set of relevant search terms used singly or in combination ( Table S1 ). The search date range was between January 2007, when the first chemistry study of e-cigarettes was published, and September 2018, when this review began. To be considered for inclusion, the publication had to (1) be written in English, (2) be published in a peer-reviewed journal, and (3) deal partly or exclusively with chemistry. Excluded sources included (1) indirect data sources, such as peer-reviewed review papers, (2) patents, and (3) conference presentations. Altogether, 283 publications remained after we applied the inclusion and exclusion criteria. Figure 1 shows how many publications were removed at each step. A full-text review screened these 283 publications to see if the publication reported any chemical quantities in e-liquids, yields from ENDS aerosols, quantities from ENDS leachables and extractables, or any combination of these sources. Our definition of chemical quantities and/or yields does not exclude qualitative studies as these papers measured whether chemicals are present or not with some publications even making estimates for what the relative quantities and/or yields are. Following the full-text review, 173 publications were relevant for this analysis. 8 – 180 Double-coding was not performed on these classifications due to resource limitations.

Sorting Procedure for Determining if a Publication Should Be Included in the Analysis

Selection of Publications that Report “Marginally” Validated Methods (MVMs)

To assess the validation status of the analytical methods in the ENDS literature accurately, the number of publications that do and do not report method validation information are determined. Each ENDS publication is checked for the presence of validation characteristics to determine if the publication reports validated methods. For this study, there is a focus on the 7 validation characteristics defined by the globally recognized ICH validation guidelines. 4 For accuracy and repeatability, the ICH guidelines also recommend a minimum number of replicates for each assessment. However, this study compares the methods reported for assessing the validation characteristics, but excludes details, such as the exact number of replicates used. As this study focuses on comparing ENDS literature to the ICH guidelines through an overall assessment to determine whether the provided method information indicates that data quality is controlled, the exclusion of these validation details does not affect the evaluation outcome of this study. For this study, the 7 method validation characteristics are as follows:

• Determining the percent recovery by adding a known amount of analyte to the sample immediately after sample collection or the earliest available time during sample preparation after collection
• Reporting the difference between the mean and true value for a certified reference material (CRM)
• Repeatability : degree of variability under the same operation conditions within a day
• Intermediate Precision : degree of variability within a laboratory across days, ideally with different analysts and, if possible, different instruments
• Reproducibility : degree of variability between laboratories

Selectivity : the ability to measure unequivocally the analyte in the presence of other components expected to be present, such as impurities, matrices, and degradants. ICH guidelines mention 3 separate ways to measure specificity but do not recommend one method over another as each way is dependent on the objective and matrix.

Limit of Detection (LOD) : the lowest amount of analyte in a sample that can be detected. ICH guidelines mention 3 separate ways to measure LOD but do not recommend one method over another.

• Limit of Quantification (LOQ) : the lowest amount of analyte in a sample that can be quantitatively determined with reasonable accuracy and precision. ICH guidelines mention 3 separate ways to measure LOQ but do not recommend one method over another.

• Minimum of 5 calibration points to establish the calibration curve
• Coefficient of correlation (r) or coefficient of determination (R 2 )
• Slope and y-intercept of the calibration curve

Range : the range between the upper and lower amounts of analyte in a sample for which the procedure has a suitable level of accuracy, precision, and linearity. The ICH guidelines recommend that the range should be at least 80% to 120% of the target concentration. However, there is not yet well-established ranges for all 28 chemicals of public interest in each matrix. Therefore, in this study, we do not analyze whether the publications follow the range recommendations in ICH guidelines.

Criteria for Determining if a Publication Reports MVMs

Ideally, an analytical method should evaluate all 7 characteristics to be interpreted as validated. However, both within and outside of the ENDS field, many publications do not report all 7 characteristics, instead relying on a subset to demonstrate proper validation. Thus, the inclusion of a certain level of validation characteristics to suggest marginal data quality (ie, the publication has the potential of reporting quality data) also is examined. By determining the publications with marginal data quality, the extent that the current ENDS literature adopts the ICH guidelines for validation containing accurate and reliable information for appropriate interpretations can be established. However, marginal data quality does not imply the publication reports adequate or acceptable quality data, but only the potential to report it. A publication is classified as reporting MVMs if it matches one of the following 4 criteria:

Reports 3 or more method validation characteristics for each method reported in the publication to measure the chemical quantities and/or yields;

States the method is validated without reporting any method validation characteristics;

Conducts the testing by an accredited chemical testing laboratory (eg, ISO 17025) and the methods reported are within the laboratory’s accreditation scope; or

Cites a previous publication from the same author(s) and the cited publication contains 3 or more method validation characteristics. Citations from different authors for method validation are not accepted as the validation characteristics generally are not transferrable between the publications.

These criteria are set for this study only and do not represent any FDA regulatory positions. The first criterion above using 3 or more method validation characteristics as classification criteria can appear arbitrary. However, it is based on the understanding that reporting 3 or more method validation characteristics in a study generally implies that the study has worked on data quality control and has a higher possibility of generating valid data. These studies would benefit from further examination to determine whether the data quality is well controlled. The second criterion was selected as the claim cannot be refuted from a literature review. The third criterion was selected as validation is a mandatory part of ISO accreditation. Lastly, the fourth criterion was selected as many papers from the same laboratory refer to a previous publication instead of repeating details that already have been published. Figure 2 details the validation sorting and shows the number of publications that matched each criterion. Figure 2 also details the number of qualitative studies identified in this analysis.

Sorting Procedure for Determining if a Publication has Marginal Validation Methods (MVMs)

Overview of Publications and Method Validation Status

There were 173 publications included in the analysis; regarding the analytical method, 97 of these report MVMs, 7 report MVMs for some but not all chemicals of public interest, and the remaining 69 do not report MVMs ( Figure 2 ). When examining the breakdown by publications reporting single or multiple chemicals, publications describing a single chemical report MVMs more frequently (63% vs. 52%; Table 1 ). Only a small fraction of the publications reporting multiple chemicals report MVMs inconsistently within the publication (6%).

The Number of Publications that Report Multiple Chemicals of Public Interest or a Single Chemical of Public Interest and the Breakdown of Their Validation Status Based on This Publication’s Definition of Marginal Validation

References 10 – 12 , 14 , 17 , 21 – 24 , 27 , 28 , 30 , 32 – 37 , 41 , 43 , 45 , 48 – 50 , 52 , 54 – 58 , 61 – 63 , 65 – 69 , 71 , 76 , 78 – 81 , 83 – 85 , 87 – 93 , 95 – 99 , 103 – 106 , 109 , 110 , 114 , 115 , 117 , 119 – 122 , 124 – 127 , 130 – 134 , 136 , 137 , 139 , 140 , 142 , 144 , 146 , 147 , 150 – 153 , 155 , 156 , 158 – 161 , 163 , 164 , 166 – 168 , 170 – 177 , 180

References 8 , 9 , 13 , 15 , 16 , 18 – 20 , 25 , 26 , 29 , 31 , 38 – 40 , 42 , 44 , 46 , 47 , 51 , 53 , 59 , 60 , 64 , 70 , 72 – 75 , 77 , 82 , 86 , 94 , 100 – 102 , 107 , 108 , 111 – 113 , 116 , 118 , 123 , 128 , 129 , 135 , 138 , 141 , 143 , 145 , 148 , 149 , 154 , 157 , 162 , 165 , 169 , 178 , 179

Breakdown of Validation Status by Each Chemical of Public Interest

When examining each chemical for the number of publications that report MVMs, major differences exist across chemicals ( Table 2 ). The chemicals that are reported by fewer than 20 publications for quantities and/or yields in each matrix were inspected to determine if the publications report MVMs. Chemicals such as 2,3-pentanedione, acrylonitrile, butyraldehyde, crotonaldehyde, NNK, NNN, and propionic acid have 60% or more of the reporting publications that report MVMs for both e-liquid and aerosol analyses. Benzyl acetate and toluene have 60% or more of the reporting publications that report MVMs for e-liquid analyses, but not for aerosol analyses. Diethylene glycol has 60% or more of the reporting publications that report MVMs for aerosol analyses but not e-liquid analyses. Acetic acid, acetoin, benzene, cadmium, chromium, diacetyl, ethyl acetate, ethylene glycol, isobutyl acetate, lead, n-butanol, and nickel have less than 60% of the publications that report MVMs for either matrix. Although cadmium, chromium, lead, and nickel are reported in leachables and extractables from metal coils in ENDS, none of the publications reported MVMs.

Number of Publications Using Marginally Validated Analytical Methods (MVMs) for Each Chemical of Public Interest (L&E: Leachables and Extractables)

Next, the chemicals that are reported by more than 20 publications for at least one of the matrices to determine if the publication reports MVMs are examined. These chemicals include acetaldehyde, acrolein, formaldehyde, glycerol, nicotine, and propylene glycol. Acetaldehyde, acrolein, formaldehyde, and nicotine all have been classified as reporting MVMs for approximately 60% and 50% of the publications that examine quantities in e-liquids and yields in aerosols, respectively. On the other hand, glycerol and propylene glycol have 36% and 35%, respectively, of the reporting publications reporting MVMs for e-liquid analyses and only 19% for aerosol analyses. Supplemental Table S2 lists details on the range of chemical quantities and yields reported, type of ENDS devices used, and analytical instrumentation utilized in the publications reporting MVMs.

Analysis of Method Validation Characteristics

In this review, we compare the method validation characteristics across all 173 publications. Detailed results are in Supplemental Table S3 . For methods measuring the chemicals of public interest in ENDS, the most commonly reported test method characteristic is LOD (47%) and the least common is specificity (9%). Overall, 25% of the publications do not report any validated characteristics. A publication with no reported validation parameters could still report MVMs by our criteria (Section 2.3 of this publication). Five combinations are present across more than 4 publications: all 7 characteristics (10, 5.8%), all characteristics except specificity (16, 9.2%), reporting LOD and LOQ (10, 5.8%), only reporting LOD (18, 10.4%), and only reporting LOQ (12, 6.9%). Many publications (64, 37%) have combinations that are infrequent, with each combination only appearing in one to 4 publications. As a large percent of combinations are infrequent, the characteristics that are most commonly reported together (overlap) are examined. To do this, the number of publications reporting 2 characteristics together are determined ( Table 3 ). Accuracy and precision are the most common overlapping characteristics along with linearity and range. These pairs being the most common is reasonable as they are frequently measured together. For example, repeated sample injections used to calculate precision also could provide accuracy while a calibration curve could provide both linearity and range.

Number of Times that 2 Method Validation Characteristics are Present in the Same Publication (LOD: Limit of Detection; LOQ: Limit of Quantification)

Reported Methods for Assessing Validation Characteristics Compared to Recommendations in the ICH Guidelines

Whereas the ICH guidelines list recommendations for all 7 method validation characteristics, the recommendations examined in-depth in this study are those for accuracy, precision, and linearity. These 3 characteristics are simple to translate and interpret across ENDS publications. In this section, the various approaches reported in the publications are compared to the recommended approaches in the ICH guidelines. These comparisons help assess the difference between the approaches reported in the current ENDS literature and those recommended by the ICH guidelines.

Table 4 lists the approaches reported in the ENDS publications to assess accuracy. Whereas the ICH guidelines recommend using certified reference material (CRM) for determining accuracy, most publications do not specify whether the reference material is certified. Thus, the number of publications that determine accuracy by using any type of reference materials are investigated. It is accepted that the data from these publications are likely less accurate than if the analysis was limited to only those publications using a CRM for determining accuracy. Most methods reporting accuracy report one of the recommendations in the ICH guidelines: (1) testing recovery by spiking a sample with a reference material after sample collection but before sample preparation or (2) using a reference material itself to test recovery. Twenty publications categorized as reporting MVMs do not clearly describe the accuracy assessment process. These publications typically fall into 2 categories: (1) the publication does not mention where in sample preparation they spike the sample to test recovery or (2) the publication does not include any accuracy assessment information. Additionally, although not examined in detail, the ICH validation guidelines are for demonstrating that the analytical procedure is suitable for intended purpose. However, one publication reports using a test cigarette to measure validation characteristics. As cigarette smoke differs in sample matrix from those of e-liquids and ENDS aerosols, this method to report accuracy is not appropriate.

Comparison of How Publications Assessed Accuracy, Precision, and Linearity

Recommended by the ICH Guidelines

ICH Guidelines recommend using a certified reference material (CRM) for comparisons; however, as most publications did not specify if they used a CRM or a reference material from a third party, the count is how many determined accuracies by comparing to any reference material.

ICH Guidelines specify reporting operating conditions, such as the personnel or instrumentation, for these elements of precision. Specifically, repeatability should use the same operating conditions while intermediate precision should use different operating conditions. However, as most publications do not report if they use the same or different operating conditions, operating condition are not included in the classification of these elements.

The ICH guidelines recommend precision be measured using both repeatability and intermediate precision, as well as reproducibility when applicable. The ICH guidelines also specify reporting operating conditions, such as personnel or instrumentation, for these elements of precision. However, as most publications do not report if the operating conditions are the same, this is not included in the classification of the precision elements. With this exclusion, 22 of the 60 publications that report precision follow the ICH recommendations ( Table 4 ). Most publications at least report one of the 3 elements (ie, repeatability, intermediate precision, and reproducibility) for method precision. However, 11 publications report using repeated injections to measure precision but do not clarify the timeframe they performed these injections. Therefore, it could not be determined if they used repeatability (repeated injections within a day), intermediate precision (repeated injections across days), or both. Furthermore, although it is not tracked thoroughly, many publications do not specify if the precision measured is method precision (including sample preparation variation but excluding sampling variations) versus instrument precision (excluding variation from sampling and sample preparations). In addition, 12 publications report precision but do not report how they measure precision.

Of the 62 publications reporting linearity, approximately one-fourth of the publications describe the ICH recommendations of reporting the slope, y-intercept, and R 2 values while using a minimum of 5 calibration points for the calibration curve ( Table 4 ). Almost all the publications that do not follow the ICH recommendation guidelines at least had one of the 3 components. However, 2 publications report evaluating the method for linearity but provide no data or methods on how the evaluation was conducted.

Overview of Results

In this analysis comparing the ENDS literature to the ICH guidelines through an overall assessment, slightly less than half of the publications reporting chemical quantities and/or yields do not meet the lenient criteria set for including marginal validation (40%) information. As these publications do not appear to have included identified method validity, the accuracy of the data reported in these studies and the reliability of using these data in scientific interpretations are, therefore, unknown. As such, these studies may not be as useful for generating reliable estimates of chemical quantities, and thus, the toxic potential of ENDS as those reporting MVMs. Currently, the only publications that report any quantities of chemical compounds or elements from leachables and extractables classify as not reporting MVMs. This finding suggests a greater need for reporting validation parameters for methods that measure constituents from ENDS leachables and extractables to ensure the publications are accurate and reliable enough to support appropriate interpretations.

Conversely, slightly more than half of the publications reporting chemical quantities and/or yields meet the criteria set for marginal validation (97 out of 173). Further investigation of the publications that report MVMs shows that only 26 (out of 173) publications report at least 6 of the method validation characteristics in the ICH guidelines. This finding suggests the prevalence of reporting fully validated methods is currently small in ENDS literature (15%). However, 40 publications (23%) reported between 3 and 5 method validation characteristics. These publications could have quality data potentially, but the publications need to be examined closely before determining if the data are accurate and reliable enough to inform appropriate interpretations for tobacco regulatory science.

In addition, 31 of the 97 publications are classified as reporting MVMs, not by describing method validation characteristics, but by citing a previous publication by the same laboratory that reports MVMs (N = 15), by using accredited laboratory that includes the target analysis in its accreditation scope (N = 5), or by making a statement that their method is valid (N = 11). The classification of these publications as reporting MVMs relies on assumptions. First, for the publications citing previous publications, there is an assumption that regarding the test method nothing has changed between the cited publication and the current publication. For the publications using an accredited laboratory, there is an assumption that the test method used in the publication is the one included in the laboratory accreditation scope and within the expiration date of the accreditation. As these publications typically do not report any information regarding the methods the laboratories are accredited to perform, the reader must determine themselves if the laboratory used validated methods. Lastly, for publications that state they are validated, there is an assumption that the authors validated their method appropriately. By providing method validation characteristics, it is easier for readers to assess both the accuracy of the data reported in these studies and how reliable the data provided is for using in scientific interpretations. A potential reason that these characteristics are not reported is the frequently tight limitations on word counts for journal articles. However, authors can submit the method validation data in supplemental materials as 6 publications reporting MVMs did.

Finally, 7 publications (4%) report multiple chemicals and have a mixture of MVMs and non-MVMs across the chemicals examined. As some chemicals are not measured by MVMs, these 7 publications are not classified as publications reporting MVMs. However, as the 7 publications do meet the MVM criteria for some chemicals, these publications are in their own categories ( Figure 2 ).

Chemicals of Public Interest

The publications reporting MVMs constitute at least 60% of the publications that measure 7 chemicals in both matrices (e-liquid and aerosol). These findings might suggest that these 7 chemicals (2,3-pentanedione, acrylonitrile, butyraldehyde, crotonaldehyde, NNK, NNN, and propionic acid) are relatively well-established compared to others. However, for all 7 of these chemicals, the total number of publications is low, suggesting a potential need for more research. For the other 21 chemicals of public interest, the number of publications that report MVMs is under 60% for at least one of the matrices. This finding may help to explain some of the disputes in the literature about the relative exposure of ENDS.

Six chemicals (acetaldehyde, acrolein, formaldehyde, glycerol, nicotine, and propylene glycol) have more than 20 publications for at least one matrix. Acetaldehyde, acrolein, formaldehyde, and nicotine have approximately 60% of the publications being classified as reporting MVMs for each matrix (50%–67%). Glycerol and propylene glycol have fewer publications being classified as reporting MVMs for each matrix (19%–36%). Thirteen publications are classified as not reporting MVM, as they rely on citing standard test methods (eg, ISO 16000, ISO 4387, CRM 74, NIOSH 2551, EPA TO-11, EPA TO-1) and/or previous validated methods from other laboratories. Whereas it is beneficial to use a standard test method, or a method validated by another laboratory initially, implementing a new test method in a new location can drastically change the method validity. Therefore, quality data generation needs an analytical method that is adequately validated for the location in which it is performed.

Comparison to ICH Guidelines

When comparing the accuracy reported in the publications to the recommendations by the ICH guidelines, 2 issues arose. First, a few papers do not report how they measured accuracy, and another small group of publications report using spiking sample recovery in assessing accuracy, but does not clearly state where the sample was spiked. These 2 groups are a minority of the publications that report accuracy. However, without these data, readers do not know if the accuracy reported is method accuracy or instrumental accuracy; therefore, one cannot determine the extent of the results reliability. Second, most publications do not specify if the reference materials used are certified, which can affect the accuracy of results. Specifically, a reference material that is not certified can be much more varied in its concentration, giving more variability and uncertainty to the accuracy value reported.

When comparing the precision and linearity reported in publications to the recommendations in the ICH guidelines, the publications tend to rely on one of the elements listed in the guidelines instead of a combination as the guidelines recommend. For precision, many publications report repeatability, intermediate precision, or reproducibility, whereas ICH guidelines recommend reporting a combination of these elements. Our assessment shows that only one-third of the publications reporting precision are this rigorous. Half of the publications reporting precision report intermediate precision, which includes instrumental drift across day. However, one-third of publications that report precision do not clearly state how precision is measured. Knowing which element of precision is used to assess the method can help readers interpret the reliability of the results. Publications that report measuring linearity frequently state how they measured linearity. Only 2 publications report measuring linearity without stating how they measured linearity. The ICH recommendations for linearity include reporting the slope, y-intercept, and R 2 values from a line containing 5 or more calibration points. However, most publications report one but not all 3. Without knowing all 3 details, the accuracy and reliability of the data can be difficult to determine. For example, a calibration curve with 2 calibration points, although perfectly linear (R 2 =1), does not provide enough information for demonstrating the data accuracy or reliability.

Limitations

This analysis has 3 major limitations. First, the classification does not have a second individual verification, which is typically frowned upon in systematic reviews, as there is a chance that a reference could be misclassified. Due to resource limitations, a second individual verification could not be performed. Second, specificity, LOD, and LOQ are strongly reliant on publications specifically stating they measured the characteristics. This reliance might explain why specificity is underreported in the analysis. However, unlike specificity, LOD, and LOQ, accuracy, precision, range, and linearity can be interpreted without specific statements stating the characteristics are measured. For example, accuracy also could be reported as percent recovery, precision can be reported as relative error of a standard sample, and range and linearity can be interpreted from calibration curves. Lastly, the MVM classification is lenient. Thus, it is possible that many publications classified as reporting MVMs in this study may not generate accurate enough data or hold up well for scientific interpretation. For example, a publication reporting only LOD, LOQ, and linearity may still report inaccurate data as there is no established accuracy, precision, or specificity. However, the objective of this analysis is to improve understanding of the nature of method validation in the publications reporting chemical quantities and/or yields, and not to determine which publications provide reliable chemical data. In other words, the focus is determining whether the publication has the potential of reporting quality data, not if the publication reports quality data that is appropriate for tobacco regulatory science interpretations.

IMPLICATIONS FOR TOBACCO REGULATION

This work highlights the need for researchers to include adequate method validation characteristics or at least a validation statement to ensure that the published methods and data can be appropriately interpreted. The study also highlights the understudied chemicals and matrices in the current literature. Altogether, these findings demonstrate that, whereas the ENDS literature is growing rapidly, the chemical quantities and/or yields published for ENDS may not be interpreted appropriately as they may lack the accuracy and reliability needed to inform the field of tobacco regulatory science.

Supplementary Material

Acknowledgements.

The author thanks Dr. Matthew Holman, Dr. Matthew Walters, Dr. Charles Feng, Dr. Colleen Rogers, and Dr. Todd Cecil for the discussions and reviews during the research process, and Dr. Cathy Backinger and Dr. Corinne Husten for their support in the preparation of this article.

Human Subjects Approval Statement : This study did not require examination by an institutional review board.

Conflicts of Interest Disclosure Statement : No authors have any conflicts of interest to report.

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the US Food and Drug Administration.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

• View on publisher site
• PDF (895.1 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections