Creating a Table Setting and Uploading Exported Haplotype(s)
Appendix A Verification Testing
This appendix covers:
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Category 1: Peak Detection and Genotyping (Reproducibility) . . . 45
Category 2: Algorithm Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Category 3: Data Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Category 4: Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Results for Category 1: Peak Detection and Genotyping
(Reproducibility) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Results for Category 2: Algorithm Testing . . . . . . . . . . . . . . . . . . . 53
Results for Category 3: Data Handling . . . . . . . . . . . . . . . . . . . . . . 54
Results for Category 4: Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . 55
Results Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
41
GeneMapper ® ID Software
Introduction
Test Plan
The test plan, which Applied Biosystems has defined as verification of the software, was designed to evaluate the performance of
GeneMapper ID Software v3.2 for the human identification communities.
Test Categories
Applied Biosystems performed verification of GeneMapper ID
Software v3.2 to assess the performance, robustness, and feature design in four specific categories:
• Peak Detection and Genotyping (Reproducibility) ( page 45 )
• Algorithm Testing (
page 46
)
• Data Handling (
page 48
)
• Workflow (
page 48
)
Evaluation
GeneMapper ID Software v3.2 verification was performed to:
• Confirm and document the functionality of the new software features
• Assess the functionality of the minor modifications made to the analysis method
Our findings demonstrate that GeneMapper ID Software v3.2 (with its default settings and panels and bin sets) is valid for forensic, paternity, and databasing analyses. GeneMapper ID Software v.3.2 accurately detects, genotypes, and performs quality checks when performing STR analysis.
42
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
User Bulletin
Materials and Methods
Materials and Methods
This section describes the AmpF lSTR ® kit, sample types, instruments, and software used for the GeneMapper ® ID Software v3.2 verification testing.
Kit
The AmpF lSTR ® Yfiler ™ PCR Amplification Kit was used to amplify each sample type for the verification testing.
Sample Types
• Single-source DNA Samples – Eighty (80) DNA samples from four (4) population groups
• Mixture Samples – Mixtures of male and male DNA were amplified. These samples were mixed in defined ratios of 1:1,
3:1, 10:1, and 15:1, respectively. Additionally, mixtures of male and female DNA were amplified. These samples were mixed in defined ratios of 1:1000, 1:2000, 1:4000, and 1:8000, respectively.
Note: The human genomic DNA samples were quantified using
Quantifiler ™ Human DNA Quantification Kit and
Quantifiler ™ Y Human Male DNA Quantification Kit. The samples were amplified using a GeneAmp® PCR System 9700 with a silver 96-well block. The recommended cycling conditions were used, as outlined in the AmpF l STR ® Yfiler ™
PCR Amplification Kit User’s Manual (PN 4358101). The amplified samples were injected on three or more
ABI P RISM
® instrument platforms.
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
43
GeneMapper ® ID Software
Instruments and
Software
Table 4
below lists the instruments, Data Collection software versions, computer operating systems, and run modules that were used to process the samples for verification testing.
Table 4 Forensic validated instrument platforms
Instrument
Data
Collection
Software v3.0
Computer Operating
System
Microsoft ® Windows NT ®
Run Module
ABI P RISM
® 310
Genetic Analyzer
ABI P RISM
® 3100
Genetic Analyzer v1.1
v2.0
Microsoft ® Windows NT ®
Microsoft ® Windows 2000
• GS STR Pop4 (1 ml) F
• GS STR Pop4 (1 ml) G5v2
• GS 36_Pop4_F
• GS 36vb_Pop4_G5
44
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
User Bulletin
Category 1: Peak Detection and Genotyping (Reproducibility)
Category 1: Peak Detection and Genotyping
(Reproducibility)
The first test category evaluated genotype concordance using singlesource population DNA samples, male:male mixture samples, and male:female mixture samples. All samples were amplified with the
Yfiler kit.
Two comparison studies were conducted to verify genotype concordance using DNA samples previously amplified with the
Yfiler kit. These data were used to compare the genotype results from data analysis with GeneScan
®
Software v3.7.1 and Genotyper
®
Software v3.7 (both running on the Windows NT operating system) to GeneMapper ID Software v3.2 (running on the Windows 2000 operating system) using the Advanced mode.
Single-Source
Population DNA
Samples
The first comparison study consisted of:
• Eighty (80) samples from the Yfiler kit population study
• Four (4) positive controls
• One (1) negative control
• Six (6) ladders
These samples were electrophoresed on the ABI P RISM
® 3100
Genetic Analyzer and consisted of 1222 alleles from the samples, 60 alleles from the controls, and 137 alleles from the ladders. The samples were examined for reproducibility.
Mixture Samples
The second comparison study consisted of mixture data produced during the Yfiler kit developmental validation. This study was conducted to verify that the same mixture samples (that is, PCR product) gave concordant genotype calls between the
GeneScan/Genotyper software and the GeneMapper ID software applications.
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
45
GeneMapper ® ID Software
The mixture samples were electrophoresed on three different instruments and consisted of five (5) male:male and five (5)
male:female mixture samples. Table 5 below lists the mixture ratios
that were used in the mixture experiment.
Table 5 Mixture ratios
Male:Male
1:1
3:1
10:1
15:1
Male:Female
1:1000
1:2000
1:4000
1:8000
Category 2: Algorithm Testing
The second test category evaluated the functionality of GeneMapper
ID Software v3.2 to accurately filter out stutter based on the Marker
Specific Stutter ratio specified in the panel and in the Allele tab of the Analysis Method Editor.
Stutter Evaluation
In the Allele tab of the Analysis Method Editor, one of the parameters that can be set to filter out stutter is the Use marker-specific stutter
ratio if available check box.
Each allele within a locus displays a percent stutter that is reproducible. The Use marker-specific stutter ratio if available check box was selected in order to use the stutter ratio information from the panel. Using GeneMapper ID Software v3.2, 27 population samples were examined and the percent stutter recorded. The 27 population samples consisted of:
• 432 (n
−4), (n−5), and (n−6) stutter peaks
• 27 DYS19 (n
−2) stutter peaks
• 27 DYS392 (n+3) stutter peaks
User Bulletin 46
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
Category 2: Algorithm Testing
Spectral Pull-Up
PQV
Laboratories that have implemented the ABI P RISM 3100 Genetic
Analyzers into their workflow to electrophorese DNA samples may observe pull-up peaks within
±2 data points (in a different color) from the main allele peak, which is causing the pull-up peak. To this end, Applied Biosystems has made modifications to the analysis method to flag spectral pull-up peaks in the range of
±2 data points.
The spectral pull-up PQV flag was evaluated to ensure that a peak meeting specified criteria for spectral pull-up is appropriately activated. The two criteria required to flag a peak as spectral pull-up are as follow:
• The peak is
≤ 5% (or as specified by the value in the Pull-up
ratio field in the Peak Quality tab of the Analysis Method
Editor) of the main allele peak causing the pull-up
• The peak is within the
±2 data points range from the main allele peak in a different color
Note: After installing GeneMapper ID Software v3.2, laboratories should create new analysis methods specifically for use with
GeneMapper ID Software v3.2. Creating new analysis methods will ensure that spectral pull-up peaks within
±2 data points from the main allele peak are flagged appropriately.
Spectral Pull-UP PQV Study
To test the functionality of the spectral pull-up PQV, twenty-eight
(28) samples from the single-source population study were electrophoresed on both the ABI P RISM
®
310 and 3100 Genetic
Analyzers (that is, the same PCR product was electrophoresed on two different instruments) and were evaluated using the following criteria:
• Each analyzed sample was evaluated for the presence of pull-up peaks by noting whether or not the spectral pull-up PQV was flagged.
• The data point (that is,
±0 data point, ±1 data point, or ±2 data points) of the pull-up peak relative to the main allele peak was recorded.
• The percent height of the pull-up peak relative to the main allele peak was recorded (that is
≤ 5% of the main allele peak causing the pull-up)
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
47
GeneMapper ® ID Software
Category 3: Data Handling
The third test category evaluated the Export Combined Table format and the size standard printing functionality.
Export Combined
Table Format
The purpose for evaluating the Export Combined Table format was to verify that samples that do not pass sizing (that is, low-quality samples) are exported along with samples that pass sizing.
Printing Labeled
Size Standard
Peaks
In GeneMapper ID Software v3.2, the ability to display and print labels on the size standards in different plot configurations was tested.
Category 4: Workflow
The fourth test category evaluated the ability of GeneMapper ID
Software v3.2 to:
• Display labeled peak assignments for all size standards
• Retain labels in both the “align by base pair” and “align by data point” views for the X-axis
Displaying
Labeled Peak
Assignments for
Size Standards
All samples within the Samples view were highlighted and only the orange dye color was selected, as shown below.
The size standards were checked to ensure that all defined size standard peaks were labeled appropriately.
Retaining Labels in Base Pair and
Data Point Views
The electropherogram plots were displayed to verify that labels associated with the peaks were retained when switching back and forth between the “align by base pair” and “align by data point” views for the X-axis.
User Bulletin 48
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
Results for Category 1: Peak Detection and Genotyping (Reproducibility)
Results for Category 1: Peak Detection and
Genotyping (Reproducibility)
Single-Source
Population DNA
Samples
The first parameter evaluated the Peak Detection and Genotyping category to assess genotyping concordance. Applied Biosystems compared the genotype results from data analysis with GeneScan
Software v3.7.1 and Genotyper Software v3.7 (both running on the
Windows NT operating system) to GeneMapper ID Software v3.2 using the Advanced algorithm.
The analysis parameter settings for the software packages were defined according to Applied Biosystems default analysis parameters with a defined peak amplitude threshold (PAT) of 50 RFUs.
As shown in
Figure 1 below, a side-by-side comparison between
Genotyper Software v3.7 and GeneMapper ID Software v3.2 was evaluated to ensure the consistency of genotype results, where the evaluation criteria was that all samples give concordant genotypes.
After the trained scientist completed data analysis, each population sample tested gave 100% concordant genotype calls between the two
software programs (see Table 6 on page 50 ).
Genotyper Software v3.7
GeneMapper ID Software v3.2
Figure 1 Example of the Yfiler kit allelic ladder analyzed with
Genotyper Software v3.7 (left) and GeneMapper ID Software v3.2 displaying concordant results (right)
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
49
GeneMapper ® ID Software
As shown in
Table 6
below, each sample type tested gave concordant genotype calls between the two software programs.
Table 6 Genotype concordance between Genotyper Software v3.7 and GeneMapper ID Software v3.2
Allele Source
Population samples
Positive and negative controls
Allelic ladders
Number of
Samples
80
5
6
Number of
Alleles
Concordant
Alleles
1222
60
100%
100%
137 100%
Mixture Samples
After the trained scientist completed data analysis, Applied
Biosystems documented three observations when comparing the results produced from GeneScan Software v3.7.1 and Genotyper
Software v3.7 (both running on the Windows NT operating system) to the results produced from GeneMapper ID Software v3.2. Of the
272 loci examined, 267 loci produced concordant results between the software packages and 5 loci/alleles produced non-concordant results. These observations are further described on
page 51
.
Figure 2
below shows the genotyping concordance results when comparing the analyzed data from GeneScan Software v3.7.1 and
Genotyper Software v3.7 (both running on the Windows NT operating system) to the results from GeneMapper ID Software v3.2
50
Figure 2 Genotyping concordance results
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
User Bulletin
Results for Category 1: Peak Detection and Genotyping (Reproducibility)
• Observation 1: 267 (98.20%) of the loci/alleles compared produced concordant genotype calls.
• Observation 2: Three (1.08%) of the loci/alleles were non-concordant because these loci/alleles fell just below the
50-RFU cutoff threshold value in GeneMapper ID Software v3.2 and were not genotyped.
• Observation 3: Two (0.72%) of the loci/alleles were non-concordant because the alleles analyzed with Genotyper
Software v3.7 did not filter the stutter or the Minus A. The label is removed from Peak A (the stutter peak) if Peak B (the true allele) meets two criteria:
– Peak B is higher than Peak A by the specified percentage, and
– Peak B is within the specified proximity size (in base pairs) range relative to Peak A
Table 7
below shows the results for observation 3.
Table 7 Observation 3: non-concordant loci
Marker/Locus
DYS439
DYS456
Allele 1
11
15
Genotyper Software v3.7
Allele 2
OL
16
Allele 3
12
OL
Allele 4
13
17
GeneMapper ID
Software v3.2
Allele 1
12
15
Allele 2
13
17
As shown in
Figure 3 and Figure 4 on page 52
, these two loci/alleles did not meet these criteria due to
−A product, resulting in the detection of a third peak (Peak C). The filtering in Genotyper
Software v3.7 compared Peak A to Peak C. GeneMapper ID Software v3.2 appropriately filtered the stutter and the shoulder peak using defined bin sets
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
51
GeneMapper ® ID Software
Genotyper Software v3.7
GeneMapper ID Software v3.2
Genotyper Software v3.7
Figure 3 DYS439 Marker
GeneMapper ID Software v3.2
Figure 4 DYS456 Marker
52
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
User Bulletin
Results for Category 2: Algorithm Testing
Results for Category 2: Algorithm Testing
Stutter Evaluation
Twenty-seven (27) population DNA samples were examined, consisting of:
• 432 (n
−4), (n−5), and (n−6) stutter peaks
• 27 DYS19 (n
−2) stutter peaks
• 27 DYS392 (n+3) stutter peaks
The results were as follows:
• All (n
−4, n−5, and n−6) stutter peaks were below the stutter percent as defined in the panel and were properly filtered.
• The DYS19 locus (n
−2) stutter peak was below the stutter percent as defined in the Allele tab of the Analysis Method
Editor and were properly filtered.
• The DYS392 locus (n+3) stutter peak was below the stutter percent as defined in the Allele tab of the Analysis Method
Editor and were properly filtered.
Spectral Pull-Up
PQV
Twenty-eight (28) single-source population samples were evaluated for spectral pull-up. For each pull-up peak, the following was recorded:
• Size
• Peak height
• Data point
• Spectral pull-up PQV activated
Seventy-eight (78) spectral pull-up peaks were identified. As shown in
Figure 5 on page 54
, 97.44% (76) of these pull-up peaks were correctly flagged by GeneMapper ID Software v3.2.
The remaining
2.56% (2) of pull-up peaks that were not flagged were produced by off-scale data. If a spectral pull-up peak is caused by an off-scale allele peak, the spectral pull-up flag is not triggered for that peak.
This is because the ratio of pull-up to off-scale peaks cannot be calculated (since the peak height for off-scale data cannot be determined).
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
53
GeneMapper ® ID Software
Applied Biosystems recommends that you carefully review all offscale yellow PQV flags. Applied Biosystems recommends that you dilute or re-run samples to obtain on-scale data to ensure the PQVs reported by data analysis are accurate and to reduce spectral separation (that is, pull-up).
Figure 5 Results for the spectral pull-up PQVs flagged
Results for Category 3: Data Handling
Export Combined
Table Format
Projects with their associated settings (that is, analysis methods, table settings, plot settings, matrices, if applicable, and size standards) consisting of samples that passed sizing as well as samples that did not pass sizing were exported. The tables were exported from the
Samples view using the Export Combined Table format. Eighty (80) samples from the Yfiler kit population study were used.
The results obtained verified that GeneMapper ID Software v3.2 is
100% functional in its ability to:
• Export projects using the Export Combined Table format
• Track all samples in an electrophoresis run (samples that do not pass sizing as well as samples that pass sizing)
User Bulletin 54
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
Results for Category 4: Workflow
Printing Labeled
Size Standard
Peaks
The purpose of this test was to verify that the size standard plots could be printed in different display configurations.
The results obtained verified that GeneMapper ID Software v3.2 is
100% functional in its ability to display and print labels on size standards in different plot configurations. For example, displaying five labeled size standards separated into five panes, using three levels of magnification.
Results for Category 4: Workflow
Displaying
Labeled Peak
Assignments
All size standards from eighty (80) DNA samples from the Yfiler kit single-source population study were examined. All peaks were correctly labeled.
Retaining Labels
The results obtained verified that GeneMapper ID Software v3.2 is
100% functional in its ability to switch back and forth between the
“align by base pair” and “align by data point” views for the X-axis.
Results Summary
Applied Biosystems verification testing demonstrates that human identification laboratories can successfully adopt GeneMapper ID
Software v3.2.
Installation Procedures and New Features for GeneMapper ® ID Software v3.2
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
55
GeneMapper ® ID Software
56
DRAFT
November 30, 2004 11:05 am, 4352543A.fm
User Bulletin
