Randall D. Penfield DIFAS User Manual

DIFAS 5.0
Differential item functioning analysis system
User’s Manual
© 2012
Randall D. Penfield
CONTENTS
Chapter 1
Getting Started with DIFAS
System requirements
Running DIFAS
A diagram of the DIFAS Main Window
Conducting analyses using DIFAS
Chapter 2
Importing Data
Preparing the data for import
Importing the data
Navigating the imported data file
Chapter 3
Conducting Analyses
Descriptives
Frequencies
DIF analyses for dichotomous items
DIF analyses for polytomous items
Differential test functioning
Differential step functioning
Missing Data
Chapter 4
Managing Output
Properties of the output
Editing the output
Saving output files
Opening output files
Printing Output
DIFAS: 5.0 User’s Manual
2
CHAPTER 1
Getting Started with DIFAS
DIFAS is a Windows based program that performs a variety of functions related to
assessing the presence of differential item functioning (DIF) in items, differential test
functioning (DTF) across all items of a test or scale, and differential step functioning
(DSF) for ordinal polytomous items. DIFAS is exclusively point-and-click, and is aimed
at providing users with the capability of conducting sophisticated DIF and DTF analyses
in a user-friendly environment.
Although a multitude of parametric and nonparametric DIF detection procedures exist,
DIFAS performs only nonparametric DIF analyses. The DIF procedures that DIFAS runs
for dichotomously scored items include:
Mantel-Haenszel chi-square
Mantel-Haenszel common log-odds ratio and estimated standard error
Standardized Mantel-Haenszel common log-odds ratio
Breslow-Day test of trend in odds ratio heterogeneity
ETS classification scheme
The DIF procedures that DIFAS runs for polytomously scored items include:
Mantel’s chi-square
Lui-Agresti cumulative common log-odds ratio and estimated standard error
Standardized Lui-Agresti cumulative common log-odds ratio
Cox’s noncentrality parameter estimator and estimated standard error
Standardized Cox’s noncentrality parameter estimator
The DTF procedures that DIFAS runs consists of estimates of the variance in the DIF
effect across the items of the test or scale.
The DSF procedures that DIFAS runs consists of estimates of the step-level common logodds ratios using the cumulative and adjacent categories dichotomization scheme.
All of the procedures described above are discussed in greater detail in Chapter 3.
In addition, DIFAS can compute: (a) item-level descriptive statistics, (b) frequencies for
the reference and focal groups at each level of the stratifying variable, and (c) the mean
of each item conditional on 10 intervals of the stratifying variable.
DIFAS: 5.0 User’s Manual
3
System Requirements
The following hardware and software is required for running DIFAS:

Microsoft Windows 95 or later, or Microsoft Windows NT 3.51 or later

24 MB RAM for Windows 95 or later, 32 MB RAM for Microsoft NT or later

Pentium 90MHz or higher
Running DIFAS
To run DIFAS, double click on the program icon (DIFAS.exe). The program itself is
contained in a single file that is about 0.22 megabytes (220 kilobytes), and thus can be
copied on a single floppy disk.
DIFAS: 5.0 User’s Manual
4
A Diagram of the DIFAS Main Window
The DIFAS Main Window has the following components:
This is the box where the type of delimiter
in the data file is specified
This box displays the name
of the currently selected file
Use these boxes to
locate desired data file
Menu for all DIF and
statistical analyses
Use this box to view
“.txt” or “.dat” files
This box displays the name
of the file currently in the
DIFAS processor
Click on this button to
import the selected file
into the DIFAS processor
File Edit Data Analyses
Select Data
Type of Delimiter
Selected File:
File Currently in Processor:
Import Selected Data File
Output
This is the output box
(where DIFAS output will be displayed)
DIFAS: 5.0 User’s Manual
5
Conducting Analyses Using DIFAS
Conducting analyses using DIFAS consists of three primary steps:
Step 1:
Import the data to be analyzed. Select the appropriate file to be imported
(the file must have either a “.txt” or a “.dat” extension) using the top
portion of the Main Window of DIFAS. Imported files must be delimited
as space-delimited, comma-delimited, or tab-delimited.
Step 2:
Select the appropriate analysis from the Analyses drop-down menu.
Step 3:
Examine the output of the results of analyses in the output box of the Main
Window of DIFAS. DIFAS adds the output of subsequent analyses
sequentially to the output displayed in the output box. The output can be
annotated, edited, or saved as a text file upon request.
DIFAS: 5.0 User’s Manual
6
CHAPTER 2
Importing Data
The first step in running DIF analyses using DIFAS is to import the desired data file into
DIFAS. DIFAS can accommodate data sets containing up to 1000 variables. There is no
theoretical limit that DIFAS places on the number of cases contained in the data file – the
limiting factor would be the memory capabilities of the computer running DIFAS.
DIFAS can only read in text and data files that have the extension “.txt” or “.dat”.
Because it will often be the case that the data of interest is being maintained in either a
statistical software package (e.g., SPSS) or a data management package (e.g., Excel), you
may have to save the data file of interest in a format that DIFAS can read. There are
some simple steps that can be taken to properly format the data file so that DIFAS can
read its contents, as outlined in the following section.
Preparing the Data for Import
Before importing the desired data file into DIFAS, it is critical that the data be formatted
properly. With respect to the values of the data, DIFAS has the following constraints:
1. DATA MUST BE NUMERIC. All data must be numeric. If any nonnumeric (e.g.,
text) entries exist, DIFAS will not be able to input the data, and an error message will
appear informing you of that.
2. ITEM RESPONSES MUST BE CODED 0, 1, 2,…, 9. All response data for items to
be analyzed for DIF must have values that set 0 equal to the lowest possible response on
the item, and all other responses consist of integer values sequentially greater than 0 (e.g.,
1, 2, 3,…9). Thus, for multiple-choice item formats, the responses must be coded as 0
(incorrect) and 1 (correct). For polytomous item formats, the responses must be coded as
0, 1, 2, …, k. Note that the highest possible item-level response category that DIFAS will
permit is 9. To summarize, for all item formats (dichotomous and polytomous), the
lowest response option must be coded 0, and each successively higher response option is
coded according to integers successively greater than 0. Any integer value greater than 9
can also be included in the data, but the DIF analyses conducted by DIFAS will only read
item-level responses between 0 and 9 - any value greater than 9 would be removed from
DIF analyses.
DIFAS: 5.0 User’s Manual
7
3. ALL DATA MUST BE INTEGER. DIFAS assumes that all data entered is integer.
Thus not only is it important for item responses to be integer in nature (as stated in #2
above), but any other variables concerning groups or external ability estimates should
also consists of integer values. Non-integer values can be read in by DIFAS, but only the
truncated values are used in analyses. An important consequence of this is that any
external measure of ability that is to be used for stratifying must be integer. As a result,
ability estimates that are initially in decimal form, should be transformed to an integer
form (as discussed below) prior to importing the data into DIFAS.
4. MISSING VALUES MUST BE DENOTED BY “.”. Missing values are permitted
by DIFAS, but must be indicated by the period symbol “.” or the absence of any value
between two delimiters. For example, the line: 0,0,.,1 indicates four responses where the
third response is missing. Alternatively, the line: 0,0,,1 could also be used to indicate
four responses where the third response is missing. Note that DIFAS will not recognize a
space as a missing value because a space is used as a delimiter in space-delimited files.
5. DATA BEGINS ON FIRST LINE OF FILE. When reading in data, DIFAS
assumes that the data begins in the first line of the file being read in. Thus, a first line
containing variable names is not permitted when importing data. For example, a spacedelimited data set containing responses of three individuals on five items might look like:
10110
11111
00010
The same data in comma-delimited form would look like:
1,0,1,1,0
1,1,1,1,1
0,0,0,1,0
And the same data in tab-delimited form would look like:
1
1
0
0
1
0
1
1
0
1
1
1
0
1
0
DIFAS labels variables as Var1, Var2, etc., and so it is often useful to have a listing of
the variable ordering and/or column numbers at your fingertips when using DIFAS.
6. EXTERNAL ABILITY ESTIMATE CAN BE INCLUDED. Because DIF analyses
are predicated upon examination of a relationship between a grouping variable and item
performance conditional on an estimate of ability, DIFAS must be instructed as to the
nature and location of the estimate of ability. By default, DIFAS uses the sum of the
responses to the items included in the DIF analysis (the summated test score) as an
internal measure of ability. If the summated test score is the desired measure of ability,
DIFAS: 5.0 User’s Manual
8
then no other estimate of ability need be included in data file. If, however, the measure
of ability is other than the summated test score (i.e., there is an external measure of
ability), then the external measure of ability must be included in the data file prior to
importing the data into DIFAS. Note, however, that DIFAS assumes all data is integer in
nature, and so if the measure of ability contains decimals, then the ability measure would
need to be transformed to eliminate the decimals. For example, if the ability measure
consists of a standard normal variable, then the user can multiply the standard variable by
1000 (if 3 decimal places are desired) and truncate the resulting variable accordingly to
remove any resulting decimal places. Negative values should not pose a problem,
provided they are integer in value. Any transformations must be conducted prior to
importing the data into DIFAS.
7. DATA FILE MUST HAVE “.txt” OR “.dat” EXTENSION. DIFAS is designed to
read in only text (*.txt) or data (*.dat) files. Text files are typically maintained in a
program such as Notepad.
8. DATA FILES MUST BE SPACE-, COMMA-, OR TAB-DELIMITED. DIFAS is
designed to read in files for which the columns (variables or items) of the data file are
delimited using either a space, comma, or tab. If the data is being maintained in a data
management program (such as Excel) or a statistical software package (such as SPSS or
SAS), the data must be converted to a delimited text file. Here are some guidelines for
creating a delimited text file from several of the more common data management and
statistical programs.
SPSS – When the desired data file is in SPSS, save the data file as a tab-delimited
(*.dat) file, and make certain that you deselect the check box labeled “Write
variable names to spreadsheet”. If this check box is not deselected, the variables
names will be written to the first line of the data file, and DIFAS cannot read in
non-numeric values. The resulting file is a tab-delimited file having the extension
“.dat”, which can be accessed by DIFAS.
Excel - When the desired data file is in Excel, save the data file as a text (tabdelimited) file. The resulting file is a text tab-delimited file having the extension
“.txt”, which can be accessed by DIFAS. Note that there cannot be any variable
names in the first row, as DIFAS can read in only numeric values, and assumes all
entries are data and are to be included in the analyses.
DIFAS: 5.0 User’s Manual
9
Importing the Data
To import a data file, use the top portion of the main window of DIFAS to select the
desired file. When selecting the desired file, follow these steps:
STEP 1: SPECIFY DELIMITER USED IN DATA FILE. Specify the delimiter used
in the file (space, comma, or tab), as listed by the options on upper left portion of
the Main Window. By default, DIFAS sets the delimiter to “space”, but you may have to
change this to comma or tab. Not changing delimiter appropriately will cause DIFAS to
not recognize the data when importing.
STEP 2: SPECIFY FILE TYPE. Specify whether the file has the extension “.txt” or
“.dat” using the drop-down list box. By default, DIFAS sets the file type to “.txt”. Note,
however, that if you are importing a file created from SPSS, you may have to select the
“.dat” extension.
STEP 3: LOCATE DESIRED DATA FILE. Using the drive list box and the directory
list box in the top portion of the Main Window, locate and select the desired file. Clicking
on the desired file will cause the file’s name to appear in the “Selected File:” box in the
upper right side of the Main Window.
STEP 4: CLICK IMPORT BUTTON. To import the data of the selected file to DIFAS,
click the button marked “Import Selected Data File” in the upper right portion of the
Main Window. If DIFAS recognizes every element of the file as being valid (i.e.,
numeric), then the box labeled “File Currently in Processor: “ will list the file name, and
a message will appear in the output box specifying the file that was imported, and the
number of cases and variables in the file. An example of such a message is as follows:
Opened the text file: C:\playdata\dif2.txt
Number of Cases: 400
Number of Variables: 12
If an invalid element exists (such as a non-numeric entry), then DIFAS will provide a
message box stating so, and listing the row and column of the file that had the invalid
value.
Here are some guidelines to follow when opening files created by common statistical and
data management programs.
1. ENSURE THAT THE CORRECT DELIMITER IS SELECTED. A common error
that occurs in importing data into DIFAS is that the proper delimiter is not selected (in
the upper left portion of the Main Window), and as a result DIFAS treats the incorrect
delimiter as text which causes an error.
2. IMPORTING DATA FROM SPSS FILES. Data from SPSS can be saved as a tabdelimited “.dat” file. In this case, the resulting “.dat” file will contain the variable names
in the first row of the data file. These names must be deleted from the file to be imported
DIFAS: 5.0 User’s Manual 10
into DIFAS because DIFAS would mistake the first line as being data. Simply delete the
fist line of the “.dat” file and save. Next, ensure that the type of delimiter in DIFAS is set
to tab-delimited. Failing to do so will result in DIFAS displaying a message that an
invalid entry was made. In addition, in order to display the *.dat files, you must change
the extension option in the top portion of DIFAS from “Text (*.txt)” to “Data (*.dat)”.
3. IMPORTING DATA FROM EXCEL FILES. Data from Excel can be saved as tabdelimited “.txt” file. In this case, ensure that in the created “.txt” file, the first row of the
file consists of data, not variable names. Next, ensure that the type of delimiter in DIFAS
is set to tab-delimited. Failing to do so will result in DIFAS displaying a message that an
invalid entry was made. In addition, in order to display the *.txt files, you must ensure
that extension option “Text (*.txt)” is selected in the top portion of the Main Window.
Navigating the Imported Data File
Once a file has been successfully imported, you may navigate through the file, one cell at
a time, using the “Navigate Data File” window contained within the Data menu. Opening
this window you will find a central box that displays the contents of any selected element
of the imported data file. By default, the initial element is that contained in the first row
and first column of the data file. By clicking “Up”, “Down”, “Left”, and “Right”, you
may view successive elements in the data file. This function permits you to ensure that
the contents of the data file are as expected.
DIFAS: 5.0 User’s Manual 11
CHAPTER 3
Conducting Analyses
The analyses offered by DIFAS are located using the Analyses Menu of DIFAS. DIFAS
includes several analysis windows that can be used to conduct a variety of analyses
related to DIF detection. The options on the Analyses Menu are as follows:





Descriptives
Frequencies
Nonparametric DIF Tests
-Dichotomous Models
-Polytomous Models
Differential Test Functioning
-Dichotomous Models
-Polytomous Models
Differential Step Functioning
-Cumulative
-Adjacent Categories
Descriptives
The Descriptives Window of the Analyses Menu computes the mean, standard deviation,
minimum, maximum, and number of valid cases for each variable selected. To obtain the
descriptive statistics, select the variables of interest by left-clicking in the check box
associated of the desired variables, and then click the OK button. An example of the
output generated by the descriptives function is given by:
DESCRIPTIVES
-----------------------------------------------------------Name
Mean
SD
Min
Max
N
-----------------------------------------------------------Var 1
1.5
0.501
1
2
400
Var 2
0.522
0.5
0
1
400
Var 3
0.507
0.501
0
1
400
Var 4
0.5
0.501
0
1
400
Var 5
0.438
0.497
0
1
400
Var 6
0.522
0.5
0
1
400
Var 7
0.47
0.5
0
1
400
Var 8
0.488
0.5
0
1
400
Var 9
0.482
0.5
0
1
400
Var 10
0.472
0.5
0
1
400
Var 11
0.502
0.501
0
1
400
------------------------------------------------------------
DIFAS: 5.0 User’s Manual 12
The use of the descriptives function may prove useful in obtaining general information
about the variables held in the DIFAS processor.
Note that by default, the “Range of Items” option is selected (in the Descriptives
Window). In this mode, a range of variables can be selected by first selecting the lower
variable of the range and then the upper variable of the range. For example, to select
variables 2 through 10, first select variable 2 and then select variable 10. This will cause
all variables between 2 and 10 to be selected. To turn off the “Range of Items” option,
click on the “Individual Items” option.
Frequencies
The Frequencies Window of the Analyses Menu computes the frequency, percentage of
the total sample, percentage of the sample having valid data, and the cumulative
percentage of each outcome for each variable selected. In addition, the Frequencies
function computes the number of cases having valid (present) values and the number of
cases having missing values. To obtain the frequencies output, select the variables of
interest by left-clicking in the check box associated of the desired variables, and then
click the OK button. An example of the output generated by the frequencies function is
given by:
FREQUENCIES AND PERCENTAGES FOR: Var 1
-------------------------------------------------Value
Freq
% Total
% Pres
Cum %
-------------------------------------------------1
200
50
50
50
2
200
50
50
100
Present
400
100
Missing
0
0
Total
400
100
--------------------------------------------------
The use of the frequencies function may prove useful in obtaining general information
about the variables held in the DIFAS processor.
Note that by default, the “Range of Items” option is selected (in the Frequencies
Window). In this mode, a range of variables can be selected by first selecting the lower
variable of the range and then the upper variable of the range. For example, to select
variables 2 through 10, first select variable 2 and then select variable 10. This will cause
all variables between 2 and 10 to be selected. To turn off the “Range of Items” option,
click on the “Individual Items” option.
DIFAS: 5.0 User’s Manual 13
DIF Analyses for Dichotomous Items
The Dichotomous Models Window conducts DIF analyses for dichotomous items by
computing the following statistics: the Mantel-Haenszel chi-square, the Mantel-Haenszel
log-odds ratio, the estimated standard error of the Mantel-Haenszel log-odds ratio, the
standardized Mantel-Haenszel log-odds ratio, the Breslow-Day chi-square test of trend in
odds ratio heterogeneity, the Combined Decision Rule classification scheme, and the ETS
classification scheme. These procedures are described later in this section.
The Dichotomous Models Window contains three lists of the variables contained in the
imported data file. The first list, called “Select Items”, is used to designate the items to
be included in the DIF analysis. The second list, called “Select Groups”, is used to
designate the variable containing the grouping information. The third list, called “Select
Stratifying”, is used to designate the variable used to stratify the individuals according to
an estimate of ability. Note that by default, DIFAS assumes that the stratification
variable is the summated test score of the items selected for the DIF analysis, as noted by
the selection of the “Stratify by Sum” option at the bottom of the “Select Stratifying” list.
However, the user may select any variable contained in the imported data file to use as
the stratifying variable.
To run DIF analyses for dichotomous items, follow these steps:
1. In the “Select Items” list (the list at the far left of the Dichotomous Models Window),
select the items to be studied for DIF. Note that by default, the “Range of Items” option is
selected. In this mode, a range of items can be selected by first selecting the lower item
of the range and then the upper item of the range. For example, to select items 2 through
10, first select item 2 and then select item 10. This will cause all items between 2 and 10
to be selected. To turn off the “Range of Items” option, click on the “Individual Items”
option.
2. In the “Select Groups” list (the list in the middle of the Dichotomous Models
Window), select the variable designating the groups of interest. DIFAS permits only the
comparison of two groups at a time. Below the variable selection box, specify the value
designating reference group members, and the value designating focal group members.
For example, the imported data file may use a “1” to designate reference group members
and a “2” to designate focal group members. Any two numeric values may be used to
designate the reference and focal groups.
3. In the “Select Stratify” list (the list at the far right of the Dichotomous Models
Window), specify the stratifying (or matching) variable. If you wish the total test score
obtained from the items selected to be included in the DIF analysis to serve as the
stratifying variable, ensure that the option “Stratify by sum” is selected. In this case, no
variable needs to be selected from the “Select Stratify” list. Note that when DIFAS
computes the total test score, any individual having a missing value is assigned a missing
value for the total test score, and thus is ignored in the analyses. Thus, if you wish a total
test score to be computed for individuals having missing values, be sure to code all
DIFAS: 5.0 User’s Manual 14
missing values as “0” before importing the data file into DIFAS. If you wish the
stratifying variable to be some other variable in the data file, click the option “Stratify by
external”, and then select the external variable from the “Select Stratify” list.
4. By default, DIFAS uses a stratum size of 1. You can, however, change this size by
specifying a stratum size in the box located in the lower right portion of the Dichotomous
Models Window. For example, if a stratum width of 2 is desired, then the user would
change the 1 to a 2 in the “Stratum Size” box.
5. DIFAS can print out the number of reference and focal group members located in each
stratum of the stratifying variable. To do so, select the “Print strata n” check box in the
lower right portion of the Dichotomous Models Window.
Once these three steps are completed, click on the OK button. The resulting output
consists of two tables – a table containing the relevant DIF statistics, and a table
containing the conditional differences in mean item score between the reference and focal
groups at ten intervals across the stratifying variable continuum (the lower and upper
limits of each of the ten intervals are presented at the top of each interval). An example
of output for a DIF analysis of ten dichotomous items is:
DIF STATISTICS: DICHOTOMOUS ITEMS
-------------------------------------------------------------------------------Name
MH CHI
MH LOR
LOR SE
LOR Z
BD
CDR
ETS
-------------------------------------------------------------------------------Var 2
0.011
-0.052
0.233
-0.223
0.034
OK
A
Var 3
1.786
0.358
0.246
1.455
0
OK
A
Var 4
0.189
0.128
0.23
0.557
2.776
OK
A
Var 5
0
-0.031
0.236
-0.131
0.03
OK
A
Var 6
0.161
-0.13
0.25
-0.52
0.043
OK
A
Var 7
0.323
-0.162
0.238
-0.681
1.156
OK
A
Var 8
0.001
0.036
0.238
0.151
2.109
OK
A
Var 9
0.013
0.053
0.229
0.231
1.11
OK
A
Var 10
0.001
-0.025
0.252
-0.099
0.016
OK
A
Var 11
0.42
-0.194
0.25
-0.776
0.202
OK
A
--------------------------------------------------------------------------------
CONDITIONAL DIFFERENCES: Intervals of size 1
---------------------------------------------------------------------------------------Lower
0
1
2
3
4
5
6
7
8
9
Upper
1
2
3
4
5
6
7
8
9
10.1
---------------------------------------------------------------------------------------Var 2
-0.07
-0.1
0.1
-0.14
0.3
-0.01
-0.07
0
-0.07
0
Var 3
0.08
0.11
0.02
0.15
-0.19
0.13
0.01
0.18
0.06
0
Var 4
0.03
-0.15
0.06
-0.28
0.08
0.22
0.09
0.14
0.03
0
Var 5
0
0
-0.12
0.12
0.08
0.01
-0.1
-0.01
0.09
0
Var 6
0.03
-0.11
0.05
0.07
-0.22
-0.09
0.09
-0.04
-0.07
0
Var 7
-0.02
0.09
0.06
0.12
-0.42
0
-0.1
-0.03
-0.04
0
Var 8
-0.02
0.05
0.03
0.17
0.36
-0.09
-0.25
-0.02
-0.07
0
Var 9
-0.03
0.08
-0.16
-0.12
0.13
0.05
0.2
-0.05
0.03
0
Var 10
0.04
0.04
0.01
-0.04
-0.06
-0.25
0.16
-0.08
0.13
0
Var 11
0.03
-0.02
-0.05
-0.05
-0.07
0.04
-0.03
-0.08
-0.07
0
----------------------------------------------------------------------------------------
Each of the DIF statistics conducted by DIFAS for dichotomous items will now be briefly
described.
DIFAS: 5.0 User’s Manual 15
Mantel-Haenszel Chi-Square (MH CHI) – The Mantel-Haenszel chi-square statistic
(Holland & Thayer, 1988; Mantel & Haenszel, 1959) is distributed as chi-square with one
degree of freedom. Critical values of this statistic are 3.84 for a Type I error rate of 0.05
and 6.63 for a Type I error rate of 0.01.
Mantel-Haenszel Common Log-Odds Ratio (MH LOR) – The Mantel-Haenszel
common log-odds ratio (Camilli & Shepard, 1994; Mantel & Haenszel, 1959) is
asymptotically normally distributed. Positive values indicate DIF in favor of the reference
group, and negative values indicate DIF in favor of the focal groups.
Standard Error of the Mantel-Haenszel Common Log-Odds Ratio (LOR SE) – The
standard error of the Mantel-Haenszel common log-odds ratio. The standard error
computed here is the nonsymmetric estimator presented by Robins, Breslow and
Greenland (1986).
Standardized Mantel-Haenszel Log-Odds Ratio (LOR Z) – This is the Mantel-Haenszel
log-odds ratio divided by the estimated standard error. A value greater than 2.0 or less
than –2.0 may be considered evidence of the presence of DIF.
Breslow-Day Chi-Square (BD) – The Breslow-Day chi-square test of trend in odds ratio
heterogeneity (Breslow & Day, 1980; Penfield, 2003) is distributed as chi-square with
one degree of freedom. Critical values of this statistic are 3.84 for a Type I error rate of
0.05 and 6.63 for a Type I error rate of 0.01. This statistic has been shown to be effective
at detecting nonuniform DIF.
Combined Decision Rule (CDR) – The combined decision rule (CDR) flags any item for
which either the Mantel-Haenszel chi-square or the Breslow-Day chi-square statistic is
significant at a Type I error rate of 0.025 (Penfield, 2003). The message OK is printed if
neither statistic is significant, and the message FLAG is printed if either statistic is
significant.
The ETS Categorization Scheme (ETS) – The ETS categorization scheme (Zieky, 1993)
categorizes items as having small (A), moderate (B), and large (C) levels of DIF.
DIF Analyses for Polytomous Items
The Polytomous Models Window of the Analyses Menu conducts DIF analyses for
polytomous items by computing the following statistic the Mantel chi-square, the LiuAgresti cumulative common log-odds ratio, the estimated standard error of the LiuAgresti cumulative common log-odds ratio, the standardized Liu-Agresti cumulative
common log-odds ratio, Cox’s estimator of the multivariate hypergeometric noncentrality
parameter, the estimated standard error of Cox’s estimator of the multivariate
hypergeometric noncentrality parameter, and the standardized value of Cox’s estimator.
These procedures are described at the end of this section.
DIFAS: 5.0 User’s Manual 16
The Polytomous Models Window contains three lists of the variables contained in the
imported data file. The first list, called “Select Items”, is used to designate the items to
be included in the DIF analysis. The second list, called “Select Groups”, is used to
designate the variable containing the grouping information. The third list, called “Select
Stratifying”, is used to designate the variable used to stratify the individuals according to
an estimate of ability. Note that by default, DIFAS assumes that the stratification
variable is the summated test score of the items selected for the DIF analysis, as noted by
the selection of the “Stratify by Sum” option at the bottom of the “Select Stratifying” list.
However, the user may select any variable contained in the imported data file to use as
the stratifying variable.
To run DIF analyses for polytomous items, follow these steps:
1. In the “Select Items” list (the list at the far left of the Polytomous Models Window),
select the items to be studied for DIF. Note that by default, the “Range of Items” option is
selected. In this mode, a range of items can be selected by first selecting the lower item
of the range and then the upper item of the range. For example, to select items 2 through
10, first select item 2 and then select item 10. This will cause all items between 2 and 10
to be selected. To turn off the “Range of Items” option, click on the “Individual Items”
option.
2. In the “Select Groups” list (the list in the middle of the Polytomous Models Window),
select the variable designating the groups of interest. DIFAS permits only the
comparison of two groups at a time. Below the variable selection box, specify the value
designating reference group members, and the value designating focal group members.
For example, the imported data file may use a “1” to designate reference group members
and a “2” to designate focal group members. Any two numeric values may be used to
designate the reference and focal groups.
3. In the “Select Stratify” list (the list at the far right of the Polytomous Models Window),
specify the stratifying (or matching) variable. If you wish the total test score obtained
from the items selected to be included in the DIF analysis to serve as the stratifying
variable, ensure that the option “Stratify by sum” is selected. In this case, no variable
needs to be selected from the “Select Stratify” list. Note that when DIFAS computes the
total test score, any individual having a missing value is assigned a missing value for the
total test score, and thus is ignored in the analyses. Thus, if you wish a total test score to
be computed for individuals having missing values, be sure to code all missing values as
“0” before importing the data file into DIFAS. If you wish the stratifying variable to be
some other variable in the data file, click the option “Stratify by external”, and then select
the external variable from the “Select Stratify” list.
Once these three steps are completed, click on the OK button. The resulting output
consists of two tables – a table containing the relevant DIF statistics, and a table
containing the conditional differences in mean item score between the reference and focal
groups at ten intervals across the stratifying variable continuum (the lower and upper
DIFAS: 5.0 User’s Manual 17
limits of each of the ten intervals are presented at the top of each interval). An example
of output for a DIF analysis of ten polytomous items is:
DIF STATISTICS: POLYTOMOUS ITEMS
-------------------------------------------------Name
Mantel
L-A LOR
LOR SE
LOR Z
-------------------------------------------------Var 2
94.397
0.948
0.1
9.48
Var 3
0.061
0.023
0.093
0.247
Var 4
0.508
-0.067
0.094
-0.713
Var 5
0.748
-0.079
0.093
-0.849
Var 6
4.55
-0.2
0.094
-2.128
Var 7
1.61
-0.119
0.093
-1.28
Var 8
1.428
-0.112
0.093
-1.204
Var 9
1.501
-0.114
0.094
-1.213
Var 10
0.543
-0.069
0.093
-0.742
Var 11
2.797
-0.156
0.093
-1.677
-------------------------------------------------CONDITIONAL DIFFERENCES: Intervals of size 3
---------------------------------------------------------------------------------------Lower
0
3
6
9
12
15
18
21
24
27
Upper
3
6
9
12
15
18
21
24
27
30.1
---------------------------------------------------------------------------------------Var 2
0.2
0.05
0.28
0.27
0.3
0.24
0.31
0.18
0.23
0.14
Var 3
0.11
-0.03
0.13
0.02
-0.11
0.12
-0.05
0
-0.12
-0.05
Var 4
0.05
0.12
0
0.08
-0.04
-0.14
-0.1
0.02
-0.01
-0.29
Var 5
0.04
-0.06
-0.01
-0.07
0
-0.04
0.11
-0.11
-0.06
-0.12
Var 6
-0.05
-0.05
-0.1
-0.02
-0.1
0.01
-0.09
-0.02
-0.04
-0.1
Var 7
-0.12
0.08
-0.13
-0.17
0.01
0.05
0.02
-0.01
-0.08
-0.12
Var 8
-0.1
-0.12
0.12
-0.03
0.01
-0.1
-0.06
-0.04
-0.02
-0.1
Var 9
0.04
0.06
-0.15
-0.03
-0.04
-0.06
-0.06
0.08
-0.01
-0.07
Var 10
-0.2
0.02
-0.01
-0.03
0.02
0.01
-0.02
-0.14
0.07
0.01
Var 11
0.07
0.02
-0.04
-0.06
-0.09
-0.08
0.02
-0.01
-0.04
0.04
----------------------------------------------------------------------------------------
Mantel Chi-Square (Mantel) - The Mantel chi-square statistic (Mantel, 1963, Zwick,
Donoghue & Grima, 1993; Zwick, Thayer & Mazzeo, 1997) is distributed as chi-square
with one degree of freedom. Critical values of this statistic are 3.84 for a Type I error
rate of 0.05 and 6.63 for a Type I error rate of 0.01.
Liu-Agresti Cumulative Common Log-Odds Ratio (L-A LOR) – The Liu-Agresti
cummulative common log-odds ratio (Liu & Agresti, 1996; Penfield & Algina, 2003) is
asymptotically normally distributed. Positive values indicate DIF in favor of the
reference group, and negative values indicate DIF in favor of the focal groups.
Standard Error of the Lui-Agresti Cumulative Common Log-Odds Ratio (LOR SE) –
The estimated standard error of the Lui-Agresti Cumulative Common Log-Odds Ratio
(Liu & Agresti, 1996).
Standardized Liu-Agresti Cummulative Common Log-Odds Ratio (LOR Z) – This is
the Liu-Agresti cumulative common log-odds ratio divided by the estimated standard
error. A value greater than 2.0 or less than –2.0 may be considered evidence of the
presence of DIF.
DIFAS: 5.0 User’s Manual 18
Cox’s Noncentrality Parameter Estimator (COX’S B) – Cox’s estimator of the
multivariate hypergeometric noncentrality parameter (Camilli & Congdon, 1999; Cox,
1958) is distributed approximately as standard normal. Positive values indicate DIF in
favor of the reference group, and negative values indicate DIF in favor of the focal
groups.
Standard Error of Cox’s Noncentrality Parameter Estimator (COX SE) – The
estimated standard error of Cox’s noncentrality parameter (Camilli & Congdon, 1999).
Standardized Cox’s Noncentrality Parameter (COX Z) – This is Cox’s noncentrality
parameter estimator divided by the estimated standard error. A value greater than 2.0 or
less than –2.0 may be considered evidence of the presence of DIF.
Differential Test Functioning
The variance of DIF effect across the items of a test or scale has been proposed as a
measure of DTF for sets of dichotomous items (Camilli & Penfield, 1997), and mixtures
of dichotomous and/or polytomous items (Penfield & Algina, 2006). The variance of
DIF effect across dichotomous items has been symbolized by 2 (tau-squared) in the
literature, and the variance of DIF effect across mixtures of polytomous and dichotomous
items has been symbolized by 2 (nu-squared) in the literature. If the test contains all
dichotomous items then select “Dichotomus Models” in the Differential Test Functioning
Menu option. If the test contains all polytomous items, or a mixture of polytomous and
dichotomous items, then select “Polytmous Models” in the Differential Test Functioning
Menu option.
To conduct DTF analyses, select the appropriate items contained in the test (i.g., the
items for which the DIF effect variance is desired), select the grouping variable, and
specify the stratifying variable. These steps mirror those described for DIF analyses
above. DIFAS will print out the following information:
1. Weighted and unweighted estimates of the DIF effect variance
2. Standard error estimators of the weighted and unweighted estimates of the DIF
effect variance
3. The ratio of each DIF effect variance estimate over its respective standard error
estimator (note that the variance estimator is not normally distributed, so the
interpretation of this ratio has not yet been given any guidelines in the literature).
Readers are referred to the papers by Camilli and Penfield (1997) and Penfield and
Algina (2006) for more detailed information about the interpretation and computations of
these estimators.
DIFAS: 5.0 User’s Manual 19
Differential Step Functioning
Differential step functioning (DSF) pertains to invariance observed at each step
underlying the polytomous response variable (a polytomous item with J score levels will
have J – 1 steps). Because DSF effects may vary in sign and/or magnitude across the
steps, examination of DSF effects can prove useful in understanding the location of the
DIF effect (i.e., which score levels are manifesting the DIF effect) and the potential
causes of the DIF effect. Estimates of DSF effects can be computed within the
Cumulative and Adjacent Categories Windows of the Analyses Menu. The cumulative
approach yields odds ratio DSF effect estimators consistent with those modeled under the
graded response model, and the adjacent categories approach yields odds ratio DSF effect
estimators consistent with those modeled under the generalized partial credit model.
Descriptions of these estimators can be found in Penfield (2007a, 2007b).
To conduct DSF analyses, select the appropriate items contained in the test (i.g., the items
for which the DSF effects are desired), select the grouping variable, and specify the
stratifying variable. These steps mirror those described for DIF analyses above. For each
item, DIFAS will print out a table containing the following information for each step of
each selected item:
1. A step-level log-odds ratio DSF effect estimator. This is denoted by CU-LOR
using the cumulative approach and AC-LOR using the adjacent categories
approach.
2. The standard error estimator of the DSF effect estimate.
3. The ratio of each DSF effect estimate over its respective standard error
estimator.
4. The Liu-Agresti common log-odds ratio DIF effect estimator (an estimate of
the item-level DIF effect) and it’s associated standard error estimate.
A sample of the DSF output for one item is shown below. This item is showing large
DSF effect for the first step.
DSF for Var 2
---------------------------------------Step
CU-LOR
SE
Z
---------------------------------------1
1.239
0.19199
6.453
2
0.401
0.15435
2.598
3
-0.23
0.21202
-1.085
---------------------------------------L-A LOR = 0.518
LOR SE = 0.131
Readers are referred to the papers by Penfield (2007a, 2007b) for more detailed
information about DSF and the interpretation and computations of these DSF effect
estimators.
DIFAS: 5.0 User’s Manual 20
Missing Data
DIFAS allows for missing data, but the way in which it deals with missing data depends
on how the stratifying variable is computed. If the total summated score is used as the
stratifying variable (the default in DIFAS), then DIFAS uses a listwise deletion procedure
for any case containing missing data. In this case DIFAS will remove any individual in
the analysis having one or more missing responses. The reason for this is that DIFAS
assumes that the total summated score is an invalid stratifying value when missing values
are present. If, however, an external variable is used as the stratifying variable, then
DIFAS will use the following deletion rules: (a) any individual for which the stratifying
variable value is missing will be omitted from the entire analysis; and (b) individuals
having a nonmissing value of the external stratifying variable, but a missing value on one
or more particular items, will be removed from the DIF analysis only for the item(s) for
which the response is missing.
When a substantial number of missing responses exist, the DIF analyst may find it useful
to compute an external stratifying variable the controls for the impact of missing data on
trait estimation. One approach is to use the mean value across just the items attempted as
the stratifying variable. In this situation, the appropriate stratifying variable would need
to be computed prior to importing the data into DIFAS. Also, because DIFAS only
stratifies on integer values, the user would need to transform the obtained mean values to
integer-level values via some form of rounding or rescaling.
DIFAS: 5.0 User’s Manual 21
CHAPTER 4
Managing Output
All output is printed in the output box on the Main Window of DIFAS. DIFAS allows
you to save the contents of the output box, open a previously saved output file, or
augment the contents of the output as you deem appropriate.
Properties of the Output
The output of DIFAS is completely text in nature. The font of the output is Courier 10point. The use of the Courier font is critical so that the output is formatted in columns
that are appropriately aligned. Although the output box can contain a large amount of
output, the space is not infinite. As a result, you should avoid storing large amounts of
output at a single time without deleting that output which is not crucial. In general, the
output box should be capable of storing at least several thousand lines of output before
experiencing capacity limitations.
Editing the Output
The output box is a simple text editor, and acts like most other data editor programs. You
may add text at any time to the output box. For example, you may want to annotate the
results providing additional information concerning the data file. Contents of the output
box may be cut, copied, and pasted using either the appropriate options of the Edit Menu,
or using the shortcuts Control-x, Control-c, and Control-v. Thus, you can cut contents of
the output box and paste the contents into an external document. Remember, however,
that if you are including the contents of the output box in an external word processing file
(such as a Word or a WordPerfect file), the output will only be aligned if the font is
Courier. Thus, you may need to change the font of the output after pasting the desired
output into the new file. In addition, to prevent output that wraps, the font size may need
to be changed in the new word processing file (10-point Courier font should be fine).
The entire contents of the output box can be erased using the Clear Output option of the
Edit Menu. Using this option immediately clears all contents of the output box, and there
is no way to retrieve any erased contents. Thus, this function should only be used if you
are completely certain that the output is to be deleted.
DIFAS: 5.0 User’s Manual 22
Saving Output Files
To save the contents of the output box, select the Save Output option of the File Menu. A
window will appear that permits you to save the file. The saved file is a text file, and has
the extension “.txt”. As a result, this file can be opened in any other word processing
application (such as Notepad, Word, or WordPerfect). Note, however, that the output
will be appropriately formatted only if the font is set to Courier in the new application.
Opening Output Files
To open the contents of the output box, select the Open Output option of the File Menu.
A window will appear that permits you to open the appropriate file. DIFAS will only
open text files having the “.txt” extension.
Printing Output
DIFAS does not have the capability to print. However, output may be printed by first
saving the output to an external file (see Saving Output Files), opening the external file in
a word processing application (such as Word, Notepad or WordPerfect), and then printing
the output from the word processing application. Note, however, that the output will be
appropriately formatted only if the font is set to Courier in the new application. Notepad
typically uses Courier as the font by default, and thus opening the saved output in
Notepad provides a simple strategy for printing DIFAS output.
DIFAS: 5.0 User’s Manual 23
REFERENCES
Breslow, N. E., & Day, N. E. (1980). Statistical methods in cancer research: Volume 1 –
The analysis of case-control studies. Lyon: International Agency for Research on
Cancer.
Camilli, G., & Congdon, P. (1999). Application of a method of estimating DIF for
polytomous test items. Journal of Educational and Behavioral Statistics, 24, 323341.
Camilli, G., & Penfield, D. A. (1997). Variance estimation for differential test
functioning based on Mantel-Haenszel statistics. Journal of Educational
Measurement, 34, 123-139.
Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. Newbury
Park, CA: Sage.
Cox, D.R. (1958). The regression analysis of binary sequences. Journal of the Royal
Statistical Society B, 20, 215-232.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the MantelHaenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129145). Hillsdale, NJ: Lawrence Erlbaum.
Liu, I-M, & Agresti, A. (1996). Mantel-Haenszel-type inference for cumulative odds
ratios with a stratified ordinal response. Biometrics, 52, 1223-1234.
Mantel, N. (1963). Chi-square tests with one degree of freedom: Extension of the
Mantel-Haenszel procedure. Journal of the American Statistical Association, 58,
690-700.
Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from
retrospective studies of disease. Journal of the National Cancer Institute, 22,
719-748.
Penfield, R. D. (2003). Application of the Breslow-Day test of trend in odds ratio
heterogeneity to the detection of nonuniform DIF. Alberta Journal of
Educational Research, 49, 231-243.
Penfield, R. D. (2007a). Assessing differential step functioning in polytomous items
using a common odds ratio estimator. Journal of Educational Measurement, 44,
187-210.
Penfield, R. D. (2007b, April). Estimating differential step functioning effects under the
graded response and generalized partial credit models. Paper presented at the
annual meeting of the National Council on Measurement in Education, Chicago.
DIFAS: 5.0 User’s Manual 24
Penfield, R. D., & Algina, J. (2003). Applying the Liu-Agresti Estimator of the
Cumulative Common Odds Ratio to DIF Detection in Polytomous Items. Journal
of Educational Measurement, 40, 353-370.
Penfield, R. D., & Algina, J. (2006). A generalized DIF effect variance estimator for
measuring unsigned differential test functioning in mixed format tests. Journal of
Educational Measurement, 43, 295-312.
Robins, J., Breslow, N., & Greenland, S. (1986). Estimators of the Mantel-Haenszel
variance consistent in both sparse data and large-strata limiting models.
Biometrics, 42, 311 – 323.
Zieky, M. (1993). Practical questions in the use of DIF statistics in item development. In
P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 337–364).
Hillsdale, NJ: Lawrence Erlbaum.
Zwick, R., Donoghue, J. R., & Grima, A. (1993). Assessment of differential item
functioning for performance tasks. Journal of Educational Measurement, 30,
233-251.
Zwick, R., Thayer, D. T., & Mazzeo, J. (1997). Descriptive and inferential procedures
for assessing differential item functioning in polytomous items. Applied
Measurement in Education, 10, 321-334.
DIFAS: 5.0 User’s Manual 25
">