Duplicate Command [DU]
By default, Suprtool copies all selected input records to the output file. The
Duplicate command determines what to do with duplicate output records. Duplicate records can be discarded, producing an output file without duplicates. Alternatively, you may be interested in seeing the duplicate records, so you can create an output file consisting solely of the duplicate records. When deciding whether an output record is a duplicate, Suprtool either compares the keys only or the entire output record.
DUPLICATE NONE | ONLY
RECORD | KEYS [num]
[ COUNT ] [ TOTAL ... ]
None
The None option removes duplicate records from the output file. Suprtool compares each output record with the previous output record. If they are not the same, the record is added to the output file. This option corresponds to the former Nodup and
Nodupkey options of the Output command.
>key 1,4
>duplicate none keys
Input Output
1111 10 1111 10
2222 25 2222 25
2222 35 3333 48
3333 48
Only
The Only option is the exact opposite of None. Only selects all output records that would not be written by the None option. When the Only option finds a record that duplicates a record already in the set, it writes that duplicate to the output file. Note that the first record is not written to the output file. Here are two examples:
>key 1,4
>duplicate only keys
Input Output Input Output
1111 10 2222 35 1111 10 2222 35
2222 25 2222 25 2222 42
2222 35 2222 35
3333 48 2222 42
Record
Suprtool has two methods for comparing output records: Record and Keys. The
Record option compares the entire output record. This option can be specified without a sort, but in that case the input file must already be sorted. Note that there are two data fields in the records in the following example, so that a comparison of the entire record yields no duplicates.
Suprtool Commands
Suprtool 5.7 for MPE• 145
146 • Suprtool 5.7 for MPE
>duplicate none record
Input Output
1111 10 1111 10
2222 25 2222 25
2222 35 2222 35
3333 48 3333 48
Keys
The Keys option compares only the sort keys to determine whether an output record is a duplicate. This option requires that at least one sort key be specified.
>sort agent {sort by agent}
>duplicate none keys
>output agents {create roster of agents}
The Keys Num option determines the level at which Suprtool compares sort keys.
This option controls which duplicate records get included in (or excluded from) the output file.
In the following example we sort by agent and by bill-date (in descending order), but only check for duplicates at the agent level.
>sort agent {sort by agent}
>sort bill-date,desc {sort by date}
>duplicate none keys 1 {only check for duplicate agents}
>output agents {create roster of agents}
Count
The Count option causes Suprtool to produce a new field in the output record with the number of occurrences of each key value. The count field is called st-count, and is an I2-type field. The Count option can only be used with Duplicate None Keys.
>key 1,4
>duplicate none keys count
Input Output
1111 10 1111 10 1
2222 25 2222 25 2 {two records for key value 2222}
2222 35 3333 48 1
3333 48
Total
The Total option allows up to 15 fields to be subtotaled for each duplicate key.
Separate the fields with spaces, not commas. The Total option can only be used with
Duplicate None Keys. A new field is created at the end of the output record for each total. Each field is called st-total-n:
>sort customer-no
>extract customer-no
>duplicate none keys total sales-qty sales-amt
The above commands will create a self-describing file with the field customer-no and the total by each customer of the sales-qty in the field st-total-1. Similarly the field st-total-2 will contain the total sales-amt by customer number.
The following data-types are chosen for each total based on the data-type of the field:
Suprtool Commands
Suprtool Commands
Data-Type
Real
IEEE all others
Total Data-Type
R4
E4
P28
Please see the P28 Fields section on how to define these fields in Cobol and
PowerHouse programs.
Note that for byte fields, there can be only digits in the field. If there are other characters such as "+", "-", or ".", then Suprtool reports an error.
You can use the Link output option to easily see the fields that Suprtool creates. For repeated fields (e.g., 6I2), the first subfield is subtotaled if you don't provide a subscript. You can combine the Count and Total options, but the Count option must appear before the Total option. You cannot combine the Total command with the
Total option of the Duplicate command.
>key 1,4
>def field,5,2,integer
>duplicate none keys total field
Input Output
1111 10 1111 1 10
2222 25 2222 2 60 {25 + 35 = 60}
2222 35 3333 4 48
3333 48
Deleting Duplicate Records
There is no direct way to delete duplicate records. Specifying both the Delete and
Duplicate commands does not delete all duplicate output records. This is because the
Delete command occurs in the input phase, as records are read and selected, but the
Duplicate command occurs in the output phase after the sort has taken place. The
Duplicate command is not part of the selection process; it is the If command that determines what records are deleted. A common mistake is to specify the Duplicate and Delete commands without an If command. In this case, Suprtool asks your permission to delete all input records. Answer "No" to this question to abort the task.
It may be possible to remove duplicates from an input dataset using Suprtool, but only through roundabout methods. These are too case-specific to document here. If you need to do this, call Robelle technical support for assistance.
Self-Describing Files
When you are using the Count or Total options, the sort information is not retained in the output file. This means that if your output file is in LINK format, there are no sort keys. If you are going to use this file from Suprlink, use the BY-clause of the
LINK command to manually specify the sort keys.
Non-Self-Describing Files
If the input file is not a self-describing file and you are using Count or Total, you also need to extract all the fields from the file. You can quickly extract the fields by defining a new field that contains the entire record. For example, if your records are
56 bytes long, then you do the following to extract all the fields:
Suprtool 5.7 for MPE• 147
148 • Suprtool 5.7 for MPE
>define entire,1,56,byte
>extract entire
Output Restrictions
When you are using the Count and Total options, the only Output formats that you can use are Data, Data-Num, Query, and Link. If you want to quickly see the Count or Total results without using a second pass, use the List Standard command.
>get dataset
>sort key-field
>duplicate none keys count total data-field
>output result,link
>list standard
>xeq
P28 fields in COBOL and PowerHouse
Some versions of COBOL do not allow a field greater than 18 digits. To use the P28 field, you must define the first five bytes of the field as filler. For example,
05 ST-TOTAL01.
10 FILLER PIC X(05).
10 TOTAL-1 PIC S9(17) COMP-3.
PowerHouse users should define this field as follows:
Define Total packed size 14
Notes
The option of Only or None must be specified before the option of Record or Keys.
Reversing the order causes a syntax error in the Duplicate command.
You cannot combine the Total command with the Total option of the Duplicate command.
It is important to note that if the field being sorted is the result of a $function, then you may not be sorting on the value of the field after the function has transformed the field. For example the following task may not give you the result you expect:
>base store,5
>get d-sales
>def cust-accountx,1,6,byte
>ext cust-accountx = $edit(cust-accout,"zzzz99"
>sort cust-accountx {sorting on transformed field before it has value from function}
>dup none keys
>output dsales
>exit
In this instance you will not be sorting on cust-accountx as transformed by the $edit function, but rather the first six bytes of d-sales. Therefore it is important to note when you are using extract to transform a field, you will not be sorting on that transformed value.
Therefore the way to do the transformation would be to either divide into two tasks or if you can logically sort on the field before the transformation in order to achieve the result, so the task could be:
Suprtool Commands
>base store,5
>get d-sales
>def cust-accountx,1,6,byte
>ext cust-accountx = $edit(cust-accout,"zzzz99"
>sort cust-account {Note sorting on source field}
>dup none keys
>output dsales
>exit or if two tasks are necessary:
>base store,5
>get d-sales
>def cust-accountx,1,6,byte
>ext cust-accountx = $edit(cust-accout,"zzzz99"
>output tempsales
>xeq
>in tempsales
>sort cust-acctx
>dup none keys
>output dsales
>exit
Suprtool Commands
Suprtool 5.7 for MPE• 149
