Conversion codes. Rush Flat Consulting OpenQM

PDF

Dokument

Annonce

Advanced QMQuery

Page 119

SORT TEX.QCH WITH CTRY EQ "US" AND WITH HS EQ "02" AND WITH YEAR GE "2012" YEAR QTR CTRY HS FOB

FOB.PREV.YR1 FOB.PREV.YR1

Year Qtr Ctry HS Code ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

2012 1 US 02 347,807,910 341,725,265 0.2

2012 2 US 02 415,144,509 431,846,739 -0.4

2012 3 US 02 190,119,391 170,331,117 1.2

2012 4 US 02 226,967,852 178,979,452 2.7

2013 1 US 02 420,003,062 347,807,910 2.1

5 record(s) listed

Well, that didn’t work! The 1.78 per cent change shown for the first quarter of 2012 has just been reduced to 0.2 per cent, rather than 1.8 per cent.

The

MR1,Z

conversion has actually moved the decimal point. This may seem an odd behaviour, but it is consistent with the way that conversion codes work.

Conversion codes

You will recall when we first set up dictionaries for the

XRATES

file that we used conversion codes of

MR22

and

MR44

. We could have simply written those as

MR2

and

MR4

respectively – because if only specify one digit, then the second digit is assumed to be the same as the first.

But what are those conversion codes really saying?

As we have already seen, the first digit is the number of decimal places to display. In the absence of any second digit, this works as follows:

Value

12345

Conversion

MR,

MR1,

MR2,

MR4,

Output

12,345

1,234.5

123.45

1.2345

That seems straightforward. Remember, in the absence of a second digit, the second digit is assumed to be same as the first. That makes those conversion codes MR00 , MR11 , MR22 , and

MR44

respectively. Which brings us to the question, what is the second digit doing.

The documentation is a little confusing here. It says that the second digit is the number of implied decimal points in the data. That is OK when the two numbers are the same, or where the input value is an integer.

Take

MR44

applied to 12345. If there are 4 implied decimal places, that means the real number is 1.2345 and we want to display to 4 decimal places. That is 1.2345 again.

Now consider

MR24

applied to 12345. The real number is 1.2345 (from above), and we want to display to 2 decimal places. That is 1.23.

What if our input number isn’t an integer? How many implied decimal places are there?

A better way to think of this is how much should the number be descaled before applying the conversion for decimal places.

For integer numbers, this is pretty much the same as outlined above. If we apply MR24 to

12345, then we descale the number by 4 decimal places to 1.2345 before displaying to 2 decimal places to get 1.23.

Now, let’s consider our issue with the percentage changes above. The first two numbers listed are 1.78 and 3.8676. We want to display these to 1 decimal place, but MR11 carries out a descaling on these numbers before displaying to 1 decimal place.

Page 120

Advanced QMQuery

We don’t want to descale these numbers – they are already scaled correctly – so the second digit needs to be ‘0’.

Change the conversion code to MR10, and try the query again.

SORT TEX.QCH WITH CTRY EQ "US" AND WITH HS EQ "02" AND WITH YEAR GE "2012" YEAR QTR CTRY HS FOB

FOB.PREV.YR1 FOB.PREV.YR1

Year Qtr Ctry HS Code ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

2012 1 US 02 347,807,910 341,725,265 1.8

2012 2 US 02 415,144,509 431,846,739 -3.9

2012 3 US 02 190,119,391 170,331,117 11.6

2012 4 US 02 226,967,852 178,979,452 26.8

2013 1 US 02 420,003,062 347,807,910 20.8

5 record(s) listed

Perfect!

Now, does it work for all the data that we have? Take the YEAR selection criteria off and try again:

SORT TEX.QCH WITH CTRY EQ "US" AND WITH HS EQ "02" YEAR QTR CTRY HS FOB FOB.PREV.YR1 FOB.PREV.YR1%

ID.SUP

00000073: Divide by zero error in dictionary expression

Item: FOB.PREV.YR1%

Filename: TEX.QCH

Id: 20091*US*02

Oops! We’ve got a ‘divide by zero’ error when processing item

20091*US*02

. Let’s think about that.

Item

20091*US*02

looks up item

20081*US*02

to get the preceding year’s value, and then calculated the percentage change from that. In this case, 20081*US*02 doesn’t exist in the file, so that value gets returned as a null value (empty string), and the percentage change calculation tries to divide by this.

The solution to this is to avoid trying to calculate a percentage change if the returned value is zero. We can put this into the I-type formula as follows:

IF (FOB.PREV.YR1 NE '') THEN (((FOB / FOB.PREV.YR1) - 1) * 100) ELSE ''

Update the formula and try again:

SORT TEX.QCH WITH CTRY EQ "US" AND WITH HS EQ "02" YEAR QTR CTRY HS FOB FOB.PREV.YR1 FOB.PREV.YR1%

ID.SUP

Year Qtr Ctry HS Code ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

2009 1 US 02 328,653,879

2009 2 US 02 330,030,602

2009 3 US 02 133,820,377

2009 4 US 02 179,094,504

2010 1 US 02 282,013,013 328,653,879 -14.2

2010 2 US 02 396,571,574 330,030,602 20.2

2010 3 US 02 143,751,110 133,820,377 7.4

2010 4 US 02 195,608,116 179,094,504 9.2

2011 1 US 02 341,725,265 282,013,013 21.2

Now the percentage change works as expected.

Let’s try this for a one-quarter percentage change. Create a dictionary similar to the one you already have for the annual change. A quick way to do this is to copy the existing dictionary item and then modify the copy:

COPY FROM DICT TEX.QCH FOB.PREV.YR1%,FOB.PREV.Q1%

1 record(s) copied.

Now, modify the new item, and change the expression to:

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 121

IF (FOB.PREV.Q1 NE '') THEN (((FOB / FOB.PREV.Q1) - 1) * 100) ELSE '' and change the heading to match. Check that it works in a query statement:

SORT TEX.QCH WITH CTRY EQ "US" AND WITH HS EQ "02" YEAR QTR CTRY HS FOB FOB.PR

Year Qtr Ctry HS Code ......FOB Value ...Prev Qtr FOB ..FOB Value

1 Qtr % Chg

2009 1 US 02 328,653,879

2009 2 US 02 330,030,602 328,653,879 0.4

2009 3 US 02 133,820,377 330,030,602 -59.5

2009 4 US 02 179,094,504 133,820,377 33.8

2010 1 US 02 282,013,013 179,094,504 57.5

2010 2 US 02 396,571,574 282,013,013 40.6

Note that no percentage change is calculated for the first quarter because it could not find a base figure to work from. Thereafter, percentage changes are correctly calculated.

6.4.3 Grouped percentage changes

So far, we are taking one record, looking up a matching record from a previous period, and then calculating a percentage change on the difference between them. That is a good start, but there are many cases where we want to calculate a percentage change between the totals of two groups – for example, what is the percentage increase in exports year on year?

Let’s start with a basic query:

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" AND WITH YEAR GE "2011" BREAK.ON YEAR "'V'"

TOTAL FOB TOTAL FOB.PREV.YR1 FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

20111*CN*02 2011 56,674,740 38,700,242 46.4

20112*CN*02 2011 78,882,851 37,411,198 110.9

20113*CN*02 2011 39,390,693 26,353,058 49.5

20114*CN*02 2011 40,315,951 33,103,077 21.8

2011 215,264,235 135,567,575

20121*CN*02 2012 92,274,894 56,674,740 62.8

20122*CN*02 2012 111,799,808 78,882,851 41.7

20123*CN*02 2012 73,815,737 39,390,693 87.4

20124*CN*02 2012 133,828,822 40,315,951 232.0

2012 411,719,261 215,264,235

20131*CN*02 2013 278,878,964 92,274,894 202.2

2013 278,878,964 92,274,894

905,862,460 443,106,704

9 record(s) listed

That has given us the percentage change for each quarter, but we don’t have the percentage changes for the each year. To do this, we need to modify our percentage change calculation and use the

CALC

keyword.

Change the

I-type

expression in

FOB.PREV.YR1%

to:

IF (TOTAL(FOB.PREV.YR1) NE '') THEN (((TOTAL(FOB) / TOTAL(FOB.PREV.YR1)) - 1) * 100) ELSE ''

Essentially, this has simply told the expression to operate on the totals of the fields on any break lines (when used with the CALC keyword). Running the query again produces exactly the same output as our initial attempt. However, if we add the

CALC

keyword to the query, we get:

Page 122

Advanced QMQuery

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" AND WITH YEAR GE "2011" BREAK.ON YEAR "'V'"

TOTAL FOB TOTAL FOB.PREV.YR1 CALC FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

20111*CN*02 2011 56,674,740 38,700,242 46.4

20112*CN*02 2011 78,882,851 37,411,198 110.9

20113*CN*02 2011 39,390,693 26,353,058 49.5

20114*CN*02 2011 40,315,951 33,103,077 21.8

2011 215,264,235 135,567,575 58.8

20121*CN*02 2012 92,274,894 56,674,740 62.8

20122*CN*02 2012 111,799,808 78,882,851 41.7

20123*CN*02 2012 73,815,737 39,390,693 87.4

20124*CN*02 2012 133,828,822 40,315,951 232.0

2012 411,719,261 215,264,235 91.3

20131*CN*02 2013 278,878,964 92,274,894 202.2

2013 278,878,964 92,274,894 202.2

905,862,460 443,106,704 104.4

9 record(s) listed

Or, if we suppress the detail:

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" AND WITH YEAR GE "2011" BREAK.ON YEAR "'V'"

TOTAL FOB TOTAL FOB.PREV.YR1 CALC FOB.PREV.YR1% DET.SUP NO.GRAND.TOTAL

Year ......FOB Value ....Prev Yr FOB .FOB Value

1 Yr % Chg

2011 215,264,235 135,567,575 58.8

2012 411,719,261 215,264,235 91.3

2013 278,878,964 92,274,894 202.2

9 record(s) listed

Perhaps you don’t want the previous year values showing:

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" AND WITH YEAR GE "2011" BREAK.ON YEAR "'V'"

TOTAL FOB CALC FOB.PREV.YR1% DET.SUP NO.GRAND.TOTAL

Year ......FOB Value .FOB Value

1 Yr % Chg

2011 215,264,235 58.8

2012 411,719,261 91.3

2013 278,878,964 202.2

9 record(s) listed

Note that this last query is a bit deceptive. The percentage change for 2013 is shown as

+202.2 per cent – but the numbers show $278.9m for 2013 and $411.7m for 2012 – which is a decrease. What is happening here is that only the first quarter for 2013 is present, and the percentage change is correctly calculated against the first quarter of 2012.

What about for a longer time period?

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" BREAK.ON YEAR "'V'" TOTAL FOB CALC

FOB.PREV.YR1% DET.SUP NO.GRAND.TOTAL

000000DA: Divide by zero error in dictionary expression

Item: FOB.PREV.YR1%

Filename: TEX.QCH

Id: 20101*CN*02

Well, our test for this error in the dictionary item didn’t work properly. Let’s try another variant. Change the I-type expression in FOB.PREV.YR1% to:

IF (TOTAL(FOB.PREV.YR1) GT 0) THEN (((TOTAL(FOB) / TOTAL(FOB.PREV.YR1)) - 1) * 100) ELSE ''

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 123

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" BREAK.ON YEAR "'V'" TOTAL FOB CALC

FOB.PREV.YR1% DET.SUP NO.GRAND.TOTAL

Year ......FOB Value .FOB Value

1 Yr % Chg

2009 140,522,200

2010 135,567,575 -3.5

2011 215,264,235 58.8

2012 411,719,261 91.3

2013 278,878,964 202.2

17 record(s) listed

Now, we are testing the totalled value against zero rather than null. That makes sense. The null value test worked fine against individual records, because we don’t store zero values.

But if we add several null values together, we get a total of zero – which is not null – and so it went through to the main calculation and hit the divide by zero error.

Arguably, the test should be ‘NE 0’ rather than ‘GT 0’ , but as we won’t experience any negative exports, both expressions will work just fine.

What about if we add more records into the query?

SORT TEX.QCH WITH CTRY EQ "CN" BREAK.ON YEAR "'V'" TOTAL FOB CALC FOB.PREV.YR1% DET.SUP

NO.GRAND.TOTAL

Year ......FOB Value .FOB Value

1 Yr % Chg

2009 3,613,047,492

2010 4,809,399,673 30.4

2011 5,863,790,222 23.8

2012 6,840,528,216 17.8

2013 2,293,298,141 35.2

1666 record(s) listed

Well, that looks OK – but if we get the calculator out, we find that the percentage change in

2010 should be 33.1 per cent. The other figures are wrong too – they are all several percentage points out.

So, what is going on? Well, it doesn’t look like a numeric overflow situation (the numbers would probably turn negative if that were the case), but we’ll change the dictionary item to eliminate that possibility. Let’s create another dictionary item named

FOB.CHG.YR1

with an expression of:

FOB – FOB.PREV.YR1

and we’ll change our percentage change expression to:

IF (TOTAL(FOB.PREV.YR1) GT 0) THEN (TOTAL(FOB.CHG.YR1) * 100 / TOTAL(FOB.PREV.YR1)) ELSE ''

The change we’ve made here is to reduce the magnitude of the working value when we multiply by 100. This will reduce the possibility of numeric overflow. Running the query after making these changes still gives us the same result.

Well, let’s try to deduce how it is getting the results that are being displayed. We’ll want a smallish data set, and we’ll display the detail records. Let’s go back to just displaying

Chapter 02 data:

Page 124

Advanced QMQuery

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "02" BREAK.ON YEAR "'V'" TOTAL FOB TOTAL

FOB.PREV.YR1 TOTAL FOB.CHG.YR1 CALC FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB Change 1 Yr FOB .FOB Value

1 Yr % Chg

20091*CN*02 2009 47,540,097 47,540,097

20092*CN*02 2009 46,094,890 46,094,890

20093*CN*02 2009 24,926,563 24,926,563

20094*CN*02 2009 21,960,650 21,960,650

2009 140,522,200 0 140,522,200

20101*CN*02 2010 38,700,242 47,540,097 -8,839,855 -18.6

20102*CN*02 2010 37,411,198 46,094,890 -8,683,692 -18.8

20103*CN*02 2010 26,353,058 24,926,563 1,426,495 5.7

20104*CN*02 2010 33,103,077 21,960,650 11,142,427 50.7

2010 135,567,575 140,522,200 -4,954,625 -3.5

Everything appears correct in this listing. Let’s add another Chapter:

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "01""02" BREAK.ON YEAR "'V'" TOTAL FOB TOTAL

FOB.PREV.YR1 TOTAL FOB.CHG.YR1 CAL FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB Change 1 Yr FOB .FOB Value

1 Yr % Chg

20091*CN*01 2009 0

20091*CN*02 2009 47,540,097 47,540,097

20092*CN*01 2009 6,128,500 6,128,500

20092*CN*02 2009 46,094,890 46,094,890

20093*CN*01 2009 5,242,213 5,242,213

20093*CN*02 2009 24,926,563 24,926,563

20094*CN*01 2009 3,465,000 3,465,000

20094*CN*02 2009 21,960,650 21,960,650

2009 155,357,913 0 155,357,913

20101*CN*01 2010 7,084,889 7,084,889

20101*CN*02 2010 38,700,242 47,540,097 -8,839,855 -18.6

20102*CN*01 2010 5,331,666 6,128,500 -796,834 -13.0

20102*CN*02 2010 37,411,198 46,094,890 -8,683,692 -18.8

20103*CN*01 2010 2,983,112 5,242,213 -2,259,101 -43.1

20103*CN*02 2010 26,353,058 24,926,563 1,426,495 5.7

20104*CN*01 2010 20,402,116 3,465,000 16,937,116 488.8

20104*CN*02 2010 33,103,077 21,960,650 11,142,427 50.7

2010 171,369,358 155,357,913 16,011,445 5.7

Now we are getting somewhere. Checking with the calculator shows that the percentage change on the 2010 total line should be 10.3 per cent. So, how did it calculate 5.7 per cent?

The top line of 2010 stands out because there is no percentage change on the detail line. That is quite correct as we’d get a divide by zero error if we tried to calculate a percentage change there. But has this affected our total? Let’s check the percentage change without this line.

(16,011,445 – 7,084,889) * 100 / 155,357,913 = 5.7%

So, the $7,084,889 of exports weren’t included in the total for the 2010 calculation. Let’s go back to the

I-type

to figure out why not. The current expression is:

IF (TOTAL(FOB.PREV.YR1) GT 0) THEN (TOTAL(FOB.CHG.YR1) * 100 / TOTAL(FOB.PREV.YR1)) ELSE ''

On the surface this looks OK, but the TOTAL functions in this expression actually determine

WHEN a data value gets added to a total. Our logic here tests the value of our denominator, and if that is positive then it does its calculation. The bit that isn’t obvious is that our numerator value only gets added to its total when that logic branch is executed. Where no value exists for last year (the denominator), then the logic takes the ELSE clause, and there is no TOTAL function there to add the current value to the relevant total. Therefore, when we get to the break line, the total for this year’s exports are missing any values for which there were no corresponding exports in the preceding year.

We need to unconditionally TOTAL the values for this year. We don’t really need to worry about last year’s figures – if they are positive, they will be added in; if they are zero, then they don’t have any impact on the total anyway.

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 125

Let’s try:

(TOTAL(FOB.CHG.YR1) * 100) / (IF (TOTAL(FOB.PREV.YR1) GT 0) THEN TOTAL(FOB.PREV.YR1) ELSE

Now, this totals the change in value, and then divides that by either last years value, or by 1.

The output from this looks like:

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "01""02" BREAK.ON YEAR "'V'" TOTAL FOB TOTAL

FOB.PREV.YR1 TOTAL FOB.CHG.YR1 CALC FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB Change 1 Yr FOB .FOB Value

1 Yr % Chg

20091*CN*01 2009 0 0.0

20091*CN*02 2009 47,540,097 47,540,097 4,754,009,

700.0

20092*CN*01 2009 6,128,500 6,128,500 612,850,00

0.0

20092*CN*02 2009 46,094,890 46,094,890 4,609,489,

000.0

20093*CN*01 2009 5,242,213 5,242,213 524,221,30

0.0

20093*CN*02 2009 24,926,563 24,926,563 2,492,656,

300.0

20094*CN*01 2009 3,465,000 3,465,000 346,500,00

0.0

20094*CN*02 2009 21,960,650 21,960,650 2,196,065,

000.0

2009 155,357,913 0 155,357,913 15,535,791

,300.0

20101*CN*01 2010 7,084,889 7,084,889 708,488,90

0.0

20101*CN*02 2010 38,700,242 47,540,097 -8,839,855 -18.6

20102*CN*01 2010 5,331,666 6,128,500 -796,834 -13.0

20102*CN*02 2010 37,411,198 46,094,890 -8,683,692 -18.8

20103*CN*01 2010 2,983,112 5,242,213 -2,259,101 -43.1

20103*CN*02 2010 26,353,058 24,926,563 1,426,495 5.7

20104*CN*01 2010 20,402,116 3,465,000 16,937,116 488.8

20104*CN*02 2010 33,103,077 21,960,650 11,142,427 50.7

2010 171,369,358 155,357,913 16,011,445 10.3

Well, that gives the correct percentage change on the break line – but is pretty ugly on those lines where there were no exports in the preceding year. We have calculated a ridiculously high percentage change for those lines. Perhaps we can hide them …. Well, yes we can.

Let’s add a range check to the expression:

OCONV(((TOTAL(FOB.CHG.YR1) * 100) / (IF TOTAL(FOB.PREV.YR1) GT 0 THEN TOTAL(FOB.PREV.YR1)

ELSE 1)), 'R-1000,1000')

This has wrapped our previous expression in an

OCONV

function, with a range checking conversion. This only allows values in the range -1,000 to +1,000. Our output now looks like:

Page 126

6.5

Advanced QMQuery

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH HS EQ "01""02" BREAK.ON YEAR "'V'" TOTAL FOB TOTAL

FOB.PREV.YR1 TOTAL FOB.CHG.YR1 CALC FOB.PREV.YR1%

TEX.QCH..... Year ......FOB Value ....Prev Yr FOB Change 1 Yr FOB .FOB Value

20091*CN*01 2009 0 0.0

20091*CN*02 2009 47,540,097 47,540,097

20092*CN*01 2009 6,128,500 6,128,500

20092*CN*02 2009 46,094,890 46,094,890

20093*CN*01 2009 5,242,213 5,242,213

20093*CN*02 2009 24,926,563 24,926,563

20094*CN*01 2009 3,465,000 3,465,000

20094*CN*02 2009 21,960,650 21,960,650

2009 155,357,913 0 155,357,913

20101*CN*01 2010 7,084,889 7,084,889

20101*CN*02 2010 38,700,242 47,540,097 -8,839,855 -18.6

20102*CN*01 2010 5,331,666 6,128,500 -796,834 -13.0

20102*CN*02 2010 37,411,198 46,094,890 -8,683,692 -18.8

20103*CN*01 2010 2,983,112 5,242,213 -2,259,101 -43.1

20103*CN*02 2010 26,353,058 24,926,563 1,426,495 5.7

20104*CN*01 2010 20,402,116 3,465,000 16,937,116 488.8

20104*CN*02 2010 33,103,077 21,960,650 11,142,427 50.7

2010 171,369,358 155,357,913 16,011,445 10.3

Excellent … but it still isn’t correct.

Compound I-types

Hiding some of the results is a pretty ugly way of doing things. Is there a better way?

Let’s think about that for a moment. Our requirements are that both the change in values, and last year’s values get accumulated into the break-point totals. And we don’t want “divide by zero” errors or spurious percentage changes.

What we need is to unconditionally total both values, and then have a conditional expression to decide what to display. We can’t do this in a single statement – but we can put more than one statement within an

I-type

. This is known as a compound

I-type

In a compound I-type , statements are separated by semi-colons. Further, you can refer to the results of earlier statements using an @n syntax (where n is the number of the statement within the

I-type

Using these rules, we can write our expression as:

TOTAL(FOB.PREV.YR1); TOTAL(FOB.CHG.YR1); IF (@1 GT 0) THEN (@2 * 100 / @1) ELSE ''

Now, the output from this is just the same as shown above. And if we now go back to our original statement where we found the errors, that looks like:

SORT TEX.QCH WITH CTRY EQ "CN" BREAK.ON YEAR "'V'" TOTAL FOB CALC FOB.PREV.YR1% DET.SUP

NO.GRAND.TOTAL

Year ......FOB Value .FOB Value

1 Yr % Chg

2009 3,613,047,492

2010 4,809,399,673 33.1

2011 5,863,790,222 21.9

2012 6,840,528,216 16.7

2013 2,293,298,141 31.3

1666 record(s) listed

Checking those percentage changes with a calculator, we find they are correct.

Incidentally, if you tried the above query with our earlier I-type where we hid the large values, you would have found that it gave the wrong percentage changes at this level. So, it is a good thing that we abandoned that expression.

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 127

Hopefully, this section has highlighted the need to CHECK YOUR RESULTS. Answers may look OK but be quite wrong.

Once you know that you have an incorrect result, it is important to understand why you got that particular result so that you can work toward the correct solution. And just because your current solution gives the correct answer some of the time, doesn’t mean it gives the correct answer all of the time.

6.6 Alternate Key Indices

Did you find that you were waiting for what seemed like a long time for your query to finish when doing the above calculations? That is because the query had to read all the items in the

TEX.QCH

file to find just that small number that were required to answer our query.

For example, in the query:

SORT TEX.QCH WITH CTRY EQ "CN" BREAK.ON YEAR "'V'" TOTAL FOB CALC FOB.PREV.YR1% DET.SUP

NO.GRAND.TOTAL

… just 1,666 records were selected out of the 411,502 records in the file. On my notebook, that query took just over seven (7.014) seconds, meaning it processed nearly 60,000 records per second. That is pretty impressive, but that is a sequential read – if it was reading in random order that rate would be likely to be significantly slower.

The problem for the query is how to minimise the time it takes to select the records required.

Once it has the 1,666 records, it doesn’t take long to sort them and display them – but the selection itself takes a long time.

Alternate key indices are one way to speed up the selection process. An alternate key index indexes the file on the value contained in a field, or provided by a dictionary item. For example, to speed up the above query, an index based on the CTRY field would enable the query to instantly select just those records with a

CTRY

value of “CN”. Let’s try that.

The basic command to make an index is

MAKE.INDEX

25 and has the syntax:

MAKE.INDEX filename field(s) {NO.NULLS}

For example:

MAKE.INDEX TEX.QCH CTRY

Added index for CTRY

Building index 'TEX.QCH CTRY'...

411502 records processed

Populating index...

Now, if we run the above query again, it takes just 0.218 seconds. That is an impressive gain.

Essentially, an alternate key index is a cross-reference file keyed on the values of the field.

QMQuery automatically takes advantage of alternate key indices where they are available.

In the above query, QMQuery reads the

CTRY

index for the

TEX.QCH

file, and gets the CN item. This is a list of item-ids that have a CTRY field of CN. From there, it is a simple matter of reading and processing those records.

Key issues for indexing are:

25 MAKE.INDEX

is actually a combination of the CREATE.INDEX

and BUILD.INDEX

commands. See the online help for more information on these commands.

Page 128

Advanced QMQuery

➢

➢ which fields should be indexed how should we construct queries to ensure that available indices are used.

A file can have up to 32 indices. That should be sufficient to restrict the item selection for most queries.

There are a couple of downsides with indices:

➢

Each index takes up additional disk space. The basic

TEX.QCH

file takes up about

14.4 MB. The CTRY index we added above takes up a further 5 MB. If we add a HS index, that takes up another 5.1 MB. If we add a third index, then the total space devoted to indices will exceed the space taken up by the data.

Indices slow down the process of writing to the file. This is because whenever an item is written to (create or update) or deleted from the file, then each index must be checked and updated if necessary.

How much do indices slow down file updates? The

TEX.QCH

file was built by importing the records form a

CSV

file to a holding file via the AccuTerm

FTD

file import utility. A QMBasic program was then used to transform the data in the holding file, and write it to TEX.QCH

Without indices, the process of transferring 24,696 records in the holding file into 24,206 items in

TEX.QCH

took 1.9 seconds. With one index defined on

TEX.QCH

, the same process took 6.6 seconds. With two indices defined, the process time increased to 18.6 seconds, while loading with three indices took 79.9 seconds. (In this case, the third index was poorly constructed and had a particularly adverse impact on performance).

In general, the more indices there are to maintain, then the greater the slowdown in update processes. The slowdown is quite apparent when making bulk updates like this. However, if you are only saving a single screen of data, then the slowdown may not be apparent at all.

So, this tells us that indexing a file gives us higher performance in selecting data from the file, at the expense of lower performance in updating the file. Further, the cost increases as we increase the number of indices – therefore, we should use indices sparingly.

Now, what is the best way to use an index. In our example so far, we have specified the exact field value we want. What about when we want a range of field values?

Say, we want to select records for China in the 2012 year. Consider the following two statements:

SORT TEX.QCH WITH YEAR EQ "2012" AND WITH CTRY EQ "CN" BY HS BREAK.ON HS TOTAL FOB DET.SUP NO.PAGE

SORT TEX.QCH WITH CTRY EQ "CN" AND WITH YEAR EQ "2012" BY HS BREAK.ON HS TOTAL FOB DET.SUP NO.PAGE

The first statement took about 6.4 seconds on my notebook, while the second took less than

0.1 seconds. The second statement is so much quicker than the first because it selects on the indexed field before the non-indexed field.

So, the first rule of using indices is to make sure that any selection puts indexed fields ahead of non-indexed fields.

Now, we’ll look at using indices to select a range of values – say 2012 exports for Chapter

02. We’ll start with the basic statement:

SORT TEX.QCH WITH HS EQ "02" AND WITH YEAR EQ "2012" BY CTRY BREAK.ON C%SHORTNAME TOTAL FOB

DET.SUP NO.PAGE

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 129

Neither of the selection fields are indexed so this statement takes just over six seconds. Note that we’ve selected the HS code first. This is because we can reduce our selected records faster this way. We only want one HS Chapter out of 99, while we want one year out of five.

By selecting on HS code first, there are far fewer second comparisons required to get our final records.

Now, should we index on

YEAR

? There are only 5 years in the database, so its a poor choice for indexing. In comparison, there are 17 YYYYQ values. That still isn’t great, but we’ll index on that in the meantime:

MAKE.INDEX TEX.QCH YYYYQ

Added index for YYYYQ

Building index 'TEX.QCH YYYYQ'...

411502 records processed

Populating index...

Running the query again gives us a similar time – just over six seconds. If we reverse the selection order (year then HS code), the time increases to 6.5 seconds.

Clearly, the indexing of YYYYQ is not having any effect on this selection. This is despite year being a function of

YYYYQ

Now, let’s specify the

YYYYQ

values directly:

SORT TEX.QCH WITH YYYYQ EQ "20121""20122""20123""20124" AND WITH HS EQ "02" BY CTRY BREAK.ON C

%SHORTNAME TOTAL FOB DET.SUP NO.PAGE

This is actually worse. Query times are now over 12 seconds.

Has the query used the index? Let’s add the

REQUIRE.INDEX

modifier to the query statement:

SORT TEX.QCH WITH YYYYQ EQ "20121""20122""20123""20124" AND WITH HS EQ "02" BY CTRY BREAK.ON C

%SHORTNAME TOTAL FOB DET.SUP NO.PAGE

OR relationship cannot be resolved with an index

Processing terminated: This query cannot be resolved with an index

In our earlier statement, the query processor did not know how to handle the selection for

YYYYQ

with indices, so reverted to scanning the whole file. This is why it took so long. Still, it is surprising that it took nearly twice as long as the query against

YEAR

that returned the same values.

How else can we specify the range of dates? Let’s try:

SORT TEX.QCH WITH YYYYQ GE "20121" AND LE "20124" AND WITH HS EQ "02" BY CTRY BREAK.ON C%SHORTNAME

TOTAL FOB DET.SUP NO.PAGE REQUIRE.INDEX

This took around 1.6 seconds. As did this:

SORT TEX.QCH WITH YYYYQ BETWEEN "20121" "20124" AND WITH HS EQ "02" BY CTRY BREAK.ON C%SHORTNAME

TOTAL FOB DET.SUP NO.PAGE REQUIRE.INDEX

So, our query times are affected by the way we write our query. Specifically, the query processor cannot handle a request to select a value that is “equal to x or y”. But it can handle an equivalent expression where it is asked to select a value that is “greater than or equal to

x and less than or equal to y” or “between x and y”.

While we have created an index on

YYYYQ

, this really isn’t a good field to index.

YYYYQ

has only 17 unique values in the file at present. That means that there are 24,206 item-ids in each index value.

CTRY

and

are much better fields to index because they have 247 and 99 unique values respectively to index on.

We’ll test that assumption now. We’ll delete the

YYYYQ

index, create an index for the HS field, and run an equivalent query:

Page 130

6.7

Advanced QMQuery

DELETE.INDEX TEX.QCH YYYYQ

Deleted index YYYYQ

MAKE.INDEX TEX.QCH HS

Added index for HS

Building index 'TEX.QCH HS'...

411502 records processed

Populating index...

Now, we’ll modify our query so that the HS selection is first:

SORT TEX.QCH WITH HS EQ "02" AND WITH YEAR EQ "2012" BY CTRY BREAK.ON C%SHORTNAME TOTAL FOB

DET.SUP NO.PAGE

This query now runs in a bit less than 0.2 seconds. That compares with 1.6 seconds when we were indexing by YYYYQ .

If you wish to see what indices have been defined for a file, use the

LIST.INDEX

command:

LIST.INDEX filename field(s) {STATISTICS} {DETAIL}

LIST.INDEX filename ALL {STATISTICS} {DETAIL}

LIST.INDEX TEX.QCH ALL

Alternate key indices for file TEX.QCH

Number of indices = 2

Index name...... En Tp Nulls SM Fmt NC Field/Expression

HS Y I Yes S R N FIELD(@ID, '*', 3)

CTRY Y I Yes S L N FIELD(@ID, '*', 2)

Or, if you wish to see which files have alternate key indices enabled against them, use the

FILE.STAT

command and look in the AK column. Note, your display output needs to be at least

132 columns wide to display this report.

Summary

Alternate key indices work by maintaining lists of item-ids for each value of an indexed field. This allows you to quickly select the item-ids for a given value of the indexed field at the expense of slower updates (because of the need to maintain the index items).

A good field to index on is one that you use frequently and has many unique values.

In a query, select on indexed fields before selecting on non-indexed fields.

When selecting on an indexed field, select for either a single value, or a range of values. Do not select for several values as the query processor does not handle this. Use the

REQUIRE.INDEX

modifier to check that your query properly uses the available indices.

Using QMBasic in I-types

Using the TEX.QCH

file, we were able to look up a previous value by calculating the ID of the item. Let’s consider how we would do the equivalent lookups in the IRATES and XRATES files.

The lookups we want to do are for one month previous, and one year previous. Let’s start with one month previous.

From any given date, we will define the one month earlier date to be the same day number of the previous month. Now, what is the one month previous date from the 31 st of March? There is no 31 st of February, but we can reason that the appropriate date is the 28 th or 29 th of

February, or more generally, the last day of February. How do we do this in a lookup? How do we do this for any month in the year?

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 131

Let’s assume we’ve solved that problem. We can now determine a date value for one month prior to our base date. Is there a value to look up on that date?

The nature of the data we are dealing with is that there are values on weekdays, but not on weekends or public holidays. So, April 30 th 2013 was a Tuesday. One month prior this date is

March 30 th 2013. This was a Saturday, and has no recorded value. Likewise, there is no data recorded on December 25 th or the 1 st of January, so we have issues with our lookups one month after these dates. So, how do we handle weekends, public holidays, and any other days that do not have recorded values?

Clearly, this is not a straightforward process of calculating a date value, and looking it up.

We need to:

➢ calculate an initial date value. Check to see whether this is a valid date, and adjust if necessary

➢ look up the data to see if data is present. If not, then step back through time until data is found, or if there is no preceding data, then return null.

These calculations don’t easily fit into an I-type expression. But they can be done easily in a

QMBasic subroutine (or function), and then we can get the

I-type

to call the subroutine using the

SUBR

function.

The function on the next page calculates a datevalue given a starting date and a parameter telling it how far away the new date is.

This should be entered into the QMBasic programs file, that we created back in Section

4.4.4.

Now use the

WED

editor to enter the program:

WED BP DATEVALUE.CALC

FUNCTION DATEVALUE.CALC(fromdate, chgval)

*****************************************************************************

* Bp DateValue.Calc - Calculate a date id, that is chgval away from fromdate.

* Author : BSS

* Created: 28 Jun 2013

* Updated: 28 Jun 2013

* Version: 1.0.0

* Pass:

* fromdate - an internal date value

* chgval - {+|-}nP where P = M or Y

* e.g. -1M One month previous

* -5M Five months previous

* 1Y One year ahead

$CATALOG LOCAL

$MODE UNASSIGNED.COMMON

COMMON /QMQ/ DFORMAT previd = ''

IF NOT(fromdate AND NUM(fromdate)) THEN RETURN previd

IF (chgval EQ '') THEN chgval = '-1M' ptype = OCONV(OCONV(chgval, 'MCA'), 'MCU') pno = OCONV(chgval, 'MC/A') dd = OCONV(fromdate, 'DD') mm = OCONV(fromdate, 'DM') yyyy = OCONV(fromdate, 'DY')

Page 132

Advanced QMQuery

BEGIN CASE

CASE (ptype EQ 'M')

mno = yyyy * 12 + mm + pno

m1 = MOD(mno, 12)

y1 = mno // 12

IF (m1 EQ 0) THEN

m1 = 12

y1 -= 1

END

CASE (ptype EQ 'Y')

m1 = mm

y1 = yyyy + pno

CASE 1

RETURN previd

END CASE

IF UNASSIGNED(DFORMAT) THEN

EXECUTE "DATE.FORMAT INFORM"

DFORMAT = @SYSTEM.RETURN.CODE

END

IF (DFORMAT) THEN dconv = 'D' ELSE dconv = 'DE' d1 = dd

LOOP

prevdate = d1:'-':m1:'-':y1

previd = ICONV(prevdate, dconv)

IF (STATUS() EQ 0) THEN EXIT

d1 -= 1

IF ((dd - d1) GT 10) THEN

previd = ''

EXIT

END

REPEAT

RETURN previd

* ------------------------------------------------------------------------- *

END

Enter the program, save it, and exit the editor. Now, we need to compile the program:

BASIC BP DATEVALUE.CALC

Compiling BP DATEVALUE.CALC

******

0 error(s)

DATEVALUE.CALC added to local catalogue

Compiled 1 program(s) with no errors

Let’s quickly look at what this program does – or is supposed to do. The main actions are:

➢

➢ set a default return value check that a valid date has been passed split the ‘chgval’ variable into a unit of movement (M or Y) and a number of units break the passed date into day, month, and year components

➢

➢ calculate test values for the month and year based on the movement specified check whether the system is set to North American or International date format and adjust the date conversion accordingly. This date format is stored in a named common so that we only need to test this once.

Start a loop

◦ form an international date string based on the test values of month and year

Getting Started in OpenQM - Part 1 © Rush Flat Consulting

Advanced QMQuery

Page 133

➢

◦

◦ try to convert this to an internal date

STATUS()

is zero, the conversion was successful and we have a valid date. Exit the loop.

Otherwise, go back a day and repeat the loop.

Return from the program.

There is also a check in the loop to make sure that it does not loop infinitely, but this should never occur anyway.

You may note that there is no check for a valid day after we have taken a day off. And you might think that if we take a day off the first day of the month, then we will reach an impossible day number. That is all true – but it can’t happen. If we started with the first of the month, there is ALWAYS a first of the preceding month. The only time when we will get invalid date conversions is at the end of the month where it is possible that our current month has more days than the preceding month.

Now, let’s test the program. Create an

I-type

dictionary item in the

XRATES

dictionary called

PREV.ID.M1

. The expression should be:

SUBR('DATEVALUE.CALC', @ID, '-1M')

Don’t bother with a conversion at this stage, and use a format of 7R . Now, let’s try it:

SORT XRATES DATE PREV.ID.M1

XRATES.... Date........ Prev ID

-1 Mth.

15346 05 JAN 2010 15315

15347 06 JAN 2010 15316 etc

15397 25 FEB 2010 15366

15398 26 FEB 2010 15367

15401 01 MAR 2010 15373

15402 02 MAR 2010 15374 etc

15426 26 MAR 2010 15398

15429 29 MAR 2010 15400

15430 30 MAR 2010 15400

15431 31 MAR 2010 15400

15432 01 APR 2010 15401

15437 06 APR 2010 15406

Look at the date values returned for the dates from 29 March through to 31 March. These all return 15400. If we look back to the end of February, we see that we don’t have date 15400 on file, but 26 February was date 15398, so date 15400 must be 28 February 2010. In other words, the calculated date for one month prior to 29 March 2010 was 28 February 2010.

Likewise, the 30 th and 31 st of March both returned 28 February also.

Now what did we just do? We called a function using the SUBR function. The SUBR function takes a variable number of arguments – we must pass the subroutine (function) name first,

followed by any arguments required by the subroutine we are calling 26 .

The

DATEVALUE.CALC

function we just created needs two arguments – a date to start from, and a distance to move. We passed @ID as the initial date value, and ‘-1M’ as the movement value – meaning go back one month. If we passed ‘-3M’, we’d go back 3 months; ‘-15M’

26 Note: If you wish to call a subroutine using the SUBR function, then you need to write the subroutine to return the answer as the first argument – but you don’t include this answer argument in the list of arguments passed in the SUBR call.

Because we are using a function, then the arguments passed match the arguments declared by the function. We are using a function here because the arguments match which makes the concept easier to understand.

Page 134

Advanced QMQuery would go back 15 months; ‘2M’ would go forward 2 months; and ‘-1Y’ would go back one year.

Note that we’ve built quite a bit of flexibility into the DATEVALUE.CALC

function. Even though we’ve only discussed finding dates for the previous month, we’ve already allowed for the possibility of finding dates months and even years into the past – or future. Nor have we restricted people to months between one and 12 – any number of months will be handled correctly.

Of course, we could add even more flexibility by adding the ability to handle day movements as well as years and months. Feel free to do so.

Now, we need to look up a currency value based on the date returned by the

PREV.ID.M1

dictionary item. We’ll create another function to do this, as the same code needs to be used by each currency.

WED BP XRATES.LOOKUP

Now, enter the program from the next page:

FUNCTION XRATES.LOOKUP(currency, dateval)

*****************************************************************************

* Bp Xrates.Lookup - Look up the value of currency at dateval. If no value,

* then look up previous value.

* Author : BSS

* Created: 28 Jun 2013

* Updated: 01 Jul 2013

* Version: 1.0.0

* Pass:

* currency - ISO currency identifier (XRATES dictionary name)

* dateval - internal date value

$CATALOGUE LOCAL xval = ''

IF NOT((dateval NE '') AND NUM(dateval)) THEN RETURN xval

OPEN 'DICT','XRATES' TO xrates.dict ELSE RETURN xval

OPEN 'XRATES' TO xrates ELSE RETURN xval

READ xdict FROM xrates.dict, currency ELSE

READ xdict FROM xrates.dict, UPCASE(currency) ELSE

RETURN xval

END

END amc = xdict<2>

IF NOT((amc NE '') AND NUM(amc)) THEN RETURN xval xid = dateval

LOOP

READ xvec FROM xrates, xid ELSE xvec = ''

xval = xvec<amc>

IF (xval NE '') THEN EXIT

xid -= 1

IF ((dateval - xid) GT 10) THEN

xval = ''

EXIT

END

REPEAT

RETURN xval

* ------------------------------------------------------------------------- *

END

Let’s look at what this does. The steps are:

Advanced QMQuery

Page 135

➢

➢ set a default return value, and check that you have passed a valid date (number) open the

XRATES

file and its dictionary read the dictionary definition of the currency you have passed. If it isn’t found then return. Get the field number from the dictionary item and check that it is a number.

Start a loop

◦ read the exchange rates for the specified date

◦

◦ see if there is an entry in the specified field number. If so, then exit the loop

Otherwise, decrement the date value.

Check that the new date value is no more than 10 days from the original date value. If OK then do another loop, else exit.

➢

Return from the program.

Now, we’ll compile it:

BASIC BP XRATES.LOOKUP

Compiling BP XRATES.LOOKUP

****

0 error(s)

XRATES.LOOKUP added to local catalogue

Compiled 1 program(s) with no errors

We’ll now use this in an I-type to retrieve the US Dollar value for one month previous. We’ll call this

USD.PREV.M1

and it will look like:

SUBR(‘XRATES.LOOKUP’, ‘USD’, PREV.ID.M1)

MR44,

US Dollar²Prev mth

Now:

SORT XRATES DATE USD PREV.ID.M1 USD.PREV.M1

XRATES.... Date........ US Dollar Prev ID US Dollar

-1 Mth. Prev mth.

15346 05 JAN 2010 0.7344 15315

15347 06 JAN 2010 0.7343 15316

15348 07 JAN 2010 0.7378 15317

15349 08 JAN 2010 0.7325 15318

15352 11 JAN 2010 0.7395 15321 etc

15376 04 FEB 2010 0.7021 15345

15377 05 FEB 2010 0.6869 15346 0.7344

15380 08 FEB 2010 0.6883 15349 0.7325

15381 09 FEB 2010 0.6823 15350 0.7325

15382 10 FEB 2010 0.6961 15351 0.7325

15383 11 FEB 2010 0.6927 15352 0.7395

We can see from this listing, that it is looking up the correct exchange rate based on the rules that we outlined at the beginning – namely, if an exchange rate doesn’t exist for the nominated date, then it will step back in time until one is found. For example, on the 10 th of

February, the lookup date is the 10 th

8 th

of January – which doesn’t exist. So, it steps back to the

of January and returns the value found for that date.

All we need to do now is create a percentage change dictionary item similar to the we created for the FOB percentage changes. We’ll call this USD.PREV.M1% and it will look like:

Page 136

Advanced QMQuery

IF (USD.PREV.M1 NE ‘’) THEN (((USD / USD.PREV.M1) – 1) * 100) ELSE ‘’

MR10,

US Dollar²% chg 1 mth

SORT XRATES DATE USD PREV.ID.M1 USD.PREV.M1 USD.PREV.M1%

XRATES.... Date........ US Dollar Prev ID US Dollar US Dollar..

-1 Mth. Prev mth. % Chg 1 mth

15346 05 JAN 2010 0.7344 15315

15347 06 JAN 2010 0.7343 15316

15376 04 FEB 2010 0.7021 15345

15377 05 FEB 2010 0.6869 15346 0.7344 -6.5

15380 08 FEB 2010 0.6883 15349 0.7325 -6.0

15381 09 FEB 2010 0.6823 15350 0.7325 -6.9

15382 10 FEB 2010 0.6961 15351 0.7325 -5.0

15383 11 FEB 2010 0.6927 15352 0.7395 -6.3

If we get the calculator out, we find that these percentage changes are calculated correctly.

Simplifying the process

All of the above seems somewhat complicated. To get a percentage change for a currency, we need to create TWO dictionary items – one to get the currency value for a preceding period, and a second to do the actual calculation. Can we simplify this process so that we only need to create one dictionary item to calculate a percentage change over a given period?

Well, of course we can. But it is useful to have gone through the individual steps, so you can see the process involved.

As always, there are multiple ways of doing things, so let’s stop for a moment and consider what we want to do.

We’ve created two functions – one to calculate a datevalue; and another to look up a currency value at or near that datevalue. If we are going to simplify things, should we amalgamate these functions into our overall calculation?

The

DATEVALUE.CALC

function is a useful general function to have. I would suggest that you keep this separate so that you can use it in other dictionary items or programs.

The

XRATES.LOOKUP

function is a bit more specific to the process of calculating percentage changes. Even so, there may be other times when you want to use this type of lookup, so we’ll keep this separate too.

That means we are going to create another function that will call both of these functions in its own calculation of a percentage change. And what we need to pass to this new function is a combination of the inputs to the two functions we have already created – a currency identifier, and a period (change) identifier. This function is shown below:

Advanced QMQuery

Page 137

FUNCTION XRATES.PC.CHANGE(currency, chgval)

*****************************************************************************

* Bp Xrates.Pc.Change - Calculated the percentage change in the currency over

* period chgval.

* Author : BSS

* Created: 01 Jul 2013

* Updated: 01 Jul 2013

* Version: 1.0.0

* Pass:

* currency - ISO currency identifier (XRATES dictionary name)

* chgval - {+|-}nP where P = M or Y

* e.g. -1M One month previous

* -5M Five months previous

* 1Y One year ahead

$CATALOGUE LOCAL pcchange = ''

OPEN 'DICT','XRATES' TO xrates.dict ELSE RETURN pcchange

READ xdict FROM xrates.dict, currency ELSE

READ xdict FROM xrates.dict, UPCASE(currency) ELSE

RETURN pcchange

END

END amc = xdict<2>

IF NOT((amc NE '') AND NUM(amc)) THEN RETURN pcchange

CALL DATEVALUE.CALC(xdate, @ID, chgval)

IF (xdate EQ '') THEN RETURN pcchange

CALL XRATES.LOOKUP(prevval, currency, xdate)

IF NOT(prevval AND NUM(prevval)) THEN RETURN pcchange nowval = @RECORD<amc> pcchange = ((nowval / prevval) - 1) * 100

RETURN pcchange

* ------------------------------------------------------------------------- *

END

You can see that we have duplicated some of the code from

XRATES.LOOKUP

where we find the field number associated with a currency identifier. After that, the DATEVALUE.CALC

function is called to get the earlier date, then the value of the exchange rate for the passed currency is retrieved by the

XRATES.LOOKUP

function. Finally, the percentage change is calculated from the two exchange rate values – the one in the current record being processed, and the one we have just looked up.

Note that this calculation simply uses the internal values of the exchange rates without converting them to external format. That isn’t particularly important here, but in other circumstances where the data items have different implied decimal precisions, then you need to consider the appropriate degree of scaling to apply to the result.

This function is clearly built around the assumption that it is going to be called from a dictionary item. This is because it is reliant on the variables

@ID

and

@RECORD

being set. If you are going to call this function from something other than QMQuery (say a QMBasic program), then it is your responsibility to ensure those variables are set.

Note that the DATEVALUE.CALC

and XRATES.LOOKUP

functions have actually been called as subroutines. A function is normally called as: answer = function-name(arg1, arg2 etc)

Page 138

6.7.1

Advanced QMQuery

Without going into a technical explanation, we’ll simply state that a function can be called as a subroutine using the following syntax:

CALL function-name(answer, arg1, arg2 etc)

Calling as a subroutine has the advantage that we don’t have to declare the function within the program. However, it has the disadvantage that we can’t directly use the function as an input to another statement or function. These issues are really beyond the scope of this book, but are covered further in “Getting Started in OpenQM – Part 2”.

Now, create a new dictionary item named GBP.PREV.M1% as follows:

SUBR(‘XRATES.PC.CHANGE’, ‘GBP’, ‘-1M’)

MR01,

UK Pound²% chg 1 mth

SORT XRATES DATE GBP GBP.PREV.M1%

XRATES.... Date........ UK Pound UK Pound...

% chg 1 mth

15346 05 JAN 2010 0.4564

15347 06 JAN 2010 0.4591

15348 07 JAN 2010 0.4606

15349 08 JAN 2010 0.4597

15352 11 JAN 2010 0.4604

15376 04 FEB 2010 0.4419

15377 05 FEB 2010 0.4358 -4.5

15380 08 FEB 2010 0.4414 -4.0

15381 09 FEB 2010 0.4381 -4.7

15382 10 FEB 2010 0.4428 -3.7

15383 11 FEB 2010 0.4441 -3.5

Once again, the calculator confirms that these percentage changes are correct.

Now we only need one dictionary item per currency to calculate the percentage change in value over a given time period.

Calculating an index value

An index value is similar to a percentage change. It expresses a current value as a proportion of a past value. However, whereas the percentage changes are calculated from a set number of periods in the past, the index value is always expressed relative to a single period in the past – the base period.

The general formula to calculate an index value is:

CURRENT * indexbase / BASE where indexbase is a usually value of 100 or 1000. If the current value is the same as the base value, then it will be given an index value of indexbase

For example, we might want to express the exchange rates relative to their value as at the beginning of February 2010. At that date, the US Dollar exchange rate was 0.7006. One year later, this exchange rate had increased to 0.7724. If we set our base period to a value of

1,000, then the February 2011 value has an index value of 1,102. In other words, the exchange rate had increased by about 10 per cent since the base period.

There is no particular difficulty in putting this into a dictionary item, but think for a moment how to include the base period value:

Advanced QMQuery

Page 139

➢

We could hard code this value into the dictionary item – but then if we change the base period, we will need to edit the dictionary item. As we have more than one exchange rate, we would need to edit each dictionary with a hard coded exchange rate.

We could enter the value into a dictionary item that always returns the @ID of the base period. We could then use this to look up the appropriate base value. We would only need to edit this one dictionary item to change the base period for all exchange rates.

This second option gives us greater flexibility, so we will use that. First, create a

BASEPERIOD dictionary. This should look like:

CT DICT XRATES BASEPERIOD

DICT XRATES BASEPERIOD

01: I

02: 15373

03:

04: Base period

05: 7R

06: S

07:

This defines the dictionary as an I-type , but gives it a constant value of 15373 which is the internal date value for the 1 st of February 2010. Perhaps you may want this value to be a bit more transparent so that you can understand what date is being referred to when you look at it. In that case, try replacing the expression with:

ICONV('01 FEB 2010', 'D')

Now, we create a dictionary to look up the value of the US Dollar in this base period. We will call this USD.BASE

CT DICT XRATES USD.BASE

DICT XRATES USD.BASE

01: I

02: TRANS('XRATES', BASEPERIOD, USD, 'X')

03: MR44,

04: USD Base

05: 7R

06: S

07:

Now, we can try these dictionary items to make sure they work: