Chapter 27
Modifying 4tH
27.1
Introduction
A good scripting language must be easy extensible. We will cover the most common extensions. You will acquire in-depth knowledge of the inner workings of 4tH. All of 4tHs functions are a toolkit in itself and can be put to your own use (especially comp_4th(), which is 4tHs most complex function).
As we proceed, there will be more files to edit and the modifications will get more complex.
Be sure you mastered the previous extensions, before you get on with the more elaborate ones. A good knowledge of C is required for most operations.
You will also note that several modules, e.g. the parser and the peephole optimizer, are not covered by this manual. First, because by design they are usually not affected by any changes to the language itself and second, they are quite complex - and consequently better left alone unless you profoundly understand what you’re doing.
27.2
Understanding 4tHs versioning
4tHs version is made up of three components:
1. Major version; this indicates the 4tH architecture number. It is incremented only if a new architecture is applied or when the minor version reaches ”99”.
2. Minor version; this indicates the 4tH bytecode number. It is incremented only if the bytecode changes or when a significant feature is added to 4tH. It is reset after an architecture change.
3. Release; this indicates the release number. It is incremented with every release within a bytecode version and reset after a bytecode change.
The combination of major version and minor version is called the bytecode version. As long as the bytecode version remains the same, the bytecode is guaranteed to work with all releases within a bytecode version. A bytecode change happens in the following cases:
• A bytecode is added or removed;
• The bytecode numbering (order) is changed;
512
CHAPTER 27. MODIFYING 4TH 513
• The behavior of a bytecode changes significantly (e.g. different input- or output parameters).
If you change the bytecode and fail to change the bytecode versioning you’re in fact disabling one of the major security mechanisms of 4tH, since it is unable to determine the compatibility of the bytecode. You may end up with a very unstable 4tH and some hard to track bugs, so that is definitely not what you want. Versioning is controlled by two
#define s in cmds_4th.h:
/* header */
#define Version4th
#define App4th
0x362
0L
Version4th is the bytecode version. Usually, you want to leave that one alone in order to see on which version of 4tH your modified 4tH is based. For that reason App4th was added. Standard 4tH always comes with App4th set to zero. App4th sets the application byte and is also part of an HX file
1
. If the application byte of an HX file differs from the one of the virtual machine, the HX code is rejected. Note that the files generated by cgen_4th() do not have any protection mechanism
2
, so the best thing is to regenerate them with every compile.
In short, if you want to experiment with your own modified 4tH, be sure to change the application byte when applicable.
27.3
A closer look at comp_4th()
As we already know, comp_4th() compiles source to H-code. First of all, we need to have a source. This is a simple character array, which is pointed to by "Source". Then an
H-code header is created.
The header is initialized by InitObject(), which calls ParseText() to get the initial size of the Code Segment. The initial size of the Code Segment is stored in the headermember "CodeSiz".
ParseText() calls two other functions: ParseDirectives(), which picks out all directives and calls ParseStrings() if need be, which parses the source for stringarguments.
If ParseStrings() encounters a ’[NEEDS’ or ’INCLUDE’ directive, it will call DoNeeds(), which will create enough room for the file to be included and read the actual file. If
DoNeeds() fails, it will set the ”ErrNo” member of the header accordingly and exit.
MakeRoom() just moves the last part of the source to the end of the reallocated space.
Since all variables are adjusted accordingly, ParseText() will pick up parsing after the
’[NEEDS’ or ’INCLUDE’ directive. It never knows the difference.
After parsing ParseText() returns the initial size of the Code Segment (the number of words). The function ParseText() sets three important variables:
V
ARIABLE
C
ONTENT
SourceStrings
SourceWords
The number of source-words
The size of the Code Segment
SourceSymbols The size of the symbol-table
Table 27.1: comp_4th() variables
1
See section 26.5.
2 see section 26.13.
CHAPTER 27. MODIFYING 4TH 514
It is imperative to know these numbers since the 4tH-environment has to size its resources.
No resizing is required until all compilation is done. All allocated resources should be large enough to contain the resulting compilant, so extending these resources should never be necessary. In the end, they are only shrunk to their actual sizes. Now we can start to allocate the compiler-resources:
• Code Segment
• Symboltable
• Controlstack
This is done by simply calling AllocResource(). Note that it is not necessary to allocate the String Segment. Strings remain in memory already allocated to the source and are just shifted to the front.
Then compilation can begin. First, the variable "Cursor" is set to the beginning of the source. Then every call to GetNextWord() sets the variable "CurrentWord" to point to the next source-word. If there are no more source-words, "CurrentWord" is set to NULL and compilation is terminated.
Now compilation really gets off. It is important to know that not all words are created equal. There are five kinds of words:
• Immediate words
• Words
• Constants
• Symbols
• Numbers
So, there are five distinct functions which handle these words:
• Immediate words are compiled by GetImmediate()
• Words are compiled by GetWord()
• Constants are compiled by GetConstant()
• Symbols are compiled by GetSymbol()
• Numbers are compiled by comp_4th()
If the first four functions fail there is one more chance that this is a valid source-word: it might be a number. So, the source-word is converted to a number in the current radix.
If this works, the number is compiled. If it doesn’t, it isn’t a valid source-word and the member "ErrNo" is set.
Compiling is done by a single function called CompileWord(). You just provide the token and its argument and CompileWord() takes care of the rest.
When all compiling is done, we can discard the symboltable and the controlstack. This is done by calling the function FreeResource(). It is called by ReallocSegs(), which shrinks the Code Segment and the String Segment to their actual sizes, and AbortCompile(), which shuts the compiler down in case of an error.
