Stanford Lecture: The Internal Details of TeX82 - Session 6 (July 29, 1982)

Key Concepts

• Semantics: The meaning and interpretation of code, distinct from syntax (structure).
• Main Control: The central driver program in Tech, acting as the "chief executive."
• Reswitch: A label within Main Control where processing can loop back to re-evaluate tokens.
• Command Codes (cur_command): Numerical identifiers for various actions Tech can perform.
• Action Procedures: Subroutines called by Main Control to perform specific semantic tasks.
• Primitives: Fundamental operations in Tech, often implemented as primitive procedures.
• Hash Table: A data structure used to store and retrieve information about primitives.
• Save Stack: A stack used for saving and restoring program state, including fixups and boundary words.
• Nest Stack: A stack used to save semantic state variables, such as the current mode, list head, and tail.
• Modes (H mode, V mode, M mode): Different operational states within Tech that dictate how lists are built and processed.
• Whatsit Nodes: Special data structures used for extensions to Tech.
• Extensions: New features added to Tech, often requiring hooks into the main program.
• Confusion: An error handling procedure in Tech used for internal consistency checks.
• H Box: A fundamental data structure in Tech representing a horizontal box.
• Scan Spec: A subroutine that scans for specifications like dimensions or keywords.
• Begin Box / End Box: Procedures that initiate and finalize the construction of a box.
• Packaging Subroutines (H pack, V pack): Procedures that convert lists into boxes.
• Hyphenation: The process of breaking words at the end of lines, a topic for a future lecture.

Semantics in Tech

This lecture delves into the semantics of the Tech typesetting system, focusing on how it interprets and acts upon the parsed syntax. Semantics is described as the "big region in the middle" of Tech's architecture, interacting with local variables, equivalence tables, and receiving tokens from syntax routines.

Math Semantics

While not the primary focus of this hour, math semantics is highlighted as a significant component, accounting for approximately 40% of the semantic code. It involves specialized data structures called "NOADs" for mathematical entities like fractions, square roots, and binary relations. A key routine, M list to H list, converts math-type lists (M list) into horizontal-type lists (H list) that the rest of Tech can process.

Main Control: The Chief Executive

The core of Tech's semantic processing lies in its "main control" procedure, also referred to as the "chief executive" or "hub of the web." This program orchestrates the execution of various subroutines.

• Structure: Main control is characterized by a large case statement that branches based on the cur_command (current command code) combined with the current mode (e.g., V mode + skip, H mode + H skip).
• Inner Loop Optimization: To ensure speed, especially during text processing, the main control procedure is highly optimized. Common operations like H mode + letter and H mode + other char are handled directly within main control to avoid procedure call overhead.
• reswitch Label: This label allows for re-evaluation of tokens without fetching a new one, useful for handling variations or disguises of commands. It also serves as a point for diagnostic tracing.
• Action Procedures: For most command/mode combinations, main control calls specific "action procedures" to perform the actual semantic task. This modularity helps manage procedure size limitations imposed by some Pascal compilers.
• Command Codes: Section 201 and subsequent sections list the command codes. Examples include charnum, math charnum, x-ray (internal code for show instructions), make box, and H move.
• Prefix Commands: Codes above max_nonprefixed_command (currently 69) are often prefix commands. These can be handled more efficiently by a single prefix_command procedure, potentially reducing the size of the main control's case statement.

Understanding Primitives and Command Codes

The lecture explains how to trace the execution of a primitive command like show box.

• Index Lookup: The index of the Tech code lists primitives alphabetically. show box primitive is found on page 430.
• Initech Program: Primitives are loaded into the hash table by the Initech program.
• primitive Procedure: A special procedure used to enter all primitives. show box is called via primitive in module 1159.
• Command and Character Codes: show box is associated with the command code x-ray and a character code show_box_code (defined as 1).
• print_command_and_char Procedure: This procedure is used for tracing, translating internal command codes (like x-ray) into symbolic names (like "show box") by looking at the cur_char code. It's designed to be robust and avoid error messages, even for unexpected inputs.

Example: show box Semantics

The process of show box is traced as a typical example of simple semantics:

1. x-ray Command: show box is recognized as cur_command = x-ray.
2. Main Control Branch: The any_mode x-ray case in main control calls the show_whatever action procedure.
3. show_whatever Procedure: This procedure checks cur_char. If it's show_box_code, it calls module 1164, show_the_current_contents_of_a_box.
4. show_the_current_contents_of_a_box: This involves scanning an 8-bit integer (for box index), printing a newline and ">", and then displaying the box's contents using show_box_depth and show_box_breadth parameters.
5. Global Variables (curval): The use of global variables like curval is discussed, highlighting the danger of them being overwritten by other procedures.

Extending Tech

Extensions to Tech are typically handled through a dedicated mechanism to minimize changes to the core program.

• Module 1209: Discusses extensions to Tech.
• Hooks: Extensions often rely on "hooks" provided in the main program.
• Extension Command Code: When possible, new features should use the Extension command code with a specific char field to identify the extension type.
• Action Procedures for Extensions: Module 1216 (any_mode_extension) and 1217 (do_extension) handle extensions.
• confusion Procedure: Used for internal consistency checks. It has a spotless global Boolean variable. If spotless is true, it prints "this can't happen"; otherwise, it prints a different message.
• Whatsit Nodes: New types of data structures for extensions. Module 1225 shows how whatsit nodes can have subtypes for different extensions (e.g., open, send, close, extend).
• Extension Lifecycle: Adding a new whatsit type requires implementing procedures for creating, printing, and freeing those nodes. Modules 1225, 1226, and 1227 cover these aspects.
• Recommendation for Vanilla Tech: Users are strongly advised to maintain a "vanilla" (unextended) version of Tech to ensure program compatibility across different installations.

Example: Processing an H Box

The lecture walks through the detailed processing of an H box command, illustrating the interplay of various components:

1. H box Specification: Tech needs to look ahead to determine if a size is specified (e.g., "to 100 points" - exact size, or "expand by 100 points" - relative size).
2. Context: H box can appear in various contexts (e.g., preceded by lower, ship out, leaders, set box). This context is stored on the Save Stack.
3. Save Stack Usage: The Save Stack stores information about the context of the box being built, including group types (e.g., H box group).
4. Make Box Command: The command code for H box is Make Box.
5. any_mode_Make_Box (Module 951): This procedure saves the context (e.g., a shift amount of 0) onto the Save Stack and calls Begin_box.
6. Begin_box (Module 958): This procedure initiates the construction of the box. It determines the type of box (H box, V box, V top) based on the cur_char code.
7. H box Primitive: The primitive for H box has the command code & H box and a char_code of V_top_code + H_mode.
8. Restricted Horizontal Mode: For H box, Tech enters restricted horizontal mode (minus H mode).
9. Scan_Spec Subroutine: This scans for the size specification (e.g., "100 points") and pushes the type of specification (0 for exact, 1 for expand) and the dimension onto the Save Stack.
10. New_Save_Level H_box_group: A boundary word is pushed onto the Save Stack to mark the beginning of an H box group.
11. Push_Nest: Semantic state variables, including the current mode, list head, tail, and space factor, are saved onto the Nest Stack. The mode is changed to minus k (e.g., minus H mode), and an empty list is initialized. The space factor is set to 1000 (normal).
12. Processing Character 'A': When the character 'A' is encountered in H mode, it's processed by Main_loop (Module 907).
13. Main_loop: Appends the character to the current H list. This involves creating a char_node and updating the tail pointer. Font information and character codes are stored.
14. Handling Spaces: If a space follows, Tech looks up the glue specification for the current font and creates a glue_node.
15. Handle_Right_Brace (Module 947): When a right brace is encountered, this procedure is called. It checks the current group type.
16. H_box_group: If the group is an H box group, it calls package_of_0.
17. package_of_0 (Module 965): This procedure unwinds the Save Stack (Unsave) to remove fixups and declarations. It then retrieves the context information (e.g., size specification) from the Save Stack.
18. H_pack: This packaging subroutine takes the constructed list and converts it into an H box node. It calculates the box's dimensions based on font metrics and glue specifications.
19. Pop_Nest: The semantic state is restored from the Nest Stack, and the previous list head is recycled.
20. Box_end Procedure: The newly created box is added to the current list in the outer mode, applying any context-specific actions (e.g., lowering, raising).

This detailed walkthrough of H box processing demonstrates how Tech manages complex structures by saving and restoring state on the Save and Nest stacks, utilizing packaging subroutines, and handling various contexts and specifications. The lecture concludes by mentioning upcoming topics like breaking paragraphs into lines and hyphenation.