Stanford Lecture: The Internal Details of TeX82 - Session 10 (July 30, 1982)

Key Concepts

• TFM (TeX Font Metric) Files: Compact files containing font information for TeX and other system programs.
• Byte Files: TeX's internal representation of files as packed 8-bit bytes (0-255).
• Big-Endian Order: A convention where the most significant byte of a multi-byte word comes first.
• Header Data: The initial fixed part of a TFM file, containing metadata like file length and font information.
• Char Info: Data within a TFM file that describes individual characters, including their width, height, depth, and italic correction.
• TFtoPL: A TeX program that converts TFM files into a symbolic, human-readable format (PL files).
• PL (Property List) Files: Symbolic representations of font information, often used for editing and debugging.
• Ligature Kern Programs: Instructions within TFM files that define how sequences of characters should be combined (ligatures) or spaced (kerns).
• Extensible Characters: Characters designed to be built from smaller parts (top, middle, bottom, repeatable) to create larger delimiters.
• Texinfo Parameters: Seven (or more) parameters embedded in TFM files that provide additional information for TeX's typesetting, such as slant, spacing, and math-specific dimensions.
• Fixed Words: A data type used in TFM files for measurements, representing fixed-point numbers with 20 bits to the right of the decimal point.
• Pixel Files: Raster image data generated by Metafont, not directly processed by TeX.

TFM File Structure and Content

This section details the internal structure and the types of information contained within TeX Font Metric (TFM) files, which are crucial for TeX's typesetting process.

TFM File Format

• Byte-Oriented Structure: TFM files are fundamentally sequences of 8-bit bytes, ranging from 0 to 255. While they can be interpreted as 32-bit words (as the number of bytes is always a multiple of 4), TeX primarily treats them as byte streams.
• Big-Endian Convention: The bytes within a word are arranged in big-endian order, meaning the most significant byte appears first.
• Header Section: The TFM file begins with a fixed header of six 16-bit words (half words).
  • File Length: The first two half words indicate the total length of the file in words.
  • Header Length: The next two half words specify the length of the header data itself.
  • Character Range: The final two half words define the beginning character (BC) and ending character (EC) codes for the font. These are 16-bit integers. The range is typically BC-1 <= EC <= 255. An empty font can be represented by BC=1 and EC=0.
• Font Metric Data: Following the header, the file contains various data fields, described in module 457. The order is generally:
  1. Length of the entire file in words.
  2. Length of the header data.
  3. Beginning character (BC).
  4. Ending character (EC).
  5. Character information (Char Info).
  6. Number of widths.
  7. Number of heights.
  8. Number of depths.
  9. Number of italic corrections.
  10. Length of ligature/kern commands.
  11. Number of kerns.
  12. Number of extension parts.
  13. Number of parameters.
• Char Info: This section provides packed information for each character within the BC-EC range. For each character, it contains indirect pointers to its width, height, depth, and italic correction values, which are stored in subsequent tables. Zeros in this section indicate missing characters.
• Widths, Heights, Depths, Italic Corrections: These sections store the actual metric values for characters.
• Ligature Kern Instructions: Defines ligatures and kerning adjustments.
• Extensible Characters: Information for characters that can be built from smaller components.
• Parameters (Texinfo): A section containing parameters used by TeX for typesetting.

Character Information (Char Info) Packing

The Char Info section is highly compressed and packed into 8-bit bytes. Each character's information is represented by a sequence of bytes:

• Width: The first byte represents the width. If the width is 0, the character is considered not present. This is an 8-bit pointer to the width table.
• Height and Depth: The next byte packs height and depth information using 4 bits for each. This limits the font to 16 distinct heights and 16 distinct depths.
• Italic Correction and Tag: The subsequent byte contains the italic correction (6 bits) and a tag (2 bits). The tag indicates special character properties:
  • Tag 0 (Vanilla): Normal character.
  • Tag 1 (Ligature Kern): Indicates a ligature or kern program is present. The remainder field points to the ligature/kern command table.
  • Tag 2 (Character List): Points to the next larger character in a sequence.
  • Tag 3 (Extensible Character): The remainder field points to an extensible recipe.

Extensible Characters

Extensible characters are designed to form larger delimiters. They are composed of four parts:

• Top: A character representing the top part.
• Middle: A character for the middle section.
• Bottom: A character for the bottom part.
• Repeatable Part: A character that can be repeated to fill the space between the top and bottom.

These parts are specified by character codes within the font. If a part is not present, its code is 0. TeX constructs the extensible character by repeating the repeatable part as needed, placing the top and bottom parts at the ends. The width of the repeatable part is used for the entire extensible character, while the widths of other parts are ignored.

Ligature Kern Programs

Ligature kern programs are stored in a table and are invoked when a character has a tag of 1. These programs consist of 4-byte commands, each with:

• Stop Bit: Indicates if the command should terminate processing.
• Op Bit: Differentiates between ligatures (Op=0) and kerns (Op=1).
• Character: The character to look for following the current character.
• Remainder/Operand: For kerns, this is an index into the kern table specifying the amount of kerning. For ligatures, it's a replacement character code.

These programs allow for complex character substitutions (e.g., 'fi' becoming a single ligature character) and fine-tuning of spacing between characters.

Texinfo Parameters

TeX assumes seven parameters for every font, stored in the TFM file. These parameters are used for various typesetting purposes:

• Slant: Used for positioning accents.
• Space, Stretch, Shrink: Control spacing in paragraphs.
• Quad: Represents the EM unit (1M).
• X-Height: Used for dimensions specified in EX units.
• Extra Space: Added after punctuation marks like periods.

These parameters can be modified at runtime by the user, allowing for dynamic adjustments to typesetting. Additional parameters can be added by font designers for specific applications or future TeX extensions.

Units and Measurements

• Design Size: A fundamental parameter in TFM files, representing the intended size of the font. All other measurements (widths, heights, depths, italic corrections, kerns, and most parameters) are expressed as multiples of the design size.
• Fixed Words: Measurements are stored as fixed-point numbers with 20 bits to the right of the decimal point. However, TeX typically uses only 4 bits to the left of the decimal point, limiting widths to 16 times the design size.
• Units: Measurements are given in units of points when the design size is specified in points.

Checksum and Design Size

• Checksum: A number computed by Metafont that changes if important font information is altered. It serves as a compatibility check between the TFM file used to generate a DVI file and the TFM file used for printing. A checksum of 0 means no test is performed.
• Design Size: The intended size of the font as conceived by the designer. It serves as the default size and the basis for all other measurements.

TFtoPL Program and PL File Format

The TFtoPL program converts TFM files into a human-readable Property List (PL) format, facilitating understanding and editing of font data.

TFtoPL Functionality

• Conversion: TFtoPL takes a TFM file as input and outputs a PL file.
• Symbolic Representation: PL files represent font properties using a structured format where each value is preceded by an indicator of its type (e.g., 'O' for octal, 'R' for real, 'D' for decimal, 'C' for character).
• Subset of PL Files: The PL files generated by TFtoPL are a subset of those accepted by the PLtoTF program, which converts PL files back into TFM format.

PL File Content

A PL file generated by TFtoPL includes:

• Header Information: Family and font name, coding scheme, design size, checksum, and a 7-bit safe flag (now obsolete).
• Texinfo Parameters: The seven standard TeX parameters (slant, spacing, etc.) with their values.
• Ligature Table: Lists ligatures and kerns, specifying character replacements and kerning amounts.
• Character Properties: For each character, it lists its width, height, depth, and italic correction if they are non-zero.
• Extensible Character Information: Details about extensible characters, including their top, bottom, and repeatable parts.
• Comments: TFtoPL automatically adds comment lines, and users can add their own.

Real-World Applications and Examples

Font Examples

• cmr10.tfm: Used as an example to demonstrate the binary dump of a TFM file, showing its length (around 250-300 words) and the distribution of character info and width/height data.
• cmathx.tfm: A math extension font used to illustrate extensible characters and "next larger" character sequences, as well as additional math-specific parameters like default rule thickness and big operator spacing.
• cmsy.tfm: A symbols font used to showcase specialized texinfo parameters for math typesetting, such as numerator/denominator positioning, superscript/subscript spacing, and delimiter sizing.

Extensible Character Construction

• Left Parenthesis and Braces: The example of a left brace demonstrates how top, middle, bottom, and repeatable parts are used to construct a large delimiter.
• Vertical Line: An example of a character composed solely of repeatable parts.
• Topless Left Parenthesis: Mentioned as an example of a character with a flat top, constructed using extensible character features.

Ligature and Kern Examples

• 'f' Ligatures: Examples like 'fi', 'ff', and 'ffi' illustrate how sequences of characters are replaced by single ligature characters.
• 'f' Kerns: The example of 'f' followed by an exclamation point shows how kerning can add space to prevent characters from colliding.
• Quote and Hyphen Ligatures: Examples of single quotes becoming double quotes, and hyphens forming en and em dashes.
• Capital Letter Kerns: Demonstrates negative kerning between capital letters like 'F' and 'A' to reduce spacing.

Texinfo Parameter Usage

• Slant for Accents: A slant of 0.25 means an accent moves 0.25 points right for every point of character height increase.
• Fraction Bars: Default rule thickness parameter is used for fraction bars.
• Big Sigma Spacing: Parameters control spacing above and below large operators.
• Delimiter Sizing: DELIM1 and DELIM2 parameters help choose appropriate delimiter sizes for display and text formulas.
• Axis Height: Used for vertical centering of bars in fractions.

Technical Details and Concepts

Byte Order and Word Interpretation

• 8-bit Bytes: The fundamental unit of data in TFM files.
• 0-255 Range: Bytes are interpreted as unsigned integers.
• 32-bit Words: TFM files can be viewed as sequences of 32-bit words, as the total number of bytes is always a multiple of 4.
• Big-Endian: The most significant byte is stored first within a word. This is contrasted with little-endian order.

Data Packing and Compression

• Char Info Packing: Height and depth are packed into a single byte using 4 bits each.
• Italic Correction and Tag: Italic correction uses 6 bits, and the tag uses 2 bits.
• Ligature Kern Command Packing: Commands use bits for stop, op, character, and remainder.

Fixed-Point Representation

• Fixed Words: Used for measurements, with 20 bits to the right of the decimal point and 12 bits to the left.
• Precision Limits: TeX limits the usable bits to the left of the decimal point to 4, effectively restricting widths to 16 times the design size.

Font Design Considerations

• Consistent Delimiter Dimensions: Font designers should ensure that extensible delimiters have consistent widths across different sizes to maintain visual harmony.
• Avoiding Rounding Errors: For critical measurements like rule thicknesses, font designers should ensure exact heights are provided rather than relying on rounded values.

Metafont and TFM Generation

• Metafont: The program used to design fonts and generate TFM files.
• Checksum Calculation: Metafont computes the checksum for TFM files.
• Pixel Files: Metafont can also produce pixel files (raster images), which are separate from TeX's processing.

Conclusion and Key Takeaways

The lecture provides a deep dive into the structure and content of TeX Font Metric (TFM) files, explaining how TeX stores and interprets font information. Key takeaways include:

• TFM files are meticulously structured binary files designed for compactness and efficiency, using byte-oriented data with specific packing schemes for various metrics.
• The header provides essential metadata, including file length, character ranges, and font identification.
• Character information is highly compressed, with clever packing of width, height, depth, and italic corrections.
• Ligature kern programs and extensible characters offer sophisticated mechanisms for character substitution, spacing adjustments, and building large delimiters.
• Texinfo parameters are crucial for TeX's math typesetting, providing context-specific dimensions and positioning information that can be controlled at runtime.
• The design size is a fundamental scaling factor, and measurements are expressed as multiples of it, allowing for flexible font scaling.
• TFtoPL and PLtoTF are essential tools for understanding, editing, and converting between the binary TFM format and human-readable PL files.
• Font designers have considerable flexibility in defining character properties, extensible characters, and additional parameters to enhance TeX's typesetting capabilities.
• The TFM format is designed to be extensible, allowing for future additions and specialized uses without altering the core structure.