The V0.3.0 goal of Keleusma is a Keleusma compiler written in Keleusma source, compiled to Keleusma bytecode, running on the Keleusma virtual machine, producing Keleusma bytecode as output. The endpoint is a fixed point. The self-hosted compiler compiled by the Rust-hosted compiler produces bytecode identical to what the Rust-hosted compiler produces from the same source. The self-hosted compiler compiled by itself reproduces its own bytecode. This article describes the strategy that will reach that fixed point.
The strategy has two parts. The first is an incremental migration method that ports the existing Rust-hosted compiler to Keleusma one stage at a time without a big-bang rewrite and without giving up a working reference at any point. The method is not Keleusma-specific. It applies to any staged pipeline whose implementation language is being changed. The second is a three-stage stream-processor pipeline architecture that matches Per Brinch Hansen’s pipeline-of-processes model and matches Keleusma’s coroutine semantics directly.
The strategy was informed by the stream-based compilers series, which covers the historical demonstrations and the mathematical foundation that the Keleusma design draws on, and by the self-hosted silicon compiler article, which develops the software-to-silicon boundary that a self-hosted software compiler sits above. The article on developments in programming language theory as a historical arc places the self-hosting concept in the broader seventy-year arc.
Self-hosting a language is the most credible demonstration that the language is expressive enough to write its own toolchain. The signal is twofold. First, it validates the surface language and the type system against a concrete, complex program of substantial size. Second, it removes a dependency. A self-hosted Keleusma can evolve without forcing every change through the Rust-hosted compiler maintainers. Teams that value a short auditable toolchain dependency graph benefit in particular.
For Keleusma specifically, the self-hosted compiler is a precondition for the V0.4.0 native code generation goal, which compiles the self-hosted compiler to native code via LLVM and links it as a static library against a Rust host. Without the V0.3.0 self-hosted compiler, V0.4.0 has nothing to compile to native code.
The self-hosting concept was treated at length for the software case in the streaming compilers series conclusion, which discusses the coalgebraic fixed-point condition that a self-hosted compiler satisfies. It was treated for the silicon case in the recent article on self-hosted silicon compilation. Keleusma sits on the software side of that boundary and offers a candidate example of a compact-toolchain language design that a self-hosting compiler could reasonably compile itself with.
The migration method is a backward variant of what Martin Fowler calls the strangler pattern, specialized for compilers by porting the emit boundary first, because that is where self-hosting efforts most often stall. The method applies when four preconditions hold.
First, a working reference implementation exists. The existing compiler keeps running throughout and serves as the equivalence oracle that every intermediate state is checked against. Without a reference there is nothing to validate against, and the method does not apply.
Second, the pipeline has clean stage boundaries. The compiler is a sequence of stages, for example tokenize, parse, analyze, generate, and emit, each consuming the previous stage’s output. Those boundaries are where the seam moves.
Third, the boundaries can be bridged. The migrating engineer can convert one stage’s output in the host language into the input the next stage expects in the target language. This is what the adapters do.
Fourth, the backend is the real risk. Producing a valid target artifact at the emit boundary is the hardest and least certain part. This is what justifies going backward.
Port the stages in reverse pipeline order. The last stage, the one that emits the target artifact, is ported first. The first stage, usually the lexer, is ported last.
The reason is risk, not convenience. Frontends are well understood and low risk. The wall is the backend. Self-hosting efforts commonly reach a state where the new compiler can read its own source but cannot yet emit a valid artifact, and they stall there. Porting the emit boundary first retires that risk before the engineer invests in the easy stages.
At any moment exactly one adapter sits at the frontier between the still-host-language upstream and the already-target-language downstream. The upstream stages run unchanged and produce their output as before. The adapter converts that output into the input the first target-language stage expects. The downstream stages, already ported, chain directly to one another.
Each time the engineer ports the next stage backward, that stage takes over the adapter’s job and the adapter disappears, and a new adapter appears one position further upstream. The seam moves through the pipeline from back to front, and there is never more than one adapter to maintain.
The seam
is easiest to see
as a picture.
The pipeline runs left to right,
H marks a stage
still in the host language,
T marks a ported target-language stage,
and the bar |
is the single adapter
at the frontier.
start H H H H H all host, no adapter yet
step 1 H H H H | T emit stage ported, adapter in front of it
step 2 H H H | T T
step 3 H H | T T T
step 4 H | T T T T
done T T T T T fully self-hosted, adapter gone
The two endpoints correspond to the two whole-compiler tombstone diagrams of a bootstrap, the compiler written in the host language at the start and the compiler written in the target language at completion, and the seam is the path between them.
The adapter’s output and its implementation are separate concerns. The output is the data contract the two adjacent stages agree on, and it is permanent. The implementation, the conversion from the host representation to the target one, is disposable. It is a prototype of the output behavior that the not-yet-written upstream stage will eventually produce for real, and writing that upstream stage is precisely what retires the adapter.
This framing sets the right quality bar. An adapter needs to be good enough to let the engineer build and validate the stage below it, not perfect, not general, and not efficient. When an adapter is getting expensive to fix, that is often the signal to stop and write the real upstream stage that will replace it. The goal is stages, not adapters.
Correctness during migration is a judgment call, and the ledger is what keeps it honest. For each corpus program, either the current chain processes it correctly, or its deviation is recorded in a ledger entry that names the specific upstream stage that will correct it. An entry records three things, the corpus program, the observed deviation from the reference, and the responsible upstream stage.
When that upstream stage lands, the deferred cases are re-run and each is confirmed resolved. A deviation that its responsible stage does not fix was never an adapter limitation. It is a real bug in the stage that already exists, and the ledger is what surfaces it. Without the ledger and the recheck, the judgment call becomes a place for stage bugs to hide until the end.
Two engineering modes apply to the process. During migration the engineer defers. The engineer prefers building the next stage to perfecting a throwaway adapter, and records every deferral in the ledger. Once all stages are ported and the adapters are gone, the engineer switches from deferred work to bug fixing. The engineer drives the ledger to empty, and the fully self-hosted pipeline must process the corpus correctly with no adapter left to defer to.
An empty ledger and a passing corpus under the fully self-hosted pipeline is the completion gate for the migration. Every intermediate deferral is a debt against it.
A further gate belongs at the end and is specific to self-hosting rather than to migration in general. The engineer compiles the compiler’s own source with the reference and with the self-hosted compiler and confirms the two artifacts agree, then compiles the compiler with itself and confirms the output is stable across successive generations. This is the staged-bootstrap fixed-point check, the same reproducibility comparison that a multi-stage bootstrap makes between its later stages. The compiler’s own source is the most demanding program it will process, and the fixed point is where a subtle miscompilation that the corpus missed will surface.
The V0.3.0 compiler
is decomposed
into three coordinated stages,
each a Keleusma loop function.
source bytes
│
▼
lexer ── yield tokens ──▶ parser ── yield declarations ──▶ codegen ── yield bytecode
The lexer consumes source bytes and yields tokens. The parser consumes tokens and yields parsed declarations, where the unit of work is a single top-level declaration rather than the whole program. The codegen stage consumes parsed declarations and yields bytecode chunks plus the auxiliary body that the wire format expects.
The decomposition matches Brinch Hansen’s pipeline-of-processes model that the stream-based compilers series covers in its Brinch Hansen article, and it matches Keleusma’s coroutine semantics directly. Each stage’s working memory is bounded by its local state plus the inter-stage buffer, both of which fit inside the per-Stream-to-Reset arena budget. The whole-program abstract syntax tree is never constructed. The parser yields each declaration as it completes. The codegen stage emits bytecode immediately and forgets the declaration. Symbol tables are per-scope and popped on scope exit.
The host application that drives the pipeline is a Rust program during the V0.3.0 line and a Keleusma program once V0.4.0 lands. The host resumes each stage as needed and handles the inter-stage flow control. The host’s responsibilities are minimal. It collects tokens emitted by the lexer, hands them to the parser, collects declarations emitted by the parser, hands them to the codegen stage, collects bytecode emitted by the codegen stage, and assembles the wire-format buffer.
Three reasons recommend this shape.
First,
it composes cleanly
with Keleusma’s existing model.
Each stage
is a loop function,
which Keleusma already admits.
The yield-and-resume protocol
is
the inter-stage communication channel.
No new language primitives
are required.
The bounded worst-case-memory-usage guarantee
falls out
for each stage independently.
Second, it matches the demonstrated prior-art model. Brinch Hansen’s compilers were written exactly this way and worked. The pattern is not speculative.
Third, it provides natural test points. Each stage can be tested in isolation by driving it with a synthesized input stream and inspecting the output stream. The Rust-hosted compiler already has a per-stage test surface that the self-hosted version can reuse with minor adaptation.
The Wirth tradition produced compilers that did not decompose into stages at all. The Turbo Pascal compiler, the Oberon compiler, and the various Modula-2 compilers ran the entire compile pipeline as a single recursive-descent parser that emitted bytecode or machine code directly during parsing. There was no token stream materialized between the lexer and the parser. The lexer was just a method on the parser that returned the next token on demand. There was no abstract syntax tree. Each syntactic construct emitted its corresponding bytecode at the point in parsing where the construct was recognized.
This is the integrated single-pass alternative that the stream-based compilers series covers across its Wirth-line and Turbo Pascal articles. Its appeal is speed. The Turbo Pascal benchmark of ten thousand to thirty thousand lines per second on a four point seven seven megahertz eight-oh-eight-eight in nineteen eighty-four, and the Oberon compiler at millions of lines per second on modern hardware, are the gold standards. The architecture has no inter-stage buffering and no stage-coordination overhead.
The V0.3.0 strategy
documents this alternative
but does not recommend it.
The reason
is that
Keleusma’s coroutine model
rewards
the decomposed pipeline shape.
Each loop function
is a natural stage,
and the bounded-worst-case-memory-usage guarantee
falls out
per stage.
An integrated single-pass compiler
in Keleusma
would either be
a single very-long loop function,
which the verifier might admit
but which is awkward to test in isolation,
or a function-call chain
that cannot use recursion,
which forces
the explicit-stack discipline
anyway.
Neither shape
is obviously better
than the pipeline.
Brinch Hansen’s pipeline-of-processes
maps directly
to Keleusma’s coroutine pipeline.
The integrated single-pass
maps to Keleusma
awkwardly.
If V0.3.0 implementation
surfaces a real reason
to prefer the integrated form,
the design is on the shelf
and the migration is straightforward.
Collapse the three loop functions
into one,
drop the inter-stage yield boundaries,
and inline the staging.
Three phases apply at the point when all three pipeline stages exist in Keleusma source. The pattern is canonical across Wirth’s Project Oberon, LLVM, Rust, and Go.
Phase A cross-compiles.
The self-hosted compiler
is written in Keleusma source
under
compiler/lexer.kel,
compiler/parser.kel,
and compiler/codegen.kel,
plus shared abstract-syntax-tree
and bytecode-encoding helpers.
The existing Rust-hosted compiler
produces
the bytecode.
The output
is a Keleusma bytecode artifact,
which the strategy calls
kelc.0.kel.bin,
that runs on the virtual machine
and accepts Keleusma source
as input.
Phase B self-compiles.
The engineer loads
kelc.0.kel.bin
into a virtual machine instance
and invokes it
against its own source files
as its input.
The output is
kelc.1.kel.bin.
If
kelc.0
is correct,
kelc.1
is byte-identical
to
kelc.0
modulo non-essential ordering
including map iteration order.
Any divergence
is a bug
in
kelc.0.
Phase C reaches the fixed point.
The engineer loads
kelc.1.kel.bin
into a virtual machine instance
and invokes it
against the same source files.
The output is
kelc.2.kel.bin.
kelc.2
must be byte-identical to
kelc.1.
The fixed point
is reached.
Validation runs
alongside Phases B and C.
Every test
in the existing regression corpus
is recompiled
under both
the Rust-hosted compiler
and kelc.1.
The bytecode outputs
must be byte-identical
modulo the same non-essential ordering.
Divergence on the corpus
is a bug
in the self-hosted compiler.
The bootstrap procedure is mechanical. It does not require additional design work. The risk in the bootstrap is not the procedure but the surface-language gap, namely that the self-hosted compiler may need features the V0.2 surface does not yet provide ergonomically. The strategy addresses this in the required-surface-features section.
Three surface-language tensions surfaced immediately when the strategy considered writing the compiler in Keleusma.
The first tension
is recursion.
The self-hosted compiler
will want to walk
recursive data structures.
Parsed declarations
contain expressions
that contain sub-expressions.
Types contain sub-types.
Keleusma forbids recursion
in fn and yield categories.
Only top-level loop
admits cyclic execution
through productive yield.
The classical resolution is to walk recursive data using explicit stacks rather than recursive function calls. Brinch Hansen’s compilers used this technique. The Wirth tradition handled the same constraint with recursive-descent parsers that exploited the fact that the recursion depth was bounded by the language’s nesting depth, not the input size. Keleusma’s recursion prohibition is stricter and requires explicit stacks. The strategy adopts the explicit-stack pattern, which requires no language surface change against V0.2.
The second tension is Hindley-Milner type inference. The Rust-hosted compiler runs Robinson unification over a constraint graph that spans an entire function. This is a multi-pass procedure within a single function. A pure single-pass compiler in the Wirth tradition does not perform this kind of inference. The surface language typically requires explicit type annotations.
The realistic V0.3.0 answer is per-function-body inference. The constraint worklist lives in the compiler-loop’s persistent data block, bounded by explicit declared limits. Analysis of the Rust-hosted compiler’s actual constraint-graph sizes established that a maximum of one thousand twenty-four type variables per function, four thousand ninety-six constraints per function, and sixteen thousand three hundred eighty-four function-body nodes covers the substantial majority of practical programs. The compiler-in-Keleusma is itself written with explicit type annotations so that the self-compilation step exercises the easy inference path.
The third tension is generics and monomorphization. Monomorphization requires the compiler to see every call site of a generic function before it can know which specializations to emit. This is fundamentally a whole-program operation. The realistic V0.3.0 answer is to keep a small specialization table in the compiler’s persistent state, across the entire compilation rather than per declaration, and emit specialized chunks lazily as new call sites are discovered. The specialization table grows with the number of distinct specializations, not with the program size. In practice this is a small bound.
A dedicated research pass addressed each of these tensions in technical detail and produced specific recommendations.
The recursion question was resolved with the work-stack pattern. Three worked examples covered the recursion shapes the compiler needs. Pre-order traversal for free-variable collection. Accumulating fold for Robinson unification. Post-order traversal for the bytecode emitter. The pattern requires no language-surface change against V0.2.
The Hindley-Milner inference question was resolved with per-function-body inference bounded by explicit declared limits. Per-function transient memory approximately one hundred thirty kibibytes, plus persistent approximately two hundred fifty kibibytes.
The symbol-table substrate question
was resolved
with three data structures.
A string interner
producing Word indices.
A sorted-array
WordMap<V>
for bulk tables including
the function table,
the type registry,
the specialization table,
and the use table.
And a linear
LocalScope
for per-scope locals.
No new language features required.
The design implements directly
on the V0.2 surface.
The byte-iteration question
was resolved
with a host-side strategy.
The host passes source
as [Byte; N].
The lexer uses array indexing.
Three host-registered natives
handle the residual Text work.
compiler::intern_bytes
returns a Word interner index.
compiler::text_from_bytes
constructs a Text
from a byte range
for diagnostic messages.
compiler::text_concat
builds composite messages.
No surface-language extension required.
The self-validation question was resolved with three-layered validation integrating into the existing test harness. Layer one is byte-identical comparison after canonicalization of the native-name table, constant pool, specialization chunks, and function-name-to-chunk-index map. A SHA-256 hash over the canonical form provides a fast pass-or-fail signal. Layer two is logical equality with diagnostic when layer one fails. It pretty-prints the bytecode and runs a structural diff against the Rust-hosted output. Layer three is behavioral equivalence over the regression corpus. Every program’s runtime output matches between Rust-hosted and self-hosted compilation.
The module-scale compilation question
was resolved
with Modula-2-style separate compilation.
Implementation files carry
the .kel extension.
Interface files carry
the .def.kel extension.
Both files
are Keleusma source.
Per-module specialization tables
reset at module boundaries.
Cross-module specializations
land on the consumer side
bounded by consumer complexity.
Three questions remain unresolved after the research pass. None are strategy blockers. Each becomes a V0.3.x or implementation-time concern.
The first is the cross-module monomorphization mechanism. The separate-compilation shape is settled. The strategy does not specify how generic functions specialize across module boundaries when modules are separately compiled. The shape of the per-module specialization table and the cross-module instantiation protocol is open.
The second is the diagnostic quality regression bound. How much diagnostic quality is acceptable to lose in V0.3.0 in exchange for the single-pass streaming architecture? The strategy’s “as good as the Rust-hosted, where possible” target is qualitative. Single-pass compilers historically have brittle error recovery, per the prior-art survey in the stream-based compilers series. Some regression is expected.
The third is a V0.2 surface adequacy audit. The universal-expressibility argument for the work-stack pattern is sound in prose but unverified in code. Before V0.3.0 implementation begins, a sample exercise compiling Robinson unification, a recursive abstract-syntax-tree walker, and a monomorphization pass against the V0.2 surface should be conducted and measured. If a load-bearing affordance is missing, the work-stack pattern needs supplementation.
The strategy draws on several traditions in language implementation.
The moving-seam incremental rewrite is Fowler’s strangler pattern, applied to compiler porting by running the seam backward so the emit boundary retires first.
Self-hosting and bootstrapping are an old tradition in language implementation. The reproducibility comparison used as the completion gate is the one a multi-stage bootstrap performs between its stages.
The pipeline-of-processes compiler architecture comes from Per Brinch Hansen. His book Brinch Hansen on Pascal Compilers published by Prentice-Hall in nineteen eighty-five is the canonical reference. His SuperPascal compiler was itself written in this style and demonstrated the pattern of stream-processor compiler in a stream-processor language. This is the architectural precedent closest to Keleusma’s intended design.
The single-pass tradition comes from Niklaus Wirth. The Pascal compiler of nineteen seventy, Modula-2 of the late nineteen seventies, and Oberon of the late nineteen eighties were each designed to be single-pass compilable, with declare-before-use rules and explicit forward declarations for mutual recursion. The Oberon compiler is approximately four thousand lines of Oberon and is published in full source in Project Oberon, by Wirth and Gutknecht, Addison-Wesley nineteen ninety-two, revised edition two thousand thirteen. Wirth’s Compiler Construction, Addison-Wesley nineteen ninety-six, is the canonical pedagogy. Wirth’s tradition demonstrates that single-pass discipline survives language evolution.
The commercial demonstration comes from Turbo Pascal one point zero through three point zero, Anders Hejlsberg’s compiler of the nineteen eighty-three through nineteen eighty-six period, written in eight-oh-eight-six assembly. Turbo Pascal compiled to memory and ran the code from memory with no traditional link step for the default in-memory build. The internals were never released as open source. Turbo Pascal is the commercial proof that single-pass compilation produces compilers fast enough to change developer workflow.
A separate line of work pursues bootstrapping with machine-checked correctness rather than migration mechanics. The CakeML verified bootstrapped compiler is the sharpest instance of that program.
Behind all of it sits Ken Thompson’s Trusting Trust argument for why the provenance of the seed matters. Thompson’s Reflections on Trusting Trust lecture, delivered as the nineteen eighty-three ACM Turing Award lecture and published in Communications of the ACM in August nineteen eighty-four, is the founding statement of the seed-provenance concern that David A. Wheeler’s Diverse Double-Compiling countermeasure of two thousand nine addresses. The recent article on self-hosted silicon compilation develops both of these in a hardware context that the software self-hosting strategy sits alongside.
The C-family tradition, namely GCC, Clang, and the Portable C Compiler line, is explicitly not relevant prior art for this strategy. Those compilers are multi-pass and abstract-syntax-tree-based. They optimize for the opposite trade-off, namely heavy optimization at the cost of compilation speed and memory. The lcc compiler, described in Fraser and Hanson’s A Retargetable C Compiler, Addison-Wesley nineteen ninety-five, is a useful counter-example to study for what multi-pass design looks like at small scale, but it is not the model V0.3.0 follows.
The brief-lang project is a contemporary language that attempted self-hosting and reached, then stalled at, the frontier V0.3.0 approaches. The strategy’s engineering choices were informed by a targeted review of that project. The observations reflect that project at a point in time and are cited for their engineering value, not as endorsement.
The frontend is the achievable part. Codegen and host output are the wall. Brief has a working compiler frontend written in Brief, including a lexer, a parser, a type checker, and a contract engine, but is not bootstrapped because the frontend runs inside a host interpreter and its backends are unfinished. The V0.3.0 work sequences codegen and the emit-to-host boundary as the high-risk stages to be de-risked first, not the lexer and parser. This reinforces the backward-migration ordering.
The output capability must be a first-class host native from the start. Brief’s self-hosted compiler could read source but had no general facility to write its output, which is a hard blocker to a true bootstrap. Keleusma’s host-native surface must expose a deliberate, bounded emit-compiled-bytecode capability to the self-hosted compiler as a designed feature, not an afterthought.
Divergent execution models
are a bootstrap hazard.
Brief maintained
a tree-walking interpreter
and a compiled backend
that drifted into
different runtime semantics,
silently miscompiling programs.
Keleusma’s bootstrap fixed point,
kelc.0 to kelc.1 to kelc.2,
is the guard against this.
The fixed point
converges
only if the self-hosted compiler’s output
is stable across stages,
so the byte-identical fixed-point check
in the bootstrap procedure
is load-bearing,
not ceremonial.
The work-stack idiom is independently validated. Brief’s compiler used explicit work-stacks and threaded state, for example iterative depth-first call-graph traversal, even though its language permits recursion. That a real compiler was written this way is external evidence that Keleusma’s recursion-free work-stack pipeline can express the compiler it needs to.
Only admit surface syntax that will actually be compiled. Brief accumulated parsed-but-uncodegened constructs, each becoming a maintenance stub that generated defects. The strategy grows the self-hosted compiler feature-complete per increment. Parse, type-check, and generate for a construct together, or not at all.
Consume every analysis result on every path. Brief repeatedly computed an analysis including liveness and convergence and then failed to consume it in one of several codegen paths, losing the benefit and creating inconsistency. A staged self-hosted pipeline must ensure each stage consumes the descriptors the prior stage produced, everywhere they apply.
V0.3.0 is complete when nine conditions hold.
First,
the compiler pipeline exists
in Keleusma source
at
compiler/lexer.kel,
compiler/parser.kel,
and compiler/codegen.kel,
plus shared abstract-syntax-tree
and bytecode-encoding helpers.
Second, the first migration step intermediate validation passes. Every program in the regression corpus compiles to byte-identical bytecode under the Keleusma-lexer-plus-Rust-parser-plus-Rust-codegen configuration as under the all-Rust baseline.
Third, the second migration step intermediate validation passes. Every program in the regression corpus compiles to byte-identical bytecode under the Keleusma-lexer-plus-Keleusma-parser-plus-Rust-codegen configuration as under the all-Rust baseline.
Fourth,
the Rust-hosted compiler
produces
kelc.0.kel.bin
from the full Keleusma source
without error.
The existing test suite
continues to pass.
Fifth,
the Phase B fixed point holds.
kelc.0.kel.bin
recompiles its own source
to produce
kelc.1.kel.bin.
kelc.1
is byte-identical to
kelc.0
modulo non-essential ordering,
formally documented.
Sixth,
the Phase C fixed point holds.
kelc.1.kel.bin
recompiles the same source
to produce
kelc.2.kel.bin,
byte-identical to
kelc.1.
Seventh,
regression corpus equivalence holds.
Every script in
examples/scripts/
and the workspace tests
compiles to byte-identical bytecode
under both
the Rust-hosted compiler
and
kelc.1.
Eighth,
the command-line frontend
gains a
--self-hosted
flag
that routes through
kelc.1
instead of the Rust-hosted compile path.
Programs compile and run
end to end.
Ninth, documentation acknowledges the self-hosted compiler as an alternative path. The Rust-hosted compiler continues to ship. V0.3.0 does not retire it.
The dual-compiler period is intentional. The Rust-hosted compiler remains the reference implementation. The self-hosted compiler is the validation that the language admits its own toolchain.
The V0.3.0 strategy combines a general-purpose migration method with a Keleusma-specific pipeline architecture. The migration method is a backward incremental rewrite that ports the emit boundary first, maintains a single moving adapter at the frontier, tracks intermediate deviations in a deferral ledger, and drives the ledger to empty at the completion gate. The architecture is a three-stage stream-processor pipeline matching Brinch Hansen’s pipeline-of-processes model and Keleusma’s coroutine semantics. Both parts draw on established compiler-implementation traditions that the stream-based compilers series covers in depth.
The endpoint is a fixed point. The self-hosted compiler compiled by itself reproduces its own bytecode. Reaching that fixed point is the V0.3.0 goal. The V0.4.0 native-code-generation goal depends on it. The V0.5 and beyond Keleusma-in-Keleusma-runtime aspiration depends on V0.4.0. Each step retires a specific dependency that the current shape carries.
The strategy is documented in full in the Keleusma repository as an incremental self-hosting reference document that states the migration method independent of Keleusma and a V0.3.0 self-hosting strategy document that applies the method to the specific case. The subproject scaffold described in the recent V0.2.2 getting-started article is where the strategy is being realized.
The eleven preceding articles of this series argued that compilation can be organised as a stream-processor operation. The historical trio in articles A189 through A191 identified the three foundational demonstrations of the discipline in Niklaus Wirth’s languages, Anders Hejlsberg’s Turbo Pascal, and Per Brinch Hansen’s pipeline architecture. The theory pair in articles A192 and A193 developed the mathematical foundation in block-structured control flow and coalgebraic productivity. The techniques trio in articles A194 through A196 covered the three foundational engineering techniques of fixup tables, declare-before-use, and scoped symbol tables. The synthesis pair in articles A197 and A198 compared the integrated and decomposed shapes and identified where the discipline breaks down.
This closing article draws the analogy in both directions. A compiler that reads its source in a stream is a stream processor. A stream processor that compiles its own source is a self-hosted stream processor. The two claims are duals of one another. The article closes with Keleusma’s compilation pipeline as the modern worked example, where the compiler is a stream processor in both directions.
The first direction of the analogy takes the compiler as its subject. A compiler $C$ maps source programs to target programs
\[C \colon \text{source} \to \text{target}.\]The stream-processor form of the compiler reads the source as a stream of tokens or bytes and produces the target as a stream of instructions or bytecode in a forward sweep
\[C_{\text{stream}} \colon A^{\omega} \to B^{\omega},\]where $A$ is the source alphabet and $B$ is the target instruction alphabet. Article A193 formalised the productivity condition that $C_{\text{stream}}$ must satisfy. A productive stream-processor compiler delivers some finite prefix of the target after consuming some finite prefix of the source, with the productivity function $m$ specifying how much output each input prefix determines.
The compiler-as-stream-processor framing is the perspective that articles A189 through A196 developed in detail. The Wirth line demonstrated the integrated form of this framing. Brinch Hansen demonstrated the decomposed form. The theory pair identified the syntactic and semantic conditions under which the framing holds. The techniques trio identified the engineering mechanisms that implement the framing in practice.
The second direction of the analogy takes the stream processor as its subject. A stream processor $P$ is a program that consumes one or more input streams and produces one or more output streams under the productivity discipline. The stream processor does not inherently compile anything. It processes values.
The question that this direction poses is what does it mean for a stream processor to compile its own source?
The answer is that a stream processor whose source is itself a stream-processor program in the same language can be compiled by another stream processor of the same shape. When the compiling stream processor and the compiled stream processor are the same program running on the same virtual machine, the arrangement is self-hosting.
The self-hosted stream-processor compiler $P_{\text{self}}$ satisfies
\[P_{\text{self}}(\text{source}(P_{\text{self}})) = P_{\text{self}},\]where $\text{source}(P_{\text{self}})$ denotes the source form of $P_{\text{self}}$ and $P_{\text{self}}$ denotes the compiled form. This is the fixed-point condition that self-hosting imposes. A compiler that compiles itself and produces its own compiled form as the result has reached the fixed point.
The fixed-point condition is non-trivial to achieve in practice because the compiler’s source must express a program whose compilation produces exactly the compiler’s current compiled form. The bootstrap process that reaches this fixed point is documented in the Wirth Oberon compiler literature and in the LLVM, Rust, and Go self-hosting histories.
The two directions of the analogy are duals in a specific technical sense. A compiler as a stream processor maps source alphabets to target alphabets under the productivity discipline
\[C_{\text{stream}} \colon A^{\omega}_{\text{source}} \to B^{\omega}_{\text{target}}.\]A stream processor as a compilable program supplies the source that compilation transforms to executable form
\[P \colon \text{state} \to \text{state}, \qquad \text{source}(P) \in A^{\omega}_{\text{source}}.\]When the two coincide, namely when $C = P$ and the source alphabet is the alphabet of the compiler’s own source language, the arrangement is self-hosted. The commutative diagram that this coincidence implies places compilation on both sides of the arrow in the coalgebraic setting of article A193.
The mathematical content of the duality is that compilation respects the productivity structure in both directions. The source-to-target direction is productive by the streaming discipline of the compiler. The compiler-source-to-compiler-behaviour direction is productive by the productivity guarantee that the source language supplies for any program written in it, including the compiler itself.
Self-hosting is the demonstration that the discipline scales to the compiler’s own complexity. A stream-processor language that cannot express its own compiler in the streaming discipline fails to demonstrate that the discipline is sufficient for programs of practical complexity. A stream-processor language that can express its own compiler in the discipline demonstrates that the discipline is sufficient for the most demanding program that the language must support, namely its own toolchain.
One demonstrated self-hosting arrangement and one in-progress target appear in the historical record of the streaming discipline. Wirth’s Oberon demonstrated the integrated shape under the Wirth language line’s strict declare-before-use discipline and closed the self-hosting endpoint. Keleusma’s V0.3.0 roadmap targets the decomposed shape under a coroutine-based discipline whose self-hosting endpoint remains in progress at the time of writing. Brinch Hansen’s SuperPascal provided the language runtime in which the pipeline-of-processes shape could be expressed directly. The SuperPascal compiler itself was implemented in standard sequential Pascal rather than in SuperPascal, per the published record. The Brinch Hansen tradition therefore did not close the self-hosting endpoint even though the language design supported it. The Rust compiler demonstrated the multi-pass whole-program-optimising shape under a fundamentally different set of architectural choices, which places Rust outside the streaming discipline’s boundary per the analysis of article A198.
The self-hosting demonstrations inside the streaming discipline share one specific structural property. The compiler, written in its own source language, is a program whose runtime behaviour respects the discipline’s productivity condition. The compiler produces target bytecode incrementally as it consumes source. The compiler holds bounded working memory regardless of the source being compiled. The compiler completes in time linear in the source size.
These properties are non-trivial to preserve when the compiled program happens to be the compiler itself, because the compiler is a complex program whose memory footprint and computational complexity must fit within the discipline’s constraints. The Wirth line’s Oberon compiler, approximately four thousand lines in Oberon source, demonstrates that the constraint is achievable. The Brinch Hansen SuperPascal language provides a runtime in which the decomposed shape can be expressed directly. The SuperPascal compiler itself was implemented in standard sequential Pascal rather than in SuperPascal. The Brinch Hansen tradition established the pipeline architecture without closing the self-hosting endpoint.
Keleusma’s compilation pipeline provides a modern worked example of the discipline at substantial engineering ambition. The pipeline is currently implemented in Rust as the production compiler and is planned to be self-hosted in the V0.3.0 milestone as a Keleusma-in-Keleusma compiler. The Keleusma-in-Rust pipeline provides the demonstration that the discipline scales to a typed, resource-bounded, coroutine-based scripting language whose target contexts include embedded audio, game engines, and high-assurance embedded control.
The Keleusma pipeline
consists of
five stages
running
in sequence
as
a compile-time pipeline
followed by
a runtime pipeline
that
respects
the streaming discipline
during execution.
The five-stage decomposition
describes
the current Rust-hosted
production compiler.
Article A197 describes
the same pipeline
under
a three-stage
coroutine projection
in which
the middle three stages
are merged
into
a single Keleusma loop function.
The two descriptions
are
the same pipeline
at different levels of granularity.
Stage one, tokenise. The source bytes are consumed by the lexer, which yields tokens one at a time. The lexer holds a small character buffer sufficient to recognise a single token plus a source-position tracker. Its working memory is constant in the source size.
Stage two, parse. The token stream is consumed by the parser, which yields declarations one at a time. The parser holds a bounded parse stack whose depth equals the current syntactic nesting depth of the source cursor, plus a fixup buffer for the currently parsed declaration.
Stage three, typecheck. Each parsed declaration is consumed by the type checker, which runs Hindley-Milner-style unification over the per-declaration constraint graph and yields a typed declaration. The constraint graph is held in arena memory whose size is bounded by the declaration’s complexity, not by the total program size. This is the per-declaration non-streaming stage identified in article A198 as the escape hatch that allows Hindley-Milner inference inside a streaming pipeline.
Stage four, monomorphise. Each typed declaration is consumed by the monomorphiser, which generates specialised versions of generic declarations for each concrete instantiation that the source requests. The monomorphiser maintains a per-specialisation table that tracks which specialisations have already been generated across the entire program. The specialisation table grows with the number of distinct specialisations, not with the source size, which is typically a small bound in practice.
Stage five, hoist and emit chunks. Each monomorphised declaration is consumed by the closure-hoister and the chunk emitter. Closure literals are hoisted to top-level synthetic chunks. Each chunk emits its bytecode in a final pass that patches forward jumps against the block-local fixup table that article A194 formalised. The output is a stream of bytecode chunks that together form the compiled module.
The pipeline respects the streaming discipline at the compilation level. Each stage consumes its input stream and produces its output stream in a productivity-preserving manner. Formally, the Keleusma compilation pipeline is the composition
\[C_{\text{Keleusma}} = \text{emit} \circ \text{monomorphise} \circ \text{typecheck} \circ \text{parse} \circ \text{tokenise},\]which is a stream processor $C_{\text{Keleusma}} \colon A^{\omega}{\text{source}} \to B^{\omega}{\text{bytecode}}$ in the sense of article A193. The emit stage performs closure hoisting before chunk emission, combining what the Keleusma implementation documents as two separate operations into a single logical stage for the purposes of this composition. The composition inherits per-stage productivity by the compositional productivity result of the theory pair, namely that a pipeline of productive stages is itself productive with composed productivity function $m = m_5 \circ m_4 \circ m_3 \circ m_2 \circ m_1$ across the five stages.
The pipeline’s total working memory respects the compositional bound of article A196,
\[M_{\text{Keleusma}} = \sum_{i=1}^{5} M_i + \sum_{i=1}^{4} b_i,\]where $M_i$ is the working memory of stage $i$ and $b_i$ is the capacity of the inter-stage buffer between stages $i$ and $i + 1$. Each per-stage term $M_i$ that corresponds to a per-declaration working set is bounded a priori by Keleusma’s worst-case memory usage analysis, which is program-independent by construction. The per-declaration portion of $M_{\text{Keleusma}}$ is therefore $O(1)$ in the source program size, matching the discipline’s overall bound. The accumulating top-level environment that tracks signatures of declarations already parsed grows with the top-level declaration count and is bounded per module under Keleusma’s separate-compilation model.
A separate structural verification stage runs after emission and validates the completed module against the block-structured verification rules that article A192 developed. The verifier is formally a distinct stream processor that operates on the emitted module rather than a stage of the compilation pipeline itself.
The compiled bytecode
respects
the streaming discipline
at
the execution level.
The bytecode
is
consumed
by
the Keleusma virtual machine
that
executes
one instruction
at a time,
yields
output values
through
the coroutine yield operation,
and resumes
through
the coroutine resume operation.
The runtime pipeline
is
itself
a stream processor
whose behaviour
respects
the productivity condition.
The self-hosting endpoint that Keleusma’s V0.3.0 roadmap targets is the coincidence of these two roles. The Keleusma-in-Keleusma compiler is a stream processor in Keleusma source that compiles Keleusma source to Keleusma bytecode. The self-hosted compiler demonstrates the discipline at its fixed-point completion.
The Keleusma pipeline sits within a lineage of self-hosted stream-processor compilers that this series has traced.
Wirth’s Oberon (nineteen eighty-seven language, nineteen ninety-two book). The Oberon compiler, written in Oberon, approximately four thousand lines of Oberon source, demonstrated the integrated shape of the streaming discipline. The compiler compiles itself in one pass under the strict declare-before-use rule.
Turbo Pascal one through three (nineteen eighty-three to nineteen eighty-five). Turbo Pascal was not self-hosted, because its compiler was written in eight-oh-eight-six assembly language, not in Pascal. Turbo Pascal was however a commercial demonstration of the streaming discipline at mass-market scale, which article A190 treated under the epistemic-policy constraint of the closed-source status.
Brinch Hansen’s SuperPascal (nineteen ninety-four).
SuperPascal
provided
a language runtime
in which
the decomposed pipeline shape
could be expressed
directly
through
parallel blocks,
forall loops,
and typed channels,
as article A191
detailed.
The SuperPascal compiler itself
was
implemented in
standard sequential Pascal
rather than in
SuperPascal,
per the published record.
Brinch Hansen’s line
established
the pipeline architecture
without closing
the self-hosting endpoint.
WebAssembly (twenty seventeen). WebAssembly codified the block-structured discipline at the bytecode level for portable execution. The WebAssembly compilers that emit this bytecode are typically multi-pass at the source-language level, but the bytecode itself respects the streaming validation discipline that article A192 developed and Watt mechanised.
Keleusma (in progress). Keleusma adopts the streaming discipline throughout its language design and targets self-hosting in its V0.3.0 milestone. The Keleusma-in-Rust production compiler implements the pipeline described above. The Keleusma-in-Keleusma self-hosted compiler is the design endpoint.
These five precedents demonstrate that the streaming discipline has been achieved across four decades by different authors in different traditions targeting different application domains. The discipline is not a niche academic exercise. It has been demonstrated repeatedly in production compilers under production constraints.
The series argued four theses.
Thesis one: the streaming discipline is a real phenomenon. The stream-processor compilation discipline is not a rhetorical framing. It has a formal mathematical foundation in Rutten’s coalgebraic productivity and a concrete syntactic characterisation in block-structured control flow. Programs compiled under the discipline inherit provable productivity guarantees that programs compiled under the multi-pass tradition do not directly support.
Thesis two: the streaming discipline is achievable at production scale. Wirth’s Oberon, Turbo Pascal, Brinch Hansen’s SuperPascal, WebAssembly’s bytecode validator, and Keleusma’s compilation pipeline all demonstrate the discipline at production scale under production constraints. The discipline does not require academic toys.
Thesis three: the streaming discipline has a specific applicability boundary. The discipline is the correct choice for embedded scripting, real-time control, safety-critical systems, interactive development, and educational contexts. It is not the correct choice for whole-program-optimised native code, polymorphism-heavy functional languages, or metaprogramming-heavy domain-specific languages. The choice between the streaming discipline and the multi-pass tradition depends on target context, and neither is universally correct.
Thesis four: self-hosting is the discipline’s demonstrative endpoint. A stream-processor language that can express its own compiler in the streaming discipline demonstrates the discipline’s sufficiency for programs of practical complexity. The self-hosting demonstrations that this series identified, namely Oberon and Keleusma’s V0.3.0 target, each close the discipline’s completeness argument for their respective language line.
The series ends here but the discipline does not. Several open questions remain for subsequent investigation.
Formal verification of stream-processor compilers. Watt’s WebAssembly mechanisation and Leroy’s CompCert both demonstrated that formal verification of compiler artefacts is feasible under current tools. A stream-processor compiler whose architectural discipline composes well with compositional verification techniques is a natural candidate for this treatment. The Keleusma V0.3.0 roadmap identifies this as a future direction without committing to a specific verification framework.
Auditability of the discipline. A stream-processor compiler whose architectural discipline composes across stage boundaries admits independent per-stage checking more readily than a multi-pass optimising compiler whose whole-program state must be inspected as a unit. Small, single-sweep verifiers of the kind that article A192 developed are easier to audit than their multi-pass counterparts. The precise mapping between architectural properties and external review criteria depends on the target context and remains future work.
Extension to non-textual source formats. The streaming discipline generalises naturally to source formats other than text, including binary intermediate representations, graph-based program representations, and domain-specific data structures. A stream-processor compiler whose source alphabet is not byte-oriented text retains the discipline’s compositional properties if the source stream respects the productivity condition.
Combined stream-and-batch pipelines. Many practical compilation workflows combine streaming per-file compilation with batch whole-program post-processing. The synthesis of the two disciplines into a single architecturally coherent compilation model is an open engineering question that articles in the multi-pass tradition have not directly addressed.
Compilation is a stream-processor operation when the source language, the target format, and the compiler architecture respect the productivity discipline that Rutten’s coalgebraic framework formalises. The historical record demonstrates that the discipline is achievable in production compilers under production constraints, from Wirth’s Oberon language in nineteen eighty-seven through Keleusma’s in-progress self-hosting in the present. The discipline has specific applicability that target contexts either match or do not, and the choice between the discipline and the multi-pass tradition depends on the target context rather than on any universal ranking of the two. The self-hosted stream-processor compiler is the fixed-point endpoint at which the compiler and its compiled programs are both stream processors of the same language, demonstrating the discipline’s sufficiency for its own implementation. This closes the series.
An earlier revision of this article made two claims that were subsequently corrected.
The SuperPascal self-hosting claim. The article previously described Brinch Hansen’s SuperPascal as one of three self-hosting demonstrations alongside Oberon and Keleusma’s V0.3.0 target. The SuperPascal compiler and interpreter were in fact implemented in standard sequential Pascal, specifically the ISO Level 1 dialect, rather than in SuperPascal. Brinch Hansen’s line established the pipeline architecture and the language design in which a self-hosted stream-processor compiler could naturally be written but did not close the self-hosting endpoint. The corrected series lists one demonstrated self-hosting arrangement, namely Oberon, and one in-progress target, namely Keleusma’s V0.3.0 roadmap. Article A191 carries the corresponding correction in the Brinch Hansen historical treatment.
The working-memory bound. The article previously stated that the compilation pipeline’s total working memory was $O(1)$ in the source program size. This was overstated. The per-declaration working set is $O(1)$ in the program size by the streaming discipline, but the accumulating top-level environment that tracks signatures of declarations already parsed grows with the top-level declaration count and is bounded per module under separate compilation. Article A196 carries the corresponding correction in the symbol table treatment.
]]>The prior articles of this series have argued for the stream-processor discipline without suggesting that the discipline is universally correct. Article A197 compared integrated and decomposed architectures within the discipline and noted that the choice depended on target context. This article closes the synthesis pair by identifying the contexts where the stream-processor discipline itself breaks down and where the multi-pass abstract-syntax-tree tradition is the correct choice.
Four language features push past the boundary of single-pass tractability. The first is whole-program optimisation, which requires information that crosses compilation-unit boundaries. The second is Hindley-Milner type inference in its unconstrained form, where the type of a local variable depends on uses that appear later in the function. The third is type-class or trait resolution, where selecting a specific implementation requires searching the whole program’s implementation space. The fourth is metaprogramming, where source-level transformations require reading and rewriting source text that has not yet been produced. Each feature has a productivity or expressiveness advantage that justifies the multi-pass cost in its target context.
Whole-program optimisation combines information from across the entire program to produce target code that is faster or smaller than per-function-optimised code. The optimisations in this class include inlining across compilation units, devirtualisation of polymorphic calls based on whole-program type analysis, and dead-code elimination that identifies functions whose result is never used by any caller.
The information required for these optimisations is not local. Inlining a function into its callers requires knowing which callers the function has, which is whole-program information. Devirtualising a polymorphic call requires knowing all types that can appear at that call site, which is whole-program information. Dead-code elimination requires knowing all uses of every function, which is whole-program information.
A stream-processor compiler that emits target code as it recognises source syntax does not have access to this whole-program information at the moment of emission. The compiler sees only the current source position and what preceded it. Any optimisation that depends on information from what follows is outside the discipline’s reach.
Formally, the optimisation cost of a whole-program pass is
\[C_{\text{WPO}}(P) = O(\lvert P \rvert \cdot \log \lvert P \rvert),\]for well-designed implementations that use efficient data structures for cross-function analysis. The cost is not $O(\lvert P \rvert^{2})$ in typical production compilers because the whole-program analysis uses structured indexing and incremental data-flow techniques. The cost is still super-linear in the program size, which is the fundamental departure from the linear-cost stream-processor bound.
The benefit of whole-program optimisation is a constant-factor speedup in the compiled code’s runtime performance. The specific factor depends on the program and the optimisation techniques. Production optimising compilers typically achieve
\[\frac{T_{\text{unoptimised}}}{T_{\text{WPO}}} \in [2, 5]\]across a range of benchmark suites, where $T_{\text{unoptimised}}$ is the runtime of code compiled without whole-program optimisation and $T_{\text{WPO}}$ is the runtime of the same code compiled with whole-program optimisation applied. The range brackets typical benchmarks rather than worst-case or best-case scenarios. For programs whose runtime dominates compilation time by several orders of magnitude, which is the case for long-running server workloads and scientific-computing programs, the multi-pass cost is a one-time investment that pays back across the program’s operational lifetime.
Hindley-Milner type inference in its classical form solves a system of type-equality constraints across the entire function body. The type of a local variable is not determined by its declaration alone. It depends on how the variable is used throughout the function.
Consider a function definition of the form
\[\lambda x. e\]where $e$ is an expression that uses $x$ in some way. The type of $x$ is determined by $e$’s constraints on it. If $e$ uses $x$ as an integer, $x$ must be of integer type. If $e$ uses $x$ as the head of a list, $x$ must be of list type. The determination requires inspecting $e$ in its entirety, which requires having parsed the whole function body before resolving $x$’s type.
A single-pass compiler cannot perform this inference because it does not have $e$ available at the time $x$’s declaration is parsed. The classical resolution requires constructing a constraint graph that spans the function body and solving it by unification after the whole body has been parsed.
Formally, for a function body containing $n$ type variables and $m$ constraints, the unification cost is
\[C_{\text{HM}}(f) = O(m \cdot \alpha(n, m)),\]where $\alpha$ is the inverse Ackermann function that arises from the union-find data structure used by efficient unification implementations. The cost is near-linear in the constraint count for each function, but requires the whole function body to be available during the analysis.
Three workarounds allow stream-processor compilers to support languages that resemble Hindley-Milner without paying the full inference cost.
Restrict inference to per-declaration scope. A compiler that requires explicit type annotations on procedure parameters and return types can resolve the type of each local variable against the annotation without whole-function analysis. This is the approach that Wirth’s languages take. The programmer supplies the types that the compiler would otherwise infer.
Bound inference to per-declaration scope with local constraint graph. A compiler that allows type inference within a single declaration but prohibits inference across declarations can solve the local constraint graph in arena memory whose size is bounded by the declaration’s complexity. This is one of the approaches that the Keleusma V0.3.0 self-hosting plan identifies as compatible with the stream-processor discipline.
Introduce inference as a separate non-streaming stage. A pipeline compiler can insert a non-streaming type-inference stage between the parser and the code generator. The type-inference stage consumes each parsed declaration in full, runs the unification algorithm on the local constraint graph, and yields a typed declaration that subsequent stages can consume in streaming fashion. This is a per-declaration non-streaming stage that does not break the pipeline’s inter-stage streaming discipline.
Type-class or trait resolution requires selecting a specific implementation of a polymorphic operation based on the types that appear at the call site. The selection involves searching the whole-program’s implementation space for an implementation that matches the required type signature.
For a call site
of the form
op(x)
where
op is
a type-class method
and
x
has type
$T$,
the compiler
must find
an instance
of the type class
declared for
type $T$.
The instance
may be declared
anywhere in
the program,
in
any imported module,
or in
any transitively imported module.
The search
requires
whole-program information.
The Haskell type-class dispatch and the Rust trait-resolution mechanism both face this constraint. Rust uses a resolution algorithm that maintains an implementation index across the entire crate graph and searches it at each polymorphic call site. A simplified model of the search cost at a call site is
\[C_{\text{trait}}(\text{call site}) = O(\lvert \text{impls}(T) \rvert),\]where $\lvert \text{impls}(T) \rvert$ is the number of implementations declared for type $T$. The actual algorithm is substantially more complex because of coherence checks, orphan rules, associated types, higher-ranked trait bounds, and blanket implementations that require additional processing per call site. For programs with many implementations per type or substantial trait-bound complexity, the cost can be substantial.
A single-pass compiler cannot support type-class resolution in this form because the implementation index requires whole-program state. The alternatives include requiring explicit dispatch at each call site, prohibiting type classes, or restricting implementations to per-declaration scope where they can be resolved locally. Each alternative sacrifices some of the ergonomic advantages that type classes provide.
Source-level metaprogramming in the tradition of Lisp macros, C-plus-plus templates, and Rust procedural macros requires the compiler to execute source-level transformations that themselves produce new source to be compiled. A macro invocation in the source expands to new source text that the compiler must then parse and compile.
The expansion can produce arbitrary code. The compiler must run the parser and sometimes the earlier compilation stages on the expanded output before resuming the compilation of the original source. The expansion can itself contain macro invocations, which require recursive expansion.
A single-pass compiler that reads each source token once cannot support this pattern without substantial machinery. The expansion output must be inserted into the input stream at the position of the macro invocation. The expanded content must be parsed before the macro’s containing construct completes. The recursion requires saving and restoring the parser state across expansion boundaries.
The stream-processor discipline therefore prohibits source-level macros in general. Restricted forms of metaprogramming that produce their expansions at compile time before the streaming pipeline starts are compatible with the discipline. The Zig compile-time evaluation model, which article A195 mentioned, is one such restricted form. The Rust declarative macro system, which uses pattern matching against the invocation syntax, is another restricted form, though Rust’s overall compilation model is already multi-pass.
Compile-time reflection allows programs to inspect the structure of their own types, functions, and modules during compilation and generate code based on the inspection results. Serialisation libraries, object-relational mappers, and schema-driven code generators frequently rely on compile-time reflection.
The reflection API requires the compiler to maintain enough type-and-structure information to answer the reflection queries. The information must be available at the reflection call site. For a reflection query that inspects a type declared later in the source, a single-pass compiler cannot answer without looking ahead.
Most modern languages that support compile-time reflection also support some form of multi-pass compilation. The reflection API sits naturally in the multi-pass architecture because the whole-program type table is already available by the time reflection queries are evaluated. Single-pass compilers that attempt to support reflection must either restrict the reflection scope to already-parsed material or add a whole-program type-collection pass before the streaming compilation begins.
The four features above share a common structural property. Each requires information that is not available from the current source position alone. The information lies either in the source that has not yet been parsed or in imported modules whose structure must be known to the current compilation.
Formally, a compilation task is inside the stream-processor discipline if the target output at position $i$ depends only on the source in positions $0$ through $i$, that is,
\[\text{output}(i) = f(\text{source}[0 \ldots i]).\]A compilation task is outside the discipline if the output at position $i$ depends on source that appears later or on whole-program information, that is,
\[\text{output}(i) = f(\text{source}[0 \ldots \infty]).\]The boundary between the two cases is sharp in theory but gradient in practice. Many languages adopt hybrid approaches that combine a stream-processor front end with a multi-pass optimisation backend, paying the multi-pass cost only for the features that require it. The hybrid form gives the output access to both the streaming source prefix and a whole-program interface index that declares identifier signatures and type declarations before compilation begins,
\[\text{output}(i) = f(\text{source}[0 \ldots i], \text{interface}[0 \ldots \infty]).\]The interface index carries whole-program declarations without whole-program bodies, which allows name resolution to proceed in a single forward pass against a pre-computed declaration table. The C-plus-plus compilation model, with its translation-unit-local compilation followed by link-time whole-program optimisation, is one instance of this hybrid. The Rust compilation model, with its crate-local compilation followed by whole-program monomorphisation and optimisation, is another.
Some features that appear to require multi-pass compilation admit restricted forms that fit the stream-processor discipline. The restrictions sacrifice some of the feature’s expressiveness in exchange for compilation-cost linearity.
Explicit type annotations in place of Hindley-Milner inference. Wirth’s languages, Go’s function signatures, Rust’s function signatures at the item level, and WebAssembly’s function types all require explicit annotations that substitute programmer effort for inference cost.
Restricted macros that expand before compilation begins. Preprocessor-style macro expansion, where the expander runs before the compiler sees the source, is compatible with streaming compilation of the post-expansion source. The overall compilation is multi-pass only if the expander pass is counted.
Whole-program information supplied by separate declaration files. Modula-2’s definition-implementation split, C-plus-plus’s header files, and Rust’s module interface files allow whole-program information to be delivered to the compiler through a pre-compiled index rather than through source scanning. The index is compact and can be loaded at compilation start, allowing the actual compilation to proceed in a single forward pass.
Per-function non-streaming stages inside a streaming pipeline. The pipeline compiler can insert non-streaming stages between the parser and the code generator that operate on one declaration at a time without requiring whole-program state. These stages break the strict streaming discipline at the per-declaration level but preserve it at the whole-program level.
The multi-pass discipline is the correct choice when one or more of the following conditions hold.
The target program runs for years of accumulated processor time. Long-running server workloads, scientific-computing programs, and production embedded systems benefit substantially from whole-program optimisation. The multi-pass compilation cost is amortised across many runs.
The source language has type-class or trait polymorphism. Modern functional languages and Rust rely on these features for ergonomic abstractions. A stream-processor compiler cannot support them without substantial restrictions that the language design did not anticipate.
The source language supports metaprogramming as a first-class feature. Lisp, Rust, Zig, and Julia all use metaprogramming extensively. A stream-processor compiler that supports these languages would have to prohibit or substantially restrict this feature.
The compilation target supports whole-program-optimised executables. Native code for performance-critical workloads benefits from link-time optimisation. JIT-compiled bytecode for dynamic languages benefits from runtime profiling and adaptive recompilation. Both patterns depend on whole-program information that the stream-processor discipline does not provide.
The four push-past conditions above identify the boundary of the stream-processor discipline. Inside the boundary lie target contexts where the discipline’s compositional properties are decisive.
For target contexts outside the boundary, including whole-program-optimised native code, polymorphism-heavy functional languages, and metaprogramming-heavy domain-specific languages, the multi-pass discipline is the correct choice. The synthesis of this series is that neither discipline is universally correct. The design decision for any specific compiler is to identify which side of the boundary the target context falls on and to choose the corresponding architecture.
The stream-processor discipline does not fit every target context. Whole-program optimisation, Hindley-Milner type inference, type-class resolution, and metaprogramming each push past the boundary of single-pass tractability by requiring information that is not available from the current source position alone. For target contexts where these features are essential or where the runtime performance benefit of whole-program analysis justifies the compilation-cost investment, the multi-pass discipline is the correct choice. For target contexts where compositional properties, bounded memory, and compilation-cycle latency matter more than these features, the stream-processor discipline holds. Article A199 closes the series with the synthesis that compilation itself is a stream-processor operation and that a self-hosted stream-processor compiler demonstrates the discipline at its strongest, using Keleusma’s V0.3.0 compilation pipeline as the modern worked example.
The prior-art trio of articles A189 through A191 identified two distinct architectures that belong to the stream-processor compilation discipline. The integrated single-pass compiler of the Wirth line and Turbo Pascal runs the entire pipeline inside a recursive-descent parser that emits target code inside its own procedures. The decomposed pipeline compiler of Per Brinch Hansen’s tradition splits the pipeline into independent stages that communicate through bounded queues. Both architectures achieve the same theoretical properties of linear compilation cost and program-independent working memory. Both have been demonstrated in production compilers by different authors in different traditions.
This article
compares
the two architectures
head to head
on
workloads
that
either architecture
can accomplish.
It identifies
where
one architecture
carries
material advantages
over the other
and
where
the choice
is
essentially
a matter of
implementation preference.
The article
closes with
the Keleusma V0.3.0
self-hosting roadmap
as
a modern worked example
of
the decomposed pipeline
approach,
showing
how
Brinch Hansen’s pattern
manifests
in
a coroutine-based
embedded scripting language
where
each pipeline stage
is
a loop function
that
communicates through
yield and resume
operations.
The integrated single-pass compiler has one main procedure, the recursive-descent parser. Inside the parser, the lexer is a method call that returns the next token on demand. The code generator is a set of methods called at the syntactic point where each construct is recognised. There is no separate lexer process, no separate parser stage, no separate code-generation phase. The three logical stages are inlined into a single call chain.
The decomposed pipeline compiler has multiple independent stages. Each stage runs as a separate process in concurrent implementations or as a coroutine in sequential implementations. Each stage holds only its local working state. The stages communicate through bounded buffers that each stage reads from its upstream neighbour and writes to its downstream neighbour.
The two shapes sit at opposite ends of one axis in the two-axis design space introduced in article A188. Both shapes sit at the same position on the other axis, namely abstract-syntax-tree-free direct emission, which distinguishes them from the multi-pass tradition of production optimising compilers.
For a workload where either architecture can produce identical target bytecode, the two shapes have comparable but not identical resource profiles.
Working memory. Both shapes achieve $O(1)$ per-declaration working set in the source program size, and both accumulate a top-level environment that grows with the top-level declaration count. The integrated compiler holds the parse stack, the symbol table, and the fixup buffer in one memory region. The decomposed compiler distributes these across the pipeline stages and adds the inter-stage buffers.
For an integrated compiler, the working memory is approximately
\[M_{\text{integrated}} \approx \lvert S \rvert + \lvert T \rvert + \lvert F \rvert,\]where $S$ is the parse stack, $T$ is the symbol table, and $F$ is the fixup buffer.
For a decomposed compiler with $k$ stages and inter-stage buffer sizes $b_i$, the working memory is approximately
\[M_{\text{pipeline}} \approx \sum_{i=1}^{k} m_i + \sum_{i=1}^{k-1} b_i,\]where $m_i$ is the working state of stage $i$. The two sums are typically of the same order of magnitude because the same total state is present in both architectures, distributed differently.
Compilation throughput. Both shapes process each source token in constant time on average. The compilation cost is linear in the program size for both. The constants differ by a factor that depends on the concrete implementation choices.
For the integrated compiler, each token travels through a single procedure call chain whose constant per-token overhead is low. For the decomposed compiler, each token crosses inter-stage buffer boundaries, which adds a small per-token buffering cost. The specific ratio
\[\frac{T_{\text{pipeline}}}{T_{\text{integrated}}} = 1 + \frac{k - 1}{L},\]where $L$ is the average number of instructions per token processed within a stage and $k$ is the number of stages, approaches one for larger $L$ and grows with smaller $L$. For a typical Wirth-style compiler with $L$ on the order of tens to hundreds of instructions per token, the pipeline overhead is under ten percent compared to the integrated form.
End-to-end latency. The two shapes differ substantially in end-to-end latency. The integrated compiler produces target output immediately as each syntactic construct is recognised, with sub-token latency. The decomposed compiler must transit each item through the buffer chain, which takes at least $k - 1$ buffer-transit times per item, as article A191 formalised. The comparative latency between the two shapes is therefore
\[\Delta L = L_{\text{pipeline}} - L_{\text{integrated}} \approx (k - 1) \cdot \tau_{\text{buffer}},\]where $\tau_{\text{buffer}}$ is the average inter-stage buffer-transit time per item. For a $k$ around four or five and $\tau_{\text{buffer}}$ on the order of microseconds under typical in-process coroutine implementations, the latency gap is on the order of tens of microseconds per item.
For compilation of programs whose bytecode output must be delivered incrementally during compilation, the integrated shape has a qualitative advantage. For compilation of programs whose bytecode output is written to a file and consumed later, the latency difference is irrelevant.
The decomposed shape has a substantial advantage in testability. Each stage can be tested in isolation by driving it with a synthesised input stream and inspecting the output stream. A test harness that exercises the parser stage does not need to run the lexer stage, because the parser receives its input from a buffer that the test can populate directly.
For a compiler with $k$ stages, the testability decomposition gives
\[\text{Test surfaces} = k \text{ per-stage surfaces} + 1 \text{ full-pipeline surface}.\]The integrated compiler has only the full-pipeline surface because its stages are inlined. A test that exercises the parser must run the lexer because the two are called in the same procedure. A test that exercises the code generator must run all upstream stages.
The per-stage testability of the decomposed shape supports several testing techniques that the integrated shape cannot use directly.
Property-based testing per stage. Random-input generation for the parser can proceed by generating token streams directly rather than generating source strings that must round-trip through the lexer. The generated inputs exercise the parser’s decision boundaries without any lexer-imposed distributional constraints.
Snapshot testing per stage. Golden-file tests that capture the output of each stage against a known input can catch regressions in individual stages independently of the other stages. A regression in the parser stage manifests as a snapshot mismatch at the parser output, which localises the failure to the parser before any downstream stage runs.
Differential testing between stages. When two different implementations of the same stage exist, they can be compared by driving both with the same input stream and comparing their output streams. The comparison requires the stage boundaries to be observable, which the decomposed shape provides by construction and the integrated shape does not.
For a compiler that must undergo formal verification or extensive independent review, the choice of architectural shape affects the verification burden.
The decomposed shape supports compositional verification. Each stage can be verified independently. The composition of correctly verified stages gives a correctly verified compiler under appropriate composition rules. The total verification cost is therefore
\[V_{\text{pipeline}} = \sum_{i=1}^{k} V_i^{\text{stage}} + V^{\text{composition}},\]where $V_i^{\text{stage}}$ is the cost of verifying stage $i$ in isolation and $V^{\text{composition}}$ is the cost of composing the individual results into an end-to-end guarantee.
The integrated shape requires whole-compiler verification. The verification argument must track the state of the parser stack, the symbol table, the fixup buffer, and the output emission throughout a single monolithic argument. The total verification cost is
\[V_{\text{integrated}} = V^{\text{monolithic}},\]which generally exceeds the sum of per-stage costs by a factor that depends on the strength of the required guarantees. The CompCert project and Watt’s WebAssembly mechanisation, mentioned in articles A188 and A192, both address verification of compiler artefacts, and both benefit from architectural decomposition where possible.
The decomposed shape admits concurrent execution of the pipeline stages on multiprocessor hardware. The integrated shape does not admit concurrent execution of its logical stages because they are inlined into a single call chain.
For the decomposed shape, under the pipeline-parallelism model of article A191, the speedup of concurrent execution against sequential execution of the same pipeline is
\[\text{speedup} = \frac{\sum_{i=1}^{k} T_i^{-1}}{\max_{i} T_i^{-1}},\]which approaches $k$ for a $k$-stage pipeline balanced across the stages and degrades to one when a single slow stage dominates. In practice, compilation stages are not perfectly balanced, and the achieved speedup is typically in the two-to-four range for a four-to-five-stage compiler. The absolute throughput gain is modest compared to the throughput of a well-tuned integrated compiler, but becomes significant for very large source files or for batch compilation of many source files.
The integrated shape can exploit concurrent execution only at a higher granularity, such as compiling multiple source files in parallel. The intra-file compilation remains serial.
Between the integrated shape and the decomposed shape with concurrent execution lies a coroutine-based decomposition that preserves the decomposed architecture without requiring concurrent execution.
Each pipeline stage
is
a coroutine
whose
yield operation
plays the role of
a bounded-buffer send
and whose
resume operation
plays the role of
a bounded-buffer receive.
The stages
run
sequentially
on
a single processor,
with
one stage
active at a time.
When
the active stage
yields,
control transfers
to the next stage,
which resumes
where it
left off.
The coroutine implementation preserves several properties of the decomposed shape. Per-stage testability holds because each coroutine can be driven independently by supplying its input stream directly. Compositional resource bounds hold because each coroutine’s working memory is bounded independently and the inter-stage buffers have fixed capacity. Sequential single-processor execution is predictable in timing and memory usage, which matters for embedded and real-time contexts.
The coroutine implementation sacrifices one property of the fully concurrent decomposed shape, namely pipeline parallelism on multiprocessor hardware. For single-processor embedded targets where concurrent execution is unavailable or undesirable, the sacrifice is irrelevant. For multi-processor production compilation contexts where throughput is critical, the coroutine implementation leaves performance on the table.
The Keleusma language is a total functional stream processor that compiles to bytecode for embedded scripting, real-time audio and game engines, and high-assurance embedded control contexts. The language’s V0.3.0 self-hosting roadmap, which the Keleusma project has published as a strategy document, adopts the decomposed-pipeline compiler architecture with the coroutine implementation just described. The recommendation is documented as a design in progress rather than a shipped result. Keleusma’s Rust-hosted compiler is the current production form, and the self-hosted Keleusma-in-Keleusma compiler is the V0.3.0 endpoint.
The proposed Keleusma
self-hosted compiler
consists of
three coordinated stages
at
the coroutine-projection level.
Each stage
is
a Keleusma loop function
in
source form.
The three-stage decomposition
is
the simplified
self-hosted coroutine projection
that
maps
one Keleusma loop function
to
one pipeline stage.
The current
Rust-hosted production compiler
uses
a finer-grained
five-stage decomposition
of
tokenise,
parse,
typecheck,
monomorphise,
and
emit-with-hoist,
as
article A199 details
in the closer.
The two decompositions
describe
the same pipeline
at different levels of granularity.
The coroutine projection
merges
the middle three
Rust stages
into
the compiler loop function
because
the coroutine model
does not
require
independent stage boundaries
between
typecheck,
monomorphise,
and
emit.
The three stages
map
one-to-one
with
Brinch Hansen’s
pipeline-of-processes model.
The Keleusma
yield operation
transmits
a value
from
the yielding stage
to
its downstream consumer.
The
resume operation,
which
Keleusma structures
as
the host’s call
to the coroutine,
delivers
the next input
to
the resuming stage.
The inter-stage buffer
is
implicitly
of capacity one
because
Keleusma’s coroutine model
delivers
one value per yield-resume cycle.
Each stage’s working memory is bounded by Keleusma’s worst-case memory usage analysis, which Keleusma calls WCMU and which compiles into a compile-time guarantee on each stage’s memory footprint. The guarantee per stage is independent of the source program being compiled, matching the bounded-working-memory bound of the trio.
The choice of the coroutine implementation over the fully concurrent decomposed shape reflects Keleusma’s target contexts. The language runs on no-standard-library plus allocation-supporting embedded hosts where single-processor execution is the norm. Pipeline parallelism is not a target design property. The coroutine implementation preserves the testability, compositional-resource, and Brinch-Hansen-lineage architectural benefits without paying the concurrency cost.
Keleusma also documents the integrated single-pass alternative for completeness and notes that the alternative sits on the shelf if implementation surfaces a material reason to prefer it. The design decision is that neither shape is obviously better for Keleusma’s target contexts, so the decomposed shape is chosen for its compositional verification properties and its per-stage testability.
The decision between the integrated shape and the decomposed shape depends on several considerations that do not always point the same way.
Verification requirements. For compilers that must undergo formal verification under a safety standard, the decomposed shape supports compositional verification and reduces the total verification burden. For compilers without such requirements, the choice is less constrained.
Concurrent execution availability. For compilers that run on multiprocessor hardware and target programs whose compilation benefits from concurrent processing, the decomposed shape admits pipeline parallelism. For compilers that run on single-processor hardware or whose targets are small enough that parallelism does not help, the choice is between the integrated shape and the coroutine-based decomposed shape.
Testing methodology. For compilers whose testing strategy relies on per-stage golden files, property-based tests of intermediate representations, or differential comparison against a reference implementation, the decomposed shape supports these techniques directly. For compilers whose testing strategy uses only end-to-end integration tests against the shipped bytecode, the decomposition adds overhead without proportionate benefit.
Implementation preference. For compilers written by a single author who prefers one shape over the other, the choice is a matter of preference provided the underlying theoretical properties hold for both. The Wirth tradition and the Brinch Hansen tradition both produced working compilers under different architectural choices, and neither tradition’s choice was uniquely correct.
The integrated single-pass compiler and the decomposed pipeline compiler both belong to the stream-processor compilation discipline and both achieve the same $O(1)$-in-program-size per-declaration working set, the same top-level environment growth proportional to top-level declaration count, and the same linear-in-program-size compilation cost. The integrated shape has a lower per-token overhead and lower end-to-end latency. The decomposed shape has per-stage testability, compositional verification, and optional pipeline parallelism. The coroutine-based implementation of the decomposed shape, demonstrated in the Keleusma V0.3.0 self-hosting roadmap, sits between the two extremes by preserving the compositional properties without requiring concurrent execution. For target contexts where formal verification, per-stage testability, or lineage alignment with Brinch Hansen’s tradition matters, the decomposed shape is the preferred choice. For target contexts where raw throughput or minimal overhead matters, the integrated shape is the preferred choice. Neither shape is universally correct. Article A198 closes the synthesis pair with an analysis of where the stream-processor discipline itself breaks down, covering whole-program optimisation, Hindley-Milner type inference, and the language features that push past the boundary of single-pass tractability.
The symbol table is the third foundational technique of the single-pass compilation discipline. Articles A194 and A195 developed the fixup mechanism that handles forward references to code addresses and the declare-before-use rule that handles forward references to identifiers. Both mechanisms depend on a specific data structure in the compiler, namely the symbol table that holds the identifiers currently visible to the parser. This article develops the symbol table in the specific form that supports the single-pass discipline, argues why scope-based popping keeps the compiler’s working memory bounded by source nesting depth rather than program size, and shows how the discipline interacts with the other techniques of the trio.
The single-pass discipline requires that the compiler hold at most the identifiers visible in the current scope stack. Identifiers declared in scopes that have closed are not carried forward. Identifiers declared in scopes that have not yet opened have not appeared yet. The symbol table’s size at any point depends on the current scope stack, not on the total number of identifiers in the program.
A block-structured source language
introduces
a new lexical scope
at each
scope-opening construct.
Scope-openers
include
function definitions,
procedure definitions,
block statements,
begin and end pairs,
and any other
construct
that admits
local declarations
that
should not leak
to
the enclosing context.
Each scope opener pushes a new empty scope onto a scope stack. Each scope closer pops the current scope off the stack, discarding all identifiers declared within it. The scope stack is therefore a stack in the strict data-structure sense, with scope openers as push operations and scope closers as pop operations.
The current symbol table is the concatenation of all scopes currently on the stack. Formally, if the scope stack at position $i$ is $[S_1, S_2, \ldots, S_d]$ where $S_1$ is the outermost scope and $S_d$ is the innermost scope of nesting depth $d$, then the symbol table at position $i$ is
\[T_i = S_1 \sqcup S_2 \sqcup \cdots \sqcup S_d,\]where $\sqcup$ denotes the merging operation that combines scopes into a single lookup table with the innermost scope taking precedence for identifier lookups.
The merging operation is not necessarily implemented as an actual concatenation in the data structure. Different implementation choices, which the Implementation Choices section below discusses, achieve the same logical merging behaviour through different concrete data structures. The abstract shape of the symbol table is what matters for the working-memory bound.
The symbol table supports four operations.
Enter scope. When a scope opener is recognised, the compiler pushes an empty scope onto the scope stack. Formally,
\[\text{enter-scope}: [S_1, \ldots, S_d] \mapsto [S_1, \ldots, S_d, \emptyset].\]The new scope starts empty. As the compiler recognises declarations inside the new scope, each declaration adds a new identifier to the top-of-stack scope.
Exit scope. When the corresponding scope closer is recognised, the compiler pops the top-of-stack scope and discards it, along with all identifiers declared inside. Formally,
\[\text{exit-scope}: [S_1, \ldots, S_d] \mapsto [S_1, \ldots, S_{d-1}].\]The popped scope is not retained in any auxiliary data structure. The memory that held its identifiers is freed for reuse by subsequent scopes.
Declare identifier. When a declaration is recognised, the compiler adds the identifier to the top-of-stack scope. Formally,
\[\text{declare}(\text{id}, e): [S_1, \ldots, S_d] \mapsto [S_1, \ldots, S_d \cup \{(\text{id}, e)\}],\]where $e$ is the identifier’s symbol-table entry containing its type, address, or other attributes.
Lookup identifier. When a reference to an identifier is parsed, the compiler walks the scope stack from innermost to outermost, returning the first matching entry. Formally,
\[\operatorname{lookup}(\text{id}): \bigcup_{k = d}^{1} \{ (\text{id}, e) : (\text{id}, e) \in S_k \} \quad \text{first match wins.}\]The innermost-first ordering implements the standard lexical-scoping semantics where inner scopes shadow outer scopes. An inner declaration of an identifier that also appears in an outer scope takes precedence inside the inner scope without modifying the outer entry.
The symbol table of a single-pass compiler following the scope-popping discipline has memory bounds that depend only on the source’s scope-nesting properties, not on the total program size.
Let $d_{\max}$ denote the maximum scope-nesting depth of the source program and $s_{\max}$ denote the maximum per-scope declaration count. Both quantities are compile-time properties of the source grammar and the specific program. The maximum symbol table size at any point during compilation is bounded by
\[\lvert T \rvert_{\max} \le d_{\max} \cdot s_{\max}.\]For a typical block-structured language, $d_{\max}$ is in the tens even for large programs, and $s_{\max}$ is bounded by the maximum per-function or per-block declaration count, typically in the low hundreds for carefully organised code. For a concrete estimate, using a typical maximum nesting depth of thirty and a maximum per-scope declaration count of two hundred,
\[\lvert T \rvert_{\max} \le d_{\max} \cdot s_{\max} \le 30 \cdot 200 = 6000 \text{ entries}.\]At sixteen to sixty-four bytes per entry, depending on the identifier length and the attribute payload, the nested-scope portion of the symbol table occupies several hundred kilobytes of memory in the pessimistic worst case and substantially less in typical use. The bound on nested scopes holds regardless of the source program size.
An important subtlety distinguishes the nested-scope portion from the outermost scope. The scope-popping discipline returns the nested-scope working set to the enclosing scope’s state whenever an inner block closes. The outermost scope, which holds top-level declarations of functions, types, constants, and module-level variables, does not pop during compilation. Its size grows monotonically with the number of top-level declarations that the source program contains, which is proportional to the source program size.
The correct per-component bound is therefore
\[\lvert T \rvert = \lvert T_{\text{nested}} \rvert + \lvert T_{\text{top-level}} \rvert,\]where $\lvert T_{\text{nested}} \rvert = O(d_{\max} \cdot s_{\max})$ is program-independent under the assumption that $d_{\max}$ and $s_{\max}$ are bounded by the source language’s grammatical constraints or by programming discipline, and $\lvert T_{\text{top-level}} \rvert = O(\lvert D_{\text{top}} \rvert)$ grows with the top-level declaration count $\lvert D_{\text{top}} \rvert$. For a separately compiled module language, $\lvert D_{\text{top}} \rvert$ is bounded per module by the largest compilation unit rather than by the whole program.
The single-pass discipline therefore bounds the per-declaration working set independently of the source program size, while the accumulating top-level environment grows with the top-level declaration count. The alternative, under which the symbol table accumulates all identifiers declared anywhere in the program and retains their attribute payloads throughout compilation, requires memory linear in the program size with substantially larger constants. The multi-pass discipline that article A188 identified as the counterpoint to the single-pass discipline generally takes this alternative, because multi-pass compilers build whole-program abstract syntax trees that carry their own identifier bindings throughout the compilation.
The lookup cost depends on both the per-scope lookup cost and the number of scopes that must be walked.
For a hash-indexed per-scope lookup, each scope’s lookup is $O(1)$. The overall lookup cost across the scope stack is therefore
\[C_{\text{lookup}} = O(d_{\max}).\]For a tree-indexed per-scope lookup, each scope’s lookup is $O(\log s_{\max})$ and the overall cost across the scope stack is
\[C_{\text{lookup, tree}} = O(d_{\max} \cdot \log s_{\max}).\]In practice, most identifier references resolve in the innermost few scopes because of programming convention. The average lookup cost is much smaller than the worst case. The compiler’s overall compilation cost therefore remains linear in the program size even under the pessimistic worst-case lookup bound.
Shadowing of identifiers across scopes is handled by the innermost-first lookup order. An inner declaration of an identifier inserts into the top-of-stack scope without disturbing outer entries. When the inner scope exits, the inner entry is discarded with the popped scope, and subsequent lookups resolve against the outer entry that was hidden during the inner scope’s lifetime. The mechanism is purely structural, requiring no explicit shadowing bookkeeping in the symbol table data structure.
Consider a Pascal-style source with a nested procedure declaration.
PROGRAM Example;
VAR x: integer;
PROCEDURE Outer(a: integer);
VAR y: integer;
PROCEDURE Inner(b: integer);
VAR z: integer;
BEGIN
z := a + b + y;
x := z
END;
BEGIN
y := a * 2;
Inner(y);
x := x + a
END;
BEGIN
x := 0;
Outer(x + 1)
END.
The compiler’s scope stack walks through five states during compilation.
State 0: at the program’s outer scope.
After parsing
PROGRAM Example
and
VAR x: integer,
the scope stack is
$[S_1]$
with
$S_1 = {x}$.
State 1: entering Outer.
Upon entering
Outer’s procedure body,
the compiler
pushes
a new scope.
The scope stack becomes
$[S_1, S_2]$.
The compiler
adds
the parameter $a$
and
the local $y$
to $S_2$,
which becomes
$S_2 = {a, y}$.
State 2: entering Inner.
Upon entering
Inner’s procedure body,
the compiler
pushes
another new scope.
The scope stack becomes
$[S_1, S_2, S_3]$
with
$S_3 = {b, z}$.
References inside
Inner’s body
resolve
against
this combined table.
The reference to $a$
inside Inner
finds
$a$
in $S_2$,
because
$S_3$
does not
contain $a$.
The reference to $y$
finds
$y$
in $S_2$.
The reference to $x$
finds
$x$
in $S_1$.
The reference to $z$
finds
$z$
in $S_3$.
State 3: exiting Inner.
When
Inner’s body
finishes,
the compiler
pops
$S_3$
from
the scope stack.
The scope stack
returns to
$[S_1, S_2]$.
The identifiers
$b$
and $z$
are
no longer
visible
in
subsequent
parsing.
State 4: exiting Outer.
When
Outer’s body
finishes,
the compiler
pops
$S_2$.
The scope stack
returns to
$[S_1]$.
The identifiers
$a$
and $y$
are
no longer visible.
State 5: the main program.
After parsing
the main program body,
the compiler
pops
$S_1$
as well
when
the END.
of
the program
is recognised.
The scope stack
is empty
at
the end of
compilation.
At no point
during
this compilation
does
the symbol table
hold
more than
seven identifiers
total.
The maximum
occurs
during
Inner’s body
compilation,
where
the combined table
holds
${x, a, y, b, z}$
plus
two procedure names
in the enclosing scopes.
The bound
is
independent of
the total size
of
the compilation unit.
The three techniques of the trio, namely fixup tables from article A194, declare-before-use from article A195, and scoped symbol tables from this article, compose to form the complete single-pass implementation of forward-reference handling.
The scoped symbol table provides the current identifier context against which identifier references resolve. The declare-before-use rule ensures that every use finds its declaration in the current symbol table without future lookahead. The fixup mechanism handles the remaining case where an identifier’s declaration exists in the symbol table but its compiled-code address is not yet known, which occurs for forward-declared procedures and mutually recursive definitions.
The three mechanisms share a common structural property. Each maintains a stack-shaped data structure whose depth is bounded by the source’s syntactic nesting depth. The fixup stack in article A194 tracks forward jumps per enclosing block. The scope stack in this article tracks identifier scopes per enclosing lexical scope. The two stacks grow and shrink together under the source language’s block-structured discipline, because scope entries and block entries tend to coincide in Wirth-style languages.
Formally, the total working memory of the compiler for these three techniques is bounded by the sum
\[M_{\text{compiler}} \le M_{\text{fixup}} + M_{\text{symtab}} + M_{\text{other}},\]where $M_{\text{fixup}}$ is bounded by the fixup-buffer nesting bound established in article A194, $M_{\text{symtab}}$ decomposes as $M_{\text{symtab}} = M_{\text{nested}} + M_{\text{top-level}}$ per the earlier section, and $M_{\text{other}}$ is a small constant covering the parser stack, the emission buffer, and other compilation state. The per-declaration working set $M_{\text{fixup}} + M_{\text{nested}} + M_{\text{other}}$ is $O(1)$ in the program size. The accumulating term $M_{\text{top-level}}$ grows with the top-level declaration count, which is bounded per module under separate compilation.
The abstract symbol table described above admits several concrete implementation choices.
Linked list of hash maps. The most direct implementation uses a linked list of hash maps. Each list element represents one scope. The head of the list is the innermost scope. Scope entry prepends a new empty hash map to the list. Scope exit removes the head element. Lookup walks the list from head to tail, returning the first match. This implementation is simple to write and easy to reason about. The lookup cost is $O(d_{\max})$ in the worst case.
Single hash map with scope IDs. A more efficient implementation maintains a single hash map whose keys are identifier names and whose values are stacks of scope-tagged entries. Each entry carries a scope ID that records which scope declared it. Scope entry increments the current scope ID counter. Scope exit iterates the hash map and removes all entries whose scope ID matches the exiting scope. Lookup returns the top-of-stack entry for the queried name. This implementation gives $O(1)$ lookup cost at the cost of slightly more complex scope-exit accounting.
Vector of vectors. The simplest in-arena implementation uses a vector of vectors, with each inner vector holding the identifiers of one scope. Scope entry appends a new empty vector to the outer vector. Scope exit truncates the outer vector. Lookup walks the outer vector from end to start and searches each inner vector linearly. This implementation has $O(d_{\max} \cdot s_{\max})$ lookup cost in the worst case but uses the minimum memory overhead and is straightforward to implement in constrained environments.
The choice among these implementations depends on the compiler’s performance requirements and the source language’s scope characteristics. For most Wirth-style languages, where nesting depth is modest and per-scope identifier count is small, the linked-list-of-hash-maps implementation is adequate.
Contemporary language implementations follow several variants of the scoped symbol table.
Interactive-development-oriented compilers. Compilers designed for interactive development, including language servers, tend to maintain whole-program identifier information in richer data structures that support features like cross-file navigation, find-usages queries, and incremental recompilation. These data structures are larger than the pure single-pass symbol table but still respect the scope-based visibility rules at compile time. The trade-off sacrifices strict single-pass compilation for tooling functionality.
Bytecode-compilation-oriented compilers. Compilers designed for producing bytecode efficiently, including WebAssembly toolchains and embedded scripting compilers, tend to follow the pure single-pass scoped-symbol-table discipline. The compilation cost matters more than the tooling functionality in these contexts. The single-pass discipline delivers the low compilation cost.
Whole-program-optimisation-oriented compilers. Compilers designed for maximum run-time performance, including production GNU Compiler Collection and Clang implementations, use scoped symbol tables during parsing and semantic analysis but retain identifier information in a persistent whole-program abstract syntax tree and intermediate representation beyond parse-time. The persistent structures enable optimisations across scope boundaries at the cost of compilation memory and time. The scope-popping discipline of this article applies to the parse-time symbol table but not to the persistent whole-program structures that these compilers carry forward.
The three categories correspond roughly to the three architectural quadrants of the two-axis design space introduced in article A188. Interactive language servers sit in the pipeline-and-AST-materialised quadrant. Bytecode compilers sit in the integrated-or-pipeline-and-AST-free quadrant. Whole-program optimising compilers sit in the multi-pass-and-AST-materialised quadrant.
The scoped symbol table with the scope-popping discipline completes the foundational trio of single-pass compilation techniques. The compiler’s working memory for identifier tracking is bounded by the source’s scope-nesting depth and per-scope declaration count, which are program-independent under the assumption that these quantities are bounded by the source language’s grammatical constraints or programming discipline. The three techniques, fixup tables, declare-before-use, and scoped symbol tables, compose to maintain the compiler’s total working memory at $O(1)$ in the program size. Article A197 opens the synthesis pair of the series with the comparison of integrated single-pass against decomposed pipeline architectures, using Keleusma’s V0.3.0 self-hosting roadmap as a modern worked example of the decomposed pipeline approach.
An earlier revision of this article stated that the symbol table’s memory footprint was $O(1)$ in the program size via the scope-popping discipline. This was overstated. Scope popping bounds the nested-scope portion of the symbol table but does not bound the outermost scope, which accumulates top-level declarations monotonically as the compiler advances through the source. The corrected framing distinguishes the per-declaration working set, which is $O(1)$ in the program size, from the accumulating top-level environment, which grows with the top-level declaration count and is bounded per module under separate compilation. Article A195 carries a compatible clarification about the monotonic growth of the outermost scope.
]]>The fixup mechanism of article A194 solves the forward-jump problem at the machine-code level. A parallel problem appears at the identifier level. When a source program references a function, a type, or a variable before the referenced entity has been declared in the source text, the compiler faces the same question that the forward-jump problem posed. How can the compiler emit code that references an identifier whose meaning is not yet known?
Two solutions are possible. The first solution is to defer identifier resolution until the whole program has been parsed, at which point all identifiers are known and all references can be resolved. This is the multi-pass discipline. It admits forward references freely at the source level and pays the cost in whole-program state and non-linear compilation that article A189 discussed as the boundary between single-pass and multi-pass architectures.
The second solution is to require the source language to declare every identifier before it is used. The compiler that reads the source in a single forward pass finds each identifier already in its symbol table by the time the identifier’s use appears. Forward identifier references are syntactically forbidden. This is the declare-before-use discipline that Wirth adopted across his language line and that this article develops as the second technique of the trio.
The declare-before-use rule states that every identifier used in a source expression must have been declared at an earlier point in the source file. The rule applies uniformly to types, constants, variables, functions, procedures, and modules. The rule is a lexical property of the source text, not a semantic property of the compiled program. A program that uses an identifier whose declaration appears later in the source is rejected at compile time regardless of whether the program would compute a correct result if the compiler could resolve the reference.
The rule has two variants. The strict variant, applied in Wirth’s Pascal and Modula-2, requires that the declaration precede the use in the source file’s top-to-bottom order. The scoped variant, applied in some later languages, requires that the declaration precede the use within the current lexical scope but permits outer scopes to be extended after inner scopes that reference them are parsed. This article concentrates on the strict variant because it is the version that most directly enables single-pass compilation.
At any point $i$ in the source text, the compiler maintains a symbol table $T_i$ containing all identifiers that have been declared in the source up to position $i$. The current-scope symbol table grows monotonically at the top level as the compiler advances through the source. Formally, for positions $i$ inside the outermost scope,
\[T_i \subseteq T_{i+1} \qquad \text{for all } i.\]Inside nested scopes, the symbol table grows and shrinks as inner scopes open and close. Article A196 describes the scope-popping discipline that returns the nested-scope portion to the enclosing scope’s state whenever an inner block closes. The monotonic growth of this article applies to the outermost scope, which holds top-level declarations and does not pop during compilation.
The declare-before-use rule states that every identifier used at position $i$ is already in $T_i$. The compiler resolves the reference by looking up the identifier in $T_i$. Identifier resolution therefore requires only the current symbol table state, not future symbol table state. Formally,
\[\operatorname{resolve}(\text{id}, i) = \operatorname{lookup}(\text{id}, T_i).\]The resolution cost is the cost of the lookup operation, which depends on the symbol table’s data structure
\[C_{\text{resolve}}(\text{id}, i) = \begin{cases} O(1) & \text{hash-indexed table,} \\ O(\log \lvert T_i \rvert) & \text{tree-indexed table.} \end{cases}\]The lookup does not require inspecting any code past position $i$. The compiler does not maintain any whole-program state to support identifier resolution.
The alternative, under which forward references are allowed at the source level, requires the compiler to either defer resolution until a whole-program pass completes or maintain a pending-reference queue that is checked at each new declaration. Both alternatives add whole-program state and increase per-identifier cost. Neither alternative is compatible with the strict stream-processor discipline that this series develops.
The declare-before-use rule has one immediate consequence that Wirth’s languages address through a syntactic escape hatch. Two procedures that call each other mutually recursively cannot both satisfy declare-before-use. At least one call must precede the callee’s declaration.
Wirth resolved this
by requiring
an explicit
forward declaration
that
introduces
the procedure’s signature
into
the symbol table
before
the procedure body appears.
The syntax
in Pascal
uses the keyword
forward
in place of
the procedure body.
The compiler
treats the forward declaration
as
a promise
that
the body
will appear later
in the source.
The forward-declared procedure occupies one entry in the symbol table starting at the position of the forward declaration. The full body, when it appears, does not add a new symbol table entry. It resolves the forward declaration by providing the compiled code for the promised body.
Formally, the symbol table entry for a forward-declared procedure carries a status flag that distinguishes the two states.
\[T_i(\text{id}) \in \{\text{forward-declared}, \text{defined}\}.\]The status transitions at the point the body appears. A call site that uses the identifier during the forward-declared phase generates a code emission that references the procedure’s future address through the same fixup mechanism that article A194 developed for forward jumps.
Consider two procedures $P$ and $Q$ that call each other mutually. The Pascal code uses a forward declaration to satisfy declare-before-use.
PROCEDURE Q(x: integer); FORWARD;
PROCEDURE P(x: integer);
BEGIN
IF x > 0 THEN
Q(x - 1)
END;
PROCEDURE Q(x: integer);
BEGIN
IF x > 0 THEN
P(x - 1)
END;
The compiler processes this source in four phases.
Phase 1: forward declaration of Q.
The compiler
parses
PROCEDURE Q(x: integer); FORWARD;
and adds
the symbol table entry
The entry carries the procedure’s signature but no compiled address yet. The address will be supplied when the body appears.
Phase 2: body of P.
The compiler
parses the body of $P$.
Inside the body,
the call Q(x - 1)
looks up
$Q$
in the symbol table
and finds it
in
the forward-declared state.
The compiler
emits
a call instruction
with
a placeholder
for
$Q$’s eventual address
and
adds
a fixup entry
for
this placeholder.
At the end of
$P$’s body,
$P$ receives
its own
symbol table entry
in the defined state.
Phase 3: body of Q.
The compiler
parses the body of $Q$.
The call P(x - 1)
looks up
$P$
in the symbol table
and finds it
in
the defined state
because $P$
was compiled
in phase 2.
The call instruction
receives
$P$’s known address
directly,
with no
fixup entry needed.
At the end of
$Q$’s body,
the compiler
resolves
$Q$’s forward declaration
by
supplying
$Q$’s
now-known address.
All fixup entries
that
reference $Q$
receive
their patched addresses.
Phase 4: end of compilation. The compilation completes with both procedures in the defined state and all fixup entries retired.
The example demonstrates that declare-before-use combined with forward declarations handles mutual recursion without any whole-program pass. The forward declaration supplies the promise that the identifier will have a defined meaning by the end of the compilation unit. The fixup mechanism supplies the mechanical bridge between the call site that emits the reference and the definition site that eventually supplies the referent’s address. The two techniques compose naturally.
The declare-before-use rule imposes a non-trivial burden on the programmer. Source code must be organised so that declarations appear before uses. For programs with substantial cross-referencing between procedures or types, the ordering is not always obvious. Programmers must either plan the source layout carefully or use forward declarations liberally.
The Wirth tradition argued that the discipline was itself a design benefit. A source file that respects declare-before-use reads naturally from top to bottom. Every reference resolves against material that the reader has already encountered. The reader never needs to scan forward to find a definition. The argument holds some empirical weight for academic programs and for carefully organised industrial code. The argument holds less weight for rapidly evolving industrial code where programmer time dominates compiler time.
Modern languages
have varied
in
their response
to
this trade-off.
Some languages,
including
Pascal,
Modula-2,
Oberon,
and
older C dialects,
maintain
the strict discipline.
Others,
including
Java,
Python,
Go,
and
modern C dialects
with
appropriate
forward declarations,
allow
scoped forward references
within
compilation units.
Still others,
including
early
functional languages
like ML
and
Haskell,
require
explicit
let rec
or equivalent
constructs
for
mutually recursive definitions
but
otherwise
follow
the declare-before-use rule.
Declare-before-use and fixup tables together form the essential pair of techniques that single-pass compilers use to handle forward references. Fixup tables handle forward references to code addresses. Declare-before-use handles forward references to identifiers. The forward-declaration escape hatch bridges the two mechanisms by allowing an identifier to be introduced into the symbol table before its code address is known, and using the fixup mechanism to resolve the code address when the body appears.
Formally, if $\iota$ is an identifier declared as forward at position $p_1$ and defined at position $p_2$ with $p_1 < p_2$, then between positions $p_1$ and $p_2$, each use of $\iota$ at position $i$ generates a fixup entry with target identity $\iota$,
\[T_i(\iota) = \text{forward-declared} \implies F' = F \cup \{(a_i, \iota)\},\]where $a_i$ is the placeholder position in the emitted output for the reference at $i$. At position $p_2$, when the body of $\iota$ completes emission, the compiler resolves all fixup entries whose target identity is $\iota$ against the body’s start address,
\[\operatorname{resolve}(\iota, a_{\iota\text{-body}}) \quad \text{at position } p_2,\]and the symbol table entry transitions to the defined state.
The composition of the two mechanisms handles mutual recursion in strict single-pass compilation without either whole-program state or non-linear compilation cost.
Contemporary language designers face the same trade-off that Wirth encountered. Strict declare-before-use enables single-pass compilation and simple compiler implementation at the cost of programmer flexibility in source organisation. Free forward references give the programmer freedom at the cost of compiler complexity and compilation-cost linearity.
Several contemporary languages have adopted positions between the two extremes.
Go requires declare-before-use within a single function body but permits forward references between top-level declarations in the same package. The Go compiler performs a preliminary pass over the top-level declarations to collect identifier signatures, then compiles the bodies in a subsequent pass that resolves references against the collected signatures. The overall shape is cheaper than a full multi-pass architecture because the preliminary pass does not need to type-check bodies or emit code.
Rust allows forward references freely within a module. The compiler performs name resolution as a distinct phase before type checking and code generation. Rust’s compilation model does not attempt strict single-pass compilation.
Zig permits declaration order to be arbitrary within a container. Compile-time evaluation uses lazy evaluation with dependency tracking, so that a compile-time expression is evaluated only when its result is needed and after any expressions it depends on have completed. Runtime code is compiled after compile-time evaluation completes for the declarations it depends on.
WebAssembly at the bytecode level uses a declare-signatures-before-any-body discipline. Every function reference in a WebAssembly module uses a function index that must be statically resolvable. The binary format places the function-index space in sections that precede the code section, so all function signatures are known before any function body is verified or compiled. Forward references between functions are resolved through the function index space, which is established before any function body is compiled. This is a specific form of the declare-before-use rule adapted to the requirements of a portable bytecode format.
The declare-before-use discipline extends across compilation-unit boundaries through the mechanism of declaration files. In C and C-plus-plus, declaration files carry the header extension and contain declarations without implementations. A source file that uses symbols defined in another compilation unit includes the declaration file for that unit, which introduces the symbol declarations into the current compilation unit’s symbol table before the uses appear.
Modula-2 formalised this discipline through the definition-implementation module split. A Modula-2 module consists of a definition module and an implementation module. The definition module contains the exported declarations, which is what importing modules see. The implementation module contains the bodies. The two modules are separate files and can be compiled independently. The importer’s compilation depends only on the definition module, not on the implementation module, which allows the importer’s compilation to proceed before the exporter’s implementation is complete.
Oberon inherited this discipline in a simpler form through the exported-marker convention. A single module file contains both declarations and implementations, and exports are marked with a single asterisk after the exported identifier. The compiler, when importing a module, scans the exported declarations without reading the implementations. The imported module’s declarations appear in the importing module’s symbol table before any use.
Modern module systems in Rust, Zig, and recent C-plus-plus modules continue this pattern. Each module publishes a declaration interface that importing modules see. The interface supports declare-before-use across module boundaries without requiring the importer to have access to the exporter’s implementation.
Declare-before-use is the language-design lever that brings identifier resolution into the single-pass discipline. The rule requires the source to declare every identifier before its use, which reduces identifier resolution to a lookup in the current symbol table. Forward declarations provide the syntactic escape hatch that allows mutual recursion without compromising the rule. The combination of declare-before-use, forward declarations, and the fixup mechanism of article A194 handles every forward reference that a block-structured source language admits, without requiring whole-program state or non-linear compilation cost. Article A196 covers the third technique of the trio, namely the symbol table data structure itself and the scope-popping discipline that keeps the compiler’s working memory bounded.
A single-pass compiler
that emits target code
as it recognises source syntax
faces
one fundamental engineering problem
that no amount of
language design can eliminate.
The problem
is the forward jump.
An if statement’s
conditional jump
must skip
past the then branch
when the condition is false,
but
the address
just past the then branch
is not known
at the moment
the conditional jump
is emitted.
The then branch
has not yet
been parsed,
so its length
is not yet known.
The compiler
must
emit the jump
without a known target
and
return later
to fill the target in.
This is the forward-jump problem. It appears in every compiler that emits target code in a forward sweep, regardless of whether the target is stack-based bytecode, register-based bytecode, or native machine code. The classical solution is the fixup table, also called the fixup buffer, the patch table, the backpatch list, or the fixup stack depending on the implementation choice. The article A189 introduced the fixup table briefly in the worked PL/0 example. This article develops the mechanism in full, argues why the fixup table’s size is bounded by the source nesting depth rather than by the program size, and shows how the mechanism generalises to break statements, early returns, and the other non-linear control-flow constructs that block-structured source languages admit.
A fixup table is a data structure that records each emitted-but-not-yet-resolved forward reference. Each entry in the table carries two pieces of information.
The first piece is the location in the emitted output where the target address needs to be written. This is the address of the operand field of the emitted jump instruction, called the placeholder position.
The second piece is the identity of the target that the placeholder will eventually refer to. For block-structured constructs where the target is always “the position just past this block”, the target identity is implicit in the fixup entry’s position on the fixup stack. For more elaborate control flow, the target identity may include a label name or a block-type tag.
Formally, a fixup entry is a pair
\[e = (a_{\text{placeholder}}, \iota_{\text{target}}),\]where $a_{\text{placeholder}}$ is the address of the operand field to be patched and $\iota_{\text{target}}$ is the target identity. For block-structured constructs, $\iota_{\text{target}}$ is often just the enclosing-block-index.
The compiler maintains a fixup buffer $F$ that holds the currently unresolved fixup entries. The buffer supports three operations.
Enqueue. When a forward jump is emitted, a new fixup entry is appended to the buffer. The placeholder position is the current output position plus the operand-field offset within the emitted instruction. The target identity is determined by the syntactic context that emitted the jump. Formally, enqueue transforms the fixup buffer by set union
\[F' = F \cup \{(a_{\text{placeholder}}, \iota_{\text{target}})\}.\]Resolve. When the target address becomes known, the compiler walks the fixup buffer, finds all entries whose target identity matches the newly known address, and patches the placeholder position at each entry with the resolved address. Resolved entries are removed from the buffer. Formally, resolve patches each matching entry against the resolved address $a_{\text{resolved}}$ and then removes them,
\[F' = F \setminus \{ e \in F : \iota_e = \iota_{\text{target}} \},\]where each removed entry $e = (a_e, \iota_e)$ has its placeholder position $a_e$ patched with $a_{\text{resolved}}$ before removal.
Terminate. At the end of compilation, the fixup buffer must be empty. Any remaining entry represents an unresolved forward reference, which is a compile-time error. The check is inexpensive.
if-then-else CompilationConsider the Pascal-style source fragment
IF x > 0 THEN
y := 1
ELSE
y := 2
END
The single-pass compiler processes this fragment in three phases, interleaving code emission with fixup table operations.
Phase 1: parse the condition.
The compiler
parses x > 0
and emits
the load and compare instructions.
At the end of the condition,
the operand stack
carries
a boolean value
representing
the condition’s result.
The compiler
emits
a conditional-jump-if-false instruction
with a placeholder
in its operand field.
Denote the placeholder position
by $a_1$.
The compiler
enqueues
the fixup entry
$(a_1, \text{else-branch})$
into the fixup buffer.
Phase 2: parse the then branch.
The compiler
parses y := 1
and emits
the corresponding
assignment instructions.
No fixup activity
occurs
during this phase
because
the then branch
has no
forward references
of its own.
After emitting
the then branch,
the compiler
emits
an unconditional jump
with a placeholder
in its operand field
to skip
over
the else branch.
Denote this placeholder
by $a_2$.
The compiler
enqueues
$(a_2, \text{end-of-if})$.
At this point, the current output position is the address just past the unconditional jump. The compiler resolves the first fixup entry $(a_1, \text{else-branch})$ by patching the operand field at $a_1$ with the current output position. The first fixup entry is removed from the buffer.
Phase 3: parse the else branch.
The compiler
parses y := 2
and emits
the assignment instructions.
After the else branch,
the compiler
resolves
the second fixup entry
$(a_2, \text{end-of-if})$
by
patching
the operand field at $a_2$
with
the current output position.
The second fixup entry
is removed
from the buffer.
At the end of
the if-then-else compilation,
the fixup buffer
returns to
its state
at the start
of
the construct.
No fixup entries
persist
past
the construct’s closer.
The two fixup entries
introduced
during
the compilation
were
retired
during
the same compilation
in strict
last-in-first-out order.
The if-then-else worked example
generalises
to
arbitrary nesting depth.
Consider
an outer if
whose then branch
contains
an inner if.
When
the compiler
reaches the inner if,
the outer if’s
first fixup entry
is
already in the buffer,
awaiting
its
resolution
at the end of
the outer then branch.
The inner if
adds
its own
fixup entries
on top of
the outer’s,
and
the compilation
proceeds
by
processing
the inner construct
completely
before
returning to
the outer.
The fixup buffer therefore behaves as a stack under block-structured compilation. Fixup entries are pushed when their forward references are emitted and popped when their targets are resolved. The stack discipline is enforced by the source language’s block structure, not by the compiler’s internal choice. A source language that admits block-structured control flow gives the fixup buffer a stack discipline by construction.
The stack discipline combined with the block-nesting bound gives a tight bound on the fixup buffer size. Let $d$ denote the current syntactic nesting depth of the parse cursor. At any point during compilation, the number of active fixup entries satisfies
\[\lvert F \rvert \le c \cdot d,\]where
$c$
is
a small constant
that depends on
the source language.
For Pascal-style
if-then-else
constructs,
$c = 2$
because
each nesting level
contributes
at most two fixup entries,
one for
the false-branch jump
and one for
the end-of-if jump.
For loop constructs
with
break statements,
$c$
may be
somewhat larger
because
multiple break statements
inside
the same loop
each contribute
their own
fixup entry.
The nesting depth $d$ is bounded a priori by the language grammar and by the source program’s maximum nesting. For realistic Wirth-language programs, $d$ rarely exceeds several dozen even for large programs. The fixup buffer size is therefore several orders of magnitude smaller than the program size in typical use. For a concrete estimate, using a typical maximum nesting depth of thirty and a per-nesting-level fixup constant of five,
\[\lvert F \rvert \le c \cdot d \le 5 \cdot 30 = 150 \text{ entries}.\]At eight to sixteen bytes per entry, the fixup buffer occupies under two kilobytes of memory regardless of the source program size. The buffer’s memory footprint is $O(d)$ in the source nesting depth and $O(1)$ in the source program size.
Each fixup entry is enqueued in constant time, resolved in constant time once the target is known, and terminated implicitly when the enclosing block closes. The per-entry cost is
\[C_{\text{fixup}} = O(1).\]The total number of fixup entries generated during compilation is bounded by the number of forward-reference emissions in the source, which is linear in the source program size. The total fixup-management cost is therefore
\[C_{\text{fixup total}}(P) = O(\lvert P \rvert),\]which contributes a constant factor to the linear-in-program-size compilation cost established in article A189.
The fixup mechanism extends straightforwardly to other block-structured control-flow constructs.
while and for loops.
A while loop
emits
a
condition check
followed by
a
conditional-jump-if-false
to a
placeholder
representing
the loop exit.
The body
is emitted next,
followed by
an
unconditional jump
back to
the condition check.
The unconditional jump’s target
is known
at the time of emission,
so no fixup is needed
for the loop-back.
The conditional-jump-if-false
fixup
is resolved
when
the end of the loop
is recognised.
The mechanism
introduces
one fixup entry
per active loop
and
is fully
subsumed by
the block-nesting bound
above.
break and break_if.
A break statement
inside a loop
emits
an unconditional jump
to
a placeholder
representing
the enclosing loop’s exit.
The jump’s target
is not known
at the time
of the break’s
emission,
so a fixup entry
is added
to the buffer.
The fixup entry
carries
a target identity
that names
the enclosing loop
by relative index.
When the enclosing loop’s end
is recognised,
the compiler
resolves
all
break fixup entries
that name
this loop
by
patching
their placeholder positions
with
the loop-exit address.
The number of
break fixup entries
active
at any point
is bounded
by the total number
of break statements
inside
the currently unclosed loops,
which is
in turn
bounded
by
the loop-body size,
which is
finite by
block-structured control flow.
Early returns.
An early return
inside a function
emits
an unconditional jump
to
a placeholder
representing
the function’s exit.
The mechanism
is
identical to
break,
with
the enclosing function
playing the role of
the enclosing loop.
Early returns
typically
add
one fixup entry per early-return statement.
Case and switch dispatch. A dispatch table that selects one of several branches based on a discriminant value emits one conditional jump per case plus one unconditional jump per case to a shared exit placeholder. Each unconditional jump adds a fixup entry that is resolved at the end of the switch. The number of active entries is bounded by the number of cases, which is finite by the source-level construct.
The fixup table mechanism
becomes
substantially more complex
in the absence of
block-structured control flow.
An unstructured source language
with
free
goto statements
and
free
labels
admits
forward references
whose targets
may
lie
across arbitrary code sections.
A fixup entry
introduced
at
one point
in
the source
may remain
in the buffer
until
its target label
is defined,
which may occur
many source lines later
in
an arbitrary part of
the program.
The fixup buffer
under this discipline
does not
behave
as a stack.
It behaves
as a
label-indexed
dictionary
that
grows
with
the number of
outstanding forward-referenced labels.
The bound on active fixup entries under unstructured control flow is therefore
\[\lvert F_{\text{unstructured}} \rvert = O(\lvert L \rvert),\]where $\lvert L \rvert$ is the number of labels in the source program with forward references. This bound is generally proportional to the program size, not to the nesting depth. The fixup buffer is no longer $O(1)$ in the program size under unstructured control flow.
The specific engineering choices that unstructured compilers make include a hash-indexed label dictionary that maps label names to either resolved addresses or lists of pending fixup entries. The dictionary lives outside the stack discipline and grows with the source program. The compilation cost per source line increases compared to the block-structured case because each label reference requires a dictionary lookup rather than a stack index.
Contemporary single-pass compilers implement the fixup table mechanism under several names and with several minor variations.
WebAssembly’s block-structured discipline admits a stack-of-fixup-lists data structure that matches the block-nesting discipline directly. Each block-opener pushes an empty fixup list onto a control stack. Each forward reference within the block appends to the top-of-stack list. Each block-closer pops the top-of-stack list and patches all its entries against the block-closer’s address. Production WebAssembly implementations, including V8 and the WebAssembly Binary Toolkit, use variations of this shape, with implementation-specific details that trade memory footprint against branch-resolution speed. The construction maps one-to-one with the block-structured discipline that WebAssembly’s type system requires.
Coroutine-based embedded scripting bytecodes that adopt block-structured control flow follow the same pattern. The fixup table lives inside the compiler’s arena or heap in a fixed-capacity buffer whose maximum size is a compile-time property of the source language grammar.
Older single-pass compilers in the Wirth line used the same mechanism implemented in their compiler source language. The Wirth Oberon compiler source in Project Oberon implements the fixup mechanism concisely across a small number of Oberon procedures, with the actual line count depending on what is counted as part of the mechanism versus part of the surrounding code-emission scaffolding. The Turbo Pascal implementation of the same mechanism was written directly in eight-oh-eight-six assembly language and remains closed-source, as article A190 discussed.
The fixup table is the essential engineering technique that makes single-pass code emission possible in a source language that admits forward references. The mechanism is straightforward, has a constant per-entry cost, and produces a buffer size bounded by the source nesting depth rather than by the program size, which is $O(1)$ in the program size for block-structured source languages. The technique generalises to break statements, early returns, and case dispatch without substantive modification. Article A195 covers the second foundational technique of the trio, namely declare-before-use ordering and forward declarations for mutual recursion, which together with the fixup mechanism close the forward-reference problem at the identifier level.
The stream-processor discipline has a mathematical structure that is independent of any particular programming language or compiler implementation. The structure belongs to coalgebra, which is the categorical dual of algebra. Where algebra studies operations that combine values of a given type to produce new values of the same type, coalgebra studies operations that decompose values of a given type into observations about them. A stream is the canonical example of a coalgebraic object. A stream processor is the canonical example of a coalgebraic morphism.
This article develops the coalgebraic view of streams, defines productivity as the formal property that distinguishes a productive stream computation from a divergent computation, and shows why block-structured control flow at the bytecode level, which article A192 introduced, places a compiled program inside the decidable subclass of productive stream processors. The exposition follows Jan Rutten’s universal coalgebra treatment and his subsequent coinductive stream calculus. The productivity discussion draws on Abel and Pientka’s copattern framework and Endrullis and colleagues’ decidability result for stream productivity.
A coalgebra over a functor $F$ is a pair $(X, c)$ where $X$ is a set, called the carrier, and $c$ is a morphism
\[c \colon X \to F(X).\]The morphism $c$ decomposes each element of $X$ into its structural components as prescribed by the functor $F$. The direction of the arrow distinguishes coalgebra from algebra. The two settings differ only in the arrow direction
\[\text{algebra:} \quad \alpha \colon F(X) \to X, \qquad \text{coalgebra:} \quad c \colon X \to F(X).\]An algebra of the functor $F$ combines $F$-structured inputs into elements of $X$. A coalgebra of the same functor decomposes elements of $X$ into $F$-structured observations. The dualisation is formal and precise under the categorical framework that Rutten’s universal-coalgebra paper develops.
The functor that describes streams over an alphabet $A$ is
\[F(X) = A \times X.\]A coalgebra for this functor has a decomposition morphism
\[c \colon X \to A \times X,\]which pairs each element of $X$ with a subsequent element of the same type. The two components of the decomposition are conventionally called
\[\operatorname{head} \colon X \to A, \qquad \operatorname{tail} \colon X \to X.\]The head of an element is the observation produced now. The tail is the state after producing that observation. Iterating the decomposition recovers an infinite sequence of $A$-valued observations, which is the stream generated by the coalgebra. Formally, the stream generated by $c$ starting at $x_0 \in X$ is the sequence
\[\operatorname{stream}(x_0) = \operatorname{head}(x_0), \operatorname{head}(\operatorname{tail}(x_0)), \operatorname{head}(\operatorname{tail}^{2}(x_0)), \ldots\]where $\operatorname{tail}^{n}$ denotes the $n$-fold composition of $\operatorname{tail}$.
Rutten’s coinductive stream calculus formulates stream definitions as behavioural differential equations. The stream derivative $s’$ is the tail of $s$, namely
\[s' = \operatorname{tail}(s).\]The initial value $[s]$ is the head of $s$, namely
\[[s] = \operatorname{head}(s).\]A stream is determined uniquely by its initial value and its derivative, in the same sense that a continuous function is determined by its value at zero and its ordinary derivative. This parallel is not merely suggestive. Rutten showed that the algebraic structure of streams over common alphabets, namely the natural numbers and the real numbers, supports a differential calculus that is formally analogous to the classical Newton-Leibniz calculus. The parallel provides a substantial proof technique for stream identities that would otherwise require explicit induction over indices.
Coinductive definition proceeds by specifying the initial value and the derivative of the defined stream in terms of the defined stream itself or other coinductively defined streams. The definition is well-formed if each observation of the defined stream can be computed in finite time from the observations of the streams it depends on. The well-formedness condition is productivity, which the next section formalises.
A stream processor $f \colon A^{\omega} \to B^{\omega}$ takes an input stream over alphabet $A$ and produces an output stream over alphabet $B$. The stream processor is productive if every finite prefix of its output is determined by a finite prefix of its input. Formally, productivity requires a monotone non-decreasing function
\[m \colon \mathbb{N} \to \mathbb{N}\]such that for every input stream $a$ and every $n \in \mathbb{N}$,
\[f(a)[0 \ldots m(n) - 1] \quad \text{depends only on} \quad a[0 \ldots n - 1].\]The function $m$ specifies how much output is determined by how much input. A productive stream processor guarantees that output eventually appears as input arrives. A non-productive stream processor may absorb arbitrarily much input without ever producing any output.
Productivity is the coinductive dual of termination. A terminating function on finite data guarantees that the function eventually returns a value after finitely many computational steps. A productive function on infinite streams guarantees that the function eventually emits an output element after finitely many computational steps. The two properties correspond under the algebra-coalgebra duality.
The coinductive-productivity dual of the well-founded recursion that admits terminating definitions on data is the guarded corecursion that admits productive definitions on codata. A corecursive definition is guarded if the recursive occurrence of the defined stream appears under a stream-constructor prefix that emits at least one output element before the recursive occurrence is reached. The guardedness condition is the syntactic sufficient condition for productivity that most proof assistants in the Rocq and Agda tradition enforce mechanically.
The stream processor $f \colon A^{\omega} \to B^{\omega}$ can be recast as a coalgebra morphism between two coalgebras of related functors.
Let $c_A \colon A^{\omega} \to A \times A^{\omega}$ be the input stream coalgebra, which decomposes an input stream into its head and tail. Let $c_B \colon B^{\omega} \to B \times B^{\omega}$ be the output stream coalgebra. A stream processor that has the productivity function $m$ gives a partial map between the state spaces whose commuting diagram expresses the input-output relationship.
For the simplest case where each input element produces exactly one output element, the productivity function is $m(n) = n$ and the stream processor $f$ respects the coalgebra structure in the sense that processing the head and taking the tail commute up to the appropriate alphabet mapping. The input functor $F_A(X) = A \times X$ and the output functor $F_B(X) = B \times X$ share the shape $F_{\bullet}(X) = \bullet \times X$ with the head type determined by the coalgebra in question.
For the general case where $m$ is non-trivial, the stream processor is a morphism under an appropriate categorical setting that accommodates variable-rate input consumption and output production. The technical machinery required to make this precise involves final coalgebras and natural transformations between the input and output stream functors, which Rutten’s papers develop in detail.
The morphism framing matters for the present series because it places stream processors inside a categorical setting where productivity is a formal property of the morphism rather than an ad-hoc implementation concern. The morphism either is productive or is not, independently of how it is concretely implemented in a particular programming language.
Productivity in general is not decidable. An arbitrary recursively enumerable stream definition may or may not be productive, and no algorithm can decide which, by a straightforward reduction from the halting problem. This is the negative side of the coinductive-productivity story.
The positive side is that suitably restricted subclasses admit decidable productivity. Endrullis, Grabmayer, Hendriks, Isihara, and Klop established in their two thousand ten Theoretical Computer Science paper that productivity is decidable for a broad class of stream definitions whose right-hand sides are built from constructors, projections, first-order functions on the alphabet, and guarded recursion. The decision procedure runs in bounded time as a function of the definition size, with the specific complexity depending on which subclass of the framework is considered.
The class that Endrullis and colleagues consider is strictly larger than the guarded-corecursion class that proof assistants enforce syntactically, but strictly smaller than the class of all recursively enumerable definitions. The class includes most stream definitions that arise in practical stream-processing programs.
Block-structured control flow at the bytecode level, which article A192 developed, places the compiled output of a bytecode program inside this decidable subclass. The argument proceeds as follows.
Block-structured control flow implies finite loop bodies. Every backward edge in the control-flow graph of a block-structured program is a loop-back whose target is the entry point of an enclosing loop. The loop body is finite, because it is the linear span of instructions between the loop entry and the loop end.
Finite loop bodies imply bounded per-iteration work. A single iteration of a loop executes a bounded number of instructions, namely the instruction count of the loop body. This bound holds at compile time and is independent of the number of iterations.
Bounded per-iteration work plus at least one output per iteration implies productivity. Consider a loop whose per-iteration work is bounded by $W_{\text{iter}}$ input elements consumed per iteration and which emits at least one output element per iteration. After $t$ iterations, the loop has consumed at most $t \cdot W_{\text{iter}}$ input elements and produced at least $t$ output elements. This gives a lower bound on the productivity function
\[m(t \cdot W_{\text{iter}}) \ge t,\]which is the productivity condition expressed in terms of the compile-time-known per-iteration bound. The function $m$ is monotone non-decreasing by construction and finite at every argument because $W_{\text{iter}}$ is finite.
The last step
is
the load-bearing one
for
the bytecode-level
productivity guarantee.
Block-structured control flow
alone
does not
guarantee productivity.
A tight loop
that
emits no output
is
still block-structured
but
not productive.
The additional condition
is
that
every loop
contains
at least one yield-like operation
that
produces output.
This condition
can be checked
in the same
single-pass validation sweep
that A192 developed,
by
requiring
each Loop
to
contain
at least one output-producing instruction
on
every control-flow path
from
the loop entry
to
the loop end.
Andreas Abel and Brigitte Pientka, in an ICFP two thousand thirteen paper, developed a unified framework for termination and productivity based on sized types and copatterns. A copattern is a pattern that appears on the left-hand side of a definition in a matching position that observes the codata being defined.
The framework distinguishes data from codata syntactically. Data is defined by constructors that appear on the right-hand side of a definition. Codata is defined by destructors that appear on the left-hand side of a definition, observing the codata being defined. The distinction mirrors the algebra-coalgebra duality at the language level.
Sized types add a size annotation to data and codata types that tracks the number of constructor or destructor applications performed so far. Termination checking for data uses the size annotation to require that recursive calls occur at strictly smaller sizes. Productivity checking for codata uses the size annotation to require that corecursive occurrences appear under a size-increasing observation.
The Abel-Pientka framework provides a formally verified foundation for termination-and-productivity checking in programming languages that distinguish data from codata. The framework has been implemented in the Agda proof assistant and in extensions of other dependent-type-theoretic languages.
For the present series, the Abel-Pientka framework matters because it demonstrates that the data-versus-codata distinction that Turner argued for in his total functional programming manifesto can be mechanically enforced at the language level. A total functional stream processor, which is what the streaming discipline of this series targets, sits at the intersection of Turner’s total functional programming and the coalgebraic stream framework.
The stream-processor compilation discipline of this series is a specific application of the coalgebraic productivity framework to the source-code-to-bytecode transformation.
The input alphabet is the source-code byte alphabet, which comprises the printable characters and whitespace of the source language. The output alphabet depends on the compiler stage. For the lexer, the output alphabet is the token type. For the parser, the output alphabet is the declaration type. For the code generator, the output alphabet is the bytecode-instruction type.
Each stage is a stream processor $f_i \colon A_i^{\omega} \to A_{i+1}^{\omega}$. The composition of the stages
\[f = f_k \circ f_{k-1} \circ \cdots \circ f_1\]is itself a stream processor $f \colon A_1^{\omega} \to A_{k+1}^{\omega}$ that takes source bytes and produces bytecode instructions. The composition is productive if each stage is productive and the productivity functions compose.
Formally, if stage $i$ has productivity function $m_i \colon \mathbb{N} \to \mathbb{N}$, then the composed pipeline has productivity function
\[m(n) = m_k(m_{k-1}(\cdots m_1(n) \cdots)).\]The composed function $m$ is monotone non-decreasing if each $m_i$ is, and finite at every $n$ if each $m_i$ is. This is the formal expression of the intuition that a pipeline of productive stages is itself productive.
The bytecode that a stream-processor compiler emits inherits the productivity property of its source language, provided the target bytecode format preserves the property under execution. Block-structured bytecode formats, including WebAssembly and the coroutine-based embedded-scripting bytecodes of article A197, preserve productivity by construction, because their control-flow constraints place compiled programs inside the decidable subclass of productive stream processors that Endrullis and colleagues characterised.
The coalgebraic-stream framework is a mathematical structure that abstracts away from implementation details. Several concerns that matter for a practical compiler do not appear directly in the framework.
Resource cost of each step. The framework requires that each productivity step be finite, but it does not bound the resource cost of the step. A compiler that takes a large but finite amount of time per output element is productive but not practically useful. Worst-case execution-time analysis, which is a distinct concern, must be layered on top of the productivity guarantee.
Amount of buffering. The framework allows the productivity function $m$ to be arbitrarily slow-growing, which means that arbitrarily much input may be consumed before any output appears. A compiler whose lexer consumes an entire source file before producing its first token is productive in the coalgebraic sense but does not respect the bounded-buffering discipline that the pipeline-of-processes architecture of article A191 requires. Bounded buffering is a stronger condition than productivity and requires additional implementation constraints.
Error handling. The framework assumes that every input produces some output. A real compiler must handle malformed input by producing error diagnostics rather than target code. The stream-processor framework extends to accommodate error handling through sum-type output alphabets that include an error variant, but the extension is a separate technical matter that this article does not develop in detail.
These limitations do not undermine the framework’s value. They locate the framework correctly within the broader theoretical landscape. The coalgebraic productivity framework provides the mathematical foundation for the streaming discipline. Practical compilers build additional guarantees on top of this foundation.
The stream-processor discipline of this series has a formal mathematical foundation in Rutten’s coalgebraic view of streams and its productivity theory. A stream processor is a coalgebra morphism. Productivity is the coinductive dual of termination. Block-structured control flow at the bytecode level places compiled programs inside the decidable subclass of productive stream processors that Endrullis and colleagues characterised. The compilation pipeline composes per-stage productivity into end-to-end productivity by function composition of the per-stage productivity functions. This closes the theory pair of the series and provides the mathematical justification for the design decisions that the earlier articles motivated historically and that the subsequent technique articles will develop implementationally. Article A194 opens the techniques trio with fixup tables and the forward-jump problem that every single-pass compiler must solve.
A bytecode format that permits arbitrary jumps between arbitrary addresses is formally equivalent to a directed graph with no structural constraints. Verifying such a format requires constructing a control-flow graph, computing a fixpoint of the type-consistency relation across every merge point, and iterating until the fixpoint stabilises. The verification cost in the worst case is super-linear in the program size, and the verification algorithm is non-trivial to prove correct.
A bytecode format
that restricts control flow
to block-structured constructs,
namely
matched pairs of
If and EndIf,
Loop and EndLoop,
and Break statements
that reference
enclosing loops
by relative index,
admits
a single-pass validator.
The verifier
walks the instruction stream
once
from start to finish,
tracks
a small amount of state,
and produces
a definitive
accept-or-reject decision
without ever
constructing a graph
or iterating to a fixpoint.
This is
the single-pass-validation property
that WebAssembly
codified
and that
the Haas et al. PLDI two thousand seventeen paper
made
the design canonical
for portable bytecode.
The theory pair of this series opens with block-structured control flow because it is the language-design lever that most directly enables the stream-processor compilation discipline. A compiler that emits block-structured control flow can also verify it in the same forward sweep. A compiler that emits arbitrary jumps cannot. The productivity guarantee that article A193 develops in coalgebraic terms would not apply to the compiled output of an unstructured-control-flow language, even if the compiler itself were structured as a stream processor.
A block-structured instruction stream consists of sequential instructions interspersed with block delimiters that come in matched pairs. The typical block-delimiter set comprises
If and EndIf,
optionally with a middle Else,
for conditional execution.Loop and EndLoop
for iteration.Break and BreakIf
for early exit from
enclosing loops.Every If
in a well-formed stream
matches
exactly one EndIf.
Every Loop
matches
exactly one EndLoop.
Every Break
appears
lexically inside
the loop
it exits.
The matching
respects nesting,
so the innermost
unclosed opener
matches
the next closer,
and the openers
form a stack
of currently open blocks.
The block structure
is a property
of the instruction stream itself,
not of the source language.
A source language
that admits
arbitrary
goto statements
can be compiled
to a block-structured target
only by
restricting the source
to eliminate
unstructured control flow,
or by
introducing
auxiliary state variables
that simulate
the unstructured jumps
inside a structured envelope.
The second option
is
formally possible
in theory,
following the classical
result of
Böhm and Jacopini,
but produces
inefficient
target code
in practice.
Most block-structured bytecode formats
therefore
constrain
the source languages
they target
rather than
simulating
unstructured control flow.
A bytecode verifier is a program that inspects a bytecode module and accepts or rejects it based on whether the module satisfies a set of structural safety properties. The typical properties include the following.
Type consistency. Every instruction that consumes values from a stack finds values of the correct type. Every instruction that produces values onto a stack leaves them in the state that subsequent instructions expect. At every merge point in the control-flow graph, the type-stack states arriving from different predecessors agree.
Control-flow integrity. Every jump target is a valid instruction address within the module. Every conditional branch falls through or takes its jump based on the runtime value of its condition, but never transfers control to an out-of-bounds address or into the middle of another instruction.
Resource bounds. The verifier may impose additional constraints on the maximum stack depth, the maximum local-variable count, or the memory footprint of the module. These constraints support resource-bounded execution in embedded or safety-critical contexts.
The verifier’s problem is to decide whether a given module satisfies these properties. The cost of the decision procedure depends on the shape of the input’s control flow.
For a block-structured instruction stream of length $n$, the verifier walks the stream once from position zero to position $n$. At each position, the verifier maintains three pieces of state.
At each instruction,
the verifier
updates these three pieces of state
in constant time
based on
the instruction’s opcode
and its operands.
An arithmetic instruction
consumes a fixed number of stack entries
and produces
a fixed number.
A block opener
pushes an entry onto $B$
and increments $d$.
A block closer
pops the matching entry
from $B$
and decrements $d$.
A Break
references
a relative loop index
that the verifier
resolves
against $B$
by counting from
the top of the stack.
The correctness invariants enforced at each step include the following.
Block-nesting invariant. At every position, the block-nesting stack matches the sequence of opened but unclosed blocks in the instruction stream:
\[B_{\text{at } i} = \operatorname{stack}(\text{unclosed openers in } [0, i)).\]Type-consistency invariant. At every position, the type stack matches the actual stack state that any execution reaching this position would produce:
\[S_{\text{at } i} = \operatorname{type-of}(\text{runtime stack at } i).\]Block-closer type-check. At every block closer, the type stack at the closer matches the type stack at the corresponding opener, adjusted for the block’s declared type signature. Formally, for a block opened at position $o$ with declared type signature $\sigma_{\text{in}} \to \sigma_{\text{out}}$ and closed at position $c$, the type stacks at the two positions share a common base $R$ and differ only in the top portion:
\[S_o = R \cdot \sigma_{\text{in}}, \qquad S_c = R \cdot \sigma_{\text{out}}.\]The base $R$ records the values on the stack below the block’s operating region, which the block’s execution does not touch.
The three invariants compose. An instruction that satisfies them locally at position $i$ produces a state that satisfies them at position $i+1$ by the same local check. The single-pass sweep verifies each position’s local check in constant time. The overall verification cost is therefore
\[C_{\text{verify}}(\text{module}) = O(n),\]where $n$ is the total instruction count. This is the best possible asymptotic cost for any verification algorithm that must inspect every instruction at least once. No fixpoint iteration is required because block-structured control flow has no non-trivial merge points. Every merge is the join at a block closer, and the type states that merge there are determined by the block’s declared type signature, not by iterative propagation.
WebAssembly is the modern canonical example of a block-structured bytecode format designed for single-pass validation. The Haas et al. two thousand seventeen paper documents the design rationale. The paper received the Programming Language Design and Implementation Distinguished Paper Award of that year in part for the observation that block-structured control flow enables single-pass validation with linear cost.
WebAssembly’s block constructs
include
block,
loop,
if with optional else,
br and br_if
for structured branches,
br_table for
switch-style multi-way branches,
and the closing end marker
for each opener.
The br instruction
takes
a relative label index
that references
an enclosing block by nesting position,
matching
the abstract structure
described above.
The type of each block
is declared
at the opener,
which fixes
the type-stack signature
that the block closer
must produce.
The declared type
serves as
the contract
that the verifier
uses to reject
non-conforming block bodies
in a single sweep.
Reference implementations
of the WebAssembly single-pass validator,
including
the wasm-validate tool
in the WebAssembly Binary Toolkit
and the type-checker portion
of the specification’s
reference interpreter,
run on the order of
one to three thousand lines
of source code
depending on
the target language
and the metrics used.
Comparable Java Virtual Machine
bytecode verifier implementations,
in production runtimes,
run substantially larger.
The size difference
reflects
the algorithmic simplification
that block structure enables.
A verifier
that does not construct
a control-flow graph,
does not iterate to fixpoint,
and does not resolve
polymorphism through subtyping
is
fundamentally smaller than
one that must do all three.
Block-structured control flow excludes several patterns that are common in unstructured bytecode formats.
Unconditional jumps
to arbitrary addresses.
An unconditional
goto instruction
that transfers control
to an arbitrary target
cannot be verified
in a single sweep
because
the verifier
would need
to know
the type-stack state
at the target
before it has been
computed.
Backward jumps that are not loop-backs. A backward jump that lands outside the current loop would require the verifier to re-examine the intervening code under the potentially different type-stack state that arrives from the new source. Block-structured formats prohibit this by construction.
Fallthrough between distant labels. Labels outside the block structure, which appear in switch-fallthrough patterns in C and in computed-goto patterns in some Fortran dialects, cannot be expressed in the block-structured discipline without introducing an auxiliary state variable.
Exception handlers that catch across arbitrary regions. Exception handling in Java and dot-NET uses try-catch blocks whose scopes may nest arbitrarily but whose control flow crosses the block boundaries in ways that break the block-structured discipline. WebAssembly addressed this by adding a structured exception mechanism as a language extension rather than as an unstructured jump.
These prohibitions are sometimes cited as limitations of block-structured formats. The alternative view, which WebAssembly’s success supports, is that the prohibited patterns are themselves liabilities that the bytecode format should not encourage, because their absence is what makes the linear-cost verifier possible. The trade-off is real but substantially favours the block-structured side for the portable-bytecode use case.
The Java Virtual Machine bytecode format predates the block-structured design and permits arbitrary control flow. The Java bytecode verifier therefore must construct a control-flow graph and iterate a type-consistency computation to fixpoint across every merge point.
The complexity of the Java bytecode verifier is higher than the WebAssembly verifier by a factor that depends on the number of merge points and the depth of the type subtyping lattice. In the worst case, the fixpoint iteration can require a number of passes proportional to the depth of the subtyping lattice. For a module with $n$ instructions and $m$ merge points, the Java bytecode verifier’s worst-case cost is on the order of
\[C_{\text{JVM verify}} = O(n \cdot m \cdot h),\]where $h$ is the maximum subtype-lattice height encountered during verification. This is an order-of-magnitude estimate rather than a tight complexity bound, because the actual verifier uses dataflow analysis whose per-instruction cost depends on implementation details. The order-of-magnitude claim is that the cost is super-linear in $n$ under the fixpoint discipline, which the block-structured verifier avoids by construction. For programs that use deep class hierarchies, $h$ is not small. Expressed as a ratio against the block-structured baseline, the multiplicative overhead of unstructured control flow is
\[\frac{C_{\text{JVM verify}}}{C_{\text{verify}}} = O(m \cdot h),\]which is the cost paid for admitting arbitrary jumps in the bytecode format.
The Java bytecode verifier has also had a significant number of security-relevant bugs over its history. These bugs partly reflect the intrinsic difficulty of implementing the fixpoint algorithm correctly. A verifier whose algorithm is a single forward sweep has fewer opportunities to introduce subtle bugs. The block-structured discipline therefore has a security advantage that is independent of its performance advantage.
Conrad Watt’s two thousand eighteen mechanised WebAssembly specification in the Isabelle proof assistant provides a machine-checked proof that the WebAssembly type system is sound. The proof establishes that well-verified WebAssembly modules cannot get stuck during execution because of a type error. This is a soundness guarantee that the informal specification alone does not provide.
The formal statement that Watt proved is the type-soundness property. For every module $M$ and every execution trace $\tau$ of $M$,
\[\operatorname{verify}(M) = \text{accept} \implies \neg \operatorname{stuck}(\tau).\]A verified module does not encounter a type error at run time. The predicate $\operatorname{stuck}$ denotes runtime execution states that cannot make progress because of a type mismatch between an instruction and the values on the stack.
Watt’s work also identified several issues in the official WebAssembly specification that had been missed by the human reviewers. The identified issues were subsequently corrected in the WebAssembly standard. The Watt result demonstrates two properties that matter for the theory pair of this series.
The first property is that formal verification of a block-structured bytecode specification is feasible with current mechanization tools. The Watt proof required substantial effort in the Isabelle framework, as any mechanization of a production specification does, but the effort was within reach of a single-researcher project. The comparable mechanization of an unstructured bytecode would require substantially more work because the fixpoint algorithm would need to be formalised as well.
The second property is that the WebAssembly specification, even after careful human review, contained subtle errors that a mechanization uncovered. This finding is a datum in favour of the position that critical specifications should be mechanized. For safety-critical or security-critical bytecode, mechanization should be part of the specification process, not an optional post-hoc verification exercise.
Block-structured control flow predates WebAssembly by decades. The Wirth-line languages of article A189 used block-structured control flow at the source level for the same reasons that make it attractive at the bytecode level. Wirth’s target machines for Oberon also used block-structured control flow at the instruction-set-architecture level, avoiding arbitrary jumps in the compiled output. The WebAssembly design made the bytecode-level block-structure discipline explicit and universal for a modern portable-bytecode format.
Several subsequent bytecode formats have adopted the block-structured discipline.
The pedagogical tradition around compiler construction has been slower to adopt the block-structured framing at the bytecode level. Standard textbooks still present bytecode verification in the Java Virtual Machine style with control-flow graph construction and fixpoint iteration as the default. The block-structured alternative, which is substantially simpler to teach and to implement, merits more prominent treatment in future editions.
Block-structured control flow at the bytecode level enables single-pass validation with cost linear in the instruction count. The alternative, arbitrary control flow, requires a control-flow-graph construction and a fixpoint iteration whose cost is super-linear in the instruction count and whose implementation has historically suffered from subtle bugs. The WebAssembly design codified the block-structured discipline for the modern portable-bytecode use case and received substantial academic recognition for the observation that the language-design constraint enables the verification-cost reduction. Conrad Watt’s mechanized proof of WebAssembly soundness provides formal-methods evidence that the design is sound and that the specification is mechanizable. Article A193 develops the mathematical framework that formalises the streaming discipline itself, in coalgebraic terms, and shows why block-structured control flow is the natural syntactic constraint that places a compiled program inside the decidable subclass of productive stream processors.
Per Brinch Hansen took a different route to the stream-processor compilation discipline than Wirth or Hejlsberg. Where the Wirth line and Turbo Pascal built the compiler as an integrated single-pass parser that emits target code inside its recursive-descent procedures, Brinch Hansen decomposed the compiler into a pipeline of independent processes communicating through bounded queues. The lexer, the parser, the semantic analyser, and the code generator each ran as a separate process. Each process consumed an input stream from its upstream neighbour and produced an output stream for its downstream neighbour. No process held the entire program. No process communicated with any process other than its immediate neighbours.
The pipeline architecture mattered for two reasons. The first reason was decomposition-for-testability. Each stage could be tested in isolation by driving it with a synthesised input stream and inspecting the output stream. The second reason was that the pipeline shape mapped directly onto the concurrent-programming discipline that Brinch Hansen had been developing since Concurrent Pascal in nineteen seventy-five. The compiler was itself a demonstration of the concurrent-programming techniques that Brinch Hansen was advocating for other domains.
Per Brinch Hansen was born in Denmark in nineteen thirty-eight and worked on operating systems and concurrent programming across a career spanning four decades. His work falls into three overlapping periods.
The first period, from the late nineteen sixties into the early nineteen seventies, covered the RC four thousand multiprogramming system at the Danish company Regnecentralen and the Operating System Principles textbook of nineteen seventy-three. This period established Brinch Hansen as one of the founders of the discipline of concurrent programming.
The second period, from the mid nineteen seventies through the mid nineteen eighties, covered the design and implementation of Concurrent Pascal in nineteen seventy-five, the Solo operating system written in Concurrent Pascal in nineteen seventy-six, Distributed Processes as a language design in nineteen seventy-eight, Edison as a teaching language in nineteen eighty-one, and the Brinch Hansen on Pascal Compilers book in nineteen eighty-five. The Pascal Compilers book is the primary source for the pipeline-of-processes compiler architecture.
The third period, from the late nineteen eighties into the two thousands, covered SuperPascal as a language for parallel scientific computing and its associated compiler tooling. Brinch Hansen retired from academic teaching in two thousand four and died in two thousand seven.
The pipeline-of-processes compiler architecture is best documented in the nineteen eighty-five book and best demonstrated in the SuperPascal compiler of the mid nineteen nineties. The rest of this article concentrates on those two artefacts.
Understanding Brinch Hansen’s compiler architecture requires some context on Concurrent Pascal. Concurrent Pascal extended Wirth’s Pascal with two concurrent-programming primitives. The first primitive was the process, a program unit with private state and its own thread of control. The second primitive was the monitor, a shared abstraction that mediated access to shared state through mutual-exclusion methods. The two primitives together allowed a Concurrent Pascal program to be structured as a collection of processes communicating through monitors.
The Solo operating system, written in Concurrent Pascal in nineteen seventy-six, consisted of multiple processes communicating through a set of monitors. The kernel of the operating system was reportedly small, on the order of a few thousand lines of Concurrent Pascal source in the shipped implementation, though the exact size varies across the reported figures. The system was tractable to reason about because each process held only its private state and communicated only through explicit monitor calls.
Brinch Hansen carried this decomposition discipline into his compiler design. In the nineteen eighty-five book the compiler is presented as a series of passes, each of which reads its input as a stream and writes its output as a stream, with the passes executed sequentially in the shipped implementation but decomposed conceptually as if they were independent processes. The passes communicate through buffered streams, not through whole-program intermediate files, which is what places the design inside the stream-processor discipline even absent concurrent execution.
The fully concurrent form, in which the passes run as actual concurrent processes communicating through bounded first-in-first-out channels, matured in SuperPascal in the mid nineteen nineties. The SuperPascal design made explicit the bounded-buffer inter-process communication model that the earlier work had presented in latent form. Each channel had a fixed capacity. A process that produced faster than its consumer could accept would block until buffer space was available. A process that consumed faster than its producer could supply would block until an item was available. The pipeline was back-pressured throughout. No process could run away from its neighbours. The two forms, the sequential pass-based form of the earlier work and the concurrent process-based form of SuperPascal, share the same architectural properties because the pass-based form is a scheduling of the process-based form onto a single processor.
A Brinch Hansen compiler consists of four or five stages arranged as a linear pipeline. The stages and their inter-stage streams follow this shape, with the dependency running left to right.
source bytes
↓
[ lexer ] ── tokens ──> [ parser ] ── declarations ──> [ semantic ] ── typed IR ──> [ codegen ] ── bytecode
The lexer consumes source bytes from a file input and yields tokens. The parser consumes tokens and yields top-level declarations one at a time. The semantic analyser consumes declarations and yields type-checked declarations with resolved identifier references and computed types. The code generator consumes type-checked declarations and yields target bytecode one chunk at a time.
Each stage holds only its local working state plus the inter-stage buffer that connects it to its downstream neighbour. The lexer’s working state is a small character buffer sufficient to recognise a single token. The parser’s working state is a bounded parse stack whose depth is the current syntactic nesting depth plus the fixup buffer for the currently parsed declaration. The semantic analyser’s working state decomposes into a nested-scope portion holding the type-environment for the current lexical scope, which is bounded by scope-nesting depth, plus a small buffer for the currently checked declaration, plus the accumulated top-level environment that resolves cross-declaration references. The top-level portion grows with the top-level declaration count and is bounded per module under separate compilation. The code generator’s working state is a small emission buffer.
The unit of work that transits each buffer depends on the buffer’s position in the pipeline. The lexer-parser buffer carries tokens, each a fixed-size record. The parser-semantic buffer carries a single top-level declaration per item. The semantic-codegen buffer carries a type-checked declaration per item. The codegen output carries bytecode chunks. Because the buffer capacity is fixed at compile time, the total memory footprint of all inter-stage buffers is a compile-time constant.
The number of pipeline stages $k$ is also a design-time constant. For a Brinch Hansen Pascal-family compiler, $k$ is typically four, lexer, parser, semantic analyser, and code generator, or five when a separate optimisation stage is introduced. The stage count does not depend on the program being compiled. Formally,
\[k = O(1) \quad \text{in } \lvert P \rvert.\]This property is what allows the memory bound to be decomposed into a program-independent per-declaration portion plus a per-stage top-level environment portion whose growth scales with the top-level declaration count rather than with the total program size.
The pipeline architecture gives a compile-time bound on the compiler’s total working memory that is tighter and simpler than the integrated-single-pass bound. Denote by $k$ the number of pipeline stages, by $m_i$ the maximum working state of stage $i$ during execution, and by $b_i$ the capacity of the inter-stage buffer between stage $i$ and stage $i+1$. The total compiler memory footprint is bounded by
\[M_{\text{compiler}} \le \sum_{i=1}^{k} m_i + \sum_{i=1}^{k-1} b_i.\]The inter-stage buffer sum is bounded at compile time by the pipeline design choice of buffer capacities. Each per-stage state $m_i$ decomposes into a per-declaration working set bounded by scope-nesting depth and buffer capacities, plus a top-level environment portion where the stage must resolve cross-declaration references, which grows with the top-level declaration count. For the semantic analyser stage in particular, the top-level environment portion is not program-independent. The compiler memory footprint therefore satisfies
\[M_{\text{compiler}} = M_{\text{per-declaration}} + M_{\text{top-level}},\]where $M_{\text{per-declaration}} = O(1)$ in the program size by construction and $M_{\text{top-level}}$ grows with the top-level declaration count, bounded per module under separate compilation.
This is the same order-of-growth pattern as the integrated-single-pass compiler of the Wirth line, where the same decomposition holds for the same reasons, but achieved through a different mechanism. The integrated single-pass compiler achieves the pattern by holding the parse stack, symbol table, and fixup buffer in a single memory region. The pipeline compiler achieves the pattern by giving each stage its own local region and connecting them through fixed-capacity buffers.
Two additional properties of the pipeline follow from the fixed-capacity buffers.
The throughput of the pipeline is limited by the throughput of the slowest stage. Denote by $T_i$ the maximum throughput of stage $i$, measured in units of work per second. The overall pipeline throughput is
\[T_{\text{pipeline}} = \min_{i \in \{1, \ldots, k\}} T_i.\]The upstream stages that can produce faster will block on their output buffers whenever the slowest stage falls behind. The downstream stages that can consume faster will block on their input buffers for the same reason. Balancing the throughput of the stages is a first-order design concern for a Brinch Hansen compiler.
The latency from the start of source input to the end of bytecode output is bounded by the sum of the maximum per-stage buffering delays. For a pipeline where each buffer holds at most $b$ units and the slowest stage processes one unit in time $\tau$, the maximum end-to-end latency is on the order of
\[L_{\text{pipeline}} \le (k - 1) \cdot b \cdot \tau.\]For small $k$ and small $b$, which is the design regime Brinch Hansen advocated, the latency is small enough that an interactive user does not perceive a delay between typing a compile command and seeing the first output.
The pipeline shape also admits concurrent execution of the stages on multiprocessor hardware. Under sequential execution, compiling a unit of work requires visiting every stage in turn, so the per-unit cost is the sum of the per-stage inverse throughputs. Under concurrent execution, the stages overlap in time, so the per-unit steady-state cost is the inverse throughput of the slowest stage. The speedup ratio is
\[\text{speedup} = \frac{T_{\text{concurrent}}}{T_{\text{sequential}}} = \frac{\sum_{i=1}^{k} T_i^{-1}}{\max_{i} T_i^{-1}}.\]For a well-balanced pipeline of $k$ stages with equal per-stage throughput, the speedup approaches $k$, which is the theoretical maximum for a linear pipeline. The speedup degrades in proportion to the imbalance between stages, reaching one when a single slow stage dominates the whole pipeline. Balancing the pipeline is therefore a first-order concern for pipeline compilers that intend to exploit concurrent execution.
SuperPascal
appeared in
nineteen ninety-three
as an extension of Pascal
for parallel scientific computing.
It added
three parallel-programming primitives
to the Wirth Pascal base language,
namely
parallel blocks
that execute their statements concurrently,
forall loops
that execute their iterations concurrently,
and channels
that transmit typed values
between concurrent components.
The channel primitive
was the direct descendant
of the bounded queues
that Brinch Hansen
had been using
in his compiler designs
for a decade.
SuperPascal provided a language runtime in which the pipeline-of-processes shape that Brinch Hansen had been advocating could be expressed directly. The SuperPascal compiler itself was implemented in standard sequential Pascal, specifically the ISO Level 1 dialect, rather than in SuperPascal. The Brinch Hansen tradition therefore did not achieve a self-hosted stream-processor compiler in the strict sense. It did establish the pipeline architecture and the language design in which such a compiler could later be written.
The pipeline architecture is formally the same shape as the Wirth Oberon result but achieved through a different discipline. Wirth’s Oberon uses an integrated single-pass style, with the compiler written as a large recursive-descent parser that emits target code during parsing. Brinch Hansen’s tradition uses a pipeline of concurrent processes, with each process written as an independent module whose input and output are typed channels. The two approaches occupy opposite ends of the two-axis design space introduced in article A188.
The SuperPascal paper, published in Concurrency Practice and Experience in nineteen ninety-four, documents the language design and reports per-stage throughput measurements. The paper is short by present-day standards, about twenty printed pages, and remains the most accessible primary source for the pipeline-of-processes compiler architecture in the Brinch Hansen tradition.
The pipeline-of-processes compiler architecture occupies the pipeline-and-abstract-syntax-tree-free quadrant of the two-axis design space introduced in article A188. The Wirth line and Turbo Pascal occupy the integrated-and-abstract-syntax-tree-free quadrant. The multi-pass abstract-syntax-tree-materialising tradition represented by GNU Compiler Collection and Clang occupies the multi-pass-and-abstract-syntax-tree quadrant. The pipeline-and-abstract-syntax-tree quadrant is populated by compilers that materialise an AST between the parser stage and later stages but still stream the AST through without ever holding the whole-program tree.
Brinch Hansen’s compilers demonstrate three architectural properties that the Wirth and Hejlsberg integrated compilers do not demonstrate as cleanly.
Independent testability of stages. A pipeline compiler can be tested one stage at a time by driving each stage with a synthesised input stream and inspecting its output stream. The test harness does not need to run the full pipeline to exercise any individual stage. An integrated compiler does not admit this factoring because the stages are inlined into the same procedure call chain.
Compositional reasoning about resources. The pipeline compiler gives a per-stage resource bound that composes to a total resource bound by summation. Reasoning about the pipeline’s total memory footprint reduces to reasoning about each stage’s local footprint plus the buffer capacities. The integrated compiler gives a single global resource bound that combines all working state in one region, which is harder to decompose for verification purposes.
Pipeline parallelism as a design option. The pipeline compiler admits concurrent execution of the stages on multiprocessor hardware without any change to the compiler’s source code. The concurrent execution does not improve throughput past the slowest-stage limit, but it does overlap the pipeline stages in time, which reduces end-to-end latency for interactive compilation. The integrated compiler cannot exploit multiprocessor hardware in this way because its stages are causally serial within the recursive-descent parse.
Brinch Hansen’s compiler architecture influenced several subsequent traditions.
The concurrent-language tradition that Brinch Hansen founded carried the pipeline-of-processes discipline forward into occam in the mid nineteen eighties, into Erlang in the late nineteen eighties, and into Go’s goroutines and channels, which were announced by Google in late two thousand nine and reached the one-point-zero release in two thousand twelve. Each of these languages adopts the bounded-buffer inter-process communication model that Concurrent Pascal and later SuperPascal codified. A compiler written in any of these languages in the pipeline-of-processes shape inherits the Brinch Hansen architectural pattern whether or not the compiler author consciously references the tradition.
The compiler-construction pedagogy tradition was more resistant to Brinch Hansen’s approach. Standard compiler textbooks in the nineteen nineties and two thousands, including Aho, Sethi, and Ullman’s Compilers, Principles, Techniques, and Tools and the successive editions that added Lam as an author, adopted the multi-pass abstract-syntax-tree tradition that Brinch Hansen was arguing against. The pipeline architecture persisted in a small academic subculture around concurrent-programming pedagogy but did not displace the AST-based standard treatment.
The coroutine-based
embedded-scripting tradition
of the twenty tens and twenty twenties
returned
to the pipeline model
under different terminology.
A compiler
written as
a series of loop functions
or coroutines
that yield to one another
through typed values
is
formally the same shape
as the Brinch Hansen pipeline,
with
yield playing the role
of a bounded-buffer send
and resume playing the role
of a bounded-buffer receive.
The coroutine implementation
does not require
concurrent execution,
but it preserves
the pipeline’s compositional properties
under sequential execution.
This variant
appears in
several present-day
resource-bounded scripting languages
and is treated
in more detail
in article A197.
Per Brinch Hansen demonstrated that a stream-processor compiler can be built as a pipeline of concurrent processes rather than as an integrated single-pass parser. The pipeline shape gives compositional resource bounds, independent per-stage testability, and optional pipeline parallelism that the integrated shape does not admit. The SuperPascal self-hosting result demonstrates the pipeline architecture at its greatest ambition, namely a stream-processor compiler implemented in a stream-processor language. The pattern has modern realisations in the concurrent-language tradition of Erlang and Go and in the coroutine-based embedded-scripting tradition that subsequent articles in this series will treat. Article A192 opens the theory pair of the series with block-structured control flow and its role in single-pass validation.
Two corrections apply to earlier revisions of this article.
The SuperPascal self-hosting claim. An earlier revision stated that the SuperPascal compiler was itself implemented in SuperPascal source, which would have placed the Brinch Hansen tradition at a self-hosting endpoint analogous to Wirth’s Oberon result. This claim was incorrect. The SuperPascal compiler and its interpreter were implemented in standard sequential Pascal, specifically the ISO Level 1 dialect, per the published record and the archived source maintained by the classic-tools SuperPascal repository. Brinch Hansen’s tradition established the pipeline architecture and the language design that made a self-hosted stream-processor compiler architecturally natural, but did not close the self-hosting endpoint. Article A199 carries the corresponding correction to the series’s self-hosting summary.
The pipeline total working memory bound. An earlier revision stated that the compiler memory footprint $M_{\text{compiler}}$ was $O(1)$ in the program size, and understated the semantic analyser’s working state as the type-environment for the current lexical scope plus a small per-declaration buffer. This was overstated. The semantic analyser stage must hold the accumulated top-level environment to resolve cross-declaration references, so that stage’s state grows with the top-level declaration count. The corrected framing decomposes $M_{\text{compiler}}$ into a per-declaration working set that is $O(1)$ in the program size plus a top-level portion that grows with the top-level declaration count and is bounded per module under separate compilation. Article A196 carries the analogous correction in the symbol table treatment.
]]>