hnq.me
all concepts
language Updated 2026-07-09 Step 1 of 6

Lexer

A beginner-friendly guide to understanding lexers and tokens.

On this page 2026-07-09
The Simple Idea Why Lexers Exist What A Token Is A Tiny Example Lexer vs Parser What Lexers Usually Handle Good Lexer Design Where Lexer Fits In The Pipeline KitJS And Kitwork Notes My Learning Notes Common Misunderstandings Related Concepts

A beginner-friendly guide to understanding lexers: the first stage that turns raw source text into meaningful pieces.

Before code can become a tree, bytecode, or runtime behavior, something has to read the characters.

That first reader is usually the lexer.

source text -> lexer -> tokens

Table of Contents

The Simple Idea

A lexer turns characters into tokens.

Characters are raw.

Tokens are meaningful pieces.

The lexer looks at source text and says:

this is a name
this is a number
this is a string
this is an operator
this is punctuation
this is the end

It does not need to understand the whole program yet.

It only needs to recognize the pieces.

Why Lexers Exist

Source code is written for humans.

It has spaces, comments, names, symbols, strings, and visual structure.

A parser does not want to deal with every individual character directly.

The lexer creates a cleaner input:

characters -> tokens

That lets the parser focus on structure:

tokens -> AST

Without a lexer, every later stage has to keep asking low-level questions:

  • is this character part of a name?
  • is this quote starting a string?
  • is this = assignment or comparison?
  • did the file end inside a string?
  • is this whitespace meaningful?

The lexer answers those questions first.

What A Token Is

A token is a meaningful piece of source text.

Common token types:

  • identifier,
  • keyword,
  • number,
  • string,
  • operator,
  • punctuation,
  • newline,
  • end of file,
  • invalid character.

Example tokens:

identifier: total
operator: =
number: 42
keyword: const
punctuation: ;

The exact token types depend on the language.

A tiny expression language may need only a few.

A JavaScript-like language needs more.

A Tiny Example

Source:

const total = price * quantity

Tokens:

keyword: const
identifier: total
operator: =
identifier: price
operator: *
identifier: quantity
end: EOF

The lexer does not decide what the whole line means.

It simply gives the parser a stream of pieces.

The parser can then ask:

is this a declaration?
what is the variable name?
what is the expression on the right side?

Lexer vs Parser

The lexer asks:

what are the pieces?

The parser asks:

how do the pieces fit together?

That separation keeps both parts smaller.

For example, the lexer can recognize price, *, and quantity.

The parser can decide that those tokens form a multiplication expression.

The lexer sees words and symbols.

The parser sees grammar.

What Lexers Usually Handle

A lexer often handles:

  • whitespace,
  • comments,
  • identifiers,
  • keywords,
  • numbers,
  • strings,
  • escape sequences,
  • operators,
  • punctuation,
  • line and column positions,
  • end-of-file markers,
  • invalid characters.

Some lexers are simple.

Some become tricky because of:

  • nested comments,
  • template strings,
  • regular expressions,
  • indentation-based syntax,
  • unicode identifiers,
  • ambiguous operators,
  • error recovery.

Even a small language needs a careful lexer.

Small mistakes here become confusing errors later.

Good Lexer Design

A good lexer should be:

  • predictable,
  • boring in the best way,
  • easy to test,
  • clear about errors,
  • careful with positions,
  • strict about what the language accepts.

Position tracking matters because users need useful error messages.

This is not helpful:

syntax error

This is better:

line 12, column 18: unexpected character "@"

The lexer is often where good diagnostics begin.

Where Lexer Fits In The Pipeline

The lexer is the first stage of a language pipeline:

source -> lexer -> tokens -> parser -> AST -> compiler -> bytecode -> VM

It is small compared with the whole system.

But it shapes everything after it.

If the token stream is clean, the parser can stay clean.

If the lexer is vague, the parser becomes noisy.

KitJS And Kitwork Notes

KitJS let me think about behavior from the HTML side. Kitwork made me go lower, all the way to text.

The lexer is where source code first stops being a string and starts becoming structured material. It breaks the file into tokens so the parser and compiler can understand it.

That taught me that a runtime begins much earlier than execution. It begins when the system decides how to read code.

My Learning Notes

I used to treat lexing as a boring pre-step.

Then I learned that the lexer is the first contract of a language.

Before a compiler can reason about meaning, the language must decide what its pieces are.

While learning from Kitwork's compiler pipeline, this became clearer:

raw text is not a program yet
tokens are the first form of understanding

That changed how I look at language design.

A lexer is not the exciting part of a runtime.

But it is the first place where the runtime's language becomes real.

Common Misunderstandings

"A lexer understands the program."

Not really. It recognizes pieces. The parser builds structure.

"Lexing is always easy."

Simple lexers can be easy. Real languages often have difficult edge cases.

"Whitespace never matters."

Sometimes it does. Some languages use whitespace as structure.

"The parser can just handle everything."

It can, but separating lexing and parsing usually makes the system easier to reason about.

"Tokens are just strings."

Tokens usually carry type, value, and position. That extra information matters.

Previous: vm

Next: ast

Related: compiler, bytecode, runtime

Search