How does a syntax highlighter work?

February 9, 2022

From the archive. I wrote this in 2022 for an earlier version of this site. It is kept as it was then, apart from the layout and a few links.

Summary

Syntax highlighting is key to make code examples easy to read and understand. This website uses Prism.js, a lightweight and very popular syntax highlighter build with JavaScript.

Since code examples are an important part of my posts, I wanted to figure out the inner workings of Prism.js.

A high-level understanding.

Before we dive into the low-level details we need to understand what happens when we apply the highlighter. According to the documentation we need to do 4 things to make Prism.js work.

1. Download the JavaScript and CSS files for our language(s) of choice
2. Add the link references to our HTML file
3. Wrap the code blocks in <pre> and <code> elements
4. Add a “language-xxxx” class to the <code> element

My approach is to add the same code twice to this HTML page, once without adding the “language-xxxx” class, and once including. By inspecting the developer tools we will learn the differences in HTML. For this post I will use Python as the example language.

import seaborn as sns
tips = sns.load_dataset("tips")

As expected the code is not highlighted, but the content does contain a border and is very small. This is caused by the default settings of the <pre> element. Let’s add the “language-python” class to the <code> element.

import seaborn as sns

tips = sns.load_dataset("tips")

The code has come alive! If you look closely you will see that not only “import”, “as”, and “tips” are highlighted, but also “.”, “=”, and “()”. Furthermore, the background of the highlighted code turned light gray. What are the changes in the HTML?

<pre>
    <code>
      import seaborn as sns tips = sns.load_dataset("tips")
    </code>
</pre>

<pre class="language-python">
    <code class="language-python">
      <span class="token keyword">import</span>
      seaborn
      <span class="token keyword">as</span>
      sns tips
      <span class="token operator">=</span>
      sns
      <span class="token punctuation">.</span>
      load_dataset
      <span class="token punctuation">(</span>
      <span class="token string">tips</span>
      <span class="token punctuation">)</span>
    </code>
  </pre>

Although the code block input is similar, the HTML completely changed.

• The <pre> block now also contains the “language-python” class
• Some elements of the code have been wrapped in <span> elements
• All these elements contain the “token” class
• The meaning of these elements is added as a class (e.g., “keyword”)

To conclude our high-level understanding, it appears that Prism.js works as follows.

1. Identify the code via the “language-xxxx” class
2. Scan the code and look for language specific elements
3. Add a <span> element to each recognized element
4. Add a “token” class to each recognized element
5. Add a class with meaning to each recognized elements
6. Link these classes to Prism.css

Now that we have an idea of what happens, we need to figure out how this happens.

Focus.

To get a low-level understanding of Prism.js we will dive into the source code. If you want to follow along make sure you download the development version. This version contains comments and sensible variables names. The minimized version is optimized to load as quickly as possible, thereby leaving out any comments and using letters as variable names.

The development script contains ~1300 lines of code if you save it with the default settings of the VSCode prettier extension. To not get lost in too much detail I focused only on the essential elements to make highlighting work. This results in 4 topics.

1. Retrieve the <code> elements and extract the textContent
2. Tokenize the textContent
3. Stringify the tokenStream to HTML and insert in into the HTML document
4. Link the inserted HTML to the CSS classes in prism.css

Each topic has its own section where we dive into the code and implementation details. Besides learning how Prism.js works we will also learn how it is able to generalize to so many different programming languages (over 100+ and counting), and how we can modify the code to our wishes.

Retrieve the <code> elements and extract the textContent.

This is the easiest part of the process. Since we add a “language-xxxx” class to the <code> element, we are able to use the querySelector method to retrieve the element of interest. Note that in Prism.js the querySelectorAll method is used since it expects more than 1 code element. However, we will keep things simple and stick to the code example from the start.

const codeElement = document.querySelector(".language-python");

The codeElement is an object that contains a lot of attributes about the HTML element. For syntax highlighting we are only interested in the textContent attribute, which is a string and looks as follows.

"\n            import seaborn as sns \n      \n            tips = sns.load_dataset(\"tips\")\n      "

Tokenize the textContent.

To better understand the process of Tokenization, let’s start with its definition. From Wikipedia we get “Tokenization is the process of demarcating and possibly classifying sections of a string of input characters.” In our case, the textContent is the string of input characters. To me, this definition is not immediately clear, so let’s jump to the output and see if we can understand it better by reverse engineering to the textContent input.

0: "\n "
1: Token {type: 'keyword', content: 'import', alias: undefined, length: 6}
2: " seaborn "
3: Token {type: 'keyword', content: 'as', alias: undefined, length: 2}
4: " sns \n \n tips "
5: Token {type: 'operator', content: '=', alias: undefined, length: 1}
6: " sns"
7: Token {type: 'punctuation', content: '.', alias: undefined, length: 1}
8: "load_dataset"
9: Token {type: 'punctuation', content: '(', alias: undefined, length: 1}
10: Token {type: 'string', content: '"tips"', alias: undefined, length: 6}
11: Token {type: 'punctuation', content: ')', alias: undefined, length: 1}
12: "\n "
length: 13

The tokenize function has 2 parameters: text and grammar. Text refers to the textContent, but grammar is a new term.

In contrast to tokenization, Wikipedia has an easier to understand definition of (lexical) grammar that explains what we are trying to achieve. If you want to better understand grammar, my advice would be to read the full article (3 minutes), but hereby the most relevant parts for this post.

In computer science, a lexical grammar is a formal grammar defining the syntax of tokens. .. The lexical grammar lays down the rules governing how a character sequence is divided up into subsequences of characters, each part of which represents an individual token. This is frequently defined in terms of regular expressions. .. Further, certain sequences are categorized as keywords - these generally have the same form as identifiers (usually alphabetical words), but are categorized separately; formally they have a different token type.

In Prism.js, the implementation of grammar is an object which holds keys that refer to a specific token type (e.g., keyword, function, Boolean, ..), and the values are regular expressions which hold the pattern to correctly identify the token type.

..
  'function': {
    pattern: /((?:^|\s)def[ \t]+)[a-zA-Z_]\w*(?=\s*\()/g,
    lookbehind: true
  },
  'class-name': {
    pattern: /(\bclass\s+)\w+/i,
    lookbehind: true
  },
  'punctuation': /[{}[\];(),.:]/
..

If you are not familiar with regular expressions, just view them as a pattern which we can try to match on any sequence of characters in a string. In our case, the string is the textContent, and this string is analyzed character for character. If the regular expression matches, a Token object is created with the identified token type (the keyword in grammar), the content, and the length of the character sequence. See the output at the beginning of this section for a visual representation.

The output of the tokenize function is either a Token or a so called tokenStream. This tokenStream provides a Token object for each recognized token type in the grammar. In the next section we will learn how this output is used to generate actual HTML code.

Please note that topics like tokenization, grammar, and regular expressions are very vast topics, which each deserve multiple blog posts on their own. Since the goal of this post is for you to understand how syntax highlighting works, and not how you can build one yourself, a thorough explanation is out of scope.

Stringify the tokenStream to HTML code and insert in into the HTML document.

In this section we will look at the stringify function, which takes a Token or Tokenstream as input, and a language. We are already familiar with the Token, and the language simply refers to the programming language we use. In our case, this is Python. Like the previous section, we will start with the output of this function.

<pre class="language-python">
    <code class="language-python">
      <span class="token keyword">import</span>
      seaborn
      <span class="token keyword">as</span>
      sns tips
      <span class="token operator">=</span>
      sns
      <span class="token punctuation">.</span>
      load_dataset
      <span class="token punctuation">(</span>
      <span class="token string">tips</span>
      <span class="token punctuation">)</span>
    </code>
</pre>

As you might have guessed, this is the same output as the HTML code we inspected in the introduction. Now that we know the output, let’s try to understand how it got this format given the tokenStream as input.

There are 2 parts in the stringify function of importance.

• Create a new object by unpacking a Token, and adding HTML elements to it
• Returning a string with HTML code based on this new object

var env = {
  type: o.type,
  content: stringify(o.content, language),
  tag: "span",
  classes: ["token", o.type],
  attributes: {},
  language: language,
};

return (
  "<" +
  env.tag +
  ' class="' +
  env.classes.join(" ") +
  '"' +
  attributes +
  ">" +
  env.content +
  "</" +
  env.tag +
  ">"
);

In our example, attributes stays empty, resulting in the HTML code you saw in the beginning of this section and in the introduction. To activate this string, all we need to do is assign it to the innerHTML attribute of the codeElement.

const codeElement = document.querySelector(".language-python");

// A lot of code as discussed in the previous sections

const stringifyOutput = stringify(tokenizeOutput, "python");
codeElement.innerHTML = stringifyOutput;

Although we covered a lot of ground, we are not done yet. Although the HTML is now ready, to actually change the style of the HTML elements we need to link the classes to the Prism CSS file.

Link the inserted HTML to the CSS classes in Prism.css.

For the last time let’s take a look at the actual code we want to highlight.

import seaborn as sns

tips = sns.load_dataset("tips")

We know from our analysis that import and as are both marked with a token and keyword class. When we search for these classes in the Prism.css file we get the following code.

.token.atrule,
.token.attr-value,
.token.keyword {
  color: #07a;
}

Unfortunately the syntax highlighter is not smart enough to depict the actual color, but the color code “#07a” matches the color of the keywords. Although this is not the most complex part of understanding syntax highlighting, it does reveal how Prism.js is capable of supporting so many languages.

When you download the code with multiple languages selected, you will see the same keys in the grammar object for each object. The only difference are the values. This is possible since many programming languages have similar building blocks like:

• Comments
• Strings
• Functions
• Booleans
• ..

Now that we understand how it works, let’s manipulate Prism.js to our liking. Instead of using the default colors, let’s use the colors from a Jupyter notebook. With the help of developers tools I managed to retrieve the colors for a keyword, function, and variable, with the following result.

Although I personally think this looks a lot better, it’s not trivial to change the color codes of each specific language to your liking. Hence, for now I will stick with the defaults.

Wrap up.

Hopefully this post (partly) demystified the inner workings Prism.js. If you want to know more about the low-level details I strongly suggest you dive into the source code yourself, or watch this Youtube video about the inner workings of compilers.

Interestingly, a lot of terms and concepts used by compilers, are also used by Prism.js. Maybe the usage of these robust programming ideas is what makes Prism.js such an effective tool, which in turn explains its popularity. Regardless, I like the functionality and enjoyed diving into the details.

No sponsored or affiliate links. If something here is new to you, the link is an easy way in.