How writing tests improved write-tight.

August 18, 2022

From the archive. I wrote this in 2022 for an earlier version of this site. It is kept as it was then, apart from the layout and a few links.

Summary

Write-tight is a Python project that helps to improve your writing. During development the project completely stopped working as intended. By writing tests I was not only able to fix the issue, but the overall design of the project improved as well.

Besides improving your project this post also argues that writing tests will make you a better programmer in general.

Write-tight is a Python project that helps to improve your writing.

Write-tight is also my first serious software project and the reason why I have not written a single post the past few months. After spending a lot of time studying web development, regular expressions, and (business) writing, I wanted to combine and apply the lessons learned from each topic.

A screenshot of the write-tight browser user interface.

More information on the project and how to use it can be found on Github. Since my goal was to share write-tight with the world, I wanted to make sure the project worked as I intended. Unfortunately this was not the case.

Write-tight did not work as intended.

To understand what was wrong with write-tight you need to first understand the four general steps to go from input to output. Note that these steps refer to the CLI version of write-tight instead of the user interface version.

  1. Extract text from a HTML page
  2. Apply the rule-based patterns on the text
  3. Update the text in case of match
  4. Return the text as HTML

My thought process was to run each pattern on the text and save all the matches in a list. To make this process more efficient I would remove duplicate matches by turning the list into a set. Finally I would search for every unique match and update the HTML with <span> elements, which in turn would be color highlighted. In pseudo code:

text = "It was a lovely party, wasn't it?" # text parsed from HTML body

patterns = [pattern_a, pattern_b, ..]
matches = []
for pattern in patterns:
    matches.append(pattern.findall(text))

matches = list(set(matches))
for match in matches:
    text.repl(match, <span class="color-me">match</span>)

return html_wrapper(text)

The code worked fine until Github pages references were inserted into the HTML file. On closer inspection the ambiguous pronouns pattern matched ‘it’ in ‘github’ and ‘write’ in the url.

ambiguous_pronouns = re.compile(r"\b(it|that|there|these|those|this)\b",
                                flags=re.IGNORECASE)

<link rel="stylesheet" href="https://ebolle.github.com/write-tight/styles.css">

The matches in ‘github’ and ‘write’ did not make sense. The \b word boundaries should skip these words, right? Since it wouldn’t be the first time I misunderstood a regular expression, I started to doubt my knowledge. Luckily, there was one way to test my knowledge while improving the overall quality of the project at the same time: to write tests.

Writing tests stimulated my thought process.

According to The Pragmatic Programmer - in my opinion one of the best books on programming - testing is not about finding bugs. Instead, the authors believe the major benefit of testing happens when you think about and write the tests, not when you run them. Since it is good practice to include actual bugs in your tests I wrote the following test.

import pytest

@pytest.mark.parametrize(
"test_input, expected",
[
    ("https://ebolle.github.io/write-tight/styles.css", []),
    ("git it thatthere this THOSE", ["it", "this", "THOSE"]),
],
)
def test_ambiguous_pronouns_word_boundaries(test_input: str, expected: str):
    assert re.findall(ambiguous_pronouns.pattern, test_input) == expected

Both tests passed, which means the pattern returns an empty list [] given the Github URL as input. After running the tests I soon figured out I made a big mistake in my thought process. By first getting all the matches, then removing all the duplicates, and then replacing the matches I ignored the index of the match.

text = "Github is truly great is it not?"
matches = ["it"] # this is correct

for match in matches:
    text.repl(match, <span class="color-me">match</span>)

print(text)
G<span class="color-me">it</span>hub is truly great is <span class="color-me">it</span> not?

After catching the mistake I completely re-thought the approach of match and replace, which significantly improved the project. Not only did the new approach need less than half of the lines of code, the code also ran faster and was easier to understand.

def match_and_replace(self, html_content: str) -> str:
    return re.sub(self.pattern, self.add_span_element, html_content)

def add_span_element(self, match: re.Match[str]) -> str:
    match_str = match.group()
    return f"<span class='{self.name}'>{match_str}</span>"

By writing tests I realized I made a mistake in my thought process. This insight eventually led to better code, and even to an overall better design. Testing and software design go hand in hand. In general, good software design makes code easier to test. In a future post I will dive deep into the topic of software design, and how it helped make write-tight more robust and easier to test.

Writing tests will make you a better programmer.

Writing a test is the ultimate way to get feedback about your understanding of the code. For write-tight I wrote the input and expected output of every pattern. Thinking about the expected output before running the test is a great mental exercise. More than once this exercise led me to re-write the pattern before even running the tests.

On a final note I would not have been able to thoroughly analyze my thought process if it wasn’t for version control. Only three years ago an experienced Data Science trainer urged me to learn git. Until this day I am grateful for that advice. Even for small personal projects I can recommend to use version control and branching strategies, and this will be the topic of my next post.

No sponsored or affiliate links. If something here is new to you, the link is an easy way in.