Improve your writing with regular expressions (part 1).

April 29, 2022

From the archive. I wrote this in 2022 for an earlier version of this site. It is kept as it was then, apart from the layout and a few links.

Summary

This is the first in a series of posts where regular expressions are leveraged to become a better writer in a business context. Based on expert resources on regular expressions and business writing I combine these topics and show how you can improve any written document.

By focusing on clear and concise writing we improve our authority as writers, thereby increasing the chance that our readers will act upon our written proposals.

Preliminary

Basic knowledge of Python and regular expressions is assumed. An introduction on using regular expressions in Python can be found on Medium.

Improving your writing is important.

Analyzing data usually results in a proposal for change, examples are:

Change is hard. Most people don’t like being told that their current way of working is wrong or inefficient. Therefore, when you communicate a written proposal of change it needs to be as concise and clear as possible. There cannot be any reason for the receiver of your proposal to ignore you, or all the hard work analyzing the data has been for nothing.

Although I only use the proposal for change as an example, better writing helps you for any written document in a professional context. From emails to powerpoint presentations, and from technical documentation to your personal blog. The definition of better writing we use for this post is to write clear and concise, and to say exactly what you mean with precision and power. This writing style is sometimes referred as ‘tight writing’, which gives your document authority and thereby increases the chance of success.

There are a few resources I frequently use in this post so I opt to list them here and refer to them in the rest of the post via superscripts0.

  1. Business writing & technical writing immersion by Paul Siegel (Udemy course)
  2. Write tight by William Brohaugh
  3. Everybody writes by Ann Handley
  4. Mastering regular expressions by Jeffrey E.F. Friedl
  5. The official re Python package documentation

Writing is sometimes referred as a fluent process where writers experience a ‘flow’ in which they write page after page. This begs the question: how can a technical and systematic subject like regular expressions help to become a better writer?

The biggest part of writing is post-writing.

The business writing course1 is taught by a mechanical engineer. He explains a ten-step process on how to write clear and concise business documents in 20% to 40% less time. To achieve this efficiency you need to divide your writing in three parts:

The percentages indicate time spend on each part. Pre-writing is the part where you analyze your future document and ask questions like:

Pre-writing scopes your document and therefore your research, saving you valuable time. It also gives focus on your audience and the goal of the document. The pre-writing phase results in a sentence outline which contains all the points you want to make in the correct order. The sentence outline serves as the skeleton of your document.

Next comes writing the draft which speaks for itself, followed by the post-writing part. Most of your time is spend here and this is where regular expressions come into play. Each of the aforementioned resources123 state that there are patterns in your writing that are indicators of ‘deadwood’. Deadwood is the unnecessarily difficult, long, or simply unnecessarily phrases or words that clog the arteries of professional writing2. Deadwood is the enemy of writing tight, and if we are able to detect and eliminate deadwood, our document will become more concise and clear.

Two patterns to improve our writing.

In this first post of the series two patterns will be discussed. The first pattern reveals the structure of the document which is part of the pre-writing phase. The second pattern is meant to eliminate deadwood.

Each pattern adheres to a specific structure so that you can easily understand the regular expression and apply it on your own documents.

At this point you might wonder if it wouldn’t be easier to use Microsoft Word or another word processing tool instead of regular expressions to improve your writing. For simple tasks like finding words that contain ‘ly’, sure. But what if you want to extract each two word combination of which the first word is a ‘to be’ word, and the second word ends on ‘ing’? That is a complex task for your typical word processing tool, but doable with regular expressions. In addition, regular expressions are expressed as code which means we can re-use, automate, and combine the regular expressions. Combining the regular expressions into one program will be the topic of a next series of posts, but first we need to write them.

Extract the first sentence of each paragraph.

The final step in the pre-writing phase is to create a sentence outline. A sentence outline consists of the points that you want to make in your document in the correct order. To be effective, key sentences need to be short - between three and ten words - and consist of one- and two syllable words that clearly state each point. Moreover they need to be either an assertion or generality. A good sentence outline helps to efficiently write a first draft and provides structure and a clear goal to your document. Every first sentence of a paragraph should be a key sentence.

The example text to demonstrate the regular expressions is taken from my last post on Syntax highlighting. Be aware that I write my posts directly in HTML in VSCode to see the structure of the text, (sub) headings, images, lists, and code blocks. Furthermore it prevents me from getting distracted by potential spelling errors. Spelling checks are performed after writing the draft. The consequence of this way of working is that the example text is raw .html. However, with a few tweaks you should be able to apply the regular expressions on other types of files.

<p class="post__header">Focus</p>

<p class="post__paragraph">
    To get a low-level understanding of Prism.js we will dive into the source
    code. If you want to follow along make sure you download the development
    version. This version contains comments and sensible variables names. The
    minimized version is optimized to load as quickly as possible, thereby leaving
    out any comments and using letters as variable names.
</p>

<p class="post__paragraph">
    The development script contains ~1300 lines of code if you save it with the
    default settings of the VSCode prettier extension. To not get lost in too much
    detail I focused only on the essential elements to make highlighting work.
    This results in 4 topics.
</p>

The HTML structure allows me to use subheadings as key sentences. Therefore I will provide two solutions to extract they key sentences. One which extracts the subheadings, and one that extracts the first sentence of each paragraph.

import re

post__header_pattern = re.compile(r'<p\s+class="post__header">(.*?)</p>', flags=re.IGNORECASE)

for match_object in re.finditer(post__header_pattern, text):
    print(match_object.group(1))
Focus

As basic knowledge of Python and regular expressions is assumed, I will not dive into details of the inner workings of the builtin regular expression library in Python. However, I will explain every detail that I believe is important to easily modify and apply the regular expression for your own texts.

The raw string notation (r'') keeps the regular expression sane. Regular expressions are a language of their own and the implementation in Python uses strings to build them. This adds a layer of complexity since you not only have to think about the regular expression, but also about the meaning of that regular expressions as a string in Python. For example, without raw string notation to escape one character you need to write four escape characters '\\\\'.5

Next we match to the opening tag of the paragraph with the ‘post__header’ class. '\s+' is a combination of special characters that ‘match one or more unicode whitespace characters’. In HTML, at least one space between the element and the class name is mandatory.

Next we capture the subheading by using parentheses around the main part of the regular expression '(.*?)'. In combination with the final part '</p>' this means ‘match any character except a newline zero or more times in a non-greedy fashion until you reach the closing tag of the paragraph’. By default special characters like ‘*’, ‘+’, and ‘?’ are greedy, which means they try to match as many text as possible.5 By combining them with ‘?’ they become non-greedy, matching as few text as possible.

By using the brackets we capture the match in a group. The re.finditer method returns an iterable of re.Match objects which in turn hold the groups in case of a match. The result in this example is ‘Focus’, which is too short and neither an assertion or generality. Hence, this post__header should be re-written. Before moving to the next regular expression we only need to discuss the flags=re.IGNORECASE part.

Flags modify the behavior of the regular expression. In this case re.IGNORECASE performs case-insensitive matching which allow to write more concise expressions. For example, the start of the regular expression without this flag would be '<[pP]\s+' since HTML does not care about case sensitivity. Other useful flags will be discussed shortly. Let’s finish this section by extracting the actual first sentence of each paragraph.

first_sentence_pattern = re.compile(r'<p\s+class="post__paragraph">(.*?\.\s+)', flags=re.IGNORECASE|re.DOTALL)

for match_object in re.finditer(first_sentence_pattern, text):
    print(match_object.group(0))
To get a low-level understanding of Prism.js we will dive into the source code.

The development script contains ~1300 lines of code if you save it with the
default settings of the VSCode prettier extension.

The part that is not grouped - not enclosed in parentheses - is previously discussed. The grouped part '(.*?\.\s+)' is new. It means ‘match any character except a newline zero or more times in a non-greedy fashion until you match a literal dot '\.' with one or more whitespace characters following '\s+'’. Without the whitespace character after the literal dot the first match would stop after ‘Prism’.

The re.DOTALL flag modifies the '.' special character to match any character including a newline. As a result the regular expression tries to match across multiple lines instead of the default line by line. Without this flag there are no matches.

Words that end on ‘ly’.

The third type of wordiness as explained in Write Tight2 is ‘The empty’. Empty modifiers can sap power from your words, examples are:

These words can add pure deadwood and are indicators or wordiness. Therefore, we want to inspect each sentence that contains a word ending on ‘ly’ to validate their presence in our documents. To match the full sentence is complex, so I will explain this regular expression in steps, starting with matching any word that ends on ‘ly’.

ly_pattern = re.compile(r'\w+ly\b')

for match_object in re.finditer(ly_pattern, text):
    print(match_object)
quickly
only

The 'w\' special character tries to match most characters that can be part of a word in any language. This means the alphabet, numbers, and the underscore. For this example we call this special character ‘the word character’. Given that definition the regular expression tries to match “a word character one or more times followed by ‘ly’ at the end of the word. The '\b' special character matches the empty string but only at the beginning or end of a word. This special character is sometimes called a word boundary.

We get the right words, but we need context to evaluate the words. The sentence is the context we want to match. The next step is to match every word that follows the word ending on ‘ly’.

ly_pattern = re.compile(r'\w+ly\b.*?[\.!?]', flags=re.DOTALL)

for match_object in re.finditer(ly_pattern, text):
    print(match_object.group(0))
quickly as possible, thereby leaving out any comments and using letters as variable names.

only on the essential elements to make highlighting work.

The new part of the regular expression is '.*?[\.!?], re.DOTALL'. After matching a word ending on ‘ly’ this new part tries to ‘match any character including a newline zero or more times until you match either a dot, exclamation mark, or question mark '[\.!?]'’. The last part between brackets is called a character class and adds another layer of functionality to the regular expression. Within the brackets, characters can have a different meaning, just like the special characters. In this case, it means match either the dot, exclamation mark, or question mark since these characters are indicators of the end of a sentence.

The last part where we try to match all the words in the sentence before the word that end on ‘ly’ is tricky. So tricky that I opted for another approach. This is a wise lesson from the book on regular expressions4: keep an open mind for easier solutions if necessary. In this case, I found that it’s easier to use a regular expression to remove all HTML tags, another regular expression to extract all sentences, and then to filter the sentences that contain ‘ly’.

tag_removal_pattern = re.compile(r'<.*[^>]>')

for match_object in re.finditer(tag_removal_pattern, text):
    print(match_object.group(0))
<p class="post__header">Focus</p>
<p class="post__paragraph">
</p>
<p class="post__paragraph">
</p>

The tag removal pattern tries to match ‘a opening tag followed by any character except a newline or a closing tag ([^>]) zero or more times until it finally reaches a closing tag for each row in the text’. The ‘?’ is left out since we want this pattern to be greedy as it includes the full post__header. Next we remove these tags with the re.sub method.

tagless_text = re.sub(tag_removal_pattern, '', text)

With the tags removed we only need to write a regular expression that is able to extract each sentence, and write a list comprehension to extract all sentences that contain ‘ly’.

sentence_pattern = re.compile(r'[A-Z].+?[\.!?]\s{1}', flags=re.DOTALL)

all_sentences = re.findall(sentence_pattern, tagless_text)

[sentece for sentece in all_sentences if 'ly' in sentece]
['The\n  minimized version is optimized to load as quickly as possible, thereby leaving\n  out any comments and using letters as variable names.\n',
 'To not get lost in too much\n  detail I focused only on the essential elements to make highlighting work.\n']

The sentence pattern regular expression tries to match ‘a capital letter followed by one or more characters including a newline (re.DOTALL) until it matches either a period, exclamation mark or question mark, followed by exactly one whitespace character’.

The two extracted sentences can be re-written for clarity. In the first sentence ‘quickly as possible’ can be replaced with ‘fast’. In the second sentence ‘only’ can be left out since ‘focused’ already implies a selected number of items. Hence, this regular expression is valuable to improve my writing.

More patterns to improve your writing.

In this post we started to become better writers by using regular expressions in Python. We examined the post__headers and every first sentence of each paragraph to validate structure, and we examined words that end on ‘ly’ to examine and eliminate deadwood. Both regular expressions provided useful insights. In the next post I am going to discuss three new patterns.

All three patterns contribute to clear and concise writing, thereby enhancing your authority as a writer, and increasing the chances that your reader will act upon your document. See you in the next post.

No sponsored or affiliate links. If something here is new to you, the link is an easy way in.