Improve your writing with regular expressions (part 2).

May 24, 2022

From the archive. I wrote this in 2022 for an earlier version of this site. It is kept as it was then, apart from the layout and a few links.

Summary

In this second and last part of the series I explain and motivate three more regular expression patterns that will improve your business writing.

We start with detecting passive voice and explain why you should almost always prefer active voice. Next we learn how to avoid subjunctive mood since it is often a source of ambiguity. Finally we learn how to detect and remove ambiguous pronouns to avoid confusion.

Preliminary

Basic knowledge of Python and regular expressions is assumed. An introduction on using regular expressions in Python can be found on Medium. Although recommended it is not necessary to read part 1 of this series first. To make sure we are all on the same page we start with a recap of the previous post.

Recap of part 1

In the previous post we discussed two patterns to become better writers:

The first pattern allows you to easily verify that each first sentence of each paragraph is a written as a ‘key sentence’. The second pattern identifies potential ‘deadwood’. Deadwood is unnecessarily difficult, long, or simply unnecessarily phrases or words that clog the arteries of professional writing. This is the opposite of what we want to achieve: to write concise and clear, also known as writing tight.

In this post we are going to discuss three new patterns:

For each regular expression pattern we will first discuss how the pattern can help you improve your writing. Next, we will discuss the regular expression pattern. Finally, we will apply the regular expression pattern to a sample text for better understanding.

Active voice ensures clarity and achieves great precision.

According to the Udemy course on business writing we should write 90% of our documents in active voice. Active voice forces you - the writer - to write clear and specific words and sentences, thereby increasing your authority. Furthermore specific details make your documents more lively and interesting. Let’s look at an example.

The software was updated yesterday.

This sentence is passive and not specific. To make it specific we can ask: ‘who updated the software?’. The answer can change the sentence from passive to active voice.

The development team updated the software yesterday.

This sentence is active, clear and more detailed. The Udemy course on business writing describes three steps to eliminate passive voice.

We recognize passive voice by a ‘to be’ verb (is, are, was, ..) followed by another verb. An example is ‘was updated’. This pattern of two words can be (partially) expressed as a regular expression.

Find and mark passive voice with regular expressions.

First we need to find a ‘to be’ verb. When found we need to extract this verb together with the following word. We then need to check if this following word is also a verb. Then, to be able to apply steps two and three of eliminating passive voice, we also need context, in this case the full sentence.

Before we start with the first step I will introduce the sample text. This text is based on one of my earlier posts and is slightly modified to include matches for all three patterns.

text = """
My general approach to learning new things is reading several recommended books on the topic and go from there.
However, HTML and CSS are easier to learn via an interactive online course, since you get immediate feedback
from the browser and hands-on experience.

After some research I stumbled upon a high-quality Udemy course on building responsive websites with HTML and CSS,
taught by Jonas Schmedtmann2. The depth of the course, the calm and knowledge of the instructor,
and the large number of hands-on projects stood out for me. It was simply an amazing course.

Although I can recommend this self-study approach, I must stress that only studying is not enough
to become skilled. Practical experience is crucial. Also, it took me 2 months to complete this course
in my spare time, so be ready to make a serious commitment if you decide to follow this path.
So why should you care about HTML and CSS?

Together with JavaScript they form the 3 languages of the internet. Having a solid understanding of how
the internet works can be a great advantage for any data professional,
and it will open doors in the digital analytics world. Solid knowledge of HTML and CSS also gives you the
power to bring ideas alive by creating tangible prototypes of websites and apps.

During the course I was learning a lot about the intricacies of HTML and CSS.
"""

The regular expression pattern to find the ‘to be verb’ and the following word:

passive_voice_pattern = re.compile(r'\b(am|are|is|was|were|been|being)\b\s{1}(.+?)\b', flags=re.IGNORECASE|re.DOTALL)

for match_object in re.finditer(passive_voice_pattern, text):
    print(match_object.group(0))
is reading
are easier
was simply
is not
is crucial
was learning

The passive_voice_pattern tries to match the word am, are, is, was, were, been, or being, followed by a single whitespace character, followed by one or more characters including a newline until a word boundary is found. The word boundary character \b matches the empty string, but only at the beginning or end of a word. The | special character means OR.

By grouping the ‘to be’ verbs (putting them between parentheses) we simplify the regular expression pattern since we only have to add the word boundaries once instead of for each word. Without the word boundaries we would also find matches like ‘this’.

Now that we have the ‘to be verb’ plus the following word, we need to check if this following word is actually a verb. Since this series is about regular expressions I figured to write another regular expression for this check. It turns out that this is a near impossible task. In case you are also as stubborn and naive as me, I invite you to try to write a generalized regular expression that checks whether a word is a verb. It sure is great practice.

Regular expressions have their limits.

As any good craftsman knows, you need to pick the right tool for the job. In this case, instead of writing an extremely complex regular expression that would detect a fraction of all verbs at most, I decided that using a large lexical database made more sense.

WordNet is a large English lexical database created by Princeton University. Nouns, verbs, adjectives, and adverbs are grouped in cognitive synonyms (synsets). WordNet holds 117.000 synsets. Although there are a lot of interesting things you can do with WordNet, I only used it to check if the second word in our match is a verb or not.

Since I was already writing the regular expressions in Python, I decided to use the Natural Language Toolkit (NLTK) as an interface to WordNet. NLTK is a vast Python package about Natural Language Processing (NLP), and the authors of the Python package also published a book with the same name. The book is from 2009 and with all the advancements in the field it might feel a bit outdated. However, I still found the book useful, enjoyable, well-written, and an overall great introduction to NLP.

Next follows the code block that incorporates the NLTK interface to WordNet and the sentence extraction pattern we already used in the first post.

from itertools import compress

from nltk.corpus import wordnet as wn

def to_be_word(text):
    passive_voice_pattern = re.compile(r'\b(am|are|is|was|were|been|being)\b\s{1}(.+?)\b', flags=re.IGNORECASE|re.DOTALL)

    match_list = []
    for match_object in re.finditer(passive_voice_pattern, text):
        match_list.append(match_object.group(0))

    return match_list

def is_verb(word):
    return bool(wn.synsets(word, pos=wn.VERB))

def sentence_extraction(text):
    sentence_pattern = re.compile(r'[A-Z].+?[\.!?]\s{1}', flags=re.DOTALL)

    return re.findall(sentence_pattern, text)

to_be_list    = to_be_word(text)
second_word   = [words.split(' ')[1] for words in to_be_list]
is_verb_list  = [is_verb(word) for word in second_word]
passive_voice = list(compress(to_be_list, is_verb_list))
all_sentences = sentence_extraction(text)

[sentence for sentence in all_sentences if re.search("|".join(passive_voice), sentence)]

We start with the to_be_word function. This function wraps the passive_voice_pattern we already discussed earlier in this post.

The is_verb function takes a word as input and returns a Boolean value (True or False). We will use the second word for each match that is returned by the to_be_word function as input. The reason to return Boolean values will be explained shortly.

The sentence_extraction function takes a text string as input and returns a list of sentences. The sentence_pattern tries to match a capital letter followed by one or more characters including a newline until it matches either a dot (.), exclamation mark, or question mark, followed by a single whitespace character. Although this pattern is still susceptible to edge cases it does a good enough job for our challenge.

Next follows a number of assignments which follow a ‘the output of function A is the input of function B and the output of function B is the input for function C ..’ pattern. In an ideal world we would use pipe operators instead of the assignments but those are not supported in Python.

The first assignment calls the to_be_word function on the sample text. This results in a list of all the potential matches of passive voice.

to_be_list

['is reading',
'are easier',
'was simply',
'is not',
'is crucial',
'was learning']

Next we extract the second word of each item in the to_be_list by calling the split method and only keeping the second word ([1]).

second_word

['reading', 'easier', 'simply', 'not', 'crucial', 'learning']

Next we check for every second word whether it is a verb by calling the is_verb function on every word in the second_word list. We wrap this in a list comprehension to get a list of Booleans.

is_verb_list

[True, False, False, False, False, True]

Next we leverage the compress function from the builtin itertools module. This function keeps items in one list based on a corresponding Boolean value in another list of equal length. We apply this on the to_be_list since we want to get the original match in the text to extract the relevant sentence.

passive_voice

['is reading', 'was learning']

The assignment of all_sentences speaks for itself. The final list comprehension returns each sentence in all_sentences if the regular expression search method matches. The "|".join method ensures that each passive voice match is concatenated by the OR operator ('|').

"|".join(passive_voice)

'is reading|was learning'

Finally, the result are the two sentences of interest.

['My general approach to learning new things is reading several recommended books on the topic and go from there. ',
'During the course I was learning a lot about the intricacies of HTML and CSS.\n']

Both sentences are long-winded and written passively. Tight writing - our ambition - is about using fewer words to say actually what we mean. The essence of the first sentence is ‘I read books to learn new things’. The second sentence can be re-written as ‘The course taught me a lot about the intricacies of HTML and CSS’ without losing any meaning.

Subjunctive mood creates ambiguity.

Subjunctive mood, of which the most known examples are would, could, and should, must be avoided when possible. Subjunctive mood suggests a condition to your writing while it might not exist. The Udemy course on business writing gives the following examples.

Poor: You should be careful with matches - implies a condition where you may be careless
Good: Be careful with matches - simply states a fact or instruction.

Poor: If you work late, you would get the next day off - creates doubt
Good: If you work late, you get the next day off - clear

The regular expression pattern of this rule is straightforward. We need to search for one of the three words and retrieve the complete sentence for context.

sm_pattern = re.compile(r'\b(would|should|could)\b', flags=re.IGNORECASE)

[sentence for sentence in all_sentences if re.search(sm_pattern, sentence)]

['So why should you care about HTML and CSS?\n']

In sm_pattern we use IGNORECASE to include cases where the sentence starts with one of the three words. The sentence returned by the pattern can be shortened without losing any meaning. ‘So why care about HTML and CSS?’ is perfectly fine.

Ambiguous pronouns cause confusion.

Ambiguous pronouns like it, these, those, and that are vague. What is it? What is that? Whose are these? Explicit writing makes your document stronger and increases your authority. Don’t let your readers guess what you mean, tell them. The Udemy course on business writing provides several good examples of confusion caused by ambiguous pronouns.

Poor: We did a study of a drug. It proves ineffective. - The study or the drug?
Good: We did a study of a drug. The study proved ineffective.

Poor: It's not worth it.
Good: The limited increase in machine speed is not worth $5,000.

A potential downside to avoiding ambiguous pronouns is that you are going to repeat yourself more often. This is the tradeoff between fewer words and being explicit and clear.

Another part of this rule is to eliminate the start of a sentence that begins with there or it, followed by a ‘to be’ verb. This combination of words at the start of a sentence also causes confusion.

Poor: There is a simple solution to our cash flow problem.
Good: Mega bank can solve our cash flow problem simply.

Let’s start with the first pattern to match it, these, those, or that. This regular expression pattern is almost identical to the previous one and also straightforward.

ap1_pattern = re.compile(r'\b(it|that|there|these|those|this)\b', flags=re.IGNORECASE)

[sentence for sentence in all_sentences if re.search(ap1_pattern, sentence)]
['My general approach to learning new things is reading several recommended books on the topic and go from there. ',
'It was simply an amazing course.\n',
'Although I can recommend this self-study approach, I must stress that only studying is not enough \nto become skilled. ',
'Also, it took me 2 months to complete this course\nin my spare time, so be ready to make a serious commitment if you decide to follow this path. ',
'Having a solid understanding of how\nthe internet works can be a great advantage for any data professional, \nand it will open doors in the digital analytics world. ']

Let’s focus on the last sentence ‘.., and it will open doors in the digital analytics world’. What will open doors? The answer is ‘having a solid understanding of how to the internet works’. Although you could repeat that to make ‘it’ explicit, I personally feel that would make the sentence too long.

The part to eliminate openings which start with there or it followed by a ‘to be’ verb is more complicated but it doesn’t introduce new regular expression characters.

ap2_pattern = re.compile(r'\b(There|It)\b\s{1}\b(am|are|is|was|were|been|being)\b')

[sentence for sentence in all_sentences if re.search(ap2_pattern, sentence)]

The ap2_pattern tries to match at the start of a sentence, hence the pattern starts with a capitalized there or it. Next we need to match exactly one white space character followed by a to be verb. The to be verbs were taken from this extensive explanation on the topic. The output consists of one sentence.

['It was simply an amazing course.\n']

This re-written sentence ‘The course was simply amazing’ does not take away any meaning but is more clear and precise.

These patterns are begging for a CLI.

Although I really enjoyed studying the theory on business writing and developing and testing the regular expression patterns in a Jupyter notebook, I am most excited about the next part: to leverage the patterns from the command line. Currently I am not sure which programming language to use and what the design of the project will be, but I know the input and output for v0.1.

wt my_next_post.html

wt refers to the ‘write-tight’ command that captures every pattern discussed. The command will open a browser and highlights every potential improvement with different colors for each pattern. When finished I will write at least one other post about my CLI adventures. Until then you can follow my progress on Github.

No sponsored or affiliate links. If something here is new to you, the link is an easy way in.