Here, we’ll work through some of the practicalities of creating and counting ngrams from text. Let’s grab the book first.
from gutenbergpy.textget \
import get_text_by_id,\
strip_headers
def get_clean_book(book_id):
"""Get the cleaned book
Args:
book_id (str|int): The book id
Returns:
(str): The full book
"""
raw_book = get_text_by_id(book_id)
book_byte = strip_headers(raw_book)
book_clean = book_byte.decode("utf-8")
return book_cleanFirst step will be getting the “unigram” frequencies.
| words in context | words to predict | total | name |
|---|---|---|---|
| 0 | 1 | 1 | unigram |
| 1 | 1 | 2 | bigram |
| 2 | 2 | 3 | trigram |
To get any counts, we need to tokenize.
Next, we can count with collections.Counter()
The “unigram” probability of a word: \[ P(w_i) = \frac{C(w_i)}{\sum_{i=1}^n C(w_i)} \]
We can get \(C(w_i)\) from the counter dictionary
The trickier thing to get is \(\sum_{i=1}^n C(w_i)\). One way to do it is with a for loop.
With total_freq, we can calculate the probability of each token, which we can store in another dictionary.
numpy is a python package that allows you to do numeric computation more easilly.
If we just had a python list of numbers, we couldn’t quickly divide each number by the sum.
We can do this with a numpy array.
There’s lots of numpy methods to make life easier when working with numbers.
While the dictionary moby_count is convenient for quickly getting the count of a token, we’ll need separate lists and arrays for:
A thing to think about is how the mathematical formula on the left is being implemented in the code on the right.
We can get a specific word’s probability like so:
We can sample a random word from the unigram distribution like so:
Making bigrams is a bit more complex. We need to get counts of each sequence of two tokens. Fortunately, nltk has a nice and convenient function for this.
This is a list of “tuples”. Tuples are kind of like lists, but you’re not able to edit them after you create them. We can use Counter() on a list of tuples just like we did a list of tokens.
[((',', 'and'), 2630),
(('of', 'the'), 1869),
(('’', 's'), 1784),
(('in', 'the'), 1122),
((',', 'the'), 916),
((';', 'and'), 857),
(('to', 'the'), 715),
(('.', 'But'), 596),
(('.', '“'), 594),
((',', 'that'), 583)]Now, getting the probability of a single token isn’t as straightforward, since we’re looking at conditional probabilities.
\[ P(w_i | w_{i-1}) = \frac{C(w_{i-1}w_i)}{\sum C(w_{i-1}w)} \]
To use concrete words for a second:
\[ P(\text{whale} | \text{the}) = \frac{C(\text{the whale})}{\sum C(\text{the }w)} \]
So, to calculate the conditional probability, we need to get
# the first word in the
# 2 word sequence
w_0 = "the"
w_0w = [
bigram
for bigram in moby_bigram_count
if bigram[0] == w_0
]
w_0w[0:10][('the', 'Whale'),
('the', 'Monstrous'),
('the', 'Less'),
('the', 'True'),
('the', 'Hand'),
('the', 'Arsacides'),
('the', 'Carpenter'),
('the', 'Cabin'),
('the', 'End'),
('the', 'First')]We can now get the specific conditional probability for the word “whale”
To randomly generate any word following “the”, we need to get the probability distribution across bigrams.
To generate a sequence, starting with a specific word, we need to encapsulate our logic above into a single function.
def generate_sequence(
bigram_count:dict,
w_0:str = "The",
n:int = 10
)->list[str]:
"""Generate a sequence of words from
a bigram model
Args:
w_0 (str): The initial token
n (int): The size of the sequence
to generate
Returns:
(list[str]): The generated sequence
"""
## start out with the seed token
sequence = [w_0]
for i in range(n):
## The new seed token should be
## the last one added
w_0 = sequence[-1]
## get all bigrams beginning
## with the seed token
w_0w = [
bigram
for bigram in bigram_count
if bigram[0] == w_0
]
## get the counts of all bigrams
C_w_0w = np.array([
bigram_count[bigram]
for bigram in w_0w
])
## get the probabilities of all bigrams
P_w_0w = C_w_0w / C_w_0w.sum()
## get the second token
## from every bigram
w_1 = [
bigram[1]
for bigram in w_0w
]
## sample a new token
chosen = np.random.choice(
w_1,
size = 1,
p = P_w_0w
)
## add the sampled token to the
## sequence
sequence.append(chosen[0])
return sequenceThe bigram sequence generator above has to start out on a specific word, and it will keep going for as many loops as we tell it.
If we wanted a generator that doesn’t need a specific word to start on, and will auto-stop when it gets to the end of a sentence, we’ll need to pre-process our data differently, so that there are special “start” and “stop” symbols, or “padding”.
moby_sent_words_padded = [
["<s>"] + sent + ["</s>"]
for sent in moby_sent_words
]
moby_sent_words_padded[500]['<s>',
'Deep',
'into',
'distant',
'woodlands',
'winds',
'a',
'mazy',
'way',
',',
'reaching',
'to',
'overlapping',
'spurs',
'of',
'mountains',
'bathed',
'in',
'their',
'hill-side',
'blue',
'.',
'</s>']sequence = ["<s>", "<s>"]
for i in range(100):
w_0 = sequence[-2:]
new = lm.generate(num_words=1, text_seed=w_0)
sequence.append(new)
if new == "</s>":
break
sequence['<s>',
'<s>',
'had',
'turned',
',',
'and',
'continually',
'set',
'in',
'a',
'calm—give',
'us',
'a',
'blue',
'hanging',
'tester',
'of',
'smoke',
',',
'illuminated',
'by',
'the',
'terms',
'of',
'my',
'own',
'voice',
'.',
'</s>']@online{fruehwald2024,
author = {Fruehwald, Josef},
title = {Making and {Counting} Ngrams in Python},
date = {2024-02-15},
url = {https://lin511-2024.github.io/notes/programming/05_ngrams.html},
langid = {en}
}