Na-Rae Han (naraehan@pitt.edu), 5/30/2017, CMU DH Summer Workshop

Preparation

Jupyter tips:

  • Shift+ENTER to run cell, go to next cell
  • Alt+ENTER to run cell, create a new cell below

More on https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/

The very basics

First code

  • Printing a string, using print().
In [ ]:
print("hello, world!")

The string type

  • String type objects are enclosed in quotation marks.
  • + is a concatenation operator.
  • Below, greet is a variable name assigned to a string value; note the absence of quotation marks.
In [ ]:
greet = "Hello, world!"
greet + " I come in peace."
  • String methods such as .upper(), .lower() transform a string.
In [ ]:
greet.upper()
  • len() returns the length of a string in the # of characters.
In [ ]:
len(greet)

Numbers

  • Integers and floats are written without quotes.
  • You can use algebraic operations such as +, -, * and / with numbers.
In [ ]:
num1 = 5678
num2 = 3.141592
result = num1 / num2
print(num1, "divided by", str(num2), "is", result)

Lists

  • Lists are enclosed in [ ], with elements separated with commas. Lists can have strings, numbers, and more.
  • Like with string, you can use len() to get the size of a list.
  • Like with string, you can use in to see if an element is in a list.
  • A list can be indexed through li[i]. Python indexes starts with 0.
  • A list can be sliced: li[3:5] returns a sub-list beginning with index 3 up to and not including index 5.
In [ ]:
li = ['red', 'blue', 'green', 'black', 'white', 'pink']
len(li)
In [ ]:
'mauve' not in li
In [ ]:
# Try [0], [2], [-1], [3:5], [3:], [:5]
li[0]

for loop

  • Using for loop, you can loop through a list of items, applying the same set of operations to each element.
  • The embedded code block is marked with indentation.
In [ ]:
for x in li :
    print(x, len(x))
print("Done!")

List comprehension

  • List comprehension builds a new list from an existing list.
  • You can filter in only certain elements, and you can apply transformation in the process.
  • Try: .upper(), len(), +'ish'
In [ ]:
[x for x in li if x.endswith('e')]
In [ ]:
[x+'ish' for x in li]
In [ ]:
[len(x) for x in li]

Dictionaries

  • Dictionaries hold key:value mappings.
  • len() on dictionary returns the number of keys.
In [ ]:
di = {'Homer':35, 'Marge':35, 'Bart':10, 'Lisa':8}
di['Homer']
In [ ]:
len(di)

Using NLTK

  • NLTK is an external module; you can start using it after importing it.

  • nltk.word_tokenize() is a handy tokenizing function out of literally tons of functions it provides.

  • It turns a text (a single string) into a list tokenized words.

In [ ]:
import nltk
In [ ]:
nltk.word_tokenize(greet)
In [ ]:
help(nltk.word_tokenize)
In [ ]:
sent = "You haven't seen Star Wars...?"
nltk.word_tokenize(sent)
  • nltk.FreqDist() is is another useful NLTK function.
  • It builds a frequency dictionary from a list.
In [ ]:
# First "Rose" is capitalized. How to lowercase? 
sent = 'Rose is a rose is a rose is a rose.'
toks = nltk.word_tokenize(sent)
print(toks)
In [ ]:
freq = nltk.FreqDist(toks)
freq
In [ ]:
freq.most_common(3)
In [ ]:
freq['rose']
In [ ]:
len(freq)

Processing a single text file

Reading in a text file

  • open(filename).read() reads in the content of a text file as a single string.
In [ ]:
myfile = 'C:/Users/zoso/Desktop/inaugural/1789-Washington.txt'  # Mac users should leave out C:
wtxt = open(myfile).read()
print(wtxt)
In [ ]:
len(wtxt)     # Number of characters in text
In [ ]:
'fellow citizens'.lower() in wtxt.lower()  # phrase as a substring
In [ ]:
'Americans' in wtxt

Tokenize text, compile frequency count

In [ ]:
# Turn off/on pretty printing (prints too many lines)
%pprint
In [ ]:
# Tokenize text
nltk.word_tokenize(wtxt)
In [ ]:
wtokens = nltk.word_tokenize(wtxt)
len(wtokens)     # Number of words in text
In [ ]:
# Build a dictionary of frequency count
wfreq = nltk.FreqDist(wtokens)
wfreq['citizens']
In [ ]:
wfreq['the']
In [ ]:
len(wfreq)      # Number of unique words in text
In [ ]:
wfreq.most_common(40)     # 40 most common words

Average sentence length, frequency of long words

In [ ]:
sentcount = wfreq['.'] + wfreq['?'] + wfreq['!']  # Assuming every sentence ends with ., ! or 
print(sentcount)
In [ ]:
# Tokens include symbols and punctuation. First 50 tokens:
wtokens[:50]
In [ ]:
wtokens_nosym = [t for t in wtokens if t.isalnum()]    # alpha-numeric tokens only
len(wtokens_nosym)
In [ ]:
# First 50 tokens, alpha-numeric tokens only: 
wtokens_nosym[:50]
In [ ]:
len(wtokens_nosym)/sentcount     # Average sentence length in number of words
In [ ]:
[w for w in wfreq if len(w) >= 13]       # all 13+ character words
In [ ]:
long = [w for w in wfreq if len(w) >= 13] 
for w in long :
    print(w, len(w), wfreq[w])               # long words tend to be less frequent

Processing a corpus

  • NLTK can read in an entire corpus from a directory (the 'root' directory).
  • As it reads in a corpus, it applies word tokenization (shown below) and sentence tokenization (not shown here).
In [ ]:
from nltk.corpus import PlaintextCorpusReader
corpus_root = 'C:/Users/zoso/Desktop/inaugural'  # Mac users should leave out C:
inaug = PlaintextCorpusReader(corpus_root, '.*txt')  # all files ending in 'txt'
In [ ]:
# .txt file names as file IDs
inaug.fileids()
In [ ]:
# NLTK automatically tokenizes the corpus. First 50 words: 
print(inaug.words()[:50])
In [ ]:
# You can also specify individual file ID. First 50 words from Obama 2009:
print(inaug.words('2009-Obama.txt')[50:])
In [ ]:
# NLTK automatically segments sentences too, which are accessed through .sents()
print(inaug.sents('2009-Obama.txt')[0])   # first sentence
print(inaug.sents('2009-Obama.txt')[1])   # 2nd sentence
In [ ]:
# How long are these speeches in terms of word and sentence count?
print('Washington 1789:', len(inaug.words('1789-Washington.txt')), len(inaug.sents('1789-Washington.txt')))
print('Obama 2009:', len(inaug.words('2009-Obama.txt')), len(inaug.sents('2009-Obama.txt')))
In [ ]:
# for-loop through file IDs and print out word count. 
for f in inaug.fileids():
    print(len(inaug.words(f)), f)

Trouble shooting

  • Unfortunately, 2005 Bush file produces Unicode encoding error.
  • Let's make a new text file from http://www.presidency.ucsb.edu/inaugurals.php
  • Copy text and paste in Notepad (Windows). Make sure to choose UTF-8 encoding and not ANSI.
  • The text files are locked; We will need to save, halt and then re-start the Python notebook.
In [ ]:
# Corpus size in number of words
print(len(inaug.words()))
In [ ]:
# Building word frequency distribution for the entire corpus
inaug_freq = nltk.FreqDist(inaug.words())
inaug_freq.most_common(100)

What next?

Take a Python course!