Creating a wordlist from text « Digital Humanities Questions & Answers

Creating a wordlist from text

(8 posts) (6 voices)

Tags:

dot.porter
Member
I would like to create a list of words that are in a single long document. Having a word count included would be helpful, too.

The purpose is to create a list of keywords that I can use to describe the items described in this document (it's a list of manuscript descriptions), with the eventual purpose of mapping those words to my own (very short) list of descriptors. But before I can map, I really need to know what words are used throughout the document.

Is there a tool I might use to do this, or some XSLT or Perl script?
Tweet this question
Posted 6 years ago Permalink
acrymble
Member

You could either use Voyant (http://voyant-tools.org/), a web-based tool. Or if you want to learn how to do that you could try the Programming Historian 2's tutorial on "Computing Frequencies" (http://programminghistorian.org/lessons).

All the best,

Adam

Posted 6 years ago Permalink
Stéfan Sinclair
Admin

Dot, is there a reason why one of the Taporware List Words tools wouldn't work for you? http://taporware.ualberta.ca/~taporware/textTools/listword.shtml

Posted 6 years ago Permalink
Laura Weakly
Member

You could also try http://tokenx.unl.edu

Posted 6 years ago Permalink

Ben Brumfield
Member

Or you can generate your own ordered CSV listing in a single--though horrible--line of Ruby:

1	IO.read('roland.txt').split(/\W+/).inject(Hash.new(0)){ \|h,x\| h[x] += 1; h }.sort_by{\|k,v\| -v}.each{ \|a\| print "\"#{a[0]}\",#{a[1]}\n" }

That will print a list of all the words in roland.txt, John O'Hagen's translation of the Song of Roland:

"the",1191
"and",675
"his",588
"of",545
"to",473
"he",409
"in",353
"And",318
"I",253
"The",236
"a",209
"him",208
"on",206
"s",205
"Roland",199

Posted 6 years ago Permalink

dot.porter
Member

Replying to @Stéfan Sinclair's post:

Only because I didn't think of it!

Posted 6 years ago Permalink
dot.porter
Member

Thanks for all the suggestions, I'll check them out and see what works best.

Posted 6 years ago Permalink
lmullen
Member
Replying to @Ben Brumfield's post:

If you have the file in plain text, you can use Unix utilities.
tr -sc '[A-Z][a-z]' '[12*]' < input.txt | sort | uniq > output.txt
Which is to say, read the file input.txt, putting each word onto one line, sort the words alphabetically, and remove any duplicates, saving the file to output.txt.

P.S. I cannot for the life of me get the forum to stop stripping characters. The third set of brackets should contain "backslash zero one two star" in numeric characters, not 12*.
Posted 6 years ago Permalink

You must log in to post.