EmmaClarke
Member
Hi,
I'm currently using MALLET to topic model 19th century journal articles.
I am running into trouble adding to the stopword list. I have tried adding my additional terms to the standard 'en' list that comes with MALLET (using Notepad++) but MALLET is not recognising these.
I also tried the method at the end of this post: http://www.jgoodwin.net/?p=1036
Neither of these have worked as I am still seeing words from the stopword list in my output.
Any ideas welcome.
joncgoodwin@gmail.com
Member
MALLET will stop running if it can't find the extra-stopwords file, so I guess that didn't happen. I'm not 100% sure about how it parses the file, but I created mine with one word per line.
Is your data from JSTOR? There may be a problem at that stage. Are the topics themselves what you'd expect other than the stopwords?
EmmaClarke
Member
My data is text extracted from JSTOR PDFs.
Yes, for the most part the topics are as I would expect.
In terms of the extra stopwords file, I created it with one word per line also / all lowercase.
I have tried placing the stop.txt file in both the C:\mallet directory and also the C:\mallet\stoplists directory.
joncgoodwin@gmail.com
Member
Did you use the Data for Research interface to download word frequency lists, or did you extract text from PDFs? It's hard to diagnose what's going on without having your data. If it's from DfR, I can download a sample and see if I have a problem as well.
EmmaClarke
Member
Thanks for your replies.
The corpus in question is Transactions of the Royal Irish Academy 1800-1899.
I did download word counts from DfR but those are not what I'm using for Topic Modeling. I'm using the full text of the articles which I extracted from JSTOR PDFs.
Examples of stopwords I was trying to include are roman numerals / vol, fig etc.
joncgoodwin@gmail.com
Member
I'd recommend using the word frequency lists, though you have to convert them using the method I described in the post you linked to in your original question. These lists are more reliable, I believe, than text extracted from PDFs would be, and smaller and easier to manage.
I also believe that the word frequency lists do a better job of removing roman numerals and publishing abbreviations, though many of those do remain. In the models I've run, OCR artifacts have tended to cluster into their own topic, which make them easy to ignore. Is that happening with yours?
Bethany Nowviskie
AdminBest Answer
via Twitter, David Mimno shares: "A new guide to Mallet stoplist options: http://mallet.cs.umass.edu/import-stoplist.php"
Just trying to connect the dots! -- Bethany
You must log in to post.
A project of the Association for Computers and the Humanities and the Chronicle of Higher Education's ProfHacker
Icons by famfamfam
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
