User-friendly, open-source/free latent semantic indexing? « Digital Humanities Questions & Answers

Digital Humanities Questions & Answers » Applications, Tools, Formats

User-friendly, open-source/free latent semantic indexing?

(3 posts) (2 voices)

Asked 5 years ago by dh.donoralibrary@gmail.com
Latest answer from dh.donoralibrary@gmail.com

Tags:

dh.donoralibrary@gmail.com
Member
I would like to use LSI (latent semantic indexing) to provide access within the public library to a local history collection which I have been scanning (usually newspaper pages or clippings; using the Paper Port software which came with our Brother all-in-one scan/fax/copy machine).

While I've read the "right tool(s)for an archive of newspaper columns" post, I do not have copyright clearance for the scanned material, nor do I seek to put it online, as most of the suggested tools seem to do. DevonThink seems to be out, as we're strictly Windows(including a few 7's)-- no Macs.

We're also perpetually broke (who isn't?), and unlikely to spend anything on this project, including renting IT help--have none in-house. Although I'm motivated to do the text cleanup of the scanned corpus, I'm not too tech-savvy, nor am I a mathematician. (I have a bad case of English-major-brain; it gets all fuzzy somewhere around the word "vector"). I still hope we can transcend our limitations and those of keyword indexing. Any help? Any suggestions? Thanks! DH
Tweet this question
Posted 5 years ago Permalink
tedunderwood
Member

I think what you're trying to do is great. LSI could help provide better search results. Moreover, as a bonus, once you've got an archive set up for LSI, it wouldn't be too challenging to use the same tools to provide other kinds of mapping services (like word associations within the corpus).

Now, how to do it practically ... I'm tempted to say that LSI isn't actually that hard to work up from scratch if you can do other kinds of coding. A "vector" is really just a sequence of numbers, for instance, and there are functions built into a lot of programming languages that will do the matrix algebra part (SVD) -- which is where it does get a bit tricky. I'm a philosophy major, but I don't find LSI very challenging in a language like R.

However, that's probably not a great answer. Taking a slightly less "from scratch" approach, I believe there's a gensim module in Python that will do LSI for you. You would still need to set up Python and load the collection and figure out how to handle a user interface; this isn't a "plug and play" solution. But if you're comfortable setting up interfaces and just shudder at matrix algebra, this might work pretty well:

http://stackoverflow.com/questions/6287411/lsi-using-gensim-in-python

Posted 5 years ago Permalink
dh.donoralibrary@gmail.com
Member

Hello, tedunderwood; please accept my apology for this delayed reply. (Mistakenly thought I'd chosen to have posts appear as email; not encouraging in this context).
Thank you for providing the Python gensim model link, and the implied confidence that I can use it. I'll see what I can make of it, and if I can't make it work, maybe I'll offer this as a project to some nearby computer major (although I'd still rather do it myself). Many thanks, DH

Posted 5 years ago Permalink

RSS feed for this topic

Reply

You must log in to post.