The right tool(s) for an archive of newspaper columns « Digital Humanities Questions & Answers

Digital Humanities Questions & Answers » Applications, Tools, Formats

The right tool(s) for an archive of newspaper columns

(9 posts) (7 voices)

Asked 7 years ago by zachwhalen
Latest answer from Wally Grotophorst

Tags:

zachwhalen
Member
Hi Folks,

I'm considering a project that is more or less on my own time, and sort of in an area of inquiry that I don't often pursue. Therefore, I hope to get some input on the right toolset to get some work done.

My great grandfather was a newspaper columnist, and his work is currently only available in microfilm and in moldy boxes at my grandmother's house. I would love to get this electronified, even if it's only for our family to use.

Omeka seems like the logical choice for organizing and eventually conveying the archive, but what else can I use to 1) make this as painless as possible and 2) give me a rich set of data for interesting analysis and text mining?

The stack of papers is pretty large -- we're looking at 35 years-worth of daily columns. The kinds of things I'd be interested in tracking would mostly be time-based (i.e. keyed to periods of time like presidencies or other historical events.

I do have some help, in siblings and cousins, for transcriptions or other tedious work, but is there some OCR approach I should consider?

This is very much in the "Maybe we could do this ..." phase, so I'd appreciate any thoughts or suggestions.

Thanks,
Zach
Tweet this question
Posted 7 years ago Permalink
mjgiarlo
Member

Hi Zach,

Neat project idea.

I don't have any ready answers about digitization or OCR, but I would suggest adding Chronicling America to your list of software packages for displaying, housing, and searching your newspapers (though Omeka is a good choice too). It's being developed by the Library of Congress as an open source application for delivery of historic newspapers. Source code and docs are here:

http://sourceforge.net/apps/trac/loc-ndnp/

And you can see the latest version demonstrated here:

http://chroniclingamerica.loc.gov/beta/

Posted 7 years ago Permalink
Ben Brumfield
Member

Replying to @zachwhalen's post:

If I were you and it was my family--and if some fairly naive assumptions about the obviousness of chronological organization and accessibility of the material hold--here's what I would do:

0: Get my copyright ducks in a row -- find out who owns the copyrights to the columns.

1: Scan the columns, being fairly careful to make sure that separate pages of the same column remain together and are differentiated from other columns.

2: Upload the scans to the Internet Archive's Community Texts project, taking some care with the metadata so that the resulting pages are well organized on the Archive.org servers. Part of the Internet Archive's derivation process involves OCR, so you'll end up with page images hosted (for free!), as well as searchable PDFs, DjVu files and eText formats.

3: Move the derivative DjVu files to WikiSource, setting up a private instance of your own if the English-language community rules are too limiting.

4: Enlist friends and family to correct and proofread the OCR using the built-in Wikisource ProofreadPage plugin. This will track workflow, allow you to flag columns for review, and display how much of the collection has been reviewed.

5: Use the built-in coolness of MediaWiki to do neat things with the column. If you're using en.wikisource.org, you can place all the columns into a category and then use the "Make a book" feature to generate publish-on-demand collections of the columns. (I've tried this with Wikipedia's cognate tool and was blown away by the quality and cost.) If you're running your own private instance of wikisource, you can hack away with the MediaWiki toolset however you like -- annotating entries in-line using wikilinks, categorizing the columns by subject, mining the mark-up a la DBPedia, or whatever else you come up with.

I'm sure that other folks will have their own suggestions, but this might be an easy way to leverage to existing tools and the communities that are built around them.

Posted 7 years ago Permalink
Patrick Murray-John

Zach,

When it comes to the transcription part, I'd add Scripto to the list of tools to evaluate.

Patrick

Posted 7 years ago Permalink
Wally Grotophorst
Member

What form are these columns in? newspaper clippings? typed pages?

Posted 7 years ago Permalink
laurie taylor
Member

Omeka's a great option for a more exhibit-style approach, but there are loads of other options if you're interested in partnering with a library. As mentioned, Chronicling America is a great place to start. It's built from a major NEH + Library of Congress project, the National Digital Newspaper Program. For it, pre-1923 papers are being digitized in massive numbers so if any of your grandfather's columns are pre-1923, then they might already be included in those projects which are done by state. If they're later, lots of states are currently grappling with how to preserve and make accessible more recent news, including born digital newspapers currently being published.

With all of this work going on, the best option could be to partner with a library to digitize the full 35 year run for the full paper. Even if the library partner can't partner for the full project, they can probably help with the OCR process (running the columns during off-cycle times) if that's helpful. Partnering with a library also means they can benefit from integrating the columns in a full newspaper digital library and they can ensure preservation. Libraries can also help with any copyright questions for the full newspaper and/or individual columns.

If dealing with the paper in print, be very careful if it's actually moldy because it can be a major health hazard (libraries digitize moldy papers with great caution - masks, hepa filters, special disposal to prevent contamination, etc).

I'm happy to help in general if I can, and there could be a collaborative opportunity if you're grandfather's columns are from Florida or the Caribbean (I'm the technical director for the Florida Digital Newspaper Library and the Caribbean Newspaper Digital Library: http://ufdc.ufl.edu/newspapers and http://dloc.com/cndl ).

Laurie

Posted 7 years ago Permalink
jmhuculak
Member

I second the copyright concerns.

I've worked for years digitizing periodicals, and as far as OCR goes, you can't find a better program than ABBYY FineReader Professional. You can only get it for Windows, so if you are using a mac, install bootcamp and run it from a Windows partition. (do not use the mac version, it is not powerful enough yet).

The reason ABBYY is so good, is that it is spectacular recognizing small text. This trait alone will save you hours of correction. Also, it allows you to export to HTML, XML, text-under-image PDF, etc.

Posted 7 years ago Permalink
Ben Brumfield
Member

Replying to @jmhuculak's post:

Note that ABBYY FineReader is incorporated into other tools as well, including some of Adobe's PDF generators and the Internet Archive's book upload derivation software.

Posted 7 years ago Permalink
Wally Grotophorst
Member

ABBYY FineReader is the engine in Devonthink Pro Office (Mac only). ABBYY Finereader software (the application) ships with ScanSnap for Mac as well.

Posted 7 years ago Permalink

RSS feed for this topic

Reply

You must log in to post.