Most effective software for building searchable digital bibliographic database?

Digital Humanities Questions & Answers » Applications, Tools, Formats

Most effective software for building searchable digital bibliographic database?

(10 posts) (6 voices)

Asked 5 years ago by shmerritt
Latest answer from shmerritt
This question has a best answer.

Tags:

shmerritt
Member
[Stéfan Sinclair (via DH Commons mentorship program) suggested that I post here. I have described this project in my "collaborator" or user profile on DH Commons at http://dhcommons.org ; direct link from the "Collaborators" section at http://dhcommons.org/users/susan-hollis-merritt and in my member profile on MLA Commons at http://commons.mla.org ; direct member link at http://commons.mla.org/members/shmerritt/; direct links are also provided in the Pinter Bibliography and CV sections on my website at http://www.susanhollismerritt.org , where there are related details.]

I am requesting advice and guidance as I explore the possibility of building a searchable digital database for my "Harold Pinter Bibliography" published in print in volumes of The Pinter Review (Tampa: The University of Tampa Press, 1987-2011). Each annual/biennial/special edition of this bibliography is published in MLA format (whatever the most current MLA format has been at the time). I would plan to continue with MLA format for consistency of documentation style throughout. (I understand that users of such a bibliography could convert easily to whatever format is desired on their end later.)

I have word-processed files (WordPerfect) for all of the 14 editions of this bibliography. The same typist working for The University of Tampa Press prepared my word-processed files into a format to be used by the printer. For several years, she converted my "Harold Pinter Bibliography" files initially to MS Word document files or various versions of PDF files for the printer. More recently, I gave her my own PDF file versions of my WordPerfect files, which I believe she also converted to Word or to another format for the printer. The typist's and printer's digital files may be accessible if I need to have them for easier digital conversion.

A problem with the typist's conversions from one word-processing software program to another has been the loss of accents, punctuation, and other typesetting features, which at times have had to be corrected in printer's proofs. I have maintained hard copies (original books and parts of books, offprints, photocopies, or print outs) and handwritten index cards (for every entry) to check as back ups of the bibliographic entries in the printed bibliographies for purposes of such error-correction of occasional printer's errors that were missed even in corrected proof.

From looking into this matter over a decade ago (when ProCite was initially popular), I believe that the printed information (even though digitized) would need to be converted into records featuring bibliographic fields (including annotations, sometimes lengthy, most often very brief) for each printed entry, and each entry would need to have key words and/or phrases (e.g., to accommodate multiple authors, editors, titles, subjects, etc.) so that it could be capable of classification by key words for searching by users. Also, there are categories and subsections of categories in each published edition of this bibliography that may need some kind of "field"/key word/phrase identification for sorting electronically.

I am not familiar with mark up language beyond HTML, but it is possible that I might be able to learn about additional mark up languages if needed (e.g., RIS Format, SML format, etc.). When I asked this question of a presenter after a DH session at the MLA Convention in Boston, she advised me to look at Drupal (which I have done somewhat), but I am not trained in using it for anything. It appears to me to be involved in the stage of actually putting an already developed database online. Another piece of advice I have received is possibly to create an EXCEL type spreadsheet which could be searched. But before I would begin doing any of this work, I would like to have a plan to use one mode of conversion (one kind of software, so as to avoid duplication of effort).

Thus, a main question that I am asking is: What would be the most effective current software package or packages to use for this entire process?

Prior to deciding this project's feasibility and how and where this database would be hosted (institutional partners/support), I need to explore such matters as: (1) the technical requirements for such conversion from printed material to searchable digital format; (2) the estimated cost of such conversion (including the costs of any commercial software packages involved (if any) and of assistance with the physical input of the records); (3) whether or not applying for grant support is warranted; and so on.

I welcome suggestions, advice, and guidance. Thank you very much.

[(updated on January 17, 2013)Since first posting this, I have been exploring some earlier DH Q&A threads and found Bethany Nowviskie's suggesting that updates are in progress for http://arts-humanities.net/, and I will be exploring the "tools" forum there; the "bibliography" section returned an error message when I tried it, so I'll try it again later.]

Susan Hollis Merritt, PhD
Founding Bibliographical Editor
The Pinter Review
http://www.susanhollismerritt.org
Tweet this question
Posted 5 years ago Permalink
Dorothea Salo
Member
Best Answer

Well, I can think of a few ways of going about this. :)

One way would be to export all your digital files to plain text, then regular-expression the living daylights out of them to turn them into RIS or other text- or markup-based format. This is fiddly, but probably not hugely difficult given the consistency in the bibliographic style; you could find a consultant to do it. The one unknown is what happens to those special characters, a point on which I can't reassure (much depends on how they were handled in the original files).

From there, you could slurp them into Zotero or Mendeley, and get the whole out onto the web through those services. (Both Zotero and Mendeley have perfectly functional export tools, so your exit strategy is clear and unworrisome.) If you'd rather run your own, VIVO or BibApp might suit.

I would be inclined to go about this a slightly different way, though. A substantial percentage of your bibliographies is probably already indexed in searchable, fielded form via one or another bibliographic database/index. Why not search all that up (I daresay some simple title and subject searches would turn up most of it!), slurp it into Zotero or Mendeley, and go from there?

Posted 5 years ago Permalink
Patrick Murray-John

Best Answer

+1 on Dorothea's idea of looking up the data in another database then putting it into Zotero or Mendeley.

A further question might be what needs to happen with the database once it exists? That is, is it publicly accessible on the web? Available to others for research upon request? Public on the web, but behind some access restrictions?

So, imagine it is happily in Zotero. That _could_ make it public on the web with a Zotero group, but the search and display is a bit limited there. To make it more accessible on the web, the next step would probably be to import the data into something like Drupal or Omeka. (I know Omeka has a plugin to import from Zotero. Not sure about Drupal).

The upshot is that depending on how you intend for the data to be used, there might be additional steps after the initial compiling of data.

Posted 5 years ago Permalink
quinnanya
Member
Best Answer

Replying to @Patrick Murray-John's post:

Yup, there's a Biblio Zotero module for Drupal that can do an import, then use the display powers of the Biblio module, and its various add-ons.

Posted 5 years ago Permalink
shmerritt
Member
Best Answer

To Dorothea, Patrick, and Quinnanya (if I may),

Thanks for all these suggestions, and I will be looking into all of them.

One way would be to export all your digital files to plain text, then regular-expression the living daylights out of them to turn them into RIS or other text- or markup-based format.

Noticing the Export Manager possibilities in EBSCO Host, e.g., I have seen this as a possibility, though with 14 editions of this bibliography published over 25 years the task involves many thousands of entries.

Related is the point:

A substantial percentage of your bibliographies is probably already indexed in searchable, fielded form via one or another bibliographic database/index.

Many of my entries came from my own searches that precede the existence of online databases like EBSCO, which now includes the online version of the MLA International Bibliography (previously I used printed versions of the latter), and precede the time when library patrons could do online searches themselves (1986-early 90s). [Prior to that I would pay for librarians with access to RLIN, e.g., at Cornell University and other research universities, to do the searches on the institution's fee basis.]

My entries are not only publications but also productions and that information I compiled from theater programs of those I attended and reading reviews and other articles and advertisements. The "Harold Pinter Bibliography" includes performances and performance reviews and productions and interviews and other programs broadcast on various media (radio, television, film, [later] podcasts, etc.) From mid 90s on, it does also include web-based or online publications.

Many of the entries that are published in international newspapers and other popular periodicals do originate in somewhat different formats in online databases like LexisNexis; I probably would not have access to their fields. To work back to my original sources would probably be more time-consuming in some cases than creating the fields needed for my purposes.

In the cases when the original entries (as noted on my notecards and printouts prior to word-processing the entries into a list format) did come from other online databases (post 1980s-early 90s) via ProQuest, EBSCO, InfoTrac, etc. [or have been included in them retroactively], the mark up is probably accessible. But, again, thousands of entries are involved and working back to where each one of them came from would probably involve conflicting formats of each of the bibliographic databases used over the period of, say, mid 1990s to the present.

A further question might be what needs to happen with the database once it exists? That is, is it publicly accessible on the web? Available to others for research upon request? Public on the web, but behind some access restrictions?

At this stage, I am uncertain of the answer to these questions. Initially, the idea is to make the full database of over 25 years both publicly accessible (free to users) on the web and also possibly open to editing by password-protected collaborator-compilers for future expansion of the database (bibliography), following whatever format is used for it. Depending on the costs of the project (if it is feasible), the costs of maintaining it on a server (if not a donated server by a university or library, e.g.) might need reimbursement through voluntary donations and/or subscription.

The 14 previously published editions of this bibliography (as published in separate lists in The Pinter Review) are not accessible via commercial vendors (like EBSCO, etc.).* I would prefer to make it publicly accessible (free to users) on the web universally (not limited only to university libraries and other subscribers to the commercial databases). How it is to be made accessible is something I can deal with once I figure out how to build the database. It seems that it would be quicker to give the material to a commercial vendor to mark up in ways to make it searchable, but that would not enable me to correct whatever errors occurred in the printing process (post proofs-correction) prior to a mark up that I and my academic collaborators had more control over.

To make it more accessible on the web, the next step would probably be to import the data into something like Drupal or Omeka. (I know Omeka has a plugin to import from Zotero. Not sure about Drupal).
The upshot is that depending on how you intend for the data to be used, there might be additional steps after the initial compiling of data.

Yup, there's a Biblio Zotero module for Drupal that can do an import, then use the display powers of the Biblio module, and its various add-ons.

It is very helpful to have these points, and I will look into what you suggest and keep them in mind in doing so. I am in the process still of exploring the Drupal links given (much of the language used there is over my head at this stage), and I will need some help with understanding them (e.g., consultant[s]).

Thank you all very much again, and I look forward to other comments that may appear here or sources of information that you think I should research.

*The Pinter Review is not available online. Its individual contents (articles and the bibliography: as a publication: author-title-subject[s]) are indexed (mostly) by the MLA International Bibliography.

Posted 5 years ago Permalink
Michael Widner
Member
Best Answer

I would look at Drupal, too. The Biblio module would allow data entry of each bibliographic record. The Feeds module would allow you to automate the process if you had all the records in a spreadsheet, csv, or some other format. Plus, since you want to control user access to edit the records, Drupal already has that ability built in with user roles. There is a bit of a learning curve, but it seems like the best tool for what you want. I'd also recommend picking up a book on Drupal if the online documentation (which is very uneven) isn't helpful. The Definitive Guide to Drupal 7 is a good reference work. With Views (another powerful Drupal module) you could create custom queries and displays of the records beyond what the Biblio module already allows.

Posted 5 years ago Permalink
shmerritt
Member

Replying to @Michael Widner's post:
Thanks very much. Will be looking into what you (and the others) suggest.

Update: Just a point of information:
I noticed recently in doing a search via the online edition of ABELL that repeatedly it gives an incorrect publication year for many of the volumes of The Pinter Review. (Apparently, its software took as a publication year dates from subtitles of volumes, not from actual publication information provided in printed title pages/copyright pages. E.g., in the entry for the most current volume, whose subtitle gives the years it covers as "2009-2011," the first of those years, 2009, is listed as the date of publication, which is incorrect, since its copyright date is actually 2011 (though it was distributed in 2012). In the past, I have found other errors in other online databases frequently used by academic researchers (when checked against actual print sources), and I would not want to perpetuate such errors by importing such erroneous data from those databases.

Posted 5 years ago Permalink
shmerritt
Member

Further update:
I have posted some more recent information pertaining to the proposed project to which this thread relates in the Day of DH 2016 (April 8) site. Please see my Day of DH 2016 blog The Harold Pinter Bibliography (Searchable Digital Database) project, especially my first post Hello world!, created on March 18, 2016.

If you have additional suggestions to add to the earlier ones, please post them here, at DH Q&A. I will greatly appreciate them. Thank you.

Posted 2 years ago Permalink
Kevin Hawkins
Member

You might be interested in a recent discussion on CODE4LIB on a similar question: https://listserv.nd.edu/cgi-bin/wa?A2=CODE4LIB;44fd042d.1604

Posted 2 years ago Permalink
shmerritt
Member

Thank you very much. That post's question seems very similar to mine indeed! I'm just now looking into the replies you've directed me to via CODE4LIB in LISTSERV. (I had to register first.)

One reply mentions Zotero, which some of the above posts suggest. The other replies seem related to library cataloguing purposes and are pretty technical in nature. But I am keeping an open mind.

So far I've been doing some practice conversion of my printed entries into Zotero in Firefox. I will be reading up on Drupal, exploring Drupal's Biblio Zotero plug in, etc., as suggested above.

Still open to further suggestions, however.

I've just applied for funding this project. Outcome will not be until December. So I expect to be researching possibilities further and, as I say, still welcome further suggestions.

[Update: the link to my website in my first post is no longer active; the archived link is susanhollismerritt.org.]

Posted 2 years ago Permalink

RSS feed for this topic

Reply

You must log in to post.