http://digitalhumanities.org/answers/forum/databases-amp-data-structures Digital Humanities Questions & Answers » Forum: Databases & Data Structures - Recent Posts en-US Sat, 06 Aug 2016 03:18:38 +0000 http://bbpress.org/?v=1.0.2 q http://digitalhumanities.org/answers/search.php http://digitalhumanities.org/answers/topic/text-network-analysis-using-gephi#post-2366 Mon, 21 Dec 2015 00:34:27 +0000 diane.southier 2366@http://digitalhumanities.org/answers/ <p>Hey!, Thank you so much for you attention! </p> http://digitalhumanities.org/answers/topic/text-network-analysis-using-gephi#post-2355 Wed, 04 Nov 2015 11:30:02 +0000 Amanda Visconti 2355@http://digitalhumanities.org/answers/ <p>Hi!</p> <p>I wrote a blog post on getting data into Gephi here: <a href="http://literaturegeek.com/2013/09/09/dataintogephi" rel="nofollow">http://literaturegeek.com/2013/09/09/dataintogephi</a>. The short version is that you need to create some kind of dataset like a spreadsheet that shows relationships among whatever you want to analyze (e.g. characters in the texts). My main focus isn't information visualization, so someone can correct me if I'm wrong, but I don't think that Gephi or its plugins are set up to import a whole text and work with it that way (i.e. I don't think there's something that lets you import a text and then highlight and pull out the characters or other things you want to visualize).</p> <p>You might be interested in some other blog posts I wrote while learning Gephi and doing some basic projects, for tutorials and a sense of what Gephi can do: <a href="http://literaturegeek.com/tag/gephi/" rel="nofollow">http://literaturegeek.com/tag/gephi/</a>. I've used Gephi to look at the types of interactions between characters in James Joyce's novel Ulysses (what characters talk to each other, and what is the depth of those interactions?), and to explore who's citing what sources in the Digital Humanities Quarterly scholarly journal.</p> <p>This recent post on cool infoviz tools and projects in the digital humanities might have some inspiration, and it also links to people with far more infoviz experience than me who probably have helpful blog posts and tutorials or slides you could search for: <a href="http://literaturegeek.com/2015/09/25/infovizlinks/" rel="nofollow">http://literaturegeek.com/2015/09/25/infovizlinks/</a> </p> <p>E.g. Miriam Posner has a post on getting started with humanities data viz: <a href="http://miriamposner.com/blog/how-did-they-make-that/#" rel="nofollow">http://miriamposner.com/blog/how-did-they-make-that/#</a><a href='/tags/network'>network</a></p> <p>E.g. Scott Weingart on when not to use data viz: <a href="http://www.scottbot.net/HIAL/?p=39600" rel="nofollow">http://www.scottbot.net/HIAL/?p=39600</a></p> <p>It was also just announced that a Gephi update is coming in December with some improvements: <a href="https://gephi.wordpress.com/2015/11/02/announcing-gephi-0-9-release-date/" rel="nofollow">https://gephi.wordpress.com/2015/11/02/announcing-gephi-0-9-release-date/</a></p> <p>I know that a big holdup for me with Gephi was figuring out that the latest version didn't work with my Mac (I had to use the second-to-latest version instead). </p> http://digitalhumanities.org/answers/topic/text-network-analysis-using-gephi#post-2354 Tue, 03 Nov 2015 17:59:59 +0000 diane.southier 2354@http://digitalhumanities.org/answers/ <p>Hi!<br /> I'm a student from Brazil, and I'm looking for help to text network analysis using Gephi graph visualization software. I have a lot of texts in portuguese to anylise, and Gephi's tools seem to be the best to reach the result I want. So how do I import text to Gephi?<br /> Thanks! </p> http://digitalhumanities.org/answers/topic/transcription-layer-on-top-of-fedorablacklight-stack#post-2326 Wed, 13 May 2015 15:36:15 +0000 elotroalex 2326@http://digitalhumanities.org/answers/ <p>What if I wanted to build a transcription layer on top of columbia lib's digital collections? we have a fedora/blacklight stack. thinking of an API. anybody out there doing it? </p> http://digitalhumanities.org/answers/topic/where-do-you-go-for-data#post-2257 Thu, 13 Nov 2014 10:33:38 +0000 thomasgpadilla 2257@http://digitalhumanities.org/answers/ <p>Where do you go for data to support your DH work aside from LAMs (library, archive, museum) specific repositories? </p> http://digitalhumanities.org/answers/topic/where-can-i-get-data-for-a-map-of-all-of-the-letters-a-historical-figure-sent#post-1939 Sat, 16 Mar 2013 18:51:18 +0000 Dorothea Salo 1939@http://digitalhumanities.org/answers/ <p>I think you might need to ask yourself "who might have compiled such a dataset, for whom, and why?" This would require at minimum looking at every single letter, so voluminous correspondents (e.g. the Adams dynasty in America) are right out. People who live all their lives in one or a very few places would produce an entirely uninteresting dataset.</p> <p>So you're looking for someone who wrote some but not too many letters, who traveled enough to make the geographic sources of those letters in some way interesting, and who is prominent enough (or whose travels, Odysseus-style, are intriguing enough) for someone to have compiled precisely that information.</p> <p>That doesn't strike me as very many people. </p> http://digitalhumanities.org/answers/topic/where-can-i-get-data-for-a-map-of-all-of-the-letters-a-historical-figure-sent#post-1933 Thu, 14 Mar 2013 15:09:40 +0000 cperryk@gmail.com 1933@http://digitalhumanities.org/answers/ <p>Hi all,</p> <p>I make maps and interactives for Slate.com. I thought it might be interesting to make a map showing all of the locations from which a historical figure sent letters. Like a simpler version of this: <a href="http://www.stanford.edu/group/toolingup/rplviz/" rel="nofollow">http://www.stanford.edu/group/toolingup/rplviz/</a></p> <p>Do any people or organizations have structured data that would be suitable for this project? I could use, for example, a dataset of an individual's letters that includes sender location, recipient location, and date.</p> <p>I would really appreciate any help on this!</p> <p>Chris Kirk<br /> Interactives Editor<br /> Slate.com </p> http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1885 Thu, 14 Feb 2013 23:02:35 +0000 Josh 1885@http://digitalhumanities.org/answers/ <p>John: Thanks for the links! As for more on what I want to do, I'll try my best to summarize it (though the idea is still in early formation/I don't necessarily know everything yet I want to do, look for, etc.). Also, the corpus, as it stands, is only 4 novels at around 800 pages, but this will increase as this author's novels are translated into English (and, in the future, I'd also like to add works from similar authors to the corpus). My ideas:</p> <p>1. Extract names, places, and titles of works from the text<br /> 2. Perform some kind of frequency rankings (within works and across corpus)<br /> 3. Visualize connections: e.g. authors &amp; their works mentioned in relation to each other, which authors are most mentioned in relation to each other, groupings of real and imaginary authors (something important and often ambiguous in these texts), and, eventually, locate overlaps in the network of this network with similar networks created by other authors.</p> <p>In other words, these novels, especially taken as a whole, embody a fairly vast historical (and sometimes fictional) network of literary figures and their works and I'd like to extract these from the corpus in order to (a) better access and analyze them, while at the same time exposing this labyrinthine network that might get lost (in its overwhelming totality) within each narrative and even more easily across multiple "distinct" works. Other questions: can narrative arrive just from, say, a network of authors? What makes metafiction more than that + annotation? etc. </p> <p>Again, these are just my initial ideas, and this is an intellectual side project but one that has implications in my daily work as a librarian working in the digital humanities; often times we need a project of our own to really acquire and retain substantive new skills. For more insight into Vila-Matas' metafictions, <a href="http://bit.ly/XHkPQ1">here's a good essay</a>. </p> http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1884 Thu, 14 Feb 2013 21:34:39 +0000 johnlaudun 1884@http://digitalhumanities.org/answers/ <p><em>Replying to @Josh Honn's <a href="http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1883">post</a>:</em></p> <p>Depending on what you are doing, and depending on how much data we are talking about, you may not need to go the TEI route. If you are on a Mac, I've written about setting up NLTK here: <a href="http://johnlaudun.org/20121230-macports-the-key-to-python-happiness/" rel="nofollow">http://johnlaudun.org/20121230-macports-the-key-to-python-happiness/</a>. The TL;DR version is here: <a href="http://johnlaudun.org/20121230-macports-for-nltk/" rel="nofollow">http://johnlaudun.org/20121230-macports-for-nltk/</a>. </p> <p>Write back with more info and I'm sure more people will kick in with help. </p> http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1883 Thu, 14 Feb 2013 18:53:08 +0000 Josh 1883@http://digitalhumanities.org/answers/ <p>Oh great, another Python book to read! Just kidding. Thanks, Korey (and Justin)! NLTK looks like the way to go. Just glancing over the extracting information chapter, and from others who have suggested TEI, it's clear that I need to, in some way, go from unstructured to structured text before I do anything else. And now back to my irregularly scheduled Python reading/learning. </p> http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1881 Thu, 14 Feb 2013 17:10:17 +0000 Korey Jackson 1881@http://digitalhumanities.org/answers/ <p>Hey Josh,<br /> Just talking to a friend of mine (Justin Joque at the spatial and numeric data library here at Michigan). One thing he mentioned was that Cytoscape might be a little more user- and spreadsheet-friendly...though one strength of Gephi is its ability to animate networks over time, if that's something you're looking for.</p> <p>More generally, he thought it sounded like you were needing to do some named entity extraction, and recommended the Python-based <a href="http://nltk.org">nltk.org</a> for that. They also host a great resource--<em>Natural Language Processing with Python</em>--at <a href="http://nltk.org/book/">nltk.org/book</a>.</p> <p>Hope that helps! </p> http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1880 Thu, 14 Feb 2013 12:09:22 +0000 Josh 1880@http://digitalhumanities.org/answers/ <p>There are plenty of tutorials on the web that are useful for learning Gephi, but I've encountered a much steeper learning curve for the steps prior—such as extracting and preparing data from literary texts to be used with Gephi. In my example, I am interested in performing a network analysis of the social networks (both real and imaginary) in the works of Enrique Vila-Matas (I'd eventually like to expand this corpus to other authors). Acquiring the digital text and working with Gephi I can do (or the latter learn), but it's the very important intermediary steps of preparing the data (extraction of names and places to getting those in a form—database? spreadsheet?—analyzable by Gephi) that I need help with. Any good reading, tutorials, or other resources out there? Other recommendations? I'm working on a Mac and have a limited knowledge of Python, so any Mac-friendly software would be a big help. Thanks! </p> http://digitalhumanities.org/answers/topic/have-you-used-mongodb-for-dh-project#post-1851 Sat, 05 Jan 2013 14:49:03 +0000 jlmcdonald@gmail.com 1851@http://digitalhumanities.org/answers/ <p>I think MongoDB has a lot to offer DH projects -- it basically incorporates some of the best features of a document-oriented database and a relational database all in one. For so much of the data often worked with in DH projects, that are sometimes squished and squeezed so much to fit a relational model that never quite works, a schemaless approach to indexing and storing data like Mongo has works very well. I especially see value for those projects that don't need a full-blown structured repository like FEDORA, but that need more than flat files or relational DBs can do.</p> <p>Having said that, the biggest thing to consider is that you say using MySQL works very well already for you. If you can't tell that there is something missing, not-quite-right, or not efficient enough with your current setup, the transition to Mongo may not be worth the effort in the long run. Mongo is pretty straightforward to set up, but it does approach data differently and doesn't use SQL at all, so when you say you'd be rewriting the backend, you'd really be rewriting it! You have to evaluate your data itself to know if the effort is worth it. Is your data tabular, or document-based? Are you mostly dealing with numbers and simple structured data (with some relationships), or heavily semantic data with complex metadata?</p> <p>Perhaps another path to consider, if you decide to rewrite your back end, is to completely decouple it from your front end. Create a thin layer that is nothing but a RESTful API -- the api talks to the DB and only spits out structured JSON on the other end. This would also likely require some tweaking of your front end, but it will A) make it so you can then very easily write multiple other front ends (mobile, TV, etc) without a hitch, and B) 3 years from now, if you decide to change out the DB layer again, it's much easier to then create a new API generator that will offer the same data structures than it is to create an all new backend interface. There are lots of starting point solutions for quickly generating RESTful APIs whether your DB is MySQL, Mongo, or something else. </p> http://digitalhumanities.org/answers/topic/have-you-used-mongodb-for-dh-project#post-1850 Sat, 05 Jan 2013 11:19:42 +0000 boybot16@gmail.com 1850@http://digitalhumanities.org/answers/ <p>We have a DH project that's currently uses MySQL for the database, and it works very well.</p> <p>There has been much recent talk about NoSQL databases, particularly MongoDB, promising flexibility and scalability. We're considering re-writing the backend. Is anyone in the DH community seriously using MongoDB?</p> <p>If you have used MongoDB for a DH project, what has been your experience with it? Biggest joy? Gotchas to look out for? Is conversion worth it for libraries and archives open source projects? </p> http://digitalhumanities.org/answers/topic/is-anyone-using-the-rbmsacrl-vocabularies-binding-terms-or-type-evidence#post-1770 Mon, 29 Oct 2012 09:21:30 +0000 martin.delaiglesia@googlemail.com 1770@http://digitalhumanities.org/answers/ <p>Using Google (why didn't this occur to me earlier?), I've been able to find some more catalogues which use the Binding Terms thesaurus:<br /> - Boston Public Library (<a href="http://www.bpl.org/" rel="nofollow">http://www.bpl.org/</a>)<br /> - Columbia University Libraries (<a href="http://library.columbia.edu/" rel="nofollow">http://library.columbia.edu/</a>)<br /> - University of Illinois at Urbana-Champaign Library (<a href="http://www.library.illinois.edu/catalog/" rel="nofollow">http://www.library.illinois.edu/catalog/</a>)<br /> - Mirlyn / University of Michigan (<a href="http://mirlyn.lib.umich.edu" rel="nofollow">http://mirlyn.lib.umich.edu</a>)<br /> - National Library of New Zealand Catalogue (<a href="http://nlnzcat.natlib.govt.nz" rel="nofollow">http://nlnzcat.natlib.govt.nz</a>)<br /> - NCSU Libraries (<a href="http://catalog.lib.ncsu.edu" rel="nofollow">http://catalog.lib.ncsu.edu</a>)<br /> - Saint Louis University Libraries (<a href="http://libcat.slu.edu" rel="nofollow">http://libcat.slu.edu</a>) </p>