Unit 10 Blog Assignment
Federated searching has been a moving target for libraries for a generation. The problem of isolated or fragmented silos of information seems to grow as digital publishing becomes more prevalent. In a related problem, libraries typically face the issue of how do they promote the added value they create for the holdings in their collections and provide and expose those efforts to their users? A lot of effort has centered on joining the library ILS to the library's subscription databases and other full text sources of digital information. The University of Arizona Libraries OPAC which successfully blurs the distinction between searching full text and searching metadata is one instance of this effort. This was not always the case, as the library OPAC traditionally only searched professionally constructed cataloging metadata. The old model was often the source of frustration for library patrons who often had expectations that full text resources would be retrieved in search results rather than just the cataloging metadata representative of that resource.
Problem is that most federated search solutions offer patrons a sometimes too generic interface that severely limits power users, who may find that searching the “native” database directly a lot more satisfying. Search options embedded in the “native” database are not available from the federated solution. Interoperablity is a key concept that makes federated searching feasible. The Dublic Core vs MARC debate that has raged for years, is in a sense a debate about interoperability. By homogenizing metadata down to fifteen common core elements and requiring that those fifteen elements be included in all instances for every record in any openarchives collection, researchers are able to search across hundreds of OAI compliant collections at once. However, while crosswalks and mapping solutions may help, sometimes discover is limited by squeezing metadata into a dc straitjacket.
Now lets take a closer look at how these concepts work in some real world OAI federated repositories.
The Openarchives.eu federated repository is an awesome federated search solution providing researchers with access to OAI-PMH compliant digital respositories from a European perspective.
A list of repositories included is displayed just be clicking the OK button next to the search box on the home page. The Openarchives.eu site indexes 2235 repositories!
The basic search indexes the descriptions of each OAI-PMH repository rather than accessing the individual digital objects within collections. I think this is a neat solution because it quickly and easily brings researchers face to face with a repository likely to contain information relevant to their topic. The researcher can then search the native digital repository with all its search features for a specific digital object. This is a more sustainable solution than harvesting all the metadata representative of every digital object in every repository. For example I searched the word “chemistry” in the Openarchives.eu search box. The search generated a list of eleven OAI-PMH chemistry repositories. A researcher can then open any of the eleven and search each repository individually.
Another great federated repository is the “Sheet Music Consortium” (SMR) which harvests metadata from seven sheet music repositories. Sheet music is an excellent barometer of popular taste and attitudes and reveals many things about the concerns and issues of the day. While international contributions are encouraged, metadata must be in English. The seven repositories contain metadata descriptions in unqualified Dublin Core for almost 100,000 digital objects. To get an appreciation for the usefulness of the SMR I performed a simple search using the word “Saratoga” which resulted in 68 digital objects or pieces of sheet music with the word Saratoga in the title. While there are many duplicates in the results list, all of the items were relevant to my digital Saratoga Springs, NY collection.
The specificity of the SMR federated repository, is attractive since it helps to speed up discovery and returns relevant results to the reasearcher of a particular topic. On the other hand, the Google approach of indexing everything, while it is very popular, because the searcher will almost always find something on point, has a significant downside. The downside is the 2 million hits syndrome where so much is returned from a search that the user just browses a few pages and ignores the rest. The Google universality approach is one that I think drives the Oaister federated respository. As with Google, universality is both a strength and a weakness of the Oaiseter.org federated repository.
The universal scope of the Oaister.org federated repository makes it similar in ambition to Google, but the lack of specifity makes searching a hit or miss effort. I wasn't nearly as successful searching Oaister.org for sheet music associated with Saratoga Springs, NY as I was when I performed the same search in the Sheet Music Consortium federated repository. This underscores how in federated repository architecture, as in other things in life, sometimes “less is more.”
Sunday, October 31, 2010
Sunday, October 24, 2010
Unit 9 cataloging consistency
Unit 9 Blog
Cataloging metadata,digital repository software and queries across repositories.
Cataloging has always been a part time do it as needed sort of chore in my career. Not having the “luxury” of a full time cataloger on my staff, when something required original cataloging I selected myself under the old labor delegation theory of: I wouldn't assign something to somebody unless I was willing and able to do it myself. Sort of... Anyway, law libraries are fairly standardized collections of serials, monographs are few, with treatises also being serial in nature, a new cataloging entry is a rare event. So while I always knew that reference work begins with cataloging and classification, it wasn't something I had to do every day. This is all in the way of justifying some of problems I am having crafting subject listings, key words, tags, and categories descriptive of each item in my Saratoga Springs digital collection.
Digital collections are extraordinarily dependent on the construction of precise standardized metadata. Browsing the shelves, persistence and luck might avail a researcher sufficiently to find a poorly cataloged printed book or report. However, without excellent metadata, digital objects are lost.
My digital objects are fairly heterogeneous, with text, images, audio, and musical scores forcing me to build a sort of polyglot metadata stew. So with only ten objects in the collection and each of those being fairly distinctive and dissimilar from one another, I haven't focused much on consistency. This week's question has me thinking a lot more about it.
Consistency is something that every digital curator will have to confront sooner or later. Most collections are so new that metadata creation is probably still in the hands of the initial cataloger.
As the years pass and digital collections grow in scale and scope, people will leave their digital collections jobs and new people will be hired and it is likely that the new cataloger will diverge from the perspective and practice of his or her predecessor. Metadata consistency will likely suffer.
Something to watch out for.
Cataloging metadata,digital repository software and queries across repositories.
Cataloging has always been a part time do it as needed sort of chore in my career. Not having the “luxury” of a full time cataloger on my staff, when something required original cataloging I selected myself under the old labor delegation theory of: I wouldn't assign something to somebody unless I was willing and able to do it myself. Sort of... Anyway, law libraries are fairly standardized collections of serials, monographs are few, with treatises also being serial in nature, a new cataloging entry is a rare event. So while I always knew that reference work begins with cataloging and classification, it wasn't something I had to do every day. This is all in the way of justifying some of problems I am having crafting subject listings, key words, tags, and categories descriptive of each item in my Saratoga Springs digital collection.
Digital collections are extraordinarily dependent on the construction of precise standardized metadata. Browsing the shelves, persistence and luck might avail a researcher sufficiently to find a poorly cataloged printed book or report. However, without excellent metadata, digital objects are lost.
My digital objects are fairly heterogeneous, with text, images, audio, and musical scores forcing me to build a sort of polyglot metadata stew. So with only ten objects in the collection and each of those being fairly distinctive and dissimilar from one another, I haven't focused much on consistency. This week's question has me thinking a lot more about it.
Consistency is something that every digital curator will have to confront sooner or later. Most collections are so new that metadata creation is probably still in the hands of the initial cataloger.
As the years pass and digital collections grow in scale and scope, people will leave their digital collections jobs and new people will be hired and it is likely that the new cataloger will diverge from the perspective and practice of his or her predecessor. Metadata consistency will likely suffer.
Something to watch out for.
Sunday, October 17, 2010
Unit 8 EPrints week 1
Comment on how this installation compared with the Drupal and DSpace installation. Indicate what configuration/branding changes you made and how easy or difficult you think customizing the look and feel of the installation is, compared with Drupal and DSpace.
I found the customization of EPrints to be very arcane. While I tried to read and comprehend the EPrints documentation, the only instructions about branding that I found helpful were the code lines on page 5 of the Unit 8 Tech Assignment that describe how to customize the index.xpage file to reflect some unique text descriptive of my digital collection. Things like replacing the default EPrint logo with a logo emblematic of my digital collection require the admin to have previously set up the necessary linux permissions through the CLI. I just didn't know how to do that , however I hope somebody in the class will share their knowledge soon so I can get to the next level . Until then, I'll have to stop and wait for more infromation. I'll need a lot more help before I can undestand and execute the branding process.
It seems that both Drupal and DSpace allow for a lot of customization without having to go back to the CLI and set up permissions. As long as the user is signed on as the admin the Drupal and DSpace GUI enables one to go about doing a lot of things that in EPrints require many CLI preliminary steps.
Another cumbersome thing about EPrints is that before changes may take effect the software requires the admin to perform additional steps also executed from the CLI like $ bin/generate_static yourarchivename. These so called housekeeping chores complicate EPrints adminstration and the bin/generate command or some variation on it is required after every EPrints reconfiguration or it won't work. On some level I guess this is similar to making changes to a website and then having to refresh the browser to display the changes.
The EPrints install was a difficult process for me, with lots of missteps and returns to prior states of the VM. In fact things got so bad for me, that I had to delete the VM from my physical disk, and then somehow I lost my Ubuntu iso file, which forced me to download a new Ubuntu iso and then install a new VM before proceeding to the install and configure the EPrints repository software. Ugh!
The 23 steps on pages 11, 12, 13 that must be performed in order to properly configure EPrints were problematic for me. I wasn't always clear whether I needed to type yes , y or hit enter. I got it wrong lots of times, but I took the precaution of saving a prior VM state so I was usually able to recover. I also encountered instances where I needed to type su user1 before I could use sudo privileges, that was confusing and I am not entirely clear as to why, and it is something I had not observed using the other repository software products.
In conclusion I found EPrints to be very unforgiving and not at all intuitive. Experimenting with EPrints isn't an option since the EPrints indexer is very finicky and liable to crash the VM if the configuration isn't perfect.
I found the customization of EPrints to be very arcane. While I tried to read and comprehend the EPrints documentation, the only instructions about branding that I found helpful were the code lines on page 5 of the Unit 8 Tech Assignment that describe how to customize the index.xpage file to reflect some unique text descriptive of my digital collection. Things like replacing the default EPrint logo with a logo emblematic of my digital collection require the admin to have previously set up the necessary linux permissions through the CLI. I just didn't know how to do that , however I hope somebody in the class will share their knowledge soon so I can get to the next level . Until then, I'll have to stop and wait for more infromation. I'll need a lot more help before I can undestand and execute the branding process.
It seems that both Drupal and DSpace allow for a lot of customization without having to go back to the CLI and set up permissions. As long as the user is signed on as the admin the Drupal and DSpace GUI enables one to go about doing a lot of things that in EPrints require many CLI preliminary steps.
Another cumbersome thing about EPrints is that before changes may take effect the software requires the admin to perform additional steps also executed from the CLI like $ bin/generate_static yourarchivename. These so called housekeeping chores complicate EPrints adminstration and the bin/generate command or some variation on it is required after every EPrints reconfiguration or it won't work. On some level I guess this is similar to making changes to a website and then having to refresh the browser to display the changes.
The EPrints install was a difficult process for me, with lots of missteps and returns to prior states of the VM. In fact things got so bad for me, that I had to delete the VM from my physical disk, and then somehow I lost my Ubuntu iso file, which forced me to download a new Ubuntu iso and then install a new VM before proceeding to the install and configure the EPrints repository software. Ugh!
The 23 steps on pages 11, 12, 13 that must be performed in order to properly configure EPrints were problematic for me. I wasn't always clear whether I needed to type yes , y or hit enter. I got it wrong lots of times, but I took the precaution of saving a prior VM state so I was usually able to recover. I also encountered instances where I needed to type su user1 before I could use sudo privileges, that was confusing and I am not entirely clear as to why, and it is something I had not observed using the other repository software products.
In conclusion I found EPrints to be very unforgiving and not at all intuitive. Experimenting with EPrints isn't an option since the EPrints indexer is very finicky and liable to crash the VM if the configuration isn't perfect.
Sunday, October 10, 2010
Unit 7 Problem searching digital collections
Each of the digital repository platforms we have read about and or installed recently, including Omeka, DSpace, and Drupal, are among other things, designed to make digital collections more accessible and or more sustainable. I'm a little concerned that the proliferation of digital repository software might be creating a profusion of digital information silos, that subverts interoperability and introduces a new layer of complexity for scholarly communications. Put another way are all of these repositories sufficiently standardized to enable interoperability and effective search and retrieval across collections? I am prompted to write this because of my experience trying to search for a digital repository similar to my Saratoga Springs, NY local history collection that is also using DSpace. It sounded like a simple assignment, but I found it really isn't. The DSpace.org website while it has a linked listing of all institutions using DSpace doesn't have a search engine that searches and retrieves the deep content from those same DSpace collections. Nor apparently is there a comprehensive DSpace repository directory similar to those constructed by Google or Yahoo that permits finding content by browsing and drilling down from Broader Term to Narrower Term or Related Terms. Even if there were such a directory it would for the most part only facilitate search at the collection level rather than at the object level.
The “Digital Library” article from Wikipedia makes this point when it reveals that "Most digital libraries provide a search interface which allows resources to be found. These resources are typically deep web (or invisible web) resources since they frequently cannot be located by search engine crawlers. Some digital libraries create special pages or sitemaps to allow search engines to find all their resources.”
The same point is made in our reading this week "Digital Repositories" by Nancy John, she writes: "One thing is certain, despite the protocols and technologies, original unscientific, non-standard Web-based exhibitions published by the world’s libraries remain far more accessible to the average user than the content-rich digital repositories they are creating. This is because they are part of the indexed, spidered, crawled open Web that is accessed via search engines. Digital repositories, for the time being, belong to the vast hidden Web,found through published citations, links and personal recommendations."
Bottom line: generally at the individual digital object level, items or nodes are largely invisible to popular search engines. True the OAI-PMH is supposed to gather sufficient metadata to allow popular search engines to provide some degree of federated searching across numerous digital repositories, but that depends entirely on the richness and depth of the metadata created for each object. At best OAI-PMH only gathers the metadata associated with each digital item, making it searchable; directly searching the full text of each individual item in the digital collection is beyond the capabilites of popular search engines. Since we are far away from anything like the semantic web, images, audio and video files remain totally dependent on metadata for discovery.
It seems to me that repositories using DSpace and other digital repository software are most effectively searched one at a time at the web site level, sometimes using the browse function or the sitemap feature. This approach is not very scalable and can hardly be said to be the best solution for improving scholarly communications. Some libraries expose their IR content to the campus through the OPAC but usually that solution is limited to that campus IR and it searches only the metadata not the full text.
This problem may grow in complexity and become even more intractable as the number of digital collections and digital objects in each digital collection continues to increase. Another problem with using digital repositories as tools of interdisciplinary communiction is that the content in many digital repositories is open only to authenticated users. We recently explored permissions and authentication at the OS and repository administration level and there are of course valid reasons for this security, not the least of which is ensuring that digital content remains authentic and unaltered. But securing content by employing user id's and passwords does prevent open access to those digital collections. The use of Captchas technology by digital collections curators is another instance of placing speed bumps in the way of automatic or robotic harvesting of metadata and content by popular search engines, which are the tool of choice for content discovery by many students and even scholars.
For now, one way to search the many thousands of Drupal, DSpace and Omeka digital collections is to first search what is already thoroughly and professionally indexed, namely scholarly peer reviewed published journals and studies found in traditional proprietary databases and then to locate by perserverance and serendipity among the embedded persistent URI footnotes and references in each relevant article those links to individual digital objects that are otherwise secured behind the high walls of specific digital collections. Of course this approach turns the digital collections movement on its head, as the digital repository is in part a response to the unacceptably high prices charged by publishers for access to journals. Using references embedded in the high priced journal article to find the "free" digital object in the IR is paradoxical at best.
The “Digital Library” article from Wikipedia makes this point when it reveals that "Most digital libraries provide a search interface which allows resources to be found. These resources are typically deep web (or invisible web) resources since they frequently cannot be located by search engine crawlers. Some digital libraries create special pages or sitemaps to allow search engines to find all their resources.”
The same point is made in our reading this week "Digital Repositories" by Nancy John, she writes: "One thing is certain, despite the protocols and technologies, original unscientific, non-standard Web-based exhibitions published by the world’s libraries remain far more accessible to the average user than the content-rich digital repositories they are creating. This is because they are part of the indexed, spidered, crawled open Web that is accessed via search engines. Digital repositories, for the time being, belong to the vast hidden Web,found through published citations, links and personal recommendations."
Bottom line: generally at the individual digital object level, items or nodes are largely invisible to popular search engines. True the OAI-PMH is supposed to gather sufficient metadata to allow popular search engines to provide some degree of federated searching across numerous digital repositories, but that depends entirely on the richness and depth of the metadata created for each object. At best OAI-PMH only gathers the metadata associated with each digital item, making it searchable; directly searching the full text of each individual item in the digital collection is beyond the capabilites of popular search engines. Since we are far away from anything like the semantic web, images, audio and video files remain totally dependent on metadata for discovery.
It seems to me that repositories using DSpace and other digital repository software are most effectively searched one at a time at the web site level, sometimes using the browse function or the sitemap feature. This approach is not very scalable and can hardly be said to be the best solution for improving scholarly communications. Some libraries expose their IR content to the campus through the OPAC but usually that solution is limited to that campus IR and it searches only the metadata not the full text.
This problem may grow in complexity and become even more intractable as the number of digital collections and digital objects in each digital collection continues to increase. Another problem with using digital repositories as tools of interdisciplinary communiction is that the content in many digital repositories is open only to authenticated users. We recently explored permissions and authentication at the OS and repository administration level and there are of course valid reasons for this security, not the least of which is ensuring that digital content remains authentic and unaltered. But securing content by employing user id's and passwords does prevent open access to those digital collections. The use of Captchas technology by digital collections curators is another instance of placing speed bumps in the way of automatic or robotic harvesting of metadata and content by popular search engines, which are the tool of choice for content discovery by many students and even scholars.
For now, one way to search the many thousands of Drupal, DSpace and Omeka digital collections is to first search what is already thoroughly and professionally indexed, namely scholarly peer reviewed published journals and studies found in traditional proprietary databases and then to locate by perserverance and serendipity among the embedded persistent URI footnotes and references in each relevant article those links to individual digital objects that are otherwise secured behind the high walls of specific digital collections. Of course this approach turns the digital collections movement on its head, as the digital repository is in part a response to the unacceptably high prices charged by publishers for access to journals. Using references embedded in the high priced journal article to find the "free" digital object in the IR is paradoxical at best.
Sunday, October 3, 2010
Unit 6 DSpace install on new Ubuntu Server VM
Unit 6 My Blog the DSpace install experience.
The install was really amazing, there were so many steps and so much text went flying by on the VM window that I found myself hypnotized by the complexity of the process. Users just see a pdf or jpg but the install configuration and administration of DSpace underscores the magnitude of the work, planning and precise execution needed to render digital objects. As I followed the install steps, my brain was focused only on the discrete task before it, and not upon the how or why a specific step was being performed. It was only after the successful installation of DSpace on my Ubuntu Server VM that I could take a breadth and look up and see the trail of virtual bread crumbs that traced the path that had just taken me so far.
As to the question could I have figured this out on my own or would I need help from a systems specialist? Despite how far I may have come from where I started way back at the first week of 672, it still would be necessary for me to consult with a specialist before I would try installing DSpace in a production environment. However, the install documentation is detailed enough so that with the DSpace technical community forum as a resource, I might only need to consult with a specialist to get over just a few speed bumps.
While I somehow initially went down the wrong path on page 6 of the install fortunately I had taken the precaution, per the advice inserted in the DSpace install document, to take snapshots of previous states of the VM so I just reverted and went forward. As I look back I am now very comfortable with the steps on pages 1 and 2 that create a new VM and install the Ubuntu Server. Many of the commands and steps after page 2 are also pretty familiar to me, for example using the linux sudo aptitude command to download code from partner repositories. I also know enough about the linux file structure to have a good idea of where all that code was deposited. These individual pieces of knowledge like the modules we added during the Drupal install have provided me with tools to better understand the DSpace install and other tasks I might confront in curating a digial collection.
The install was really amazing, there were so many steps and so much text went flying by on the VM window that I found myself hypnotized by the complexity of the process. Users just see a pdf or jpg but the install configuration and administration of DSpace underscores the magnitude of the work, planning and precise execution needed to render digital objects. As I followed the install steps, my brain was focused only on the discrete task before it, and not upon the how or why a specific step was being performed. It was only after the successful installation of DSpace on my Ubuntu Server VM that I could take a breadth and look up and see the trail of virtual bread crumbs that traced the path that had just taken me so far.
As to the question could I have figured this out on my own or would I need help from a systems specialist? Despite how far I may have come from where I started way back at the first week of 672, it still would be necessary for me to consult with a specialist before I would try installing DSpace in a production environment. However, the install documentation is detailed enough so that with the DSpace technical community forum as a resource, I might only need to consult with a specialist to get over just a few speed bumps.
While I somehow initially went down the wrong path on page 6 of the install fortunately I had taken the precaution, per the advice inserted in the DSpace install document, to take snapshots of previous states of the VM so I just reverted and went forward. As I look back I am now very comfortable with the steps on pages 1 and 2 that create a new VM and install the Ubuntu Server. Many of the commands and steps after page 2 are also pretty familiar to me, for example using the linux sudo aptitude command to download code from partner repositories. I also know enough about the linux file structure to have a good idea of where all that code was deposited. These individual pieces of knowledge like the modules we added during the Drupal install have provided me with tools to better understand the DSpace install and other tasks I might confront in curating a digial collection.
Subscribe to:
Posts (Atom)