The National Archives Labs

The Discovery service – UPDATED

We have released an update to our new, beta search facility.The Discovery service

Advanced search

The Discovery service remains in beta, or test, form, but we have added an advanced search function, the first in a series of updates that will see this new service refined and improved to fully meet the needs of users before it is implemented on The National Archives main website.

Your feedback

We’ll also be holding sessions on site to demonstrate the service and invite user feedback. The next session will take place at Kew from 14.00-16.00 on 7 September. Please email  discovery@nationalarchives.gov.uk for more information and to let us know if you’d like to take part.

We are releasing improvements as they are developed, rather than all at once, so we can continue to get your feedback and use it to further develop and refine this new service.

Each new release is prioritized based on your feedback and feature requests so please keep using Discovery and giving us feedback!

We’ve listed additions and bug fixes below. Each item is followed by a reference number, eg [1397] – please include the relevant number in your feedback, to help us prioritise our response and refinements.

Additions

Advanced search

  • Pre-select subjects [1397]
  • Search within i.e. WO, WO 95 [1371]
  • Boolean search [1396]
  • Change number of results displayed

Search results page

  • Search terms highlighted on search results page [1399]

Details page

  • Additional fields included [1448]

Bug fixes

  • Fixed titles being cut off mid word on search results page [1404]
  • Fixed items not showing description or title in browse view [1466]

Our next release in early August will include:

  • Download search results as CSV, HTML or XML
  • Context snippet shown on details page
  • Improvements to date range search, numbers and search results ordering

We need your help to develop and improve the system, so please let us know through the Comments box below what you think and what you would like to see.

You can also tweet your comments @UkNatArchives with the hashtag #tnadiscovery.

Comments (43)

  • Martin Tolley

    Just on a quick run through this seems to work and to work very quickly which is good. But for me the results take up a large amount of screen real-estate, it’s a case of forever scrolling, next paging etc. I’d much prefer to see more results, maybe more briefly at the one go.

  • Maureen Lister

    Just had a quick half hour look. Found all of the people that i have already obtained documents for. In a couple of cases they were much easier to find than when I first looked for them. As many of my ancestors have very common names, the advanced search will be interesting, also the search in a collection facility. this could hopefully save a lot of scrolling through time.

  • Chris Keene

    Hi
    New service looks great

    Would you mind sharing the software you used to build this on?

    Work with IT in a Library so always interested in what tools and services are used to create this sort of service.

    Thanks

    The National Archives reply:

    Chris – the Discovery System is an integration platform built using Service Oriented Architecture principles. This enables system extensibility, by writing reusable components, and integration with other systems.

    The software uses Microsoft .NET platform technologies and written in C#. The data store is a high-performance, document-based, open source MongoDB database. Search and taxonomy (subjects) functionality are powered by Autonomy indexes.

    The web user interface is Microsoft .NET MVC(Model-View-Controller) – for displaying search results, browsing, and displaying document details. The map functionality is powered by OpenLayers and JQuery.

    The core of the system is the Enterprise Service Bus (Neuron), which enables communication between all parts of the system.

    We hope this is useful, thanks for your interest.

  • Chris Willis

    It took me quite a while to notice the refinements box. Very useful. – Chris W

  • Chris Willis

    I feel a “regular expression” search, perhaps limited to a single word would be useful.

    E.g. Sithwood near Woking (“lost” in the standard work) occurs with many spellings. Although the possibilities might be greater a very large proportion can be subsumed in

    [SC][iy][td]+h?e?(w|uu)o+d+e?

    which assuming that the plusses represent just one or two (rather than the strict one or more), represents 10 choices -> 1024 possible spellings. (how about Invernessshire?)

    Chris W

  • Chris Willis

    I feel a “regular expression” search, perhaps limited to a single word would be useful.

    E.g. Sithwood near Woking (“lost” in the standard work) occurs with many spellings. Although the possibilities might be greater a very large proportion can be subsumed in

    [SC][iy][td]+h?e?(w|uu)o+d+e?

    which assuming that the plusses represent just one or two (rather than the strict one or more), represents 10 choices -> 1024 possible spellings. (how about Invernessshire?)

    Sorry. I should have added that * wildcarding is not very useful here, because of the resemblance to parts of this to other common forms such as South, Suth, wold and ward.

    Chris W

  • Peter Trott

    Tested this out with a couple of series I know well. I liked the way Discovery presented the results, particularly the introduction of highlighted best matches, with alternatives. This approach is very useful when searching for a known name (I research a period in history for a regimental museum). However when I changed to another research methodology I use, that is to trawl the records at an item level (I think that’s the name), so WOxx/xx level looking for say the records for a given year &/or location for a battalion or regiment then the way Discovery presents the results as folders with more limited “browse from here” than the current system is a major weakness. Not all research is conducted using a known name or targeted at an individual. In fact the level of detail in the results presented by current system when “browse from here” is selected appears to have reduced over the last 18 months with the introduction of the “Order and View Options” (very useful for digitised records but not for original items. More detail in the results please where original items are concerned.

    I have experience in online product development and usability so happy to help as and when I can with this.

  • Geoffrey Skelsey

    Thanks.

    I’m impressed (although initially sceptical that the existing catalogue access needed upheaval). It’s quick and ingenious and I like the ‘refinements’. Look forward to extended use.

  • Family Research – English, Scottish and Irish Genealogy » The Discovery service

    [...] product, but rather a solid base from which we can build, develop and enhance for years to come. for more click here Incoming LinksJessica's Genejournal: 107th Edition of the Carnival of [...]

  • Nigel Neil

    I agree with Martin Tooley, above, that an abbreviated version of the content would be useful, as in current Catalogue search. More to the point, an option box as to how results are to be presented is vital, just as with many online shops. Some people might prefer ‘relevance’, my preference is sometimes for date order, more often for numerical order by series, sub-series, then item. For example, as in the A2A portal, where results are ordered by Record Office and then by collection within it (on the right). The user can sub-select specific collections to study, and even search again later and items already seen by highlighted differently from those yet to be studied.

    You could usefully create easy links to research guidance documents for particular collections and subject areas.

    I would VERY much like to see a process of digitising and uploading (aas PDF) searchroom draft lists and similar finding aids for collections for which item-level formal cataloguing is not likely to be available for many years.

  • Tony Wilson

    My first impression is that the new system is currently a useful supplement to the existing catalogue search facility but not an adequate replacement.

    The main thing I miss is the “download search results as c.s.v.” feature of the current catalogue. This is the easiest way to avoid the time-consuming scrolling.

    The “refine” feature is useful, but can throw up bizarre associations when titles are truncated.

    It would be useful to have an indication of the availability of a digitized record at the first search level as done by the Australian National Archives. Likewise, it would be useful to have “digitized items only” as a refinement option.

    The order of presentation of the of the search results seems to be based on some sort of “relevance” ranking. Would it be possible to provide a facility to sort the results by Class, Series and Sub-series?

    These comments are based on a brief first exploration. I look forward to learning more of the features and capabilities and testing the planned developments.

    The National Archives reply:

    Thank you for sharing your observations and preferences – we hope you will check back and comment as we develop this service.

  • Evelyn Bromwich

    I was blessed to receive 13 more wills that I had not had before.Whoppee but I wish that one would tell a small tidbit with them. I also, was disapointed when I put in things I have actually researched before and they were not there. Benjamin Bromwich or John Bromwich etc of Birmingham. I feel that the fees for the copying and sending over seas should come down and they should be sent just like the wills. Much easier and faster. In fact birth, marriage, death certs would be welcomed that way. And to see the original I am unable to do because RC, SD is a long way from the National Archives…sigh..

    The National Archives reply:

    Thank you for your comments. Please note that The National Archives does not hold birth/marriage/death certificates. For help in searching for them (elsewhere) please see http://www.nationalarchives.gov.uk/records/birthmarriagedeathcertificates.htm .

  • Moira Draper

    Much easier than last time – well done!

  • Barbara Butler

    This new Discovery service still does not seem to find all the documents available. eg. “Walter Butler” 1697-1722, only shows 3 documents, yet I have obtained ten documents from National Archives after discovering their existence and reference numbers from outside sources. Will all of these documents eventually show up in an online search of the Discovery service, as it is improved in the future? If these documents are referenced in books, one would think that they should be on the National Archives reference database, and therefore show up in the Discovery search, even though they do not show up in the old catalogue system search facility. If someone orders an unlisted document it should then automatically be added to the system to make it easier for other researchers to find.

    The National Archives reply:

    Thank you for your feedback, we are aware of some of the issues with the Discovery service. Our users play a very valuable part in helping us develop the website and our resources, so do continue to check back and let us know what you think.

  • Martin Brady

    The old online ADM 175 Catalogue entry (Coastguard Records) was arranged so it was possible to follow the sequence of the documents; the new system makes it necessary to scroll through all the entries, arranged in what appears to be a random form. Bring back the old catalogue entry!!!!

    The National Archives reply:

    Thanks very much for your feedback. The Discovery service is designed to eventually replace the Catalogue and DocumentsOnline. This first release of the Discovery service is what we call ‘beta’; it is not a finished product and we hope you will continue to help test it.

  • Chris Morry

    The search engine is fast, as others have said, and appears to be isolating all the articles in the inventory for which the search term is valid. But it also includes many others that are only vaguely similar and not relevant. But at least the “hits” are ranked in order of likely relevance. I did try to order a copy of one of the articles that was isolated (PROB 31/1025/280) but the ordering service did not recognise this reference and would not proceed with the order for some reason.

    Overall this is a worthwhile advancement.

  • Matthew Camp

    I’d like to still be able to define a series from the outset. IE, if I know I want to find the reference for an Officer’s papers and they’re going to be in WO 374 I want to be able to specify that the search is only in that series at stage one rather than have to drill down through numerous filters. Looks like you need a ‘basic’ search and an ‘advanced’ search option.

    Also agree that an option for downloading as a CSV would be great.

    Keep up the good work.

  • Colin Jones

    On a quick check it would appear that the new system is not as good as the current catelogu search. I tried the word “Blackpole” and it only found one of the two files. MT6/2496/2 was found but not AVIA53/1.

  • carrie carnes

    I’m still trying to get at E179 tax rolls for Gloucestershire (and other counties that exchanged parishes with Gloucestershire until 1931, etc.) until 1362. Every time I type in E179, I get a lot of stuff from C; if I type in C74, which is supposed to be 1362, I get a lot of much later stuff. How am I supposed to be able to use this new catalogue?
    Thank you.
    Sincerely,Carrie Carnes

  • Reginald Rhodes

    Excellent idea, only searching for members of the Rhodes family in Reigate (Rhodes at Reigate) I get Rhodes for the whole country, and there are many… followed by everything happening to everybody in Reigate ! Should finda way to couple two names.

  • Graham Sturdy

    Searching for the retirement of an ancestor from the Metropolitan Police, I found that the date range would not accept a date later than 1900. Also auto fill put my post code in the Captcha code box. Confusing.

  • julie

    Looks good so far, it would be helpful to refine even further to reduce the number of hits by allowing specific date periods, rather than the current tick-box dates e.g. 1780-1815 vs having to tick for the 2 centuries that cover this period.

  • Tony

    Advanced Search looks good. Results seem to be in random sequence – would prefer them in document reference sequence or date sequence, or even better, a choice. Would also prefer document reference more prominent in results.

  • Laura Berry

    Just had a play around searching for names in series WO 97 as an example and noticed that the full description does not give any indication of what kind of records series WO 97 holds (unless you click the ‘Browse from this record’ option), whereas the old Catalogue breaks down a description for the department, series and sub-series on the ‘full details’ page, so you can see these are Soldiers’ Service Documents from Royal Hospital Chelsea. I guess there must be more examples like this. On the whole though I really like the Advanced Search and look forward to seeing more improvements.

    The National Archives reply:

    Thank you for all your feedback and suggestions for improvements and refinements. We will use this to inform our development of this resource. Please do continue to help test our ‘beta’ product.

  • Trevon

    I’m so glad that the itnernet allows free info like this!

  • deli

    I’m impressed (although initially sceptical that the existing catalogue access needed upheaval). It’s quick and ingenious and I like the ‘refinements’. Look forward to extended use.

  • william gordon

    browse facility 1 class tells me all i need to know

  • Colin Withers

    Are there any plans to introduce variation matching, so someone does not have to do separate searches on Smith, Smithe, Smyth, Smythe, etc? Most genealogy sites employ this so variations in name or placename spellings do not all require separate searches.

    The National Archives reply:

    There is currently a piece of work being done to look into name searching and the possibility of developing a name gazetteer, (along with the placename gazetteer currently in development).

  • Post – genealogy, genealogical, family tree research, reviews. | Empty Nest Genealogy

    [...] new Discovery Service will make it easier for everyone – novice to expert – to search and use the [...]

  • peter clark

    I put General Gordon into the search box and got 1,239,283 results!I put “General Gordon” in and then got what I was looking for, 126 of them. Surely most enquirers would expect the first to yield what the second did (if you are assuming unsophisticated enquirers I suggest they would not put in the quotes.

    The National Archives reply:

    Thank you for your feedback. We have included search tips which pop up when you click in the search box (on the right, in yellow) – these do state that putting words in inverted commas searches for an exact phrase. The initial search will merely have produced results containing these words. It displays in order of exact phrase followed by near. The filters can be used to narrow down search results. Users can also click on the advanced search button and this page clearly lays out how these words are being searched for – ie whether for the exact phrase or all or any of these search terms.

  • Chris Willis

    I am still unconvinced I like it, but IR58 is certainly improved by the inclusion of the district in each record.

    Wd it be possible to sort the results. The new record number wd be quite adequate.

    It makes it much easier to understand a block of similar (or sd be similar) records.

    Chris W

  • Colin Greenstreet

    Topic: User enrichment of Discovery records with standardized georeferences

    The Humanities Research Institute at Sheffield, together with the Institute of Historical Research and MOLA in London have recently launched a raterized version of John Rocque’s 1748 map of London. The Centre for Metropolitan History, working with MOLA, plan a downloadable rasterization of William Morgan’s

    Part of the project brief for Locating London’s Past was the public release of the Geocoder the project developed to facilitate automatic and manual matching of geographic data related to names and places in the associated project data. See http://www.locatinglondon.org/static/Geocoder.html

    What thinking is taking place within the Discovery Project team about the incorporation of georeferences to the search results delivered up by the Discovery search tool?

    Specifically, would TNA/Discovery consider building a facility for users to upload suggested georeferences to the metadata of individual records, using predefined geospatial definitions which might be incorporated into the Geocoder or into another tool?

    I am aware that this is a complicated subject, both technically and conceptually, given the migration of urban and rural “centres” within a physical location, and complex changes to administrative boundaries (such as London parishes). However, having attended the IHR Digital Seminar yesterday at Senate House, I came away with a sense that archives and academics need to coordinate on georeferencing. Otherwise there will need to be a big after the fact concordance project to link up different georeferencing initiatives around specific datasets.

  • Colin Greenstreet

    Topic: tagging

    It would be useful to understand from a technical perspective how the Tagging function has been defined and implemented, and is then used in your search methodology.

    For example, can tags be grouped into hierachies by the user (or automatically by some tag crawl function), e.g. “Inventory: vintner: St Bartholomew the Exchange” within “Inventory: distributive trades”. Or do you need to multiple tag: “Vintner”, “Distributive trades”, “St Barthoomew the Exchange”?

    How does the search function then use tags to rank possible results?

    Related to tagging, is there a facility to add as a tag, or as someother user contributed data, a link to another National Archives record, or indeed to a non-NA record discoverable through A2A (and thus through Discovery)?

    The National Archives reply:

    Thanks for contacting us. I’ve passed your comments to the Discovery team – they will be able to tell us more.

  • John Chandler

    The new system looks very helpful.
    From my trials it understandably majors on Titles of Files. In many there is a contents list and as a second stage I would see that as opening “The Box”

Leave a comment




Comment validation by @