Simulating Information Retrieval Test Collections

Simulating Information Retrieval Test Collections
Author :
Publisher : Springer Nature
Total Pages : 162
Release :
ISBN-10 : 9783031023231
ISBN-13 : 3031023234
Rating : 4/5 (31 Downloads)

Book Synopsis Simulating Information Retrieval Test Collections by : David Hawking

Download or read book Simulating Information Retrieval Test Collections written by David Hawking and published by Springer Nature. This book was released on 2022-06-01 with total page 162 pages. Available in PDF, EPUB and Kindle. Book excerpt: Simulated test collections may find application in situations where real datasets cannot easily be accessed due to confidentiality concerns or practical inconvenience. They can potentially support Information Retrieval (IR) experimentation, tuning, validation, performance prediction, and hardware sizing. Naturally, the accuracy and usefulness of results obtained from a simulation depend upon the fidelity and generality of the models which underpin it. The fidelity of emulation of a real corpus is likely to be limited by the requirement that confidential information in the real corpus should not be able to be extracted from the emulated version. We present a range of methods exploring trade-offs between emulation fidelity and degree of preservation of privacy. We present three different simple types of text generator which work at a micro level: Markov models, neural net models, and substitution ciphers. We also describe macro level methods where we can engineer macro properties of a corpus, giving a range of models for each of the salient properties: document length distribution, word frequency distribution (for independent and non-independent cases), word length and textual representation, and corpus growth. We present results of emulating existing corpora and for scaling up corpora by two orders of magnitude. We show that simulated collections generated with relatively simple methods are suitable for some purposes and can be generated very quickly. Indeed it may sometimes be feasible to embed a simple lightweight corpus generator into an indexer for the purpose of efficiency studies. Naturally, a corpus of artificial text cannot support IR experimentation in the absence of a set of compatible queries. We discuss and experiment with published methods for query generation and query log emulation. We present a proof-of-the-pudding study in which we observe the predictive accuracy of efficiency and effectiveness results obtained on emulated versions of TREC corpora. The study includes three open-source retrieval systems and several TREC datasets. There is a trade-off between confidentiality and prediction accuracy and there are interesting interactions between retrieval systems and datasets. Our tentative conclusion is that there are emulation methods which achieve useful prediction accuracy while providing a level of confidentiality adequate for many applications. Many of the methods described here have been implemented in the open source project SynthaCorpus, accessible at: https://bitbucket.org/davidhawking/synthacorpus/


Simulating Information Retrieval Test Collections Related Books

Simulating Information Retrieval Test Collections
Language: en
Pages: 162
Authors: David Hawking
Categories: Computers
Type: BOOK - Published: 2022-06-01 - Publisher: Springer Nature

DOWNLOAD EBOOK

Simulated test collections may find application in situations where real datasets cannot easily be accessed due to confidentiality concerns or practical inconve
Simulating Information Retrieval Test Collections
Language: en
Pages: 184
Authors: DAVID. BILLERBECK HAWKING (BODO. THOMAS, PAUL.)
Categories: History
Type: BOOK - Published: 2020-09-04 - Publisher: Synthesis Lectures on Informat

DOWNLOAD EBOOK

Test Collection Based Evaluation of Information Retrieval Systems
Language: en
Pages: 143
Authors: Mark Sanderson
Categories: Computers
Type: BOOK - Published: 2010-06-03 - Publisher: Now Publishers Inc

DOWNLOAD EBOOK

Use of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 19
String Processing and Information Retrieval
Language: en
Pages: 288
Authors: Shunsuke Inenaga
Categories: Computers
Type: BOOK - Published: 2016-09-20 - Publisher: Springer

DOWNLOAD EBOOK

This book constitutes the refereed proceedings of the 23rd International Symposium on String Processing and Information Retrieval, SPIRE 2016, held in Beppu, Ja
Methods for Evaluating Interactive Information Retrieval Systems with Users
Language: en
Pages: 246
Authors: Diane Kelly
Categories: Database management
Type: BOOK - Published: 2009 - Publisher: Now Publishers Inc

DOWNLOAD EBOOK

Provides an overview and instruction on the evaluation of interactive information retrieval systems with users.