Identifying duplicate content using statistically improbable phrases

Mounir Errami, Zhaohui Sun, Angela C. George, Tara C. Long, Michael A. Skinner, Jonathan D. Wren, Harold R. Garner

Research output: Contribution to journalArticle

15 Citations (Scopus)

Abstract

Motivation: Document similarity metrics such as PubMed's 'Find related articles' feature, which have been primarily used to identify studies with similar topics, can now also be used to detect duplicated or potentially plagiarized papers within literature reference databases. However, the CPU-intensive nature of document comparison has limited MEDLINE text similarity studies to the comparison of abstracts, which constitute only a small fraction of a publication's total text. Extending searches to include text archived by online search engines would drastically increase comparison ability. For large-scale studies, submitting short phrases encased in direct quotes to search engines for exact matches would be optimal for both individual queries and programmatic interfaces. We have derived a method of analyzing statistically improbable phrases (SIPs) for assistance in identifying duplicate content. Results: When applied to MEDLINE citations, this method substantially improves upon previous algorithms in the detection of duplication citations, yielding a precision and recall of 78.9% (versus 50.3% for eTBLAST) and 99.6% (versus 99.8% for eTBLAST), respectively.

Original languageEnglish (US)
Article numberbtq146
Pages (from-to)1453-1457
Number of pages5
JournalBioinformatics
Volume26
Issue number11
DOIs
StatePublished - May 13 2010

Fingerprint

Search Engine
Search engines
MEDLINE
Citations
PubMed
Program processors
Publications
Databases
Duplication
Query
Metric
Text
Similarity

ASJC Scopus subject areas

  • Biochemistry
  • Molecular Biology
  • Computational Theory and Mathematics
  • Computer Science Applications
  • Computational Mathematics
  • Statistics and Probability
  • Medicine(all)

Cite this

Errami, M., Sun, Z., George, A. C., Long, T. C., Skinner, M. A., Wren, J. D., & Garner, H. R. (2010). Identifying duplicate content using statistically improbable phrases. Bioinformatics, 26(11), 1453-1457. [btq146]. https://doi.org/10.1093/bioinformatics/btq146

Identifying duplicate content using statistically improbable phrases. / Errami, Mounir; Sun, Zhaohui; George, Angela C.; Long, Tara C.; Skinner, Michael A.; Wren, Jonathan D.; Garner, Harold R.

In: Bioinformatics, Vol. 26, No. 11, btq146, 13.05.2010, p. 1453-1457.

Research output: Contribution to journalArticle

Errami, M, Sun, Z, George, AC, Long, TC, Skinner, MA, Wren, JD & Garner, HR 2010, 'Identifying duplicate content using statistically improbable phrases', Bioinformatics, vol. 26, no. 11, btq146, pp. 1453-1457. https://doi.org/10.1093/bioinformatics/btq146
Errami M, Sun Z, George AC, Long TC, Skinner MA, Wren JD et al. Identifying duplicate content using statistically improbable phrases. Bioinformatics. 2010 May 13;26(11):1453-1457. btq146. https://doi.org/10.1093/bioinformatics/btq146
Errami, Mounir ; Sun, Zhaohui ; George, Angela C. ; Long, Tara C. ; Skinner, Michael A. ; Wren, Jonathan D. ; Garner, Harold R. / Identifying duplicate content using statistically improbable phrases. In: Bioinformatics. 2010 ; Vol. 26, No. 11. pp. 1453-1457.
@article{280286ebed534ccd87288e924acd62aa,
title = "Identifying duplicate content using statistically improbable phrases",
abstract = "Motivation: Document similarity metrics such as PubMed's 'Find related articles' feature, which have been primarily used to identify studies with similar topics, can now also be used to detect duplicated or potentially plagiarized papers within literature reference databases. However, the CPU-intensive nature of document comparison has limited MEDLINE text similarity studies to the comparison of abstracts, which constitute only a small fraction of a publication's total text. Extending searches to include text archived by online search engines would drastically increase comparison ability. For large-scale studies, submitting short phrases encased in direct quotes to search engines for exact matches would be optimal for both individual queries and programmatic interfaces. We have derived a method of analyzing statistically improbable phrases (SIPs) for assistance in identifying duplicate content. Results: When applied to MEDLINE citations, this method substantially improves upon previous algorithms in the detection of duplication citations, yielding a precision and recall of 78.9{\%} (versus 50.3{\%} for eTBLAST) and 99.6{\%} (versus 99.8{\%} for eTBLAST), respectively.",
author = "Mounir Errami and Zhaohui Sun and George, {Angela C.} and Long, {Tara C.} and Skinner, {Michael A.} and Wren, {Jonathan D.} and Garner, {Harold R.}",
year = "2010",
month = "5",
day = "13",
doi = "10.1093/bioinformatics/btq146",
language = "English (US)",
volume = "26",
pages = "1453--1457",
journal = "Bioinformatics",
issn = "1367-4803",
publisher = "Oxford University Press",
number = "11",

}

TY - JOUR

T1 - Identifying duplicate content using statistically improbable phrases

AU - Errami, Mounir

AU - Sun, Zhaohui

AU - George, Angela C.

AU - Long, Tara C.

AU - Skinner, Michael A.

AU - Wren, Jonathan D.

AU - Garner, Harold R.

PY - 2010/5/13

Y1 - 2010/5/13

N2 - Motivation: Document similarity metrics such as PubMed's 'Find related articles' feature, which have been primarily used to identify studies with similar topics, can now also be used to detect duplicated or potentially plagiarized papers within literature reference databases. However, the CPU-intensive nature of document comparison has limited MEDLINE text similarity studies to the comparison of abstracts, which constitute only a small fraction of a publication's total text. Extending searches to include text archived by online search engines would drastically increase comparison ability. For large-scale studies, submitting short phrases encased in direct quotes to search engines for exact matches would be optimal for both individual queries and programmatic interfaces. We have derived a method of analyzing statistically improbable phrases (SIPs) for assistance in identifying duplicate content. Results: When applied to MEDLINE citations, this method substantially improves upon previous algorithms in the detection of duplication citations, yielding a precision and recall of 78.9% (versus 50.3% for eTBLAST) and 99.6% (versus 99.8% for eTBLAST), respectively.

AB - Motivation: Document similarity metrics such as PubMed's 'Find related articles' feature, which have been primarily used to identify studies with similar topics, can now also be used to detect duplicated or potentially plagiarized papers within literature reference databases. However, the CPU-intensive nature of document comparison has limited MEDLINE text similarity studies to the comparison of abstracts, which constitute only a small fraction of a publication's total text. Extending searches to include text archived by online search engines would drastically increase comparison ability. For large-scale studies, submitting short phrases encased in direct quotes to search engines for exact matches would be optimal for both individual queries and programmatic interfaces. We have derived a method of analyzing statistically improbable phrases (SIPs) for assistance in identifying duplicate content. Results: When applied to MEDLINE citations, this method substantially improves upon previous algorithms in the detection of duplication citations, yielding a precision and recall of 78.9% (versus 50.3% for eTBLAST) and 99.6% (versus 99.8% for eTBLAST), respectively.

UR - http://www.scopus.com/inward/record.url?scp=77953896957&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77953896957&partnerID=8YFLogxK

U2 - 10.1093/bioinformatics/btq146

DO - 10.1093/bioinformatics/btq146

M3 - Article

VL - 26

SP - 1453

EP - 1457

JO - Bioinformatics

JF - Bioinformatics

SN - 1367-4803

IS - 11

M1 - btq146

ER -