Asserting the Patent Phrase Similarity Dataset


Patent paperwork usually use authorized and extremely technical language, with context-dependent phrases that will have meanings fairly totally different from colloquial utilization and even between totally different paperwork. The method of utilizing conventional patent search strategies (e.g., key phrase looking out) to look via the corpus of over 100 million patent paperwork might be tedious and end in many missed outcomes as a result of broad and non-standard language used. For instance, a “soccer ball” could also be described as a “spherical recreation gadget”, “inflatable sportsball” or “ball for ball sport”. Moreover, the language utilized in some patent paperwork might obfuscate phrases to their benefit, so extra highly effective pure language processing (NLP) and semantic similarity understanding can provide everybody entry to do a radical search.

The patent area (and extra common technical literature like scientific publications) poses distinctive challenges for NLP modeling as a consequence of its use of authorized and technical phrases. Whereas there are a number of generally used general-purpose semantic textual similarity (STS) benchmark datasets (e.g., STS-B, SICK, MRPC, PIT), to the perfect of our data, there are at present no datasets centered on technical ideas present in patents and scientific publications (the considerably associated BioASQ problem accommodates a biomedical query answering process). Furthermore, with the persevering with progress in measurement of the patent corpus (tens of millions of recent patents are issued worldwide yearly), there’s a must develop extra helpful NLP fashions for this area.

As we speak, we announce the discharge of the Patent Phrase Similarity dataset, a brand new human-rated contextual phrase-to-phrase semantic matching dataset, and the accompanying paper, offered on the SIGIR PatentSemTech Workshop, which focuses on technical phrases from patents. The Patent Phrase Similarity dataset accommodates ~50,000 rated phrase pairs, every with a Cooperative Patent Classification (CPC) class as context. Along with similarity scores which can be usually included in different benchmark datasets, we embody granular ranking lessons just like WordNet, corresponding to synonym, antonym, hypernym, hyponym, holonym, meronym, and area associated. This dataset (distributed below the Inventive Commons Attribution 4.0 Worldwide license) was utilized by Kaggle and USPTO because the benchmark dataset within the U.S. Patent Phrase to Phrase Matching competitors to attract extra consideration to the efficiency of machine studying fashions on technical textual content. Preliminary outcomes present that fashions fine-tuned on this new dataset carry out considerably higher than common pre-trained fashions with out fine-tuning.

The Patent Phrase Similarity Dataset

To higher prepare the subsequent technology of state-of-the-art fashions, we created the Patent Phrase Similarity dataset, which incorporates many examples to deal with the next issues: (1) phrase disambiguation, (2) adversarial key phrase matching, and (3) onerous unfavorable key phrases (i.e., key phrases which can be unrelated however obtained a excessive rating for similarity from different fashions ). Some key phrases and phrases can have a number of meanings (e.g., the phrase “mouse” might discuss with an animal or a pc enter gadget), so we disambiguate the phrases by together with CPC lessons with every pair of phrases. Additionally, many NLP fashions (e.g., bag of phrases fashions) won’t do effectively on information with phrases which have matching key phrases however are in any other case unrelated (adversarial key phrases, e.g., “container part” → “kitchen container”, “offset desk” → “desk fan”). The Patent Phrase Similarity dataset is designed to incorporate many examples of matching key phrases which can be unrelated via adversarial key phrase match, enabling NLP fashions to enhance their efficiency.

Every entry within the Patent Phrase Similarity dataset accommodates two phrases, an anchor and goal, a context CPC class, a ranking class, and a similarity rating. The dataset accommodates 48,548 entries with 973 distinctive anchors, break up into coaching (75%), validation (5%), and take a look at (20%) units. When splitting the information, the entire entries with the identical anchor are stored collectively in the identical set. There are 106 totally different context CPC lessons and all of them are represented within the coaching set.

Anchor Goal Context Score Rating
acid absorption absorption of acid B08 actual 1.0
acid absorption acid immersion B08 synonym 0.75
acid absorption chemically soaked B08 area associated 0.25
acid absorption acid reflux disorder B08 not associated 0.0
gasoline mix petrol mix C10 synonym 0.75
gasoline mix gasoline mix C10 hypernym 0.5
gasoline mix fruit mix C10 not associated 0.0
faucet meeting water faucet A22 hyponym 0.5
faucet meeting water provide A22 holonym 0.25
faucet meeting faculty meeting A22 not associated 0.0
A small pattern of the dataset with anchor and goal phrases, context CPC class (B08: Cleansing, C10: Petroleum, gasoline, gasoline, lubricants, A22: Butchering, processing meat/poultry/fish), a ranking class, and a similarity rating.

Producing the Dataset

To generate the Patent Phrase Similarity information, we first course of the ~140 million patent paperwork within the Google Patent’s corpus and mechanically extract necessary English phrases, that are usually noun phrases (e.g., “fastener”, “lifting meeting”) and purposeful phrases (e.g., “meals processing”, “ink printing”). Subsequent, we filter and preserve phrases that seem in not less than 100 patents and randomly pattern round 1,000 of those filtered phrases, which we name anchor phrases. For every anchor phrase, we discover the entire matching patents and the entire CPC lessons for these patents. We then randomly pattern as much as 4 matching CPC lessons, which turn out to be the context CPC lessons for the precise anchor phrase.

We use two totally different strategies for pre-generating goal phrases: (1) partial matching and (2) a masked language mannequin (MLM). For partial matching, we randomly choose phrases from the complete corpus that partially match with the anchor phrase (e.g., “abatement” → “noise abatement”, “materials formation” → “formation materials”). For MLM, we choose sentences from the patents that comprise a given anchor phrase, masks them out, and use the Patent-BERT mannequin to foretell candidates for the masked portion of the textual content. Then, the entire phrases are cleaned up, which incorporates lowercasing and the removing of punctuation and sure stopwords (e.g., “and”, “or”, “mentioned”), and despatched to professional raters for evaluation. Every phrase pair is rated independently by two raters expert within the know-how space. Every rater additionally generates new goal phrases with totally different scores. Particularly, they’re requested to generate some low-similarity and unrelated targets that partially match with the unique anchor and/or some high-similarity targets. Lastly, the raters meet to debate their scores and provide you with closing scores.

Dataset Analysis

To guage its efficiency, the Patent Phrase Similarity dataset was used within the U.S. Patent Phrase to Phrase Matching Kaggle competitors. The competitors was extremely popular, drawing about 2,000 opponents from world wide. Quite a lot of approaches had been efficiently utilized by the highest scoring groups, together with ensemble fashions of BERT variants and prompting (see the complete dialogue for extra particulars). The desk under exhibits the perfect outcomes from the competitors, in addition to a number of off-the-shelf baselines from our paper. The Pearson correlation metric was used to measure the linear correlation between the anticipated and true scores, which is a useful metric to focus on for downstream fashions to allow them to distinguish between totally different similarity scores.

The baselines within the paper might be thought of zero-shot within the sense that they use off-the-shelf fashions with none additional fine-tuning on the brand new dataset (we use these fashions to embed the anchor and goal phrases individually and compute the cosine similarity between them). The Kaggle competitors outcomes reveal that by utilizing our coaching information, one can obtain vital enhancements in contrast with current NLP fashions. We now have additionally estimated human efficiency on this process by evaluating a single rater’s scores to the mixed rating of each raters. The outcomes point out that this isn’t a very straightforward process, even for human consultants.

Mannequin Coaching Pearson correlation
word2vec Zero-shot 0.44
Patent-BERT Zero-shot 0.53
Sentence-BERT Zero-shot 0.60
Kaggle 1st place single Tremendous-tuned 0.87
Kaggle 1st place ensemble Tremendous-tuned 0.88
Human 0.93
Efficiency of widespread fashions with no fine-tuning (zero-shot), fashions fine-tuned on the Patent Phrase Similarity dataset as a part of the Kaggle competitors, and single human efficiency.

Conclusion and Future Work

We current the Patent Phrase Similarity dataset, which was used because the benchmark dataset within the U.S. Patent Phrase to Phrase Matching competitors, and reveal that by utilizing our coaching information, one can obtain vital enhancements in contrast with current NLP fashions.

Further difficult machine studying benchmarks might be generated from the patent corpus, and patent information has made its method into lots of as we speak’s most-studied fashions. For instance, the C4 textual content dataset used to coach T5 accommodates many patent paperwork. The BigBird and LongT5 fashions additionally use patents by way of the BIGPATENT dataset. The supply, breadth and open utilization phrases of full textual content information (see Google Patents Public Datasets) makes patents a singular useful resource for the analysis neighborhood. Prospects for future duties embody massively multi-label classification, summarization, data retrieval, image-text similarity, quotation graph prediction, and translation. See the paper for extra particulars.


This work was doable via a collaboration with Kaggle, Satsyil Corp., USPTO, and MaxVal. Due to contributors Ian Wetherbee from Google, Will Cukierski and Maggie Demkin from Kaggle. Due to Jerry Ma, Scott Beliveau, and Jamie Holcombe from USPTO and Suja Chittamahalingam from MaxVal for his or her contributions.


Please enter your comment!
Please enter your name here