Overview of Patent Retrieval Task
at NTCIR-3
(2001/4 ~ 2002/10)
Organizers:
Makoto Iwayama (TIT / Hitachi)
Atsushi Fujii (Univ. of Tsukuba / JST)
Noriko Kando (NII)
Akihiko Takano (NII)
Collaboration with:
JIPA (Japan Intellectual Property Association)
Data provider:
PATOLIS Co.
What is Patent Retrieval Task
- Evaluation workshop for Patent Retrieval -
• Evaluating/comparing patent retrieval
techniques of participating systems
• Investigating evaluation methods of
patent retrieval techniques
• Constructing/providing test collections
for patent retrieval
• Providing a forum for research groups
of patent retrieval
How to Construct Test Collection
(Basic Idea)
assessors
(human experts)
search target
(doc.
collection)
system1
search
topic
system2
relevance
judgment
search
results1
search
results2
pooled
results
pooling
runs
Test Collection
evaluation (recall/precision)
relevant
docs.
Why Patent
• Patent information processing is a good test bed
of researches on information access
– Real task, real user, real information need
– Variety of tasks (IR, CLIR, Filtering, Summarization,
Text Mining)
• Unique document collection which is different
from other well-studied document collections
– Searching patents may be different from searching
news article (in TREC, CLEF) or searching technical
abstracts (in NTCIR).
Characteristics of Patent Documents
• Structured documents
– claims, purposes, effects, embodiments, etc.
• Unusual style for claims (esp. Japanese)
– Many sub-topics in single long sentence
• Large variation in document length
– The longest patent in our collection has
30,000 unique words !
Characteristics of Patent Documents
(cont)
• Many technical terms
• Many new terms (defined by applicants)
• General terms in claims
ex. floppy disk → external storage
• Classifications
– IPC(International Patent Classification)
– FI(File Index) and FT(File Forming Term) for
Japanese patents
Characteristics of Patent Retrieval
• Various search purposes
– High recall is necessary in some situations
– Claim interpretation is necessary in some
situations
• Difference between industries
– Chemical formulas/materials are important in
chemistry
– Images/diagrams are important in machinery
– IPC is primal in some industries, but not in some
others (ex. business method patents)
Main Task (Technical Survey)
non-expert
(manager)
expert
(patent searcher)
clipping
+
reading
news articles
memo
for supplementing,
focusing, etc.
• patent survey before
product development
• patents as technical
documents (cf. as legal
documents)
• cross-DB retrieval
How to Construct Test Collection
(Basic Idea)
assessors
(human experts)
search target
(doc.
collection)
system1
search
topic
system2
relevance
judgment
search
results1
search
results2
pooled
results
pooling
runs
evaluation (recall/precision)
relevant
docs.
Example of Search Topic (JA)
<TITLE>
<ARTICLE>
<SUPPLEMENT>
<DESCRIPTION>
<NARRATIVE>
<CONCEPT>
<PI>
<TOPIC>
<NUM>0004</NUM>
<LANG>JA</LANG>
<PURPOSE>技術動向調査</PURPOSE>
<TITLE>バーコードなどの符号を比較し優劣を判定する装置</TITLE>
<ARTICLE>
<A-DOC>
<A-DOCNO>JA-981031179</A-DOCNO>
<A-LANG>JA</A-LANG>
<A-SECTION>社会</A-SECTION>
<A-AE>無</A-AE>
<A-WORDS>189</A-WORDS>
<A-HEADLINE>エポック社の特許侵害訴訟、バンダイが敗訴--東京地裁</A-HEADLINE>
<A-DATE>1998-10-31</A-DATE>
<A-TEXT> カードゲームの特許を侵害されたとして、玩具(がんぐ)製造会社のエポック社がバン
ダイに2億6400万円の損害賠償を求めた訴訟で、東京地裁は30日、約1億1400万円の支
払いを命じた。 森義之裁判長は、バンダイが1992年7月~93年3月に製造・販売した小型
ゲーム機「スーパーバーコードウォーズ」のキー操作などの機能について「エポック社が持つ特許
の技術的範囲に属する」と指摘した。</A-TEXT>
</A-DOC>
</ARTICLE>
<SUPPLEMENT>バーコードなどを読み込み、これに基づく数値を比較して勝敗を決定していればよい。
</SUPPLEMENT>
<DESCRIPTION>バーコードなどの符号を複数読み込ませ、これら符号に対応する数値を比較するこ
とにより、これらの優劣/勝敗の判定を行うことで対戦を行う装置にはどのようなものがあるか。
</DESCRIPTION>
<NARRATIVE>「スーパーバーコードウォーズ」とは、小型ゲーム機の一種であり、キャラクターな
どが描かれたカードに記録されたバーコードを読み込ませ、プレーヤーが攻撃や防御などのキー操
作を行うことで、半リアルタイムに対戦を行うものである。符号の例としては、バーコードや磁気
コードなどがあるが、これらに限定するものではない。</NARRATIVE>
<CONCEPT>符号 バーコード コード 優劣 勝敗 比較 判定</CONCEPT>
<PI>PATENT-KKH-G-H01-333373</PI>
</TOPIC>
Search Topics
• 31 Japanese topics created by JIPA
• English, Korean and Chinese (simplified
and traditional) topics translated from
Japanese
How to Construct Test Collection
(Basic Idea)
assessors
(human experts)
search target
(doc.
collection)
system1
search
topic
system2
relevance
judgment
search
results1
search
results2
pooled
results
pooling
runs
evaluation (recall/precision)
relevant
docs.
Patent Collections (from PATOLIS Co.)
Type
Format
Language
Years
Number of
documents
Bytes
kkh
jsh
paj
Full text
Free
Japanese
98,99
697,262
abstract
SGML-like
Japanese
95 - 99
1,706,154
abstract
SGML-like
English
95 - 99
1,701,339
18,139M
1,883M
2,711M
kkh: Publication of unexamined patent applications
jsh: JAPIO Patent Abstracts
paj: Patent Abstracts Japan
Relationships between Patent
Collections
kkh: (98,99)
Full texts with
author’s abstracts
(in Japanese)
Modification of the original abstracts
(by JAPIO experts)
length normalization around 400 words
and term normalization
jsh: (95-99)
Abstracts
(in Japanese)
Translation
(by JAPIO experts)
paj: (95-99)
Abstracts
(in English)
How to Construct Test Collection
(Basic Idea)
assessors
(human experts)
search target
(doc.
collection)
system1
search
topic
system2
relevance
judgment
search
results1
search
results2
pooled
results
pooling
runs
evaluation (recall/precision)
relevant
docs.
Runs
• Mandatory runs
– Search topic fields:
<ARTICLE> + <SUPPLEMENT> only (i.e. cross-DB)
– Search target:
full texts (98, 99)
– manual or automatic retrieval
– mono-lingual or cross-lingual retrieval
• Optional runs
– TREC-styled ad-hoc runs recommended
search from <DESCRIPTION> (+ <NARRATIVE>)
How to Construct Test Collection
(Basic Idea)
assessors
(human experts)
search target
(doc.
collection)
system1
search
topic
system2
relevance
judgment
search
results1
search
results2
pooled
results
pooling
runs
evaluation (recall/precision)
relevant
docs.
Manual Search before Pooling
JIPA (Japan Intellectual
Property Association)
by systems and experts
relevant
docs.
assessors
(human experts)
by experts only
by systems only
preliminary
search
evaluation
relevant
docs.
search
results1
original
pool
systems
search
results2
pooling
skimming
relevance
judgments
skimmed
pool
relevant
docs.
Evaluation
• Recall/Precision graph
• Average precision
• R-precision
Submitted Runs
• 36 runs from 8 groups
– 6 groups from Japan, 2 groups from
overseas
– 5 groups from universities, 3 groups from
companies
– 20 mandatory runs, 16 optional runs
– 5 cross-lingual runs (EJ) from 2 groups
Recall/Precision
-- mandatory, A -A, mandatory
0.8
0.7
0.6
LAPIN4(A)
DTEC1(M)
precision
0.5
DOVE4(M)
brklypat1(A)
0.4
daikyo(M)
SRGDU5(A)
0.3
IFLAB6(A)
brklypat3(A,E)
0.2
A: auto
M: manual
0.1
E: English topics
0
0
0.1
0.2
0.3
0.4
0.5
recall
0.6
0.7
0.8
0.9
1
Recall/Precision
-- mandatory, A+B -A+B, mandatory
0.8
0.7
0.6
LAPIN4(A)
DOVE4(M)
precision
0.5
DTEC1(M)
daikyo(M)
0.4
brklypat1(A)
SRGDU3(A)
0.3
IFLAB6(A)
brklypat3(A,E)
0.2
A: auto
M: manual
0.1
E: English topics
0
0
0.1
0.2
0.3
0.4
0.5
recall
0.6
0.7
0.8
0.9
1
Recall/Precision
-- optional, A -A, optional
0.8
0.7
LAPIN1(A,TDNC)
brklypat2(A,DN)
0.6
DOVE3(A,DN)
DOVE2(A,D)
precision
0.5
SRGDU6(A,DN)
IFLAB2(A,D)
IFLAB3(A,DN)
0.4
IFLAB7(A,T)
brklypat4(A,DN,E)
0.3
A: auto
M: manual
0.2
T: TITLE
D: DESCRIPTION
N: NARRATIVE
C: CONCEPT
0.1
E: English Topics
0
0
0.1
0.2
0.3
0.4
0.5
recall
0.6
0.7
0.8
0.9
1
Recall/Precision
-- optional, A+B -A+B, optional
0.8
0.7
LAPIN1(A,TDNC)
DOVE3(A,DN)
0.6
brklypat2(A,DN)
DOVE2(A,D)
precision
0.5
IFLAB2(A,D)
SRGDU4(A,DN)
IFLAB4(A,DN)
0.4
IFLAB7(A,T)
brklypat4(A,DN,E)
0.3
A: auto
M: manual
0.2
T: TITLE
D: DESCRIPTION
N: NARRATIVE
C: CONCEPT
0.1
E: English Topics
0
0
0.1
0.2
0.3
0.4
0.5
recall
0.6
0.7
0.8
0.9
1
Comparable Runs from the
same group (brkly)
precision
0.8
0.7
ad-hoc run from Japanese
0.6
cross-DB run from Japanese
0.5
ad-hoc run from English
0.4
cross-DB run from English
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
recall
0.6
0.7
0.8
0.9
1
Some Results
• Cross-DB retrieval is more difficult than
ad-hoc retrieval
• Cross-lingual retrieval (EJ) is more
difficult than Mono-lingual retrieval (JJ)
• Pseudo relevance feedback may be
effective
Refer to our SIGIR2003 paper (to appear)
for more detail of glass box evaluations
Median of Mean Average Precisions
0.8
A
A+B
0.7
median of average precisions
0.6
0.5
0.4
0.3
0.2
0.1
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
topic ID
Human Experts vs. Systems
- breakdown of relevant documents 450
BS
AS
BJ
AJ
BU
AU
400
number of documents
350
300
250
200
150
100
50
0
BS
AS
BJ
AJ
BU
AU
1
2
3
4
1 34 44 2
0
5 27 2
10 7
2
2
4
0
0
0
7 13 4
4
22 13 6 10
5
6
7
8
9 10 11 12 13
15 43 9 29 7
3
6 15 0
0
0 55 0
1
0
1 57 6
2
0
9 42 10 3
0 47 22
0
0 23 10 17 11 1 173 12
6
0
5 33 7
4
3 29 5
15 12 27 23 18 15 4 101 22
14 15 16 17
6
2
2 15
37 0
2
0
2
8
4
9
10 1
1
2
10 3
1
8
17 5 15 5
topic ID
18 19 20 21 22
5
4
0
1
0
6
9
1
0
1
36 0
5
0
0
11 0
0
2
4
18 2
1
1
0
17 36 4
8
4
23 24
1
7
1 10
2 33
2 108
4 16
4 72
25
15
49
3
12
6
49
26
17
11
16
6
16
12
27 28 29 30 31
26 0
0
1 16
5 35 0
5 15
18 3
0
2 19
7
7
0
1 16
38 7
5
7
4
19 40 6
3 16
Human Experts vs. Systems
- recall of the “A” relevant documents experts found Recall of AJ+AU
1
0.9
0.8
0.7
recall
0.6
all
mandatory
mandatory (auto)
0.5
0.4
0.3
0.2
0.1
0
0
100
200
300
400
500
ranking
pooling (rank = 30)
600
700
800
900
1000
Results 2
• Medians of Mean Average Precisions
vary topic by topic.
• Numbers of relevant documents
systems/experts found vary topic by
topic.
Proposal-based task was also
supported
• CRL
– Using the Diff Command in Patent
Documents
• TIT
– Rhetorical Structure Analysis of Japanese
Patent Claims using Cue Phrases
Summary
• The first evaluation workshop for patent
retrieval
• Three kinds of patent collections
• Technical survey task
– Cross-DB / Cross-lingual retrieval
• Relevance judgment by expert patent
searchers (JIPA)
Patent Retrieval Task at NTCIR-4
(currently on “Dry Run”)
• Joint project with JIPA
• 10 years’ Japanese full texts (5 years’
for search target)
• 10 years’ English Abstracts
• Invalidity search from claims
• Feasibility task for automatically
creating “Patent Map”
ダウンロード

Japan Intellectual Property Association