Usability of Keyword-driven Schema-agnostic Search - A Comparative Study of Keyword Search, Faceted...

KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)

1

Usability of Schema-agnostic SearchESWC 2010 paper presentation

Thanh Tran1, Tobias Mathäß2, Peter Haase2

1 AIFB (KIT)2 fluid Operations GmbH


Outline

Motivation - Complex Information Needs Overview - Schema-agnostic Search Frontend Conceptual Study of Schema-agnostic Search

Approaches from Process-oriented Point of View What are the steps? What are the actions required by the user? What is the state of the art?

Experimental Study of Schema-agnostic Search Systems using Task-based Experiments

2


Motivating Scenario: Complex information needs on the Web of Data

Growing number of triples and links Web of linked data Complex information needs (entity search, fact

search, and particularly relation search)

Tran Thanh: Schema-agnostic Search3

Mary is a novice computer science student at KIT. She is eager to learn more about this vast research field and decided to find information about research work of researchers at AIFB.

How to express complex information needs? Formal structured query languages require knowledge

about the underlying data schemas Web data is dynamically evolving

z) rk(y,researchWo AIFB) name(x, x), worksAt(y )researcher type(y,.,).( yxz


Contributions

We identified a particular class of search paradigms that we refer to as schema-agnostic search frontends

From a process-oriented viewpoint, we conducted a theoretical study of four widely used approaches representing the class of keyword-driven schema-agnostic search

We implemented these approaches under one common framework and carried out an experimental evaluation to assess efficiency, effectiveness and usability



Schema-agnostic Search Frontend

Schema-agnostic frontend the user does not know the schema

Schema-agnostic search approaches Application-specific form-based interfaces Generic form-based interfaces: faceted search & browsing Graphical query interfaces Natural language interfaces IR-style keyword search DB-style keyword search

– Computing queries vs. computing complex results – Auto-completion: query vs. result completion

5


Keyword-driven Schema-agnostic Search Process

Search paradigm from the process point of view A search process that starts with the use of

keywords followed by result inspection, refinement and further browsing, and ends with the retrieval of final results.1. IR-style keyword search + resource-based browsing2. IR-style keyword search + faceted-based browsing3. DB-style keyword search + result completion4. DB-style keyword search + query completion



Keyword Search

QueryCompletion

ResultCompletion

IR-style Keyword Search + Resource-based Browsing


Keyword Search Result Set Inspection

ResourceInspection

Resource-basedNavigation

Facet-basedResult Set Inspection

Facet-basedExpansion & Refinement

Query SetInspection

Further Inspection, Browsing and Retrieval

Result Tuple Set Inspection

Search Initial Inspection FacetedSearch

“Information about research work of researchers at AIFB”

Querytype(y,researcher), worksAt(y,x), name(x,AIFB), publication(y,z)

Keywordsk1=publication, k2=researcher, K3=aifb

Result SetElena SimperlAndreas HarthRudi StuderThanh Tran

Result SetAIFB’s Publication 1AIFB’s Publication 2

FacetsPublicationNameAddressProject

Result SetTran’s Publication 3Tran’s Publication 4

Result Tuple SetPublication 1 AIFB HarthPublication 2 AIFB StuderPublication 3 AIFB TranPublication 4 AIFB Tran

ResourceTran

k2,k3

k1,k2,k3

Further Inspection, Browsing and RetrievalSearch Initial Inspection


Keyword Search

QueryCompletion

ResultCompletion

IR-style Keyword Search + Faceted Browsing



ResourceInspection




Query SetInspection












ResourceTran

k2,k3

k1,k2,k3



Keyword Search

QueryCompletion

ResultCompletion

DB-style Keyword Search + Result Completion



ResourceInspection




Query SetInspection












ResourceTran

k2,k3

k1,k2,k3



Keyword Search

QueryCompletion

ResultCompletion

IR-style Keyword Search + Faceted Browsing



ResourceInspection




Query SetInspection












ResourceTran

k2,k3

k1,k2,k3



Experimental Study: Objectives

Given complex needs, we aimed to assess whether users are able to address them

(effectiveness) how much effort they have to invest to address

them (efficiency) and how is their experience as a whole (to

obtain initial usability results)



Experimental Study: Information Workbench

Vorlesung Knowledge Discovery - Institut AIFBFacets

Term Completions

Keywords

Query & Resultcompeltions


Experimental Study: Information Workbench

13


Experimental Study: Tasks

Nine tasks of varying complexity All tasks were solvable Tasks aborted when not solvable with reasonable

effort / when more than 3 minutes are needed


Task Description Search Type

Task 1 Find The Beatles (the band) Entity Search

Task 4 Find the occupation of Barack Obama Fact Search

Task 9 Find all people whose birthplace is Albany, together with their deathplace

Relation Search


Experimental Study: Overall Setting

19 participants DBpedia together with YAGO as a domain-

independent dataset Top-10 queries for query completion and top-25

results for result completion Standard commodity PCs, Intel Pentium Core2Duo

with 2 x 2Ghz CPU and 3 GB RAM All actions taken by the participants and the

system responses were recorded as screencasts Study results based on these screencasts and

questionnaires Tran Thanh: Schema-agnostic Search15


Experimental Study: Effectiveness


Entity-Queries Fact-Queries Tuple-Queries -

10.00

20.00

30.00

40.00

50.00

Percentage of Unsolved Tasks

keyword searchfaceted searchresult completionquery translation

No difference for entity search Slight difference for fact search Largest difference for relation search

Keyword search requires large browsing effort to go through all entities DB-style query completion best (5 % failure) DB-style result completion requires inspecting large number of results

& asking queries with very specific keywords


Experimental Study: Efficiency


Keywordsearch Faceted Search Result-Completion Query-Completion -

20.00

40.00

60.00

80.00

100.00

120.00

Average Time Needed (in seconds)Entity-QueriesFact-QueriesTuple-QueriesOverall

Time needed for keyword search and faceted search increases rapidly with the complexity of the information needs.

No significant increase for completion-based approaches Slight differences: result completions faster than query

completions in some cases


Experimental Study: Usablity


0

4

8

12

Most popular

very easy

pretty easy

fair pretty hard

very hard

0123456789

10

Ease of Query Completion

very easy

pretty easy

fair pretty hard

very hard

0123456789

10

Ease of Result Completion


Conclusions

Schema-agnostic search is essential for searching the large and evolving Web of data!

Keyword-driven schema-agnostic search is natural evolution of Web search, from simple document search to entity search to complex relation search that address complex information needs Simple IR-style keyword search is sufficient for simple

information needs (entity and fact search) Faceted search not superior when need is specific Query- and result completion have advantages for

complex needs (relation search) Query completion more effective and usable while

result completion slightly more efficient



Thanks!

Tran Duc [email protected]

http://sites.google.com/site/kimducthanh/

Date post:	11-May-2015
Category:	Technology
Upload:	thanh-tran
View:	334 times
Download:	0 times

Usability of Keyword-driven Schema-agnostic Search - A Comparative Study of Keyword Search, Faceted...

Technology