Publications - Current Year

We examine the ability of large language models (LLMs) to generate salient
(interesting) negative statements about real-world entities; an emerging
research topic of the last few years. We probe the LLMs using zero- and k-shot
unconstrained probes, and compare with traditional methods for negation
generation, i.e., pattern-based textual extractions and knowledge-graph-based
inferences, as well as crowdsourced gold statements. We measure the correctness
and salience of the generated lists about subjects from different domains. Our
evaluation shows that guided probes do in fact improve the quality of generated
negatives, compared to the zero-shot variant. Nevertheless, using both prompts,
LLMs still struggle with the notion of factuality of negatives, frequently
generating many ambiguous statements, or statements with negative keywords but
a positive meaning.

Despite their impressive scale, knowledge bases (KBs), such as Wikidata,
still contain significant gaps. Language models (LMs) have been proposed as a
source for filling these gaps. However, prior works have focused on prominent
entities with rich coverage by LMs, neglecting the crucial case of long-tail
entities. In this paper, we present a novel method for LM-based-KB completion
that is specifically geared for facts about long-tail entities. The method
leverages two different LMs in two stages: for candidate retrieval and for
candidate verification and disambiguation. To evaluate our method and various
baselines, we introduce a novel dataset, called MALT, rooted in Wikidata. Our
method outperforms all baselines in F1, with major gains especially in recall.

Fact-centric question answering (QA) often requires access to multiple,
heterogeneous, information sources. By jointly considering several sources like
a knowledge base (KB), a text collection, and tables from the web, QA systems
can enhance their answer coverage and confidence. However, existing QA
benchmarks are mostly constructed with a single source of knowledge in mind.
This limits capabilities of these benchmarks to fairly evaluate QA systems that
can tap into more than one information repository. To bridge this gap, we
release CompMix, a crowdsourced QA benchmark which naturally demands the
integration of a mixture of input sources. CompMix has a total of 9,410
questions, and features several complex intents like joins and temporal
conditions. Evaluation of a range of QA systems on CompMix highlights the need
for further research on leveraging information from heterogeneous sources.

Questions on class cardinality comparisons are quite tricky to answer and
come with its own challenges. They require some kind of reasoning since web
documents and knowledge bases, indispensable sources of information, rarely
store direct answers to questions, such as, ``Are there more astronauts or
Physics Nobel Laureates?'' We tackle questions on class cardinality comparison
by tapping into three sources for absolute cardinalities as well as the
cardinalities of orthogonal subgroups of the classes. We propose novel
techniques for aggregating signals with partial coverage for more reliable
estimates and evaluate them on a dataset of 4005 class pairs, achieving an
accuracy of 83.7%.

Models for conversational question answering (ConvQA) over knowledge graphs
(KGs) are usually trained and tested on benchmarks of gold QA pairs. This
implies that training is limited to surface forms seen in the respective
datasets, and evaluation is on a small set of held-out questions. Through our
proposed framework REIGN, we take several steps to remedy this restricted
learning setup. First, we systematically generate reformulations of training
questions to increase robustness of models to surface form variations. This is
a particularly challenging problem, given the incomplete nature of such
questions. Second, we guide ConvQA models towards higher performance by feeding
it only those reformulations that help improve their answering quality, using
deep reinforcement learning. Third, we demonstrate the viability of training
major model components on one benchmark and applying them zero-shot to another.
Finally, for a rigorous evaluation of robustness for trained models, we use and
release large numbers of diverse reformulations generated by prompting GPT for
benchmark test sets (resulting in 20x increase in sizes). Our findings show
that ConvQA models with robust training via reformulations, significantly
outperform those with standard training from gold QA pairs only.

Large Language Models (LLMs) have taken Knowledge Representation -- and the
world -- by storm. This inflection point marks a shift from explicit knowledge
representation to a renewed focus on the hybrid representation of both explicit
knowledge and parametric knowledge. In this position paper, we will discuss
some of the common debate points within the community on LLMs (parametric
knowledge) and Knowledge Graphs (explicit knowledge) and speculate on
opportunities and visions that the renewed focus brings, as well as related
research topics and challenges.

General-purpose knowledge bases (KBs) are a cornerstone of knowledge-centric
AI. Many of them are constructed pragmatically from Web sources, and are thus
far from complete. This poses challenges for the consumption as well as the
curation of their content. While several surveys target the problem of
completing incomplete KBs, the first problem is arguably to know whether and
where the KB is incomplete in the first place, and to which degree.
In this survey we discuss how knowledge about completeness, recall, and
negation in KBs can be expressed, extracted, and inferred. We cover (i) the
logical foundations of knowledge representation and querying under partial
closed-world semantics; (ii) the estimation of this information via statistical
patterns; (iii) the extraction of information about recall from KBs and text;
(iv) the identification of interesting negative statements; and (v) relaxed
notions of relative recall.
This survey is targeted at two types of audiences: (1) practitioners who are
interested in tracking KB quality, focusing extraction efforts, and building
quality-aware downstream applications; and (2) data management, knowledge base
and semantic web researchers who wish to understand the state of the art of
knowledge bases beyond the open-world assumption. Consequently, our survey
presents both fundamental methodologies and their working, and gives
practice-oriented recommendations on how to choose between different approaches
for a problem at hand.

The widespread usage of latent language representations via pre-trained
language models (LMs) suggests that they are a promising source of structured
knowledge. However, existing methods focus only on a single object per
subject-relation pair, even though often multiple objects are correct. To
overcome this limitation, we analyze these representations for their potential
to yield materialized multi-object relational knowledge. We formulate the
problem as a rank-then-select task. For ranking candidate objects, we evaluate
existing prompting techniques and propose new ones incorporating domain
knowledge. Among the selection methods, we find that choosing objects with a
likelihood above a learned relation-specific threshold gives a 49.5% F1 score.
Our results highlight the difficulty of employing LMs for the multi-valued
slot-filling task and pave the way for further research on extracting
relational knowledge from latent language representations.

Questions on class cardinality comparisons are quite tricky to answer and
come with its own challenges. They require some kind of reasoning since web
documents and knowledge bases, indispensable sources of information, rarely
store direct answers to questions, such as, ``Are there more astronauts or
Physics Nobel Laureates?'' We tackle questions on class cardinality comparison
by tapping into three sources for absolute cardinalities as well as the
cardinalities of orthogonal subgroups of the classes. We propose novel
techniques for aggregating signals with partial coverage for more reliable
estimates and evaluate them on a dataset of 4005 class pairs, achieving an
accuracy of 83.7%.

Recommender systems are most successful for popular items and users with
ample interactions (likes, ratings etc.). This work addresses the difficult and
underexplored case of supporting users who have very sparse interactions but
post informative review texts. Our experimental studies address two book
communities with these characteristics. We design a framework with
Transformer-based representation learning, covering user-item interactions,
item content, and user-provided reviews. To overcome interaction sparseness, we
devise techniques for selecting the most informative cues to construct concise
user profiles. Comprehensive experiments, with datasets from Amazon and
Goodreads, show that judicious selection of text snippets achieves the best
performance, even in comparison to ChatGPT-generated user profiles.

Structured knowledge bases (KBs) are an asset for search engines and other
applications, but are inevitably incomplete. Language models (LMs) have been
proposed for unsupervised knowledge base completion (KBC), yet, their ability
to do this at scale and with high accuracy remains an open question. Prior
experimental studies mostly fall short because they only evaluate on popular
subjects, or sample already existing facts from KBs. In this work, we perform a
careful evaluation of GPT's potential to complete the largest public KB:
Wikidata. We find that, despite their size and capabilities, models like GPT-3,
ChatGPT and GPT-4 do not achieve fully convincing results on this task.
Nonetheless, they provide solid improvements over earlier approaches with
smaller LMs. In particular, we show that, with proper thresholding, GPT-3
enables to extend Wikidata by 27M facts at 90% precision.

Structured knowledge bases (KBs) are a foundation of many intelligent
applications, yet are notoriously incomplete. Language models (LMs) have
recently been proposed for unsupervised knowledge base completion (KBC), yet,
despite encouraging initial results, questions regarding their suitability
remain open. Existing evaluations often fall short because they only evaluate
on popular subjects, or sample already existing facts from KBs. In this work,
we introduce a novel, more challenging benchmark dataset, and a methodology
tailored for a realistic assessment of the KBC potential of LMs. For automated
assessment, we curate a dataset called WD-KNOWN, which provides an unbiased
random sample of Wikidata, containing over 3.9 million facts. In a second step,
we perform a human evaluation on predictions that are not yet in the KB, as
only this provides real insights into the added value over existing KBs. Our
key finding is that biases in dataset conception of previous benchmarks lead to
a systematic overestimate of LM performance for KBC. However, our results also
reveal strong areas of LMs. We could, for example, perform a significant
completion of Wikidata on the relations nativeLanguage, by a factor of ~21
(from 260k to 5.8M) at 82% precision, usedLanguage, by a factor of ~2.1 (from
2.1M to 6.6M) at 82% precision, and citizenOf by a factor of ~0.3 (from 4.2M to
5.3M) at 90% precision. Moreover, we find that LMs possess surprisingly strong
generalization capabilities: even on relations where most facts were not
directly observed in LM training, prediction quality can be high.