Publications

Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric interpretation of these imbalances, often prioritising increased image attention while giving less consideration to the roles of the other modalities. In this study, we evaluate a more holistic, system-mediated account, which attributes these imbalances to functionally redundant system weights that reduce attention to image and textual inputs. We show that this framework offers a useful empirical perspective on the yes-bias, a common form of hallucination in which VLMs indiscriminately respond 'yes'. Causally redistributing attention from the system modality to image and textual inputs substantially suppresses this bias, often outperforming existing approaches. We further present evidence suggesting that system-mediated attention imbalances contribute to the yes-bias by encouraging a default reliance on coarse input representations, which are effective for some tasks but ill-suited to others. Taken together, these findings firmly establish system attention as a key factor in VLM hallucination and highlight its potential as a lever for mitigation.

Large Language Models (LLMs) have demonstrated impressive potential in
translating natural language (NL) instructions into program code. However, user
instructions often contain inherent ambiguities, making it challenging for LLMs
to generate code that accurately reflects the user's true intent. To address
this challenge, researchers have proposed to produce multiple candidates of the
program code and then rerank them to identify the best solution. In this paper,
we propose CodeRSA, a novel code candidate reranking mechanism built upon the
Rational Speech Act (RSA) framework, designed to guide LLMs toward more
comprehensive pragmatic reasoning about user intent. We evaluate CodeRSA using
one of the latest LLMs on a popular code generation dataset. Our experiment
results show that CodeRSA consistently outperforms common baselines, surpasses
the state-of-the-art approach in most cases, and demonstrates robust overall
performance. These findings underscore the effectiveness of integrating
pragmatic reasoning into code candidate reranking, offering a promising
direction for enhancing code generation quality in LLMs.

It has been frequently observed that human speakers align their language use
with each other during conversations. In this paper, we study empirically
whether large language models (LLMs) exhibit the same behavior of
conversational adaptation. We construct a corpus of conversations between LLMs
and find that two LLM agents end up making more similar syntactic choices as
conversations go on, confirming that modern LLMs adapt their language use to
their conversational partners in at least a rudimentary way.

Idioms present a unique challenge for language models due to their non-compositional figurative meanings, which often strongly diverge from the idiom's literal interpretation. This duality requires a model to learn representing and deciding between the two meanings to interpret an idiom in a figurative sense, or literally. In this paper, we employ tools from mechanistic interpretability to trace how a large pretrained causal transformer (LLama3.2-1B-base) deals with this ambiguity. We localize three steps of idiom processing: First, the idiom's figurative meaning is retrieved in early attention and MLP sublayers. We identify specific attention heads which boost the figurative meaning of the idiom while suppressing the idiom's literal interpretation. The model subsequently represents the figurative representation through an intermediate path. Meanwhile, a parallel bypass route forwards literal interpretation, ensuring that a both reading remain available. Overall, our findings provide a mechanistic evidence for idiom comprehension in an autoregressive transformer.

Natural language processing (NLP) models often replicate or amplify social bias from training data, raising concerns about fairness. At the same time, their black-box nature makes it difficult for users to recognize biased predictions and for developers to effectively mitigate them. While some studies suggest that input-based explanations can help detect and mitigate bias, others question their reliability in ensuring fairness. Existing research on explainability in fair NLP has been predominantly qualitative, with limited large-scale quantitative analysis. In this work, we conduct the first systematic study of the relationship between explainability and fairness in hate speech detection, focusing on both encoder- and decoder-only models. We examine three key dimensions: (1) identifying biased predictions, (2) selecting fair models, and (3) mitigating bias during model training. Our findings show that input-based explanations can effectively detect biased predictions and serve as useful supervision for reducing bias during training, but they are unreliable for selecting fair models among candidates.

Implicit discourse relation recognition (IDRR) -- the task of identifying the
implicit coherence relation between two text spans -- requires deep semantic
understanding. Recent studies have shown that zero- or few-shot approaches
significantly lag behind supervised models, but LLMs may be useful for
synthetic data augmentation, where LLMs generate a second argument following a
specified coherence relation. We applied this approach in a cross-domain
setting, generating discourse continuations using unlabelled target-domain data
to adapt a base model which was trained on source-domain labelled data.
Evaluations conducted on a large-scale test set revealed that different
variations of the approach did not result in any significant improvements. We
conclude that LLMs often fail to generate useful samples for IDRR, and
emphasize the importance of considering both statistical significance and
comparability when evaluating IDRR models.

Large Language Models (LLMs) can generate text by transferring style
attributes like formality resulting in formal or informal text. However,
instructing LLMs to generate text that when spoken, is more intelligible in an
acoustically difficult environment, is an under-explored topic. We conduct the
first study to evaluate LLMs on a novel task of generating acoustically
intelligible paraphrases for better human speech perception in noise. Our
experiments in English demonstrated that with standard prompting, LLMs struggle
to control the non-textual attribute, i.e., acoustic intelligibility, while
efficiently capturing the desired textual attributes like semantic equivalence.
To remedy this issue, we propose a simple prompting approach,
prompt-and-select, which generates paraphrases by decoupling the desired
textual and non-textual attributes in the text generation pipeline. Our
approach resulted in a 40% relative improvement in human speech perception, by
paraphrasing utterances that are highly distorted in a listening condition with
babble noise at a signal-to-noise ratio (SNR) -5 dB. This study reveals the
limitation of LLMs in capturing non-textual attributes, and our proposed method
showcases the potential of using LLMs for better human speech perception in
noise.