United States

This study investigates the ability of Large Language Models (LLMs) to understand and interpret Chinese Xiehouyu, a unique linguistic phenomenon in Chinese involving a riddle and sometimes an explicit answer that requires complex reasoning to arrive at the correct non-literal meaning. 
We first create a new evaluation benchmark, X-Riddles, with 345 annotated xiehouyu of three categories: polysemy, homophony, and direct, and 671 sentences containing these xiehouyu.
We then evaluate LLMs through a series of experiments focusing on their proficiency in matching the riddle with the answer and explaining the linguistic phenomena involved. 
Our results indicate that while LLMs perform well on high-frequency xiehouyu, their accuracy drops significantly on low-frequency ones, especially those involving homophonic puns, whereas for human subjects, the gap between high- and low-frequency ones is much smaller.
This suggests that unlike humans, LLMs cannot perform reasoning based on linguistic and world knowledge for low-frequency xiehouyu. 
Their high performance on high-frequency ones may result from memorizing relevant information in the training data rather than actual reasoning. 
Our findings expose LLMs&#39; inability in understanding the non-literal meaning in xiehouyu and contributes to the growing literature on LLMs&#39; comprehension of the nuances of figurative language.

NAACL 2025

X-Riddles: Evaluating the Reasoning Abilities of LLMs with Chinese Xiehouyu Riddles

xiehouyu

idiomatic language

model evaluation

reasoning

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

AAAI 2022

This study investigates the ability of Large Language Models (LLMs) to understand and interpret Chinese Xiehouyu, a unique linguistic phenomenon in Chinese involving a riddle and sometimes an explicit answer that requires complex reasoning to arrive at the correct non-literal meaning. 
We first create a new evaluation benchmark, X-Riddles, with 345 annotated xiehouyu of three categories: polysemy, homophony, and direct, and 671 sentences containing these xiehouyu.
We then evaluate LLMs through a series of experiments focusing on their proficiency in matching the riddle with the answer and explaining the linguistic phenomena involved. 
Our results indicate that while LLMs perform well on high-frequency xiehouyu, their accuracy drops significantly on low-frequency ones, especially those involving homophonic puns, whereas for human subjects, the gap between high- and low-frequency ones is much smaller.
This suggests that unlike humans, LLMs cannot perform reasoning based on linguistic and world knowledge for low-frequency xiehouyu. 
Their high performance on high-frequency ones may result from memorizing relevant information in the training data rather than actual reasoning. 
Our findings expose LLMs' inability in understanding the non-literal meaning in xiehouyu and contributes to the growing literature on LLMs' comprehension of the nuances of figurative language.

poster

### Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Welcome to the 2025 meeting of the Nations of the Americas Chapter of the Association for Computational Linguistics! I am proud to help organize the first NAACL conference to carry the new name of our organization, one that emphasizes inclusion for all of the Americas. I am also pleased to welcome you to Albuquerque, New Mexico, a state whose unique blend of cultural influences will make for an excellent backdrop for NAACL 2025, especially with this year’s special theme on NLP in a Multicultural World. 
**[Continue reading...](https://drive.google.com/file/d/1jX-qGhqVSZZCIrAnJaz798pflu5Irrdn/view?usp=sharing)**

*- Colin Cherry, Google, NAACL 2025 General Chair* 

[![](https://assets.underline.io/markdown_image/1/image/b087f8a4dc5816d6e1a6514e59c59ac3.png)](https://drive.google.com/file/d/1T96GzPqObXrMln2BMByCSXTSizjTg69P/view?usp=sharing)

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

To access NAACL 2025 event page you are required to register. Please follow [**this link**](https://2025.naacl.org/registration/registration/) to register. Access will depend on your registration type.

Please register!

Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

This paper presents a research thesis proposal to develop a generalizable Native American language identification system, the first of its kind. Despite their cultural and historical significance, Native American languages remain entirely unsupported by major commercial language identification systems. This omission not only underscores the systemic neglect of endangered languages in technological development, but also highlights the urgent need for dedicated, community-driven solutions. We propose a two-pronged approach: (1) systematically curating linguistic resources across all Native American languages for robust training, and (2) tailored data augmentation to generate synthetic yet linguistically coherent training samples. As proof of concept, we extend an existing rudimentary Athabaskan language classifier by integrating Plains Apache, an extinct Southern Athabaskan language, as an additional language class. We also adapt a data generation framework for low-resource languages to create synthetic Plains Apache data, highlighting the potential of data augmentation. This proposal advocates for a community-driven, technological approach to supporting Native American languages.

What is it? Towards a Generalizable Native American Language Identification System

SFT (Supervised Fine-Tuning) and PO (Preference Optimization) are key methods for enhancing language models and aligning them with human preferences. However, scaling preference datasets for PO training is challenging, leading AI customer support systems to rely on SFT. To address this, we propose a new methodology to generate customer preference datasets without human intervention based on public dataset for SFT. Our approach includes sentiment analysis to classify responses as aligned or not aligned with customer preferences, fine-tuning models to generate aligned responses, and applying advanced sampling and evaluation techniques to ensure diversity and quality. Ultimately, we generated 1,174 customer preference datasets based on 357 test datasets, and through experiments, we confirmed that the AI customer support system trained on these datasets is capable of carefully considering customer emotions and generating professional and appropriate responses.

Sentimatic: Sentiment-guided Automatic Generation of Preference Datasets for Customer Support Dialogue System

This paper presents a federated learning system with differential privacy for hate speech detection, tailored to low-resource languages. By fine-tuning pre-trained language models, ALBERT emerged as the most effective option for balancing performance and privacy. Experiments demonstrated that federated learning with differential privacy performs adequately in low-resource settings, though datasets with fewer than 20 sentences per client struggled due to excessive noise. Balanced datasets and augmenting hateful data with non-hateful examples proved critical for improving model utility. These findings offer a scalable and privacy-conscious framework for integrating hate speech detection into social media platforms and browsers, safeguarding user privacy while addressing online harm.

Privacy-Preserving Federated Learning for Hate Speech Detection

CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.
However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.
In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.
The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code will be publicly available.

Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation

Past vocabulary learning techniques identify the relevant vocabulary prior to training based on statistical and entropy-based assumptions that largely neglect the role of model training. Empirically, we observe that the trained translation models are induced to use a byte-pair encoding (BPE) vocabulary subset distinct from the original BPE vocabulary while leading to performance improvements upon retraining with the induced vocabulary. In this paper, we first analyze this difference in the context of neural machine translation by examining the vocabulary and entropy shifts during self-training -- where each iteration creates a labeled dataset by pairing source sentences with the model's predictions to define a new vocabulary. Based on our findings, we propose *self-vocabularizing training*, which iteratively self-selects vocabulary that is both smaller in size and more optimal with up to 1.49 BLEU improvement. Moreover, we found that using deeper model architectures leads to both an increase in the use of unique tokens and a significant reduction in the vocabulary size by 6-8%.

Self-Vocabularizing Training for Neural Machine Translation

We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the ``Generative Pretrained Thoughtformer'' (GPTHF), a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. GPTHF retains GPT’s architecture, modifying only token interactions via dynamic sparse attention masks. 


Our experiments show that GPTHF achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized GPT models in the low-size regime. This is achieved through a unique generation method that caches and reuses sentence embeddings, allowing significant portions of the input to bypass large parts of the network.

Text Compression for Efficient Language Generation

When developing language technology, researchers have routinely turned to transfer learning to resolve the data scarcity conundrum presented in low-resource languages. As far as we know, this study is the first to evaluate the amount of documentation needed for transfer learning, specifically the smallest vocabulary size needed to create a sentence embedding space. In adopting widely spoken languages as a proxy for low-resource languages, our experiments show that the relationship between a sentence embedding's vocabulary size and performance is logarithmic with performance leveling at a vocabulary size of 25,000. It should be noted that this relationship cannot be replicated across all languages and this level of documentation does not exist for many low-resource languages. We do observe, however, that performance accelerates at a vocabulary size of <= 1000, a quantity that is present in most low-resource language documentation. These results can aid researchers in understanding whether a low-resource language has enough documentation necessary to support the creation of a sentence embedding and language model.

How many words does it take to understand a low-resource language?

A two-part affine approximation has been found to be a good approximation for transformer computations over certain subject-object relations. Adapting the Bigger Analogy Test Set, we show that the linear transformation W s , where s is a middle-layer representation of a subject token and W is derived from model derivatives, can accurately reproduce final object states for many relations. This linear technique achieves 90% faithfulness on morphological relations, with similar findings across languages and models. Our results suggest that some conceptual relationships in language models, such as morphology, are readily interpretable from latent space and are sparsely encoded by cross-layer linear transformations.

Linear Relational Decoding of Morphology in Language Models

Accurate detection of offensive language is essential for a number of applications related to social media safety. There is a sharp contrast in performance in this task between low- and high-resource languages. In this paper, we adapt fine-tuning strategies for the downstream task of offensive language detection to a previously unexplored low-resourced language, Sinhala. Using this approach, we introduce four models: ''Subasa-XLM-R'', which incorporates an intermediate Pre-Finetuning step using Masked Rationale Prediction. Two variants of ''Subasa-Llama'' and ''Subasa-Mistral'', are fine-tuned versions of Llama and Mistral, respectively, with a task-specific strategy. We evaluate our models on the SOLD benchmark dataset for Sinhala offensive language detection. All our models outperform existing baselines. Subasa-XLM-R achieves the highest Macro F1 score (0.84) surpassing state-of-the-art Large Language Models (LLMs) like GPT-4o when evaluated on the same SOLD benchmark dataset under zero-shot settings. The models and code are publicly available.

Subasa - Adapting Language Models for Low-resourced Offensive Language Detection in Sinhala

The frontier of large language model (LLM) development has largely been substantiated by knowledge-intensive tasks specified in English. In this proposed thesis, I argue for the key role that multilinguality occupies in the development of practical and knowledgeable LLMs. Importantly, I show that informed use of multilinguality enhances LLMs' capabilities, and our understanding thereof.


First, I consider practical methods to improve LLM's performance on standard natural language processing (NLP) tasks by leveraging their existing multilingual knowledge. Then, I investigate the underlying multilingual knowledge of LLMs with two benchmarks: on complex reasoning, and on territorial disputes. These benchmarks reveal LLMs' inconsistent performance across languages. I then design efficient techniques, both at inference-time and training-time, to address these discrepancies. I close with a proposed work extending the geopolitical benchmark to the retrieved augmented generation scenario.

Next from NAACL 2025

What is it? Towards a Generalizable Native American Language Identification System

Similar lecture

BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES