United States

While Large Language Models (LLMs) can generate fluent and convincing responses, they are not necessarily correct. This is especially apparent in the popular decompose-then-verify factuality evaluation pipeline, where LLMs evaluate generated text by decomposing it into individual, valid claims. Factuality evaluation is especially important for medical answers, since incorrect medical information could seriously harm the patient. However, existing factuality systems are a poor match for the medical domain, as they are typically only evaluated on objective, entity-centric, formulaic texts such as biographies and historical topics. This differs from condition-dependent, conversational, hypothetical, sentence-structure diverse, and subjective medical answers, making decomposition into valid facts challenging. We propose MedScore, a new pipeline to decompose medical answers into condition-aware valid facts and verify against in-domain corpora. Our method extracts up to three times as many valid facts as existing methods, reducing hallucination and vague references, and retaining condition-dependency in facts. We also find MedScore is generalizable to non-medical domains without any specific tuning. The resulting factuality score substantially varies by decomposition method, verification corpus, and used backbone LLM, highlighting the importance of customizing each step for reliable factuality evaluation by using our generalizable and modularized pipeline for domain adaptation.

ACL 2026

MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification

medical chatbot evaluation

open-ended question answering

factuality evaluation

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. <br>
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) <br><br><br>
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>
<br><br><br><br>

To access this event page, you need to log in with the **email address you registered with**. <br>Access credentials will be sent to your email from Underline -  subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

Registration Required

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

workshop paper

Reproducibility of human evaluations in Natural Language Processing remains a critical open challenge. This paper presents a third independent replication of the human evaluation from Yao et al. (2022), which assessed an automated Question-Answer Generation (QAG) system for children's storybooks against a baseline system and human-authored ground truth, across three criteria --- Readability, Question Relevance, and Answer Relevance --- using five NLP-literate annotators. Our replication confirms the main findings of the original study: the QAG system outperforms the baseline on Readability and Question Relevance, and Ground Truth ranks highest across all criteria. System rankings are preserved across all three criteria, with the exception of a statistically non-significant difference in Answer Relevance. This holds true despite a severe drop in inter-annotator agreement for Readability. We further document several methodological concerns, some unreported in prior replications, including data quality issues and evaluation design limitations identified during our pilot study.

ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children’s Storybooks

Evaluating long-form answers from retrieval-augmented generation (RAG) systems remains challenging: human evaluation is expensive, while automatic metrics must reliably capture answer completeness. The AutoNuggetizer framework addresses this by decomposing evaluation into atomic facts (nuggets) and using LLMs for both nugget creation and assignment. The original study validated this approach on open-ended TREC RAG queries; however, it remains unclear whether the same cost-quality tradeoffs hold for structurally different tasks. We reproduce AutoNuggetizer on seven RAG systems over the QAMPARI list-QA benchmark, where answers consist of discrete entities and omissions are more directly measurable. Our results directionally reproduce the original findings: fully automatic evaluation preserves run-level rankings, assignment-only automation yields stronger agreement than end-to-end automation, and LLM-based assignment is highly concordant with human labels while being modestly stricter. These findings support the use of AutoNuggetizer for comparative evaluation beyond open-ended RAG, while also identifying systematic biases in automatic nugget creation and assignment.

Do Nugget-Based Evaluation Patterns Generalize to List-QA?

Recent work in NLP has probed large language models for their understanding of cultural norms across countries. However, this work typically considers distributional patterns, ignoring group consensus or possible multicultural environments within a country. In this work, we leverage cultural consensus theory (CCT) from cultural anthropology to model such multidimensional nuance. Applying CCT to the World Values Survey (WVS) across 10 countries and 12 domains, we demonstrate that models frequently misrepresent cultural structures by either failing to form cohesive consensus or severely over-regularizing consensus. Through explicit representation of intra-group variance, CCT provides actionable diagnostics to evaluate when models reflect true human diversity versus algorithmic homogenization.

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Bangla memes are widely used on social me dia to express humor and social commentary, yet computational analysis of gender bias in Bangla memes remains largely unexplored. In this work, we present a multimodal framework for detecting gender bias in Bangla memes by jointly analyzing textual and visual con tent. We construct a dataset of 6,846 Bangla and Banglish code-mixed memes annotated into three categories: male-biased, female biased, and neutral. For textual representa tion, we use BanglishBERT, while visual fea tures are extracted using ConvNeXt, and the two modalities are fused for final classifica tion. Our best-performing model, ConvNeXt + BanglishBERT, achieves accuracy of 0.67 and an F1-score of 0.63, outperforming sev eral multimodal baselines. The results demon strate the effectiveness of multimodal learn ing for understanding culturally nuanced and code-mixed meme content in low-resource lan guages.

Behind the Laughter: Uncovering Gender Bias in Code-Mixed Bangla Memes

Bias and fairness risks in Large Language Mod els (LLMs) vary substantially across deploy ment contexts, yet existing approaches lack systematic guidance for selecting appropriate evaluation metrics. We present a decision framework that maps LLM use cases, charac terized by a model and population of prompts, to relevant bias and fairness metrics based on task type, whether prompts contain protected attribute mentions, and stakeholder priorities. Our framework addresses toxicity, stereotyp ing, counterfactual unfairness, and allocational harms, and introduces novel metrics based on stereotype classifiers and counterfactual adapta tions of text similarity measures. We release an open-source Python library, langfair, for practical adoption. Extensive experiments on use cases across five LLMs and five prompt populations demonstrate that fairness risks can not be reliably assessed from benchmark per formance alone: results on one prompt dataset likely overstate or understate risks for another, underscoring that fairness evaluation must be grounded in the specific deployment context.

Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs

Multilingual Natural Language Understanding (NLU) systems often struggle to adapt when new languages or new semantic labels are in troduced with only a few annotated examples. This challenge is particularly pronounced for low-resource languages, where limited super vision and evolving label spaces make conven tional joint-label classification approaches un stable. Most existing multilingual NLU models treat each language–semantic pair as an inde pendent class, entangling linguistic and seman tic representations and hindering few-shot adap tation. We propose Dual-Axis Compositional Few-Shot Learning, a framework that explicitly factorizes the representation space into linguis tic and semantic embedding axes, enabling in dependent modeling of language variation and domain–intent semantics. Joint representations are constructed compositionally through mul tiplicative interaction of axis-specific embed dings, allowing controlled adaptation when ei ther the language set or the semantic label space evolves. The framework integrates factorized prototype learning, axis-structured contrastive alignment, and disentanglement regularization using HSIC-based statistical independence and Jacobian-based cross-axis decorrelation. Ex periments on six low-resource Indic languages spanning Indo-Aryan and Dravidian families (Hindi, Bengali, Sanskrit, Assamese, Tamil, and Telugu) demonstrate strong performance under two structured generalization regimes. The model achieves 81.12% accuracy when adapting to few-shot languages with known semantics and 63.5% accuracy when learning new semantic classes from few-shot examples, along with an accuracy of 89.56% on known languages and seen semantics. These results show that axis-factorized representations en able stable compositional generalization, offer ing a promising direction for scalable multilin gual NLU in linguistically diverse low-resource settings.

Dual-Axis Compositional Contrastive Few-Shot Learning using Prototypes Across Linguistic and Semantic Dimensions for Indic Low-Resource Multilingual NLU

Automatic text simplification (ATS) aims to en hance language accessibility for various target groups, particularly persons with intellectual disabilities. Recent advancements in large lan guage models (LLMs) have substantially im proved the quality of machine-generated text simplifications, however, existing LLM-based ATS systems do not incorporate preference feedback during post-training, resulting in a lack of personalization tailored to the specific needs of target group persons. In this work, we propose an ATS personalization framework using direct preference optimization (DPO). Specifically, we post-trained LLM-based ATS models using human feedback collected from persons with intellectual disabilities, reflecting their preferences of paired text simplifications generated by mainstream LLMs. Our pipeline for developing personalized LLM-based ATS systems encompasses data collection, model se lection, supervised fine-tuning (SFT) and DPO post-training, and result evaluation. Our find ings underscore the necessity of active partici pation of target group persons in designing per sonalized inclusive AI solutions aligned with human preferences.

Evaluating Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities

This paper presents a methodology that uses LLMs to align multilingual offensive lexi cons at the sense level. Lexicons of differ ent structures and origins in Arabic, Bulgar ian, Modern Greek, French, and Italian have been aligned directly without pivoting through English. The Modern Greek lexicon is LLM generated, and the other four lexicons are WordNet-compatible. For inter-language align ment of senses, an LLM-as-a-judge rubric was used over lemma–definition–example triples. The LLM makes 2.87M pairwise comparisons and yields 31 strict global-sense categories. The paper discusses the challenges involved in sense alignment tasks. The resource is avail able to support downstream applications such as Machine Translation and cross-lingual hate speech detection.

From Form to Meaning: Interlingua Sense-Alignment of Offensive Language with LLMs

Large Language Models (LLMs) perform strongly on many language tasks, but their capa bility in structurally constrained, accessibility critical modalities such as Braille remains un clear. We evaluate state-of-the-art LLMs on bidirectional Korean–Braille translation using a human-annotated dataset. Despite expecta tions that multilingual, instruction-tuned mod els can generalize to Braille via text repre sentations, we find consistently poor, unstable outputs and substantial disagreement with hu man judgments. These results point to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns. In con trast, supervised fine-tuning of a small model (T5-small) on the same data yields large and stable gains over zero-shot and prompted LLM baselines across standard metrics (SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr). Our findings reveal a systematic lim itation of current LLMs and demonstrate the effectiveness of modest task-specific supervi sion.

I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs

We investigate the role of large language mod els (LLMs) in promoting gender-inclusive lan guage by evaluating their ability to rewrite bi ased text and generate counterfactual narratives across multiple languages. We introduce a shared task with two subtasks: gender-inclusive rewriting and counterfactual generation. The task covers five languages English, German, Spanish, Tamil, and Kannada reflecting diverse grammatical gender systems and sociocultural contexts. We release curated word-level and sentence-level datasets to support controlled inclusive generation. A total of 50 teams reg istered for the shared task, and around 8 teams submitted results. Submissions are evaluated using a hybrid framework combining rubric based automatic scoring with expert human judgment. Finally, we provide an overview of participating systems and discuss key findings and challenges observed across languages.

Content not yet available

Downloads

Next from ACL 2026

ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children’s Storybooks

Similar lecture

MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES

Content not yet available

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2026

ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children’s Storybooks

Similar lecture

MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES

Downloads