United States

Reproducibility of human evaluations in Natural Language Processing remains a critical open challenge. This paper presents a third independent replication of the human evaluation from Yao et al. (2022), which assessed an automated Question-Answer Generation (QAG) system for children&#39;s storybooks against a baseline system and human-authored ground truth, across three criteria --- Readability, Question Relevance, and Answer Relevance --- using five NLP-literate annotators. Our replication confirms the main findings of the original study: the QAG system outperforms the baseline on Readability and Question Relevance, and Ground Truth ranks highest across all criteria. System rankings are preserved across all three criteria, with the exception of a statistically non-significant difference in Answer Relevance. This holds true despite a severe drop in inter-annotator agreement for Readability. We further document several methodological concerns, some unreported in prior replications, including data quality issues and evaluation design limitations identified during our pilot study.

ACL 2026

ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children’s Storybooks

children's storybooks

replication study

question-answer generation

evaluation methodology

human evaluation

reproducibility

natural language processing

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

ACL 2022

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

Reproducibility of human evaluations in Natural Language Processing remains a critical open challenge. This paper presents a third independent replication of the human evaluation from Yao et al. (2022), which assessed an automated Question-Answer Generation (QAG) system for children's storybooks against a baseline system and human-authored ground truth, across three criteria --- Readability, Question Relevance, and Answer Relevance --- using five NLP-literate annotators. Our replication confirms the main findings of the original study: the QAG system outperforms the baseline on Readability and Question Relevance, and Ground Truth ranks highest across all criteria. System rankings are preserved across all three criteria, with the exception of a statistically non-significant difference in Answer Relevance. This holds true despite a severe drop in inter-annotator agreement for Readability. We further document several methodological concerns, some unreported in prior replications, including data quality issues and evaluation design limitations identified during our pilot study.

workshop paper

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. 
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) 
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

Registration Required

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

Evaluating long-form answers from retrieval-augmented generation (RAG) systems remains challenging: human evaluation is expensive, while automatic metrics must reliably capture answer completeness. The AutoNuggetizer framework addresses this by decomposing evaluation into atomic facts (nuggets) and using LLMs for both nugget creation and assignment. The original study validated this approach on open-ended TREC RAG queries; however, it remains unclear whether the same cost-quality tradeoffs hold for structurally different tasks. We reproduce AutoNuggetizer on seven RAG systems over the QAMPARI list-QA benchmark, where answers consist of discrete entities and omissions are more directly measurable. Our results directionally reproduce the original findings: fully automatic evaluation preserves run-level rankings, assignment-only automation yields stronger agreement than end-to-end automation, and LLM-based assignment is highly concordant with human labels while being modestly stricter. These findings support the use of AutoNuggetizer for comparative evaluation beyond open-ended RAG, while also identifying systematic biases in automatic nugget creation and assignment.

Do Nugget-Based Evaluation Patterns Generalize to List-QA?

Recent work in NLP has probed large language models for their understanding of cultural norms across countries. However, this work typically considers distributional patterns, ignoring group consensus or possible multicultural environments within a country. In this work, we leverage cultural consensus theory (CCT) from cultural anthropology to model such multidimensional nuance. Applying CCT to the World Values Survey (WVS) across 10 countries and 12 domains, we demonstrate that models frequently misrepresent cultural structures by either failing to form cohesive consensus or severely over-regularizing consensus. Through explicit representation of intra-group variance, CCT provides actionable diagnostics to evaluate when models reflect true human diversity versus algorithmic homogenization.

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Bangla memes are widely used on social me dia to express humor and social commentary, yet computational analysis of gender bias in Bangla memes remains largely unexplored. In this work, we present a multimodal framework for detecting gender bias in Bangla memes by jointly analyzing textual and visual con tent. We construct a dataset of 6,846 Bangla and Banglish code-mixed memes annotated into three categories: male-biased, female biased, and neutral. For textual representa tion, we use BanglishBERT, while visual fea tures are extracted using ConvNeXt, and the two modalities are fused for final classifica tion. Our best-performing model, ConvNeXt + BanglishBERT, achieves accuracy of 0.67 and an F1-score of 0.63, outperforming sev eral multimodal baselines. The results demon strate the effectiveness of multimodal learn ing for understanding culturally nuanced and code-mixed meme content in low-resource lan guages.

Behind the Laughter: Uncovering Gender Bias in Code-Mixed Bangla Memes

Bias and fairness risks in Large Language Mod els (LLMs) vary substantially across deploy ment contexts, yet existing approaches lack systematic guidance for selecting appropriate evaluation metrics. We present a decision framework that maps LLM use cases, charac terized by a model and population of prompts, to relevant bias and fairness metrics based on task type, whether prompts contain protected attribute mentions, and stakeholder priorities. Our framework addresses toxicity, stereotyp ing, counterfactual unfairness, and allocational harms, and introduces novel metrics based on stereotype classifiers and counterfactual adapta tions of text similarity measures. We release an open-source Python library, langfair, for practical adoption. Extensive experiments on use cases across five LLMs and five prompt populations demonstrate that fairness risks can not be reliably assessed from benchmark per formance alone: results on one prompt dataset likely overstate or understate risks for another, underscoring that fairness evaluation must be grounded in the specific deployment context.

Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs

Multilingual Natural Language Understanding (NLU) systems often struggle to adapt when new languages or new semantic labels are in troduced with only a few annotated examples. This challenge is particularly pronounced for low-resource languages, where limited super vision and evolving label spaces make conven tional joint-label classification approaches un stable. Most existing multilingual NLU models treat each language–semantic pair as an inde pendent class, entangling linguistic and seman tic representations and hindering few-shot adap tation. We propose Dual-Axis Compositional Few-Shot Learning, a framework that explicitly factorizes the representation space into linguis tic and semantic embedding axes, enabling in dependent modeling of language variation and domain–intent semantics. Joint representations are constructed compositionally through mul tiplicative interaction of axis-specific embed dings, allowing controlled adaptation when ei ther the language set or the semantic label space evolves. The framework integrates factorized prototype learning, axis-structured contrastive alignment, and disentanglement regularization using HSIC-based statistical independence and Jacobian-based cross-axis decorrelation. Ex periments on six low-resource Indic languages spanning Indo-Aryan and Dravidian families (Hindi, Bengali, Sanskrit, Assamese, Tamil, and Telugu) demonstrate strong performance under two structured generalization regimes. The model achieves 81.12% accuracy when adapting to few-shot languages with known semantics and 63.5% accuracy when learning new semantic classes from few-shot examples, along with an accuracy of 89.56% on known languages and seen semantics. These results show that axis-factorized representations en able stable compositional generalization, offer ing a promising direction for scalable multilin gual NLU in linguistically diverse low-resource settings.

Dual-Axis Compositional Contrastive Few-Shot Learning using Prototypes Across Linguistic and Semantic Dimensions for Indic Low-Resource Multilingual NLU

Automatic text simplification (ATS) aims to en hance language accessibility for various target groups, particularly persons with intellectual disabilities. Recent advancements in large lan guage models (LLMs) have substantially im proved the quality of machine-generated text simplifications, however, existing LLM-based ATS systems do not incorporate preference feedback during post-training, resulting in a lack of personalization tailored to the specific needs of target group persons. In this work, we propose an ATS personalization framework using direct preference optimization (DPO). Specifically, we post-trained LLM-based ATS models using human feedback collected from persons with intellectual disabilities, reflecting their preferences of paired text simplifications generated by mainstream LLMs. Our pipeline for developing personalized LLM-based ATS systems encompasses data collection, model se lection, supervised fine-tuning (SFT) and DPO post-training, and result evaluation. Our find ings underscore the necessity of active partici pation of target group persons in designing per sonalized inclusive AI solutions aligned with human preferences.

Evaluating Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities

This paper presents a methodology that uses LLMs to align multilingual offensive lexi cons at the sense level. Lexicons of differ ent structures and origins in Arabic, Bulgar ian, Modern Greek, French, and Italian have been aligned directly without pivoting through English. The Modern Greek lexicon is LLM generated, and the other four lexicons are WordNet-compatible. For inter-language align ment of senses, an LLM-as-a-judge rubric was used over lemma–definition–example triples. The LLM makes 2.87M pairwise comparisons and yields 31 strict global-sense categories. The paper discusses the challenges involved in sense alignment tasks. The resource is avail able to support downstream applications such as Machine Translation and cross-lingual hate speech detection.

From Form to Meaning: Interlingua Sense-Alignment of Offensive Language with LLMs

Large Language Models (LLMs) perform strongly on many language tasks, but their capa bility in structurally constrained, accessibility critical modalities such as Braille remains un clear. We evaluate state-of-the-art LLMs on bidirectional Korean–Braille translation using a human-annotated dataset. Despite expecta tions that multilingual, instruction-tuned mod els can generalize to Braille via text repre sentations, we find consistently poor, unstable outputs and substantial disagreement with hu man judgments. These results point to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns. In con trast, supervised fine-tuning of a small model (T5-small) on the same data yields large and stable gains over zero-shot and prompted LLM baselines across standard metrics (SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr). Our findings reveal a systematic lim itation of current LLMs and demonstrate the effectiveness of modest task-specific supervi sion.

I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs

We investigate the role of large language mod els (LLMs) in promoting gender-inclusive lan guage by evaluating their ability to rewrite bi ased text and generate counterfactual narratives across multiple languages. We introduce a shared task with two subtasks: gender-inclusive rewriting and counterfactual generation. The task covers five languages English, German, Spanish, Tamil, and Kannada reflecting diverse grammatical gender systems and sociocultural contexts. We release curated word-level and sentence-level datasets to support controlled inclusive generation. A total of 50 teams reg istered for the shared task, and around 8 teams submitted results. Submissions are evaluated using a hybrid framework combining rubric based automatic scoring with expert human judgment. Finally, we provide an overview of participating systems and discuss key findings and challenges observed across languages.

Insights from Multilingual Gender Inclusive Language Generation Shared Task

Gender bias in multilingual language gener ation systems poses serious ethical and so cial issues, especially in languages with com plex morphology. In this study, we propose a lightweight multilingual approach that employs instruction-guided fine-tuning of the mT5 small transformer model for gender-inclusive language generation. The framework accom modates five languages: English, German, Spanish, Tamil, and Kannada. The approach uses the task-prefix rewriting method to trans form gender-specific sentences to their gender neutral versions. The training data from differ ent languages is combined into a single multi lingual dataset for sequence-to-sequence fine tuning. Beam search decoding with repetition constraints is used during inference to improve the quality of the output. The system’s perfor mance is measured using GIFI, semantic simi larity, and an overall combined score across all languages. Experimental results show that the system can eliminate gender-biased language while retaining semantic meaning in part across languages.

Downloads

Next from ACL 2026

Do Nugget-Based Evaluation Patterns Generalize to List-QA?

Similar lecture

It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2026

Do Nugget-Based Evaluation Patterns Generalize to List-QA?

Similar lecture

It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES

Downloads