publications

Refereed Publications

2026

1. ACL

   Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models

   Ryoma Kumon and Hitomi Yanaka

   In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2026

   Abs DOI

   While language models demonstrate sophisticated syntactic capabilities, the extent to which their internal mechanisms align with cross-constructional principles studied in linguistics remains poorly understood. This study investigates whether models employ shared neural mechanisms across different syntactic constructions by applying causal interpretability methods at a granular level. Focusing on filler-gap dependencies and negative polarity item (NPI) licensing, we utilize activation patching to identify the functional roles of specific attention heads and MLP blocks. Our results reveal a highly localized and shared mechanism for filler-gap dependencies located in the early to middle layers, whereas NPI processing exhibits no such unified mechanism. Furthermore, we find that these mechanisms identified by activation patching generalize to out-of-distribution, while distributed alignment search, a supervised interpretability method, is susceptible to overfitting on narrow linguistic distributions. Finally, we validate our findings by demonstrating that the manipulation of the identified components improves model performance on acceptability judgment benchmarks.

2. ACL SRW

   Analysis of the Neglect-Zero Effect in Large Language Models

   Jin Tanaka, Daiki Matsuoka, Ryoma Kumon, and 1 more author

   In Proceedings of the 64rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Jul 2026

   Abs

   We investigate the extent to which the language processing of LLMs resembles human cognitive processes, focusing on a human cognitive bias called the neglect-zero effect. This effect refers to the human tendency to ignore zero-models, which are configurations that render a proposition vacuously true by virtue of an empty set. We focus on two types of inference driven by the neglect-zero effect, and examine how LLMs process the inference by comparing their behavior with that in inference not involving the neglect-zero effect. For this purpose, we employ a paradigm using structural priming, where recent exposure to a preceding sentence (the prime) facilitates the processing of a subsequent sentence (the target) due to their structural similarity. We prepare primes so that they force LLMs to consider the zero-model, and analyze whether they also consider zero-models in the target. The results indicate the possibility that the neglect-zero effect does not arise within the LLMs analyzed in this study.

3. ICLR

   Neuron-Level Analysis of Cultural Understanding in Large Language Models

   Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, and 1 more author

   In The Fourteenth International Conference on Learning Representations, Apr 2026

   Abs

   As large language models (LLMs) are increasingly deployed worldwide, ensuring their fair and comprehensive cultural understanding is important. However, LLMs exhibit cultural bias and limited awareness of underrepresented cultures, while the mechanisms underlying their cultural understanding remain underexplored. To fill this gap, we conduct a neuron-level analysis to identify neurons that drive cultural behavior, introducing a gradient-based scoring method with additional filtering for precise refinement. We identify both culture-general neurons contributing to cultural understanding regardless of cultures, and culture-specific neurons tied to an individual culture. These neurons account for less than 1% of all neurons and are concentrated in shallow to middle MLP layers. We validate their role by showing that suppressing them substantially degrades performance on cultural benchmarks (by up to 30%), while performance on general natural language understanding (NLU) benchmarks remains largely unaffected. Moreover, we show that culture-specific neurons support knowledge of not only the target culture, but also related cultures. Finally, we demonstrate that training on NLU benchmarks can diminish models’ cultural understanding when we update modules containing many culture-general neurons. These findings provide insights into the internal mechanisms of LLMs and offer practical guidance for model training and engineering.

2025

1. NAACL

   Analyzing the Inner Workings of Transformers in Compositional Generalization

   Ryoma Kumon and Hitomi Yanaka

   In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Apr 2025

   Abs DOI

   The compositional generalization abilities of neural models have been sought after for human-like linguistic competence.The popular method to evaluate such abilities is to assess the models’ input-output behavior.However, that does not reveal the internal mechanisms, and the underlying competence of such models in compositional generalization remains unclear.To address this problem, we explore the inner workings of a Transformer model byfinding an existing subnetwork that contributes to the generalization performance and by performing causal analyses on how the model utilizes syntactic features.We find that the model depends on syntactic features to output the correct answer, but that the subnetwork with much better generalization performance than the whole model relies on a non-compositional algorithm in addition to the syntactic features.We also show that the subnetwork improves its generalization performance relatively slowly during the training compared to the in-distribution one, and the non-compositional solution is acquired in the early stages of the training.

2. GeBNLP

   Intersectional Bias in Japanese Large Language Models from a Contextualized Perspective

   Hitomi Yanaka, Xinqi He, Lu Jie, and 6 more authors

   In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Aug 2025

   Abs DOI

   An growing number of studies have examined the social bias of rapidly developed large language models (LLMs). Although most of these studies have focused on bias occurring in a single social attribute, research in social science has shown that social bias often occurs in the form of intersectionality—the constitutive and contextualized perspective on bias aroused by social attributes. In this study, we construct the Japanese benchmark inter-JBBQ, designed to evaluate the intersectional bias in LLMs on the question-answering setting. Using inter-JBBQ to analyze GPT-4o and Swallow, we find that biased output varies according to its contexts even with the equal combination of social attributes.

3. GeBNLP

   JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models

   Hitomi Yanaka, Namgi Han, Ryoma Kumon, and 5 more authors

   In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Aug 2025

   Abs DOI

   With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue.Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated.In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs.The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase.In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. Our dataset is available at https://github.com/ynklab/JBBQ_data.

4. EMNLP

   Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset

   Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, and 1 more author

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Nov 2025

   Abs DOI

   Large language models (LLMs) exhibit social biases, prompting the development of various debiasing methods. However, debiasing methods may degrade the capabilities of LLMs. Previous research has evaluated the impact of bias mitigation primarily through tasks measuring general language understanding, which are often unrelated to social biases. In contrast, cultural commonsense is closely related to social biases, as both are rooted in social norms and values. The impact of bias mitigation on cultural commonsense in LLMs has not been well investigated. Considering this gap, we propose SOBACO (SOcial BiAs and Cultural cOmmonsense benchmark), a Japanese benchmark designed to evaluate social biases and cultural commonsense in LLMs in a unified format. We evaluate several LLMs on SOBACO to examine how debiasing methods affect cultural commonsense in LLMs. Our results reveal that the debiasing methods degrade the performance of the LLMs on the cultural commonsense task (up to 75% accuracy deterioration). These results highlight the importance of developing debiasing methods that consider the trade-off with cultural commonsense to improve fairness and utility of LLMs.

2024

1. ACL Findings

   Evaluating Structural Generalization in Neural Machine Translation

   Ryoma Kumon, Daiki Matsuoka, and Hitomi Yanaka

   In Findings of the Association for Computational Linguistics: ACL 2024, Aug 2024

   Abs DOI

   Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures.Since it is regarded as a desired property of neural models, recent work has assessed compositional generalization in machine translation as well as semantic parsing.However, previous evaluations with machine translation have focused mostly on lexical generalization (i.e., generalization to unseen combinations of known words).Thus, it remains unclear to what extent models can translate sentences that require structural generalization (i.e., generalization to different sorts of syntactic structures).To address this question, we construct SGET, a machine translation dataset covering various types of compositional generalization with control of words and sentence structures.We evaluate neural machine translation models on SGET and show that they struggle more in structural generalization than in lexical generalization.We also find different performance trends in semantic parsing and machine translation, which indicates the importance of evaluations across various tasks.

Non-Refereed Publications

2026

1. NLP

   LLMの知能は人間と同様の構造を持つか？：人間の知能理論を用いた評価スコアの統計分析

   Namgi Han, 九門涼真, and 谷中瞳

   In 言語処理学会第32回年次大会論文集, Mar 2026
2. NLP

   言語モデルにおける共通統語メカニズムの分析

   九門涼真 and 谷中瞳

   In 言語処理学会第32回年次大会論文集, Mar 2026
3. NLP

   LLMの多言語内部表現における意味空間と言語成分の構造

   山本泰成, 九門涼真, Danushka Bollegala, and 1 more author

   In 言語処理学会第32回年次大会論文集, Mar 2026

2025

1. NLP

   構成的汎化におけるTransformerの内部機序の分析

   九門涼真 and 谷中瞳

   In 言語処理学会第31回年次大会論文集, Mar 2025

   若手奨励賞 PDF

   若手奨励賞

2. NLP

   大規模言語モデルにおける社会的バイアスの抑制と文化的常識の理解のトレードオフの分析

   山本泰成, 九門涼真, Danushka Bollegala, and 1 more author

   In 言語処理学会第31回年次大会論文集, Mar 2025

   PDF
3. NLP

   日本語LLMに含まれる交差バイアスと有害性の評価に向けて

   谷中瞳, Sunjin Oh, Xinqi He, and 6 more authors

   In 言語処理学会第31回年次大会論文集, Mar 2025

   PDF

2024

1. NLP

   ニューラル機械翻訳モデルにおける構成的汎化能力の評価

   九門涼真, 松岡大樹, and 谷中瞳

   In 言語処理学会第30回年次大会論文集, Mar 2024

   委員特別賞 PDF

   委員特別賞