RuCoLA

Dataset &amp;&nbsp;Baseline

Paper

Leaderboard

file

info

model

teamName

Submit

Error

In&nbsp;recent years, natural language processing systems have rapidly improved in&nbsp;quality for a&nbsp;number of&nbsp;tasks, many of&nbsp;which involve concepts as&nbsp;difficult as&nbsp;common sense or&nbsp;even general world knowledge. This trend was enabled by&nbsp;the emergence of&nbsp;large-scale self-supervised pretraining methods that formed the backbone of&nbsp;mainstream language models like BERT or&nbsp;GPT-3. Such models have surpassed human performance on&nbsp;canonical NLU benchmarks and proved capable of&nbsp;generating texts hardly distinguishable from those written by&nbsp;humans.

Background

Despite these impressive results, modern language models are still far from perfect, particularly for the Russian language. Although passages from generative models may seem human-like at&nbsp;first glance, they tend to&nbsp;be&nbsp;rife with hallucinated facts or&nbsp;contradictory information. Furthermore, a&nbsp;growing number of&nbsp;studies have reported that even the largest language models do&nbsp;not properly capture various linguistic phenomena and have limited ability to&nbsp;make fine-grained judgments about the correct use of&nbsp;language.

Shortcomings of language models

With that in&nbsp;mind, we&nbsp;designed RuCoLA as&nbsp;a&nbsp;benchmark for evaluating the linguistic competence of&nbsp;Russian language models. RuCoLA follows the general concept of&nbsp;linguistic acceptability: unlike grammatical correctness, which relates to&nbsp;the structure of&nbsp;language, acceptability denotes whether the utterance would be&nbsp;considered natural by&nbsp;a&nbsp;native speaker. Thus, a&nbsp;grammatical sentence can be&nbsp;unacceptable (e.g., "Colorless green ideas sleep furiously"), but an&nbsp;acceptable sentence has to&nbsp;be&nbsp;grammatical. Similarly to&nbsp;GLUE-style and probing benchmarks (e.g., GLUE, Russian SuperGLUE and RuSentEval), it&nbsp;can be&nbsp;used to&nbsp;compare the general language understanding capabilities of&nbsp;neural networks or&nbsp;to&nbsp;analyze and improve the fluency and consistency of&nbsp;text generation models.

Why we&nbsp;created RuCoLA

For any questions about RuCoLA or&nbsp;this website, write to&nbsp;[contact@rucola-benchmark.com](mailto:contact@rucola-benchmark.com) or&nbsp;create an&nbsp;issue in&nbsp;our [GitHub repository](https://github.com/RussianNLP/RuCoLA).

You can also join the [official Telegram chat](https://t.me/RuCoLA_benchmark) to&nbsp;discuss your ideas with others and to&nbsp;follow the latest updates of&nbsp;the project.

The Russian Corpus of&nbsp;Linguistic Acceptability (RuCoLA) is&nbsp;a&nbsp;dataset consisting of&nbsp;Russian language sentences with their binary acceptability judgements. It&nbsp;includes expert-written sentences from linguistic publications and machine-generated examples.

The corpus covers a&nbsp;variety of&nbsp;language phenomena, ranging from syntax and semantics to&nbsp;generative model hallucinations. We&nbsp;release RuCoLA to&nbsp;facilitate the development of&nbsp;methods for identifying errors in&nbsp;natural language and create a&nbsp;public leaderboard to&nbsp;track the progress made on&nbsp;this problem.