Welcome!
This is the coopetition leaderboard for Learning from Textbooks (LEFT). The task is to classify statements about U.S. history as true or false based on the information given in a relevant textbook.
The paper describing the LEFT task is available in the ACL Anthology proceedings of EMNLP 2021. To cite, use:
@inproceedings{ciosici-etal-2021-perhaps,
title = "Perhaps {PTLM}s Should Go to School {--} A Task to Assess Open Book and Closed Book {QA}",
author = "Ciosici, Manuel and
Cecil, Joe and
Lee, Dong-Ho and
Hedges, Alex and
Freedman, Marjorie and
Weischedel, Ralph",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.493",
pages = "6104--6111",
abstract = "Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be {\textasciitilde}50{\%}. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook{'}s content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5{'}s pre-training), yields at best minor improvement (56{\%}), suggesting that the PTLM may not have {``}understood{''} the textbook (or perhaps misunderstood the questions). Performance is better ({\textasciitilde}60{\%}) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).",
}
Public goods are available to all without payment.
Philadelphia served as the base for British operations for most of the Revolutionary War.
Evaluation is done in three modes:
All participants must submit results for the prior knowledge mode. This is intended to make the closed-book and open-book results more comprehensible.
In each mode, submissions will be scored on their accuracy with respect to the true/false labels for the statements. Accuracy will be over all statements scored. Accuracy will be evaluated for each textbook separately, on both a dev set and a test set for each textbook. Accuracy will be presented as the percentage of statements accurately assessed.
The textbooks used are the OpenStax textbooks
American Government 2e
and U.S. History,
which are licensed under a Creative Commons Attribution 4.0 International License.
The columns accuracy (AmGov-*) refer to the American Government 2e dev and test split accuracy results,
while the columns accuracy (USHist-*) refer to the U.S. History dev and test split accuracy results.
This is the data for the competition. It is to be used responsibly.
This section describes four things:
Inputs are given in a zip file containing one file for each column on the leaderboard, i.e.
amgov_dev.jsonl for AmGov-Dev (American Government 2e, dev set)amgov_test.jsonl for AmGov-Test (American Government 2e, test set)ushist_dev.jsonl for USHist-Dev (U.S. History, dev set)ushist_test.jsonl for USHist-Test (U.S. History, test set)Note that the dev data does not include labels. This is for ethical reasons; for a detailed explanation, see the "Ethical Considerations" section of the LEFT paper. However, while test set submissions are limited to one per week, dev set submissions are effectively unlimited (the limit is 100 per day).
In addition, the input zip file includes plaintext versions of each textbook chapter, with one file for each
chapter, intended for use only in the closed-book and open-book evaluation modes.
The American Government 2e chapter files are named amgov_chapter_XX.txt (where XX = 01, 02, 03, ..., 17)
while the U.S. History chapter files are named ushist_chapter_YY.txt (where YY = 01, 02, 03, ..., 32).
Each input will be in JSON Lines format, like so:
{"id": 435, "statement": "A U.S. senator is not an agent of political socialization.", "book": "american_government", "chapter": 6, "gold_ir": "The most prominent agents of socialization are family and school. Other influential agents are social groups, such as religious institutions and friends, and the media."}
{"id": 574, "statement": "Under the Articles of Confederation, the government could not conduct foreign affairs.", "book": "american_government", "chapter": 2, "gold_ir": "The Articles of Confederation satisfied the desire of those in the new nation who wanted a weak central government with limited power. Ironically, however, their very success led to their undoing. It soon became apparent that, while they protected the sovereignty of the states, the Articles had created a central government too weak to function effectively.One of the biggest problems was that the national government had no power to impose taxes. To avoid any perception of \"taxation without representation,\u201d the Articles of Confederation allowed only state governments to levy taxes. To pay for its expenses, the national government had to request money from the states, which were required to provide funds in proportion to the value of the land within their borders."}
{"id": 657, "statement": "The right to privacy has been controversial because most U.S. citizens today believe the government should be allowed to outlaw birth control.", "book": "american_government", "chapter": 4, "gold_ir": "The legal landscape changed dramatically as a result of the 1973 ruling in Roe v. Wade,67 in which the Supreme Court decided the right to privacy encompassed a right for women to terminate a pregnancy, at least under certain scenarios. The justices ruled that while the government did have an interest in protecting the \"potentiality of human life,\u201d nonetheless this had to be balanced against the interests of both women's health and women's right to decide whether to have an abortion."}
...
In the open-book mode, each input object will also have a "gold_ir" key,
the value of which is a textbook passage chosen by a human as relevant to the statement.
Note that the statements and chapters used are the same for all three evaluation modes. Thus, there is only one zip file you need to download — the one for the open-book mode. In the preexisting knowledge mode you should ignore the chapters and gold IR; in the closed book mode you should use the chapters only for training and ignore gold IR; and in the open book mode you can use any of the provided inputs.
The model's prediction output should also be in JSON Lines format, like so:
{"id": 435, "label": true}
{"id": 574, "label": true}
{"id": 657, "label": false}
...
Download the LEFT input data here: left_input_data.zip
Submissions should follow the same structure as the input, i.e. a submission should be a flat zip file, with one output file for each JSONL input file. Each output file should give the model's predictions for the statements described in the corresponding input file. All submissions should therefore contain the following files:
amgov_dev.jsonlamgov_test.jsonlushist_dev.jsonlushist_test.jsonlThe provided starting kit illustrates how to read the inputs, how to structure the output, and how to zip the resulting outputs for submission to the leaderboard.
Download the starting kit here: starting_kit.zip
First, fill out the following form:
Submissions missing any of these fields will not be scored.
Next, you'll need to provide us your filled-out form and submission ZIP files. Submission ZIP files must be named according to the evaluation mode they pertain to:
_pk.zip._cb.zip._ob.zip.Submissions not following these criteria will not be scored until you specify which ZIP corresponds to which mode.
Note that the first submission for a given team and method must include system output for the prior knowledge split. For example, suppose your team's approach is to prompt GPT-3 with answer-only prompting. The first time your team submits output from GPT-3 with answer-only prompting, you must submit prior knowledge output. In subsequent submissions with the same approach, you may include only closed-book or open-book output.
E-mail your filled-out form and submission zips
to jcecil@isi.edu
or manuelc@isi.edu with the subject line
LEFT leaderboard submission.
The textbooks used are the OpenStax textbooks American Government 2e and U.S. History, which are licensed under a Creative Commons Attribution 4.0 International License. The full-text content of the two textbooks is available in the public data package.
All participants must submit results for the prior knowledge mode. This is intended to make the closed-book and open-book results easily comparable.
By submitting to this competition, you consent to the public release of your submission's score through the leaderboard at the task organizers' discretion. Scores may include, but are not limited to, automatic or manual qualitative judgments, comparisons between your method's performance and that of other submissions to the leaderboard, whether in aggregate or individually. You accept that the ultimate decision of the metric choice and score value is that of the leaderboard's organizers.
You further agree that task organizers are under no obligation to release scores and that scores may be withheld at the task organizers' discretion. For example, the organizers might withhold scores if the organizers deem the submission was erroneous, incomplete, deceptive, or broke any of the rules in this document or stated elsewhere on the leaderboard.
You further agree that your system may be named according to the team name provided at the time of submission or a shorthand determined by the leaderboard organizers.
You further agree to respect the terms and licenses under which the data is released. The LEFT data for this leaderboard comprises several materials licensed under Creative Commons Attribution License v4.0: the full text of textbooks, statements derived from the textbooks' review questions, and statements derived from assessment items provided by Lumen Learning.
The top three entries are highlighted in green except for baseline results. Baselines are highlighted in gold.
| Rank | Submitter (Affiliation) | Date of Last Entry | Method Name | Accuracy (AGDev, %) | Accuracy (AGTest, %) | Accuracy (USDev, %) | Accuracy (USTest, %) | Project URL |
|---|---|---|---|---|---|---|---|---|
| 1 | USC ISI East [Baseline] | 2023-11-10 | Human Perf. (upper) | - | 85.05 | - | 85.68 | https://docs.google.com/document/d/1lNZJVeEhFbTiwExfd-ZYMAWepnbnKMlVJOeAgpbnY8g/edit?usp=sharing |
| 2 | USC ISI East | 2023-04-04 | GPT-4 | 85.48 | 87.85 | 85.77 | 82.04 | https://docs.google.com/document/d/1vVVsYStc5GUr2gBg7ek7oCLKsgu_RqQ2w9EeQ5Vh8Ro/edit?usp=sharing |
| 3 | USC ISI East | 2022-07-18 | GPT-3 DaVinci | 70.43 | 73.83 | 66.06 | 68.45 | https://docs.google.com/document/d/1BqRHSPOoweXtGoBFitsB2HD1ZTuOet932KVQN0NQHeQ/edit?usp=sharing |
| 4 | USC ISI East [Baseline] | 2021-09-09 | Random baseline | 51.61 | 51.40 | 54.01 | 50.73 | |
| 5 | USC ISI East [Baseline] | 2021-09-09 | Always false | 50.00 | 50.00 | 50.00 | 50.00 | |
| 6 | USC ISI East [Baseline] | 2021-09-09 | Always true | 50.00 | 50.00 | 50.00 | 50.00 | |
| 7 | USC ISI East | 2021-09-09 | T5+BoolQ -ctx | 51.08 | 49.53 | 50.36 | 50.00 | |
| 8 | USC ISI East | 2021-09-09 | GPTNeo -ctx | 52.69 | 48.13 | 51.09 | 49.27 | |
| 9 | USC ISI East [Baseline] | 2023-11-10 | Human Perf. (lower) | - | 42.06 | - | 32.28 | https://docs.google.com/document/d/1lNZJVeEhFbTiwExfd-ZYMAWepnbnKMlVJOeAgpbnY8g/edit?usp=sharing |
| Rank | Submitter (Affiliation) | Date of Last Entry | Method Name | Accuracy (AGDev, %) | Accuracy (AGTest, %) | Accuracy (USDev, %) | Accuracy (USTest, %) | Project URL |
|---|---|---|---|---|---|---|---|---|
| 1 | USC ISI East | 2021-09-09 | GPTNeo +pt -ctx | 50.00 | 55.14 | 50.73 | 49.76 | |
| 2 | USC ISI East [Baseline] | 2021-09-09 | Random baseline | 51.61 | 51.40 | 54.01 | 50.73 | |
| 3 | USC ISI East | 2021-09-09 | T5+BoolQ +pt -ctx | 56.45 | 52.34 | 50.73 | 50.00 | |
| 4 | USC ISI East [Baseline] | 2021-09-09 | Always false | 50.00 | 50.00 | 50.00 | 50.00 | |
| 5 | USC ISI East [Baseline] | 2021-09-09 | Always true | 50.00 | 50.00 | 50.00 | 50.00 |
| Rank | Submitter (Affiliation) | Date of Last Entry | Method Name | Accuracy (AGDev, %) | Accuracy (AGTest, %) | Accuracy (USDev, %) | Accuracy (USTest, %) | Project URL |
|---|---|---|---|---|---|---|---|---|
| 1 | USC ISI East | 2023-04-04 | GPT-4+REALM | 93.55 | 93.46 | 89.05 | 91.99 | https://docs.google.com/document/d/1vVVsYStc5GUr2gBg7ek7oCLKsgu_RqQ2w9EeQ5Vh8Ro/edit?usp=sharing |
| 2 | USC ISI East | 2023-03-31 | GPT-4+Gold IR | 94.09 | 92.99 | 89.05 | 91.50 | https://docs.google.com/document/d/1vVVsYStc5GUr2gBg7ek7oCLKsgu_RqQ2w9EeQ5Vh8Ro/edit?usp=sharing |
| 3 | USC ISI East [Baseline] | 2023-11-10 | Human Perf. (upper) | - | 88.32 | - | 90.53 | https://docs.google.com/document/d/1lNZJVeEhFbTiwExfd-ZYMAWepnbnKMlVJOeAgpbnY8g/edit?usp=sharing |
| 4 | USC ISI East | 2022-07-18 | GPT-3 DaVinci +REALM | 83.87 | 86.45 | 83.21 | 86.41 | https://docs.google.com/document/d/1BqRHSPOoweXtGoBFitsB2HD1ZTuOet932KVQN0NQHeQ/edit?usp=sharing |
| 5 | USC ISI East | 2022-07-18 | GPT-3 DaVinci | 83.87 | 86.45 | 82.85 | 86.17 | https://docs.google.com/document/d/1BqRHSPOoweXtGoBFitsB2HD1ZTuOet932KVQN0NQHeQ/edit?usp=sharing |
| 6 | USC ISI West | 2022-08-29 | T0pp | 80.11 | 81.78 | 78.47 | 81.07 | https://huggingface.co/bigscience/T0pp |
| 7 | USC ISI East | 2022-05-19 | RoBERTa-MNLI+goldIR | 80.65 | 81.78 | 77.74 | 78.16 | https://docs.google.com/document/d/18ypfm2J4VIe34VHi4ACN1NCZYtOG0G4gd0WYj85xykY/edit?usp=sharing |
| 8 | USC ISI East | 2021-10-28 | UnifiedQA-11B+goldIR | 77.42 | 80.84 | 70.80 | 68.93 | https://docs.google.com/document/d/16z0K8A0TUrXLSHZ5-AtUiZjHLxjsViaeyrFWaNe7ZcA/edit?usp=sharing |
| 9 | USC ISI East [Baseline] | 2023-11-10 | Human Perf. (lower) | - | 72.43 | - | 73.30 | https://docs.google.com/document/d/1lNZJVeEhFbTiwExfd-ZYMAWepnbnKMlVJOeAgpbnY8g/edit?usp=sharing |
| 10 | USC ISI East | 2022-06-17 | T5-MNLI+sbertIR | 68.82 | 75.23 | 65.33 | 70.63 | https://docs.google.com/document/d/12gRiQZ7-jjg3GnpF2kEONCPtOFYdbYPEJDVusWXELQ0/edit?usp=sharing |
| 11 | USC ISI East | 2022-06-17 | RoBERTa-MNLI+sbertIR | 70.97 | 71.96 | 66.06 | 70.63 | https://docs.google.com/document/d/18ypfm2J4VIe34VHi4ACN1NCZYtOG0G4gd0WYj85xykY/edit?usp=sharing |
| 12 | USC ISI East | 2021-09-09 | T5+BoolQ +goldIR | 71.51 | 74.30 | 68.61 | 68.69 | |
| 13 | USC ISI East | 2021-09-09 | T5+BoolQ +pt +goldIR | 58.60 | 63.08 | 57.66 | 66.26 | |
| 14 | USC ISI East | 2021-10-28 | T5-3B+BoolQ +REALMIR | 68.82 | 65.42 | 60.22 | 62.86 | https://docs.google.com/document/d/1QghHWgNUEaR7QzDB2oDwCB0l3Oy7Ci3aO4q_UPNNDzo/edit?usp=sharing |
| 15 | USC ISI East | 2021-09-09 | T5+BoolQ +sBERT | 60.22 | 61.21 | 55.47 | 59.95 | |
| 16 | USC ISI East | 2021-09-09 | T5+BoolQ +pt +sBERT | 55.91 | 52.80 | 52.19 | 56.31 | |
| 17 | USC ISI East [Baseline] | 2021-09-09 | Random baseline | 51.61 | 51.40 | 54.01 | 50.73 | |
| 18 | USC ISI East [Baseline] | 2021-09-09 | Always false | 50.00 | 50.00 | 50.00 | 50.00 | |
| 19 | USC ISI East [Baseline] | 2022-01-21 | Always true | 50.00 | 50.00 | 50.00 | 50.00 |