{"repo":"lechmazur/nyt-connections","free":true,"listed":false,"github":"https://github.com/lechmazur/nyt-connections","clone":"git clone https://github.com/lechmazur/nyt-connections.git","description":"Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words","language":"Python","stars":238,"topics":["benchmark","evaluation","llm","llms-benchmarking","puzzles","reasoning","testing","claude","gemini-pro","gpt-5"],"license":null,"category":"ai-agents","readme_excerpt":"Extended Version This benchmark evaluates large language models (LLMs) using 940 NYT Connections puzzles, with additional words included to increase difficulty. As of Feb 4, 2025, there is a new version of the benchmark. The standard NYT Connections benchmark is nearing saturation, with o1 scoring 90.7 and o3, along with other reasoning models, expected this year. The current rules require knowing only three categories, letting the fourth fall into place. To increase difficulty, Extended Connections adds up to four extra trick words to each puzzle. We double-check that none of the added words fit into any category used in the corresponding puzzle. New puzzles have expanded the total from 436 to 940 as of Feb 2, 2026. Headline results use quadratic-v1 scoring: a puzzle with g exact groups contributes (g / 4)² , so 0, 1, 2, 3, and 4 groups earn 0%, 6.25%, 25%, 56.25%, and 100%, respectively. The leaderboard score is the mean of those per-puzzle values. To reduce visual noise, scoreboard chart labels omit the implied word reasoning from positive parenthetical effort levels: for example, (high reasoning) is displayed as (high) and (xhigh reasoning) as (xhigh) . The explicit negative mode remains (no reasoning) . Model IDs and the canonical display names used in tables are unchanged. Chart: Extended Version Scoreboard charts apply the current report-time suppression roster. The leaderboard tables retain every full-coverage result, including models omitted from charts, so historica","default_branch":null,"files":null,"tree":[],"storefront":"/r/lechmazur","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lechmazur/nyt-connections/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}