跳到正文
arXiv cs.CL· Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes, Karim Ali·· 2 天前AI 评分38

PerfArena:按生成代码性能对 LLM 排名

PerfArena: Ranking LLMs by Code Performance

AI 导读

开源可复现的评测平台 PerfArena 按生成代码的性能对 LLM 排名,同时报告正确性、延迟、峰值内存和能耗。平台已收录 13 个 SOTA LLM 在 7 种编程语言下的 9,455 个正确且完成测量的解,覆盖 CLBGComp 的 10 个计算密集型基准和 LeetCodeComp 的 99 道题。

正文 · 原文

View PDF HTML (experimental)

Abstract:Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.
Subjects: Performance (cs.PF); Computation and Language (cs.CL); Software Engineering (cs.SE)
Cite as: arXiv:2611.18030 [cs.PF]
  (or arXiv:2611.18030v1 [cs.PF] for this version)
  https://doi.org/10.48550/arXiv.2611.18030

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Rui Rua Dr. [view email]
[v1] Fri, 13 Nov 2026 10:58:12 UTC (391 KB)

来源:arXiv cs.CL · arxiv.org