arXiv cs.CL· Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, Yuan-Fang Li·· 2 天前AI 评分31
基于提示词的大模型语义解析鲁棒性研究:Codex 实证分析
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex
AI 导读
针对基于提示词的大代码模型 Codex 的对抗鲁棒性实证研究显示,这类 SOTA 代码语言模型对精心构造的对抗样本依然脆弱。由于对抗训练对大规模模型不可行(需大量算力和领域内语义解析数据的人工标注),作者提出了无需大量标注数据或高算力即可提升鲁棒性的方法。该研究被 EACL2023 主会接收。
正文 · 原文
Abstract:Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models trained on code have demonstrated superior performance in generating these representations compared to traditional unimodal language models, which are trained on downstream tasks. Despite these advancements, existing fine-tuned neural semantic parsers are susceptible to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, \codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples. To address this challenge, we propose methods for improving robustness without the need for significant amounts of labeled data or heavy computational resources.
| Comments: | Accepted at EACL2023 (main) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2301.12868 [cs.CL] |
| (or arXiv:2301.12868v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2301.12868 arXiv-issued DOI via DataCite |
Submission history
From: Terry Yue Zhuo [view email]
[v1]
Mon, 30 Jan 2023 13:21:00 UTC (3,624 KB)
[v2]
Mon, 6 Feb 2023 07:06:53 UTC (3,625 KB)
[v3]
Thu, 9 Mar 2023 11:01:02 UTC (3,625 KB)
[v4]
Fri, 13 Nov 2026 11:26:17 UTC (47 KB)
来源:arXiv cs.CL · arxiv.org