CRAFT-E：可解释可编辑的神经符号框架，实现基于可供性的主动物体检索

View PDF HTML (experimental)

Abstract:Assistive robots operating in unstructured environments must reason not only about what objects are, but how they can be used, including non-canonical and improvised uses. This requires grounding language-based action queries to objects that afford the requested function, are visually present, and can be physically retrieved. Existing approaches often rely on black-box models, fixed affordance labels, or direct object predictions, limiting transparency and controllability. We introduce CRAFT-E, a modular neuro-symbolic framework that composes an LLM-derived verb-property-object knowledge graph with CLIP-based visual alignment and energy-based grasp reasoning. The system exposes interpretable grounding paths, incorporates grasp feasibility into affordance inference, and supports targeted graph edits without retraining. We also construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. Across static scenes, real-world trials involving 20 verbs and 39 objects, ImageNet-based functional retrieval, and COCO-Tasks transfer, CRAFT-E provides a competitive training-free prior for affordance-grounded retrieval while exposing component-level diagnostics and graph-level interventions. Under staged and full-pipeline perception, CRAFT-E matches strong flat LLM priors in several settings while adding failure attribution and editable reasoning paths. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an inspectable and customizable framework for affordance-grounded object retrieval.

Submission history

From: Sathyanarayanan Aakur [view email]
[v1] Wed, 3 Dec 2025 19:54:27 UTC (2,024 KB)
[v2] Sun, 25 Oct 2026 21:35:41 UTC (1,999 KB)