Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences

计算机科学 任务(项目管理) 偏爱 行为模式 人机交互 人工智能 模棱两可 概括性 剪辑 心理学 软件工程 经济 微观经济学 管理 程序设计语言 心理治疗师
作者
Lin Guan,Karthik Valmeekam,Subbarao Kambhampati
出处
期刊:Cornell University - arXiv
标识
DOI:10.48550/arxiv.2210.15906
摘要

Generating complex behaviors that satisfy the preferences of non-expert users is a crucial requirement for AI agents. Interactive reward learning from trajectory comparisons (a.k.a. RLHF) is one way to allow non-expert users to convey complex objectives by expressing preferences over short clips of agent behaviors. Even though this parametric method can encode complex tacit knowledge present in the underlying tasks, it implicitly assumes that the human is unable to provide richer feedback than binary preference labels, leading to intolerably high feedback complexity and poor user experience. While providing a detailed symbolic closed-form specification of the objectives might be tempting, it is not always feasible even for an expert user. However, in most cases, humans are aware of how the agent should change its behavior along meaningful axes to fulfill their underlying purpose, even if they are not able to fully specify task objectives symbolically. Using this as motivation, we introduce the notion of Relative Behavioral Attributes, which allows the users to tweak the agent behavior through symbolic concepts (e.g., increasing the softness or speed of agents' movement). We propose two practical methods that can learn to model any kind of behavioral attributes from ordered behavior clips. We demonstrate the effectiveness of our methods on four tasks with nine different behavioral attributes, showing that once the attributes are learned, end users can produce desirable agent behaviors relatively effortlessly, by providing feedback just around ten times. This is over an order of magnitude less than that required by the popular learning-from-human-preferences baselines. The supplementary video and source code are available at: https://guansuns.github.io/pages/rba.

科研通智能强力驱动
Strongly Powered by AbleSci AI
更新
PDF的下载单位、IP信息已删除 (2025-6-4)

科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
yue发布了新的文献求助10
2秒前
2秒前
胡须应助淼队采纳,获得30
3秒前
木木木袁袁袁完成签到,获得积分20
3秒前
寒梅恋雪完成签到,获得积分10
4秒前
自觉的凛完成签到,获得积分10
4秒前
天天快乐应助源来是洲董采纳,获得10
4秒前
75986686完成签到,获得积分10
5秒前
7秒前
学好久完成签到 ,获得积分10
8秒前
a24017完成签到,获得积分10
11秒前
YG完成签到,获得积分10
12秒前
yue完成签到,获得积分10
16秒前
淼队完成签到,获得积分10
17秒前
17秒前
落叶解三秋完成签到,获得积分10
18秒前
Crystal完成签到 ,获得积分10
21秒前
小小酥完成签到,获得积分10
21秒前
等待蚂蚁完成签到 ,获得积分10
22秒前
zgt01发布了新的文献求助10
22秒前
心心完成签到 ,获得积分10
23秒前
123完成签到,获得积分10
24秒前
温超完成签到,获得积分10
24秒前
量子星尘发布了新的文献求助10
24秒前
25秒前
26秒前
Menta1y完成签到,获得积分10
26秒前
czzlancer完成签到,获得积分10
27秒前
汶溢完成签到,获得积分10
27秒前
xsss完成签到,获得积分10
28秒前
TAN完成签到,获得积分10
28秒前
通通通发布了新的文献求助10
29秒前
liudw完成签到,获得积分10
29秒前
丹丹子完成签到 ,获得积分10
30秒前
时光完成签到,获得积分10
30秒前
31秒前
充电宝应助vsvsgo采纳,获得10
33秒前
123完成签到 ,获得积分10
35秒前
Ammr完成签到 ,获得积分10
35秒前
无限的依波完成签到,获得积分10
35秒前
高分求助中
【提示信息,请勿应助】关于scihub 10000
Les Mantodea de Guyane: Insecta, Polyneoptera [The Mantids of French Guiana] 3000
徐淮辽南地区新元古代叠层石及生物地层 3000
The Mother of All Tableaux: Order, Equivalence, and Geometry in the Large-scale Structure of Optimality Theory 3000
Handbook of Industrial Diamonds.Vol2 1100
Global Eyelash Assessment scale (GEA) 1000
Picture Books with Same-sex Parented Families: Unintentional Censorship 550
热门求助领域 (近24小时)
化学 材料科学 医学 生物 工程类 有机化学 生物化学 物理 内科学 纳米技术 计算机科学 化学工程 复合材料 遗传学 基因 物理化学 催化作用 冶金 细胞生物学 免疫学
热门帖子
关注 科研通微信公众号,转发送积分 4038235
求助须知:如何正确求助?哪些是违规求助? 3575992
关于积分的说明 11374009
捐赠科研通 3305760
什么是DOI,文献DOI怎么找? 1819276
邀请新用户注册赠送积分活动 892662
科研通“疑难数据库(出版商)”最低求助积分说明 815022