斯坦福炒虾机器人Mobile ALOHA的关键技术：动作分块算法ACT的原理解析
时间：2024-05-03 15:25:44 来源：网络cs 作者：利杜鹃 栏目：广告工具 阅读：
1.1 ALOHA + ACT解决现有机器人昂贵且难以做精确任务的问题
Stanford University、UC Berkeley、Meta等研究者为了让低成本和不太精确的硬件也能完成这些复杂操作，提出了一个低成本系统，该系统可以通过定制远程操作接口收集实际演示，从而进行端到端的模仿学习(We present a low-cost system that performs end-to-end imitation learning directly from real demonstrations,collected with a custom teleoperation interface)
为了解决这些挑战，该团队开发了一种动作分块算法，即Action Chunking with Transformers (ACT)，它基于Transformer在动作序列上生成模型并允许机器人学习现实世界中6项困难任务(例如打开半透明调味品杯和插入电池)，成功率达80-90%(这是其演示地址)
Designing a model accurate enough for planning would require significant research and task specific engineering efforts. In contrast, the policy of nudging and opening the cup is much simpler, since a closed-loop policy can react to different positions of the cup and lid rather than precisely anticipating how it will move in advance
说白了，咱们不是去生成一个轻推和打开杯子的视频，不用去描绘过程，咱们只是要完成一个具体的目标(结果说话)，而完成某个具体的目标有策略、有方法就行
Small errors in the predicted action can incur large differences in the state, exacerbating the “compounding error” problem of imitation learning [47, 64, 29]. To tackle this, we take inspiration from action chunking, a concept in psychology that describes how sequences of actions are grouped together as a chunk, and executed as one unit [35].在他们的案例中，策略预测了接下来k个时间步的目标关节位置，而不仅仅是一次一步。这通过k折减少了任务的有效视界，减轻了复合误差
In our case, the policy predicts the target joint positions for the next k timesteps, rather than just one step at a time. This reduces the effective horizon of the task by k-fold, mitigating compounding errors.
总之，预测动作序列也有助于解决时间相关的干扰因素，例如难以用马尔可夫单步策略建模的演示中的停顿
Predicting action sequences also helps tackle temporally correlated confounders [61], such as pauses in demonstrations that are hard to model with Markovian single-step policies.为了进一步提高策略的平滑性，本文提出了时间集成，更频繁地查询策略，并在重叠的动作块上进行平均。他们使用Transformers实现动作分块策略，并将其训练为条件VAE (CVAE)，以捕获人类数据中的可变性。他们将该方法命名为Action Chunking with Transformers(ACT)，并发现它在一系列模拟和现实世界的精细操作任务上显著优于以前的模仿学习算法
To further improve the smoothness of the policy, we propose temporal ensembling, which queries the policy more frequently and averages across the overlapping action chunks. We implement action chunking policy with Transformers [65], an architecture designed for sequence modeling, and train it as a conditional VAE (CVAE) [55, 33] to capture the variability in human data. We name our method Action Chunking with Transformers (ACT), and find that it significantly outperforms previous imitation learning algorithms on a range of simulated and real-world fine manipulation tasks.
为了规避这些问题，之前的工作以离线的方式生成合成校正数据[16,29,70]。虽然它们仅限于可获得低维状态的设置，或特定类型的任务，如抓取。由于这些限制，需要从不同的角度来解决复合误差问题，好与高维视觉观察兼容因此建议通过动作分块(action chunking)来减少任务的有效视界，即预测一个动作序列而不是单个动作，然后跨重叠的动作块进行集成，以产生既准确又平滑的轨迹
具体而言，这4个Logitech C922x网络摄像头，每个流输出480×640 RGB图像
其中两个网络摄像头安装在跟随机器人手腕上，以提供夹具的近距离视角(allowing for a close-up view of the grippers)
剩下的两个相机分别安装在前方和顶部位置，遥控操作和数据记录均以50Hz频率进行中间是“手柄和剪刀”机制和定制夹具的详细视图根据上面的原则1、4和5，建立了一个双手平行颚夹持器设置与两个ViperX 6-DoF机器人手臂，上图右侧列出了ViperX 6dof机器人的技术规格
出于价格和维护方面的考虑，不使用灵巧手。使用的ViperX臂具有750克和1.5米跨度的工作有效载荷，精度为5-8毫米
之所以使用领导者的关节位置而不是跟随者的关节位置，是因为通过低级PID控制器施加力量时，这些力量是由“它们之间的差异”隐式定义决定的
It is important to use the leader joint positions instead of the follower’s, because the amount of force appliedis implicitly defined by the difference between them, through the low-level PID controller
观察结果由跟随机器人当前关节位置和4个摄像机图像馈入组成(The observations are composed ofthe current joint positions of follower robots and the image feedfrom 4 cameras)接下来，训练ACT根据当前观察结果预测未来动作序列(we train ACT to predict the sequence offuture actions given the current observations)
这里一个动作对应于下一个时间步中两只手的目标关节位置(An action here corresponds to the target joint positions for both arms in the next time step)
直观地说，ACT试图模仿人类操作员在当前观察结果下，在接下来的时间步中会做什么(ACT tries to imitate what a human operator would do in the following time steps given current observations)然后，Dynamixelmotors内部的低级、高频PID控制器跟踪这些目标关节位置。在测试时，加载实现了最低验证损失策略，并在环境中推出。出现的主要挑战是复合错误，其中包括来自之前动作的错误导致了超出训练分布范围的状态
These target joint positions are then tracked bythe low-level, high-frequency PID controller inside Dynamixelmotors. At test time, we load the policy that achieves the lowest validation loss and roll it out in the environment. Themain challenge that arises is compounding errors, where errorsfrom previous actions lead to states that are outside of training distribution.
wefix the chunk size to be k: every k steps, the agent receivesan observation, generates the next k actions, and executes theactions in sequence.This implies a k-fold reduction in the effective horizon of the task.
具体来说，该策略模拟
而不是。换言之，单步策略将对抗与时间相关的干扰因素，例如人类演示的过程中间出现暂停 ，因为行为不仅取决于状态还取决于时间步长。通过采用动作分块方法可以缓解这种混淆
Concretely,%20the%20policymodels πθ(at:t+k|st) instead%20of πθ(at|st). Specifically,%20a%20single-step%20policy%20would%20struggle%20with%20temporally%20correlated%20confounders,%20such%20as%20pauses%20in%20the%20middle%20of%20ademonstration%20[61],%20since%20the%20behavior%20not%20only%20depends%20onthe%20state,%20but%20also%20the%20time%20step. Action%20chunking%20can%20mitigate这样做可以使不同的动作块相互重叠，在给定的时间步长上产生多个预测动作(This%20makes%20different%20action%20chunks%20overlap%20with%20each%20other,and%20at%20a%20given%20time%20step%20there%20will%20be%20more%20than%20one%20predict%20edaction)，并提出了一个时间集成方法来组合这些预测结果
啥意思呢，如上图所示
在t=0时生成了t=0,1,2,3，四个动作步骤，比如 %20A%20B%20C%20D
%20在t=1时生成了 %20 t=1,2,3,4，四个动作步骤，比如 %20 E%20F%20G%20H
%20在t=2时生成了 %20 %20 %20t=2,3,4,5，四个动作步骤，比如 %20 %20I %20J %20K%20L
%20在t=3时生成了 %20 %20 %20 %20 t=3,4,5,6，四个动作步骤，比如 %20 %20M%20N%20O%20P
%20在t=3时最终采用什么动作，由t=0,t=1,t=2,t=3这四段进行指数加权平均，来得到最终的结果
%20其时间集成通过加权平均对这些预测进行处理，采用指数加权方案，其中表示最早动作的权重(Our temporal%20ensemble%20performs%20a%20weighted%20average%20over%20these%20predictions%20with%20anexponential%20weighting%20scheme wi =%20exp(−m ∗i),%20where w0is%20the%20weight%20for%20the%20oldest%20action. )
%20新观察到达时速度由参数决定，较小的意味着更快地纳入新观察(The%20speed%20for%20incorporatingnew%20observation%20is%20governed%20by m,%20where%20a%20smaller m meansfaster%20incorporation)
%20需要注意的是，与典型平滑方法不同(当前动作与相邻时间步中的动作被聚合在一起，即：A%20B%20C%20D%204个动作，那样会引入偏差)，他们只将同一时间步内的预测动作进行聚合：即D%20G%20J%20M 4个动作
We%20note%20that%20unlike%20typical%20smoothing,where%20the%20current%20action%20is%20aggregated%20with%20actions%20in%20adjacent%20time%20steps,%20which%20leads%20to%20bias,%20we%20aggregate%20actions%20predicted%20for%20the same time%20step.
%20该过程没有额外训练成本，只需增加推理计算所需时间。实践证明，动作分块和时间集成对ACT模型都至关重要，它们能够产生准确且平滑流畅的运动效果
This%20procedure%20also%20incurs%20no%20additional%20training%20cost,%20only%20extra%20inference-time%20computation. In%20practice,we%20find%20both%20action%20chunking%20and%20temporal%20ensembling%20to%20beimportant%20for%20the%20success%20of%20ACT,%20which%20produces%20precise%20andsmooth%20motion%20 编码器的输入是来自演示数据集的当前关节位置，和长度为k的目标动作序列，前面加上一个类似于BERT的习得的“[CLS]”token，从而这形成了一个k+2长度的输入
The inputs to the encoder are the current joint positions and the target action sequence of length k from thedemonstration dataset, prepended by a learned “[CLS]” tokensimilar to BERT. 通过编码器之后，使用“[CLS]”对应的特征来预测“风格变量”z的均值和方差之后，CVAE解码器(即策略)，通过z和当前观测(图像+机器人关节位置)的条件来预测动作序列(即接下来的k个动作)。在测试时，我们将z设置为先验分布的均值，即0以确定解码
For faster training in practice, we leave outthe image observations and only condition on the proprioceptiveobservation and the action sequence. The CVAE decoder, i.e.the policy, conditions on both z and the current observations(images + joint positions) to predict the action sequence. Attest time, we set z to be the mean of the prior distribution i.e.zero to deterministically decode.
整个模型被训练成maximize演示动作块的对数似然，即
The%20whole%20model%20is%20trained%20to%20maximize%20P%20the%20log-likelihood%20of%20demonstration%20action%20chunks,%20i.e.minθ−st,at:t+k ∈D log πθ(at:t+k |st)
%20标准VAE目标有两项:重建损失和将编码器正则化为高斯先验的项。直观地说，更高的β将导致z中传递的信息更少
with%20the%20standard%20VAEobjective%20which%20has%20two%20terms:%20a%20reconstruction%20loss%20and%20a%20termthat%20regularizes%20the%20encoder%20to%20a%20Gaussian%20prior. Following%20[23],we%20weight%20the%20second%20term%20with%20a%20hyperparameter β.
%20当然，我们使用ResNet图像编码器、transformer%20encoder，和transformer%20decoder来实现CVAE解码器，直观地说
%20transformer编码器综合了来自不同相机视角、关节位置和风格变量的信息(different%20camera%20viewpoints,%20the%20joint%20positions,%20and%20the%20style%20variable)
%20transformer解码器生成连贯的动作序列(generates%20a%20coherent%20action%20sequence)%20
Intuitively, the transformer encoder synthesizes information from different camera viewpoints, the joint positions, and the style variable, and the transformer decoder generates a coherentaction sequence. The observation includes 4 RGB images, eachat 480 ×640 resolution, and joint positions for two robot arms(7+7=14 DoF in total). 输出：动作空间是两个机器人的绝对关节位置，一个14维向量。因此，通过动作分块，策略在给定当前观测的情况下输出一个k ×14张量(每个动作都被定义为一个14维的向量，所以k个动作自然便是一个k ×14张量)
The action space is the absolute jointpositions for two robots, a 14-dimensional vector. Thus withaction chunking, the policy outputs a k ×14 tensor giventhe current observation.
通过一个线性层linear layer2，把joints投影到嵌入维度的关节位置(14维到512维)：embedded joints嵌入动作序列embedded action sequence
通过另一个线性层linear layer1，把k × 14的action sequence投影到嵌入维度的动作序列(k × 14维到k × 512维) 首先，对于每一个图像观察，其皆被ResNet18处理以获得一个特征图(将480张×640 ×3 RGB图像转换为15张×20 ×728特征图)
然后flatten化以获得一个特征序列(300×728)
这些特征用线性层linear layer5投影到嵌入维度(300×512)
为了保留空间信息，再添加一个2D正弦位置嵌入(即Sinusoidal PosEmb)
其次，对所有4张图像重复此操作，得到的特征序列维度为1200 ×512
Repeating this for all 4 images gives a feature sequence of 1200 × 512 in dimension.
接着，将来自每个摄像机的特征序列连接起来，用作transformer encoder的输入之一，对于另外两个输入：当前的关节位置joints和“风格变量”z，它们分别通过线性层linear layer6、linear layer7从各自的原始维度(14、15)都统一投影到512
最终，the input to the transformer encoder is 1202×512(相信你很快反应出来了，连接此三：4张图像的特征维度1200 ×512、关节位置joins的特征维度1×512，风格变量z的特征维度1×512)transformer解码器的输入有两个方面
一方面，transformer解码器的“query”是第一层固定的正弦位置嵌入，即如上图右下角所示的position embeddings(fixed)，其维度为k ×512
二方面，transformer解码器的交叉注意力(cross-attention)层中的“keys”和“values”来自上述transformer编码器的输出
即如论文中所述，The transformer decoder conditions on the encoder output through cross-attention, where the input sequence is a fixed position embedding, with dimensions k × 512, and the keys and values are coming from the encoder.
从而，transformer解码器在给定编码器输出的情况下预测动作序列
以下是更多细节
transformer解码器的输出维度是k ×512，然后用MLP向下投影到k ×14，对应于接下来k个步骤的预测目标关节位置
This gives the transformer decoder an output dimension of k × 512, which is then down-projected with an MLP into k × 14, corresponding to the predicted target joint positions for the next k steps.
最终，我们使用L1损失进行重建，而不是更常见的L2损失：我们注意到，L1损失导致对动作序列进行更精确的建模(We use L1 loss for reconstruction instead of the more common L2 loss: we noted that L1 loss leads to more precise modeling of the action sequence. )
我们还注意到，当使用delta关节位置作为动作而不是目标关节位置时，性能会下降(We also noted degraded performance when using delta joint positions as actions instead of target joint positions)
BC-ConvMLP is the simplest yet most widely used baseline [69, 26], which processes the current image observations with a convolutional network, whose output features are concatenated with the joint positions to predict the action.BeT [49]也利用Transformer作为架构，但有两个关键区别：
(1)没有对动作进行分块处理，该模型预测给定“观测历史之下”的一个动作
no action chunking: the model predicts one action given the history of observations
(2)图像观测由单独训练的冻结视觉编码器预处理，即感知和控制网络没有联合优化
the image observations are pre-processed by a separately trained frozen visual encoder. That is, the perception and control networks are not jointly optimized.RT-1 [7]是另一种基于Transformer的架构，从“过去观测中固定长度的历史中”预测一个动作
RT-1 [7] is another Transformerbased architecture that predicts one action from a fixed-length history of past observations.
BeT和RT-1都离散化了动作空间：输出为离散箱分类分布，在BeT情况下还添加了连续偏移量。而我们的方法ACT直接预测连续动作，这是出于精确操作所需精度驱使
Both BeT and RT-1 discretize the action space: the output is a categorical distribution over discrete bins, but with an added continuous offset from the bincenter in the case of BeT. Our method, ACT, instead directly predicts continuous actions, motivated by the precision required in fine manipulation.最后，VINN [42]是一种非参数方法，在测试时假设可以访问演示数据。给定新的观察数据时，它检索具有最相似视觉特征的k个样本，并返回一个加权操作
Lastly, VINN [42] is a non-parametric method that assumes access to the demonstrations at test time. Given a new observation, it retrieves the k observations with
the most similar visual features, and returns an action using weighted k-nearest-neighbors.
且其采用了一个经过预训练的ResNet作为视觉特征提取器，并在无监督学习的演示数据上进行微调。我们通过精心调整超参数，使用立方体转移方法对这四种先前方法进行了优化
The visual feature extractor is a pretrained ResNet finetuned on demonstration data with unsupervised learning. We carefully tune the hyperparameters of these four prior methods using cube transfer.
斯坦福Mobile ALOHA在被推出之前，其实在23年Q1便已有了ALOHA，所谓ALOHA，即是A Low-cost Open-source Hardware System for Bimanual Teleoperation
如此文《大模型机器人发展史：从VoxPoser、RT2到斯坦福Mobile ALOHA、Google机器人》的第四部分“4.1.2 Mobile ALOHA的前置工作：ALOHA与ACT”节所述
Mobile ALOHA其实是在23年ALOHA的工作基础上迭代优化出来的，不是一蹴而就，以下是关于ALOHA的一系列重要信息
ALOHA项目地址：Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware论文地址：Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
这是其解读，论文中首次系统阐述了作为“无论是最新系统Mobile ALOHA还是其前身系统ALOHA中的关键技术”：即动作分块算法ACT代码地址：GitHub - tonyzhaozh/aloha
该代码仓库友情提醒：
To build ALOHA, follow the Hardware Assembly Tutorial and the quick start guide below.
To train imitation learning algorithms, you would also need to install ACT.硬件安装指南：ALOHA 🏖️ Tutorial (文档标题为：ALOHA 🏖️: A Low-cost Open-source Hardware for Bimanual Teleoperation)基于动作分块算法ACT的训练代码：GitHub - tonyzhaozh/act
所以，一个低成本的开源硬件系统ALOHA，可手动远程操作，是怎么一步步来的呢？
在前两年，让机器人去完成一些抽象精细的操作任务，比如穿线扎带或开槽电池，是比较困难的
一方面，因为这些任务需要比较高的精准度、协调性以及闭环视觉反馈。通常情况下，执行这些任务需要使用高端机器人、精确传感器或者仔细校准设备，而且成本昂贵且难以设置Stanford University、UC Berkeley、Meta等研究者为了让低成本和不太精确的硬件也能完成这些复杂操作，提出了一个低成本系统，该系统可以通过定制远程操作接口收集实际演示，从而进行端到端的模仿学习(We present a low-cost system that performs end-to-end imitation learning directly from real demonstrations,collected with a custom teleoperation interface)
为了解决这些挑战，该团队开发了一种动作分块算法，即Action Chunking with Transformers (ACT)，它基于Transformer在动作序列上生成模型并允许机器人学习现实世界中6项困难任务(例如打开半透明调味品杯和插入电池)，成功率达80-90%(这是其演示地址)
根据该团队发表的这篇论文《Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware》可知
他们训练了一个端到端的策略，该策略直接将来自商品网络相机的RGB图像映射到动作(we therefore train an end-to-end policy that directly maps RGB images from commodity web cameras to the actions.)。这种从像素到动作的转换方法特别适用于精细操作，因为精细操纵通常涉及具有复杂物理特质的对象，这样学习操纵策略比建模整个环境要简单得多以调味品杯为例：模拟轻推杯子时的接触，以及撬开盖子时的变形，都会引起大量的物理变化。与其耗费大量的研究和具体任务的工程努力去设计一个足够精确的模型进行具体的规划，不如转化为执行上要简单很多的策略制定，即确定轻推和打开杯子的策略或方法，因为策略可以对杯子和盖子的不同位置做出反应，而不是提前精确预测它将如何移动Designing a model accurate enough for planning would require significant research and task specific engineering efforts. In contrast, the policy of nudging and opening the cup is much simpler, since a closed-loop policy can react to different positions of the cup and lid rather than precisely anticipating how it will move in advance
说白了，咱们不是去生成一个轻推和打开杯子的视频，不用去描绘过程，咱们只是要完成一个具体的目标(结果说话)，而完成某个具体的目标有策略、有方法就行
1.1.1 模仿学习及其挑战：Action Chunking with Transformers(ACT)
系统有了，数据也好办(高质量的人类演示可以让系统学习人类的灵巧，因此，可让低成本但灵巧的遥操作系统ALOHA做数据收集)，但训练一个端到端的策略(end-to-end policy)可没那么容易，因为即使是高质量的演示，在处理需要精度和视觉反馈的任务时，对模仿学习来说也是一个重大挑战
预测动作中的小误差会引起状态的大差异，加剧模仿学习的“复合误差”问题。为了解决这个问题，他们从动作分块(action chunking)中获得灵感，这是心理学中的一个概念，描述了如何将一系列动作组合在一起作为一个块，最终作为一个单元执行Small errors in the predicted action can incur large differences in the state, exacerbating the “compounding error” problem of imitation learning [47, 64, 29]. To tackle this, we take inspiration from action chunking, a concept in psychology that describes how sequences of actions are grouped together as a chunk, and executed as one unit [35].在他们的案例中，策略预测了接下来k个时间步的目标关节位置，而不仅仅是一次一步。这通过k折减少了任务的有效视界，减轻了复合误差
In our case, the policy predicts the target joint positions for the next k timesteps, rather than just one step at a time. This reduces the effective horizon of the task by k-fold, mitigating compounding errors.
总之，预测动作序列也有助于解决时间相关的干扰因素，例如难以用马尔可夫单步策略建模的演示中的停顿
Predicting action sequences also helps tackle temporally correlated confounders [61], such as pauses in demonstrations that are hard to model with Markovian single-step policies.为了进一步提高策略的平滑性，本文提出了时间集成，更频繁地查询策略，并在重叠的动作块上进行平均。他们使用Transformers实现动作分块策略，并将其训练为条件VAE (CVAE)，以捕获人类数据中的可变性。他们将该方法命名为Action Chunking with Transformers(ACT)，并发现它在一系列模拟和现实世界的精细操作任务上显著优于以前的模仿学习算法
To further improve the smoothness of the policy, we propose temporal ensembling, which queries the policy more frequently and averages across the overlapping action chunks. We implement action chunking policy with Transformers [65], an architecture designed for sequence modeling, and train it as a conditional VAE (CVAE) [55, 33] to capture the variability in human data. We name our method Action Chunking with Transformers (ACT), and find that it significantly outperforms previous imitation learning algorithms on a range of simulated and real-world fine manipulation tasks.
1.1.2 行为克隆(Behavioral cloning, BC)中为何要引入ACT
模仿学习可以让机器人直接向专家学习，而行为克隆(Behavioral cloning, BC)是最简单的模仿学习算法之一，将模仿作为从观察到行动的监督学习
BC的一个主要缺点是复合误差，以前时间步长的误差累积并导致机器人偏离其训练分布，导致难以恢复状态[47,64]。这个问题在精细操作设置[29]中尤为突出。减轻复合错误的一种方法是允许额外的政策上的交互和专家修正，如DAgger[47]及其变体[30,40,24]然而，对于遥操作界面[29]，专家注释可能是耗时且不自然的。人们也可以在演示采集时注入噪声以获得具有纠正行为[36]的数据集，但对于精细操作而言，这种噪声注入会直接导致任务失败，降低遥操作系统的灵巧度为了规避这些问题，之前的工作以离线的方式生成合成校正数据[16,29,70]。虽然它们仅限于可获得低维状态的设置，或特定类型的任务，如抓取。由于这些限制，需要从不同的角度来解决复合误差问题，好与高维视觉观察兼容因此建议通过动作分块(action chunking)来减少任务的有效视界，即预测一个动作序列而不是单个动作，然后跨重叠的动作块进行集成，以产生既准确又平滑的轨迹
1.2 硬件套装：ALOHA——低成本的开源硬件系统，用于手动远程操作
我们已经知道市面上已有类似达芬奇外科手术机器人或ABB YuMi这样的机器人，但成本一般比较昂贵
他们团队则转向低成本硬件(很类似七月在一系列大模型应用上的探索，侧重小团队 低成本 大效果)，例如每个成本约为5k美元的手臂，并寻求使它们能够执行高精度、闭环任务。他们的遥操作设置最类似于Kim等人的[32]，也使用了人类遥控者和跟随者机器人之间的关节空间映射与之前的系统不同，他们没有使用特殊的编码器、传感器或加工部件。他们只使用现成的机器人和少量3D打印部件来构建他们的系统，让非专业人士在不到2小时内组装好它具体而言，其具备以下五个特点
低成本：整个系统应该在大多数机器人实验室的预算之内，可与单个工业手臂相媲美通用性：它可以应用于广泛的与现实物体的精细操作任务人性化：系统应该直观、可靠、易于使用可修复：当设置不可避免地出现故障时，研究人员可以轻松修复设置易于搭建：研究人员可以快速组装，材料来源容易所以，在设计一个远程操作系统时，没有将VR控制器或摄像头捕获的手部姿势映射到机器人的末端执行器姿势，即任务空间映射(Instead of mapping the hand pose captured by a VR controller or camera to the end-effector pose of the robot, i.e. task-space mapping)，而是使用来自同一家公司制造的小型机器人WidowX的直接关节空间映射，成本为3300美元
如下图所示：
具体而言，这4个Logitech C922x网络摄像头，每个流输出480×640 RGB图像
其中两个网络摄像头安装在跟随机器人手腕上，以提供夹具的近距离视角(allowing for a close-up view of the grippers)
剩下的两个相机分别安装在前方和顶部位置，遥控操作和数据记录均以50Hz频率进行中间是“手柄和剪刀”机制和定制夹具的详细视图根据上面的原则1、4和5，建立了一个双手平行颚夹持器设置与两个ViperX 6-DoF机器人手臂，上图右侧列出了ViperX 6dof机器人的技术规格
出于价格和维护方面的考虑，不使用灵巧手。使用的ViperX臂具有750克和1.5米跨度的工作有效载荷，精度为5-8毫米
且该机器人模块化，维修简单：在电机出现故障的情况下，低成本的Dynamixel电机可以轻松更换。这种机器人可以以5600美元左右的价格购买到现货。然而，OEM的手指不够通用，无法处理精细的操作任务。因此，设计了自己的3D打印“透明”手指，并将其贴合在夹持胶带(gripping tape)上
再补充说明下
该系统中，用户通过反向驱动较小的WidowX(“领导者”)来远程操作，其关节与较大的ViperX(“追随者”)同步。在开发设置时，注意到使用关节空间映射(joint-space mapping)比使用任务空间有一些好处
精细操作通常需要在机器人的singularities附近操作，在他们的例子中，机器人有6个自由度，没有冗余。现成的逆运动学(inverse kinematics，IK)在这种情况下经常失效。另一方面，关节空间映射保证了关节限制内的高带宽控制，同时也需要更少的计算和减少延迟leader机器人的重量可以防止用户移动过快，同时也可以抑制小的振动。注意到使用关节空间映射比手持VR控制器在精确任务上表现更好。为了进一步提高遥操作体验，设计了一个3d打印的“手柄和剪刀(handle and scissor)”机构，可以改装到领导者机器人上
它减少了操作人员反向驱动电机所需的力，并允许连续控制夹持器，instead of binary opening or closingWe also design a rubber band load balancing mechanism that partially counteracts the gravity on the leader side。它减少了操作人员所需的努力，并使更长的遥操作时间(如>30分钟)成为可能(项目网站中包含了有关设置的更多细节)其余设置包括一个尺寸为20×20mm的铝挤压机器人笼，由交叉钢缆加固(The rest of the setup includes a robot cage with 20×20mm aluminum extrusions, reinforced by crossing steel cables)
第二部分 动作分块算法ACT的原理解析
2.1 从整体流程上整体理解：Action Chunking with Transformers(ACT)
为了在新任务上训练ACT，首先使用ALOHA收集人类演示
之所以使用领导者的关节位置而不是跟随者的关节位置，是因为通过低级PID控制器施加力量时，这些力量是由“它们之间的差异”隐式定义决定的
It is important to use the leader joint positions instead of the follower’s, because the amount of force appliedis implicitly defined by the difference between them, through the low-level PID controller
观察结果由跟随机器人当前关节位置和4个摄像机图像馈入组成(The observations are composed ofthe current joint positions of follower robots and the image feedfrom 4 cameras)接下来，训练ACT根据当前观察结果预测未来动作序列(we train ACT to predict the sequence offuture actions given the current observations)
这里一个动作对应于下一个时间步中两只手的目标关节位置(An action here corresponds to the target joint positions for both arms in the next time step)
直观地说，ACT试图模仿人类操作员在当前观察结果下，在接下来的时间步中会做什么(ACT tries to imitate what a human operator would do in the following time steps given current observations)然后，Dynamixelmotors内部的低级、高频PID控制器跟踪这些目标关节位置。在测试时，加载实现了最低验证损失策略，并在环境中推出。出现的主要挑战是复合错误，其中包括来自之前动作的错误导致了超出训练分布范围的状态
These target joint positions are then tracked bythe low-level, high-frequency PID controller inside Dynamixelmotors. At test time, we load the policy that achieves the lowest validation loss and roll it out in the environment. Themain challenge that arises is compounding errors, where errorsfrom previous actions lead to states that are outside of training distribution.
2.1.1 动作分块：将同一时间步内的预测动作进行聚合
为了以一种与像素到动作策略兼容的方式来解决模仿学习中的复合错误，寻求减少高频收集的长轨迹的有效视域(To combat the compounding errors of imitation learning in away that is compatible with pixel-to-action policies,we seek to reduce the effective horizon of long trajectories collected at high frequency)
对此，受到了动作分块的启发，这是一个神经科学概念，其中个体动作被分组在一起并作为一个单元执行，从而更有效地存储和执行[We are inspired by action chunking,a neuro science concept where individual actions are grouped together and executed as one unit, making them more efficientto store and execute]
wefix the chunk size to be k: every k steps, the agent receivesan observation, generates the next k actions, and executes theactions in sequence.This implies a k-fold reduction in the effective horizon of the task.
具体来说，该策略模拟
Concretely,%20the%20policymodels πθ(at:t+k|st) instead%20of πθ(at|st). Specifically,%20a%20single-step%20policy%20would%20struggle%20with%20temporally%20correlated%20confounders,%20such%20as%20pauses%20in%20the%20middle%20of%20ademonstration%20[61],%20since%20the%20behavior%20not%20only%20depends%20onthe%20state,%20but%20also%20the%20time%20step. Action%20chunking%20can%20mitigate这样做可以使不同的动作块相互重叠，在给定的时间步长上产生多个预测动作(This%20makes%20different%20action%20chunks%20overlap%20with%20each%20other,and%20at%20a%20given%20time%20step%20there%20will%20be%20more%20than%20one%20predict%20edaction)，并提出了一个时间集成方法来组合这些预测结果
啥意思呢，如上图所示
在t=0时生成了t=0,1,2,3，四个动作步骤，比如 %20A%20B%20C%20D
%20在t=1时生成了 %20 t=1,2,3,4，四个动作步骤，比如 %20 E%20F%20G%20H
%20在t=2时生成了 %20 %20 %20t=2,3,4,5，四个动作步骤，比如 %20 %20I %20J %20K%20L
%20在t=3时生成了 %20 %20 %20 %20 t=3,4,5,6，四个动作步骤，比如 %20 %20M%20N%20O%20P
%20在t=3时最终采用什么动作，由t=0,t=1,t=2,t=3这四段进行指数加权平均，来得到最终的结果
%20其时间集成通过加权平均对这些预测进行处理，采用指数加权方案
%20新观察到达时速度由参数
%20需要注意的是，与典型平滑方法不同(当前动作与相邻时间步中的动作被聚合在一起，即：A%20B%20C%20D%204个动作，那样会引入偏差)，他们只将同一时间步内的预测动作进行聚合：即D%20G%20J%20M 4个动作
We%20note%20that%20unlike%20typical%20smoothing,where%20the%20current%20action%20is%20aggregated%20with%20actions%20in%20adjacent%20time%20steps,%20which%20leads%20to%20bias,%20we%20aggregate%20actions%20predicted%20for%20the same time%20step.
%20该过程没有额外训练成本，只需增加推理计算所需时间。实践证明，动作分块和时间集成对ACT模型都至关重要，它们能够产生准确且平滑流畅的运动效果
This%20procedure%20also%20incurs%20no%20additional%20training%20cost,%20only%20extra%20inference-time%20computation. In%20practice,we%20find%20both%20action%20chunking%20and%20temporal%20ensembling%20to%20beimportant%20for%20the%20success%20of%20ACT,%20which%20produces%20precise%20andsmooth%20motion%20
2.2.2%20Modeling%20human%20data与ACT的执行
%20上文1.2节我们解决了硬件系统的问题，1.3节我们解决了算法优化的问题，接下来，该细化数据的问题了
%20实话说，让机器人从嘈杂的人类演示中学习并不容易，在相同的观察下，人类可以使用不同的轨迹来解决任务。在精度不那么重要的区域，人类也会更随机。因此，对于政策来说，关注高精度重要的区域是很重要的。该团队通过训练他们的动作分块策略作为一个生成模型来解决这个问题
%20最终将策略训练为条件变分自编码器(CVAE，用类似BERT的编码器实现)，以生成以当前观察为条件的动作序列。CVAE有两个组件：一个CVAE编码器和一个CVAE解码器。CVAE编码器只用于训练CVAE解码器(策略)，在测试时被丢弃
%20具体而言，如下图所示
The inputs to the encoder are the current joint positions and the target action sequence of length k from thedemonstration dataset, prepended by a learned “[CLS]” tokensimilar to BERT. 通过编码器之后，使用“[CLS]”对应的特征来预测“风格变量”z的均值和方差之后，CVAE解码器(即策略)，通过z和当前观测(图像+机器人关节位置)的条件来预测动作序列(即接下来的k个动作)。在测试时，我们将z设置为先验分布的均值，即0以确定解码
For faster training in practice, we leave outthe image observations and only condition on the proprioceptiveobservation and the action sequence. The CVAE decoder, i.e.the policy, conditions on both z and the current observations(images + joint positions) to predict the action sequence. Attest time, we set z to be the mean of the prior distribution i.e.zero to deterministically decode.
整个模型被训练成maximize演示动作块的对数似然，即
The%20whole%20model%20is%20trained%20to%20maximize%20P%20the%20log-likelihood%20of%20demonstration%20action%20chunks,%20i.e.minθ−st,at:t+k ∈D log πθ(at:t+k |st)
%20标准VAE目标有两项:重建损失和将编码器正则化为高斯先验的项。直观地说，更高的β将导致z中传递的信息更少
with%20the%20standard%20VAEobjective%20which%20has%20two%20terms:%20a%20reconstruction%20loss%20and%20a%20termthat%20regularizes%20the%20encoder%20to%20a%20Gaussian%20prior. Following%20[23],we%20weight%20the%20second%20term%20with%20a%20hyperparameter β.
%20当然，我们使用ResNet图像编码器、transformer%20encoder，和transformer%20decoder来实现CVAE解码器，直观地说
%20transformer编码器综合了来自不同相机视角、关节位置和风格变量的信息(different%20camera%20viewpoints,%20the%20joint%20positions,%20and%20the%20style%20variable)
%20transformer解码器生成连贯的动作序列(generates%20a%20coherent%20action%20sequence)%20
2.2%20从ACT执行的详细架构图与详细步骤进一步理解ACT
%20以下是论文中关于上述训练过程的更多细节
%202.2.1 采样数据
如下图所示
Intuitively, the transformer encoder synthesizes information from different camera viewpoints, the joint positions, and the style variable, and the transformer decoder generates a coherentaction sequence. The observation includes 4 RGB images, eachat 480 ×640 resolution, and joint positions for two robot arms(7+7=14 DoF in total). 输出：动作空间是两个机器人的绝对关节位置，一个14维向量。因此，通过动作分块，策略在给定当前观测的情况下输出一个k ×14张量(每个动作都被定义为一个14维的向量，所以k个动作自然便是一个k ×14张量)
The action space is the absolute jointpositions for two robots, a 14-dimensional vector. Thus withaction chunking, the policy outputs a k ×14 tensor giventhe current observation.
2.2.2 推断z
然后，我们使用下图右侧黄色所示的CVAE编码器推断风格变量z
编码器的输入是
[CLS]token，它由随机初始化的学习权值组成嵌入关节位置embedded joints通过一个线性层linear layer2，把joints投影到嵌入维度的关节位置(14维到512维)：embedded joints嵌入动作序列embedded action sequence
通过另一个线性层linear layer1，把k × 14的action sequence投影到嵌入维度的动作序列(k × 14维到k × 512维)
以上三个输入最终形成(k + 2)×embedding_dimension的序列，即(k + 2) × 512，并用transformer编码器进行处理
最后，只取第一个输出，它对应于[CLS]标记，并使用另一个线性网络来预测分布的均值和方差，将其参数化为对角高斯分布。使用重新参数化获得
2.2.3%20预测动作序列
%20接下来，我们尝试从CVAE解码器中获得预测的动作，即策略(预测动作序列)
%20然后flatten化以获得一个特征序列(300×728)
这些特征用线性层linear layer5投影到嵌入维度(300×512)
为了保留空间信息，再添加一个2D正弦位置嵌入(即Sinusoidal PosEmb)
其次，对所有4张图像重复此操作，得到的特征序列维度为1200 ×512
Repeating this for all 4 images gives a feature sequence of 1200 × 512 in dimension.
接着，将来自每个摄像机的特征序列连接起来，用作transformer encoder的输入之一，对于另外两个输入：当前的关节位置joints和“风格变量”z，它们分别通过线性层linear layer6、linear layer7从各自的原始维度(14、15)都统一投影到512
最终，the input to the transformer encoder is 1202×512(相信你很快反应出来了，连接此三：4张图像的特征维度1200 ×512、关节位置joins的特征维度1×512，风格变量z的特征维度1×512)transformer解码器的输入有两个方面
一方面，transformer解码器的“query”是第一层固定的正弦位置嵌入，即如上图右下角所示的position embeddings(fixed)，其维度为k ×512
二方面，transformer解码器的交叉注意力(cross-attention)层中的“keys”和“values”来自上述transformer编码器的输出
即如论文中所述，The transformer decoder conditions on the encoder output through cross-attention, where the input sequence is a fixed position embedding, with dimensions k × 512, and the keys and values are coming from the encoder.
从而，transformer解码器在给定编码器输出的情况下预测动作序列
以下是更多细节
transformer解码器的输出维度是k ×512，然后用MLP向下投影到k ×14，对应于接下来k个步骤的预测目标关节位置
This gives the transformer decoder an output dimension of k × 512, which is then down-projected with an MLP into k × 14, corresponding to the predicted target joint positions for the next k steps.
最终，我们使用L1损失进行重建，而不是更常见的L2损失：我们注意到，L1损失导致对动作序列进行更精确的建模(We use L1 loss for reconstruction instead of the more common L2 loss: we noted that L1 loss leads to more precise modeling of the action sequence. )
我们还注意到，当使用delta关节位置作为动作而不是目标关节位置时，性能会下降(We also noted degraded performance when using delta joint positions as actions instead of target joint positions)
以下是算法1和算法2中ACT的训练和推理。该模型有大约80M个参数，为每个任务从头开始训练。训练在单个11G RTX 2080 Ti GPU上需要大约5个小时，在同一台机器上的推理时间约为0.01秒
2.3 ACT与其他模仿学习方法的比较
我们将ACT与之前的4种模仿学习方法进行比较
BC-ConvMLP是最简单但最广泛使用的基线[69, 26]，它采用卷积网络处理当前图像观测，并将其输出特征与关节位置连接，以预测动作BC-ConvMLP is the simplest yet most widely used baseline [69, 26], which processes the current image observations with a convolutional network, whose output features are concatenated with the joint positions to predict the action.BeT [49]也利用Transformer作为架构，但有两个关键区别：
(1)没有对动作进行分块处理，该模型预测给定“观测历史之下”的一个动作
no action chunking: the model predicts one action given the history of observations
(2)图像观测由单独训练的冻结视觉编码器预处理，即感知和控制网络没有联合优化
the image observations are pre-processed by a separately trained frozen visual encoder. That is, the perception and control networks are not jointly optimized.RT-1 [7]是另一种基于Transformer的架构，从“过去观测中固定长度的历史中”预测一个动作
RT-1 [7] is another Transformerbased architecture that predicts one action from a fixed-length history of past observations.
BeT和RT-1都离散化了动作空间：输出为离散箱分类分布，在BeT情况下还添加了连续偏移量。而我们的方法ACT直接预测连续动作，这是出于精确操作所需精度驱使
Both BeT and RT-1 discretize the action space: the output is a categorical distribution over discrete bins, but with an added continuous offset from the bincenter in the case of BeT. Our method, ACT, instead directly predicts continuous actions, motivated by the precision required in fine manipulation.最后，VINN [42]是一种非参数方法，在测试时假设可以访问演示数据。给定新的观察数据时，它检索具有最相似视觉特征的k个样本，并返回一个加权操作
Lastly, VINN [42] is a non-parametric method that assumes access to the demonstrations at test time. Given a new observation, it retrieves the k observations with
the most similar visual features, and returns an action using weighted k-nearest-neighbors.
且其采用了一个经过预训练的ResNet作为视觉特征提取器，并在无监督学习的演示数据上进行微调。我们通过精心调整超参数，使用立方体转移方法对这四种先前方法进行了优化
The visual feature extractor is a pretrained ResNet finetuned on demonstration data with unsupervised learning. We carefully tune the hyperparameters of these four prior methods using cube transfer.
如本文开头所说的， 至于ACT的代码剖析、部署实践，请见下一篇文章《逐行解读ACT：机器人模仿学习之动作分块算法ACT的代码剖析》
参考文献与推荐阅读
https://tonyzhaozh.github.io/aloha/Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware(阅读笔记)..本文链接：https://www.kjpai.cn/news/2024-05-03/164505.html，文章来源：网络cs，作者：利杜鹃，版权归作者所有，如需转载请注明来源和作者，否则将追究法律责任！
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。