@article{shafiullah2023bringing,
title={On bringing robots home},
author={Shafiullah, Nur Muhammad Mahi and Rai, Anant and Etukuru, Haritheja and Liu, Yiqian and Misra, Ishan and Chintala, Soumith and Pinto, Lerrel},
journal={arXiv preprint arXiv:2311.16098},
year={2023}
}
Dataset
Homes of New York (HoNY)
22 homes
216 environments
5620 trajectories
13 hours
1.5 million frames
Homes of New York (HoNY) is a dataset containing 13 hours of interactions at 22 different homes of New York City collected with the Stick. The dataset contains RGB and depth videos at 30 fps, as well as full action annotations for 6D pose of the gripper as well as the gripper's opening angle normalized between (0, 1).
Model
Home Pretrained Representations (HPR)
Home Pretrained Representation (HPR) is a model pre-trained on the HoNY dataset that we used to initialize a robot policy to perform a new task in a novel enviroment. HPR is a ResNet-34 model trained on the HoNY dataset using the MoCo-v3 self-supervised learning objective.
During deployment,we used HPR to initialize a policy, the trunk of which was simply our pretrained ResNet-34 model followed by two linear layers on top.
🤗 Get the model at HuggingfaceOr if you are using 🤗 Pytorch Image Models (TIMM), you can simply start using it in a couple of lines:
import timm
model = timm.create_model(
"hf-hub:notmahi/dobb-e",
pretrained=True
)
import timm
model = timm.create_model("hf-hub:notmahi/dobb-e", pretrained=True)
Paper
On Bringing Robots Home
