ELLIS header
University of Stuttgart Logo
Max Planck Institute for Intelligent Systems Logo

A Soft Thumb-sized Vision-based Sensor with Accurate All-round Force Perception

Huanbo Sun, Katherine J. Kuchenbecker, Georg Martius

Nature Machine Intelligence, 4, 2022.


Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer vision techniques; however, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized three-dimensional haptic sensor named Insight, which continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-moulded on a stiff frame to guarantee sensitivity, robustness and soft contact. Furthermore, Insight uniquely combines photometric stereo and structured light using a collimator to detect the three-dimensional deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of three-dimensional contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, a force magnitude accuracy of around 0.03 N and a force direction accuracy of around five degrees over a range of 0.03–2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts.



@article{sun22_NMI, title = {A Soft Thumb-sized Vision-based Sensor with Accurate All-round Force Perception}, author = {Sun, Huanbo and Kuchenbecker, Katherine J. and Martius, Georg}, journal = {Nature Machine Intelligence}, volume = {4}, organization = {Max Planck Institute for Intelligent Systems}, year = {2022}, doi = {10.1038/s42256-021-00439-3} }