DAVATaR: Depth-Aware Visually Augmented Tactile Learning Across the Air-Water Interface

Abstract

Many everyday and field robotics tasks require manipulating objects that are submerged in water — from bathing assistance and dish washing to environmental water sampling. This setting, submerged manipulation, is uniquely hard: the robot operates from air while its end-effector reaches into water, so the wrist camera observes through the air–water interface, where refraction, reflection, and foam distort visual cues, and conventional tactile sensors place electronics where water makes them unreliable.

We introduce DAVATaR, a depth-aware visually augmented tactile robot learning framework that addresses both sides of the problem: fluidically innervated multi-modal tactile sensing that keeps electronics out of the water while providing contact-force and immersion-depth signals, and submerged-domain visual augmentation that builds robustness to water-induced artifacts. A multimodal policy fuses both streams to predict operational-space control actions. Across diverse submerged tasks, DAVATaR is substantially more robust than vision-only baselines, especially under adversarial visual conditions.

Tasks

Results

Each task is evaluated in a normal setting and in adversarial visual settings that increase foam, bubbles or turbidity, water-depth variation, and reflection. Average — the mean success rate across both settings — is the main result.

Across all evaluated tasks, DAVATaR achieves the highest mean success rate.

• Highest overall. DAVATaR reaches 71% mean success, versus 41% without tactile and 55% without augmentation. In the normal setting several policies make progress; the gap becomes clearest once the visual stream is corrupted by foam, glare, and depth changes.
• Tactile & depth recover hidden physical state. Contact-force and immersion-depth sensing supply what vision cannot reliably infer — how deep the syringe is under foam, whether the fingers reached the tank bottom, whether the object is actually grasped — lifting adversarial success from 25% to 61%.
• Tactile alone is not enough when the view is distorted. Reflection can hide visual evidence and refraction changes apparent geometry, so without submerged-domain visual augmentation success reaches only 37% under the same adversarial conditions.