Abstract

Multi-stream robot manipulation policies achieve unparalleled sample efficiency and generalization by modeling actions relative to environmental reference frames. However, existing approaches typically assume these frames to be strictly exogenous. This causal assumption collapses in dynamic settings, such as when a single robot arm manipulates a moving object or when two arms coordinate, where each arm effectively becomes part of the dynamic environment of the other. We propose DynaMAC, a lightweight, policy-agnostic framework that resolves this causal limitation while preserving the sample efficiency, computational speed, and flexibility of multi-stream policies, DynaMAC treats the opposite arm as a dynamic task parameter, thereby providing a unified formulation for dynamic manipulation and bimanual coordination without requiring an explicit leader-follower relationship. To rigorously evaluate these capabilities, we introduce DynaBench, a novel benchmark for robot manipulation in dynamic environments. Across both dynamic environments and bimanual manipulation tasks, DynaMAC outperforms leading probabilistic and generative baselines by over 35 percentage points while requiring 20 times fewer samples. Crucially, DynaMAC generalizes zero-shot from static demonstrations to dynamic environments, substantially simplifying data collection and establishing an elegant bridge toward human-robot collaboration.

Overview

DynaMAC unifies manipulation in dynamic environments, bimanual coordination, and multi-agent cooperation under the umbrella of dynamic multi-strem learning.
  • Sample-efficient: only 5 demos needed.
  • Fast: fitting a policy in less than one minute.
  • Interpretable: easy to interpret, debug, and extend.
  • Probabilistic: enables effective inference-time steering, like
  • Zero-Shot Transfer: demonstrate in static environments, execute in dynamic ones.
  • Robust to Pertubations: dynamically re-coordinates if either agent is perturbed.
  • Human-Robot Collaboration through our reactive framework.
Overview

Manipulation in Dynamic Environments

Multi-Stream Policy Learning models the trajectory distribution from multiple perspectives and leverages explicit geometry, enabling efficient generalization across task instances.

But this approach requires task-parameters to be exogenous, i.e. not effected by the policy itself. Instead, only the effect of the task parameter on the robot is modeled. If an object is manipulated by the robot, this assumption is violated leading to policy collapse. To effectively act in dynamic environments, we therefore need to disentangle these two causal directions.

We detect kinematic links between robot and task objects using the precision of the policy streams. The streams corresponding to kinematically linked objects exhibit temporarilly very high precision. We temporarilly mask these streams to prevent policy collapse, enabling multi-stream learning with dynamic task parameters.

MultiStreamLearning KinematicAnalysis

Bimanual Manipulation

We reduce bimanual manipulation to dynamic multi-stream learning. Each arm is treated as a dynamic task parameter for the other arm. This allows us to model the interaction between the two arms without requiring an explicit leader-follower relationship. Explicit leader-follower relationships require priviliged information and break if the dominant arm changes during a long-horizon task. Our approach dynamically assings temporary leader-follower relationships based on the demonstrated behavior. It also allows for dynamically re-coordination if either arm is perturbed.

Image 1 Image 2

Human-Robot Interaction

DynaMAC's ability to react dynmically to inconsistent behavior of a collaborator makes it suitable for real-world applications where human partners may not always behave as expected.

Image 1

DynaBench

We introduce DynaBench, a novel benchmark for robot manipulation in dynamic environments. Built on RLBench, DynaBench provides a diverse set of task, automatic demonstration generation, and compatability with other RLBench extensions.
Image 1 Image 2

Video

Code

For academic usage a software implementation of this project based on PyTorch will soon be realesed in our GitHub repository and is released under the GPLv3 license. For any commercial purpose, please contact the authors.

Model downloads will soon be available below.

Publications

If you find our work useful, please consider citing our paper:

Jan Ole von Hartz, Abhinav Valada, Joschka Boedecker

One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments
Under review for publication, 2026.
(PDF) (BibTeX)

Authors

Jan Ole von Hartz

Jan Ole von Hartz

University of Freiburg

Abhinav Valada

Abhinav Valada

University of Freiburg

Joschka Boedecker

Joschka Boedecker

University of Freiburg