PUPPET: Presence, Embodiment, and Teleoperation

Abstract

Remote access to robotics hardware does not by itself guarantee usable experimental data. A teleoperated trial can complete the task and still produce an incomplete or inconsistent record. PUPPET aims to solve this problem by presenting a cyber-physical framework that opens a physical robotics laboratory through a continuously synchronized VR replica of the robot, surrounding environment, and task objects. In addition, PUPPET integrates a trial validation pipeline built into the experimental workflow. We evaluated PUPPET in a within-subject study with 30 participants, who performed the same table-to-shelf object-transfer task in person, through 2D teleoperation, and through PUPPET. VR reduced post-learning completion time by 27.7% over 2D (68.7 s vs. 95.0 s), statistically indistinguishable from in-person execution (72.1 s). It also produced more validation-ready (85.6% vs. 66.0%) and intervention-free (89.7% vs. 67.2%) trials. With PUPPET we show that modality shapes the remote task performance and the reusability of the resulting experimental record.

Why it matters

Illustration: a user in a VR headset works inside a virtual replica of a lab, while the real TIAGo robot performs the same action in the physical lab — PUPPET gives remote users an immersive virtual representation of the physical robotics laboratory, so they interact with the real robot through the replicated VR environment.

Real robots and laboratory setups are expensive, geographically fixed, and hard to replicate, so running and repeating real-robot experiments still largely depends on being in the lab. Remote laboratories, digital twins, and VR teleoperation have lowered that barrier, but they are mostly judged on task performance, interface usability, or control quality.

PUPPET starts from a different observation: successful teleoperation is not the same as successful experimentation. A remotely controlled trial can finish the task and still leave behind data that is incomplete, inconsistent, or unusable. In many experiments the state of the workspace is part of the data too, such as where the object was, how it moved, and what surrounded it.

PUPPET gives remote researchers access to a real lab through a synchronized VR environment that holds the robot, the task objects, and the workspace in one virtual space. Data collection and validation run as part of the workflow, so every trial is judged both on whether the task succeeded and on whether its record is complete, consistent, and reusable.

System design

PUPPET connects a physical laboratory and its virtual replica through three layers: interaction and control, synchronization, and integrated data collection and validation. A remote user enters the virtual lab, controls the physical TIAGo through its virtual counterpart, and performs an object-transfer task in the real workspace.

PUPPET architecture: the physical laboratory with OptiTrack, server, WiFi 6, VR interface and TIAGo; the interaction and control loop; the Unity virtual laboratory; and the data collection and validation layer — The physical and virtual laboratories are connected through interaction and control, synchronization, and an integrated data collection and validation layer. Letters correspond to the components below; numbered arrows show the main control and data flows.

AVR interfaceMeta Quest 3 with haptic controller straps: 6-DOF head tracking, hand tracking, joystick input, and vibro-tactile feedback.
BServerRenders the virtual lab, hosts the robot control interfaces, and closes the control loop at 100 Hz.
CWiFi 6A commercial access point carries commands to the robot in the lab.
DTIAGo robotDifferential-drive base, 7-DOF torque-controlled arm, head-mounted RGB-D camera, and an onboard ROS control stack.
EOptiTrackMotion capture at 120 Hz with sub-millimeter accuracy tracks the robot base and the task objects.
FVirtual laboratoryUnity and the Meta VR SDK reproduce the full lab, including walls, tables, shelves, TIAGo, and the task object.
GSynchronizationThe NatNet ROS driver and ROS–TCP connector place every tracked object in the virtual scene in real time.
HData collection & validationROS bags, multi-view video, metadata, and motion capture are stored per trial and validated automatically.

30.6 ms

command-to-motion

Median VR command-to-motion feedback latency.

7.15 ms

physical-to-virtual

Median delay from physical motion to the virtual update.

100 Hz

control loop

Rate at which the server closes the control loop.

120 Hz

motion capture

OptiTrack 6-DOF poses of the robot and objects, sub-millimeter.

Embodied control

Inside the virtual laboratory, control is embodied: the user's arm becomes the TIAGo arm and the user's head becomes the TIAGo head. The whole experimental workspace serves as both the teleoperation interface and a live view of robot state, sensor data, and task progress.

01

Head → TIAGo head

Headset orientation drives the robot's head, so the viewpoint follows the user's gaze. Users can look one way while reaching another.

02

Hand → arm

The tracked hand pose drives the end-effector directly, with motion planning keeping the arm inside its kinematic limits. The gripper takes discrete open and close commands.

03

Joystick → base

The controller joystick drives TIAGo's mobile base, so one task covers navigation, manipulation, and the full teleoperation pipeline.

Data validation pipeline

The streams that keep the virtual lab synchronized are also written to disk as the experimental record. Each trial stores a ROS bag with joint states, commanded velocities, base odometry, gripper state, end-effector pose, and the OptiTrack poses of the robot and task object, together with trial metadata and multi-view video at 30 fps. Validation runs automatically when a trial closes and produces a per-trial report.

No established standard exists for validating teleoperated robotics data, so PUPPET uses four quality dimensions drawn from prior work on teleoperation and demonstration quality: completeness, temporal continuity, signal plausibility, and execution consistency. They are implemented as six rule families.

Rule	Quality dimension	Check	Criteria	Outcome
R1a	Completeness	Record completeness	Required metadata, files, topics, telemetry columns, and minimum sample count must be present.	Fail
R1b	Completeness	Validation metadata	Missing manual- or video-validation information is flagged for inspection.	Review
R2	Temporal continuity	Telemetry continuity	Required streams > 50% missing, end-effector pose > 90% missing, or largest timestamp gap > 1 s.	Review
R3	Signal plausibility	Object tracking	Successful trials whose object path is < 0.02 m, or movement < 0.005 m on all axes.	Review
R4	Signal plausibility	Robot motion	Expected gripper, arm, torso, or end-effector motion is too small, or joint jerk exceeds 5 × 10⁴.	Review
R5	Execution integrity & consistency	Technician intervention	More than two minor interventions, any major intervention in T2–T3, or inconsistent intervention metadata.	Review
R6	Execution integrity & consistency	Cross-trial consistency	Repeated-trial variation in completion time, path length, gripper chatter, or joint jerk exceeds predefined limits.	Review

Validation rules implementing the four data-quality dimensions used in the study.

Pass no quality issues detected Review complete, but anomalous Fail record is unrecoverable

User study

30 participants from STEM backgrounds did the same mobile-manipulation task under three conditions, in counterbalanced order (balanced Latin square). They used TIAGo to pick up a plastic cylinder from a table and place it on a shelf, starting 1.7 m away. Each condition had three repetitions: T1 served as the learning trial, and T2–T3 were the post-learning trials used for the primary analysis.

Baseline

In-person

Participants operated TIAGo while physically in the lab, with direct sight of the robot. Keyboard for the arm, joystick for the base.

Conventional remote

2D

Remote operation using three camera feeds on a single monitor. Keyboard for the arm, joystick for the base.

PUPPET

VR

Remote operation through PUPPET's synchronized virtual lab, using a Meta Quest 3 headset and haptic controller straps.

The robot, task, and ROS control pipeline were the same in every condition, so the comparison isolates the access modality itself. The study tested three hypotheses:

H1
Task effectiveness. PUPPET supports more effective remote task execution than 2D teleoperation: comparable completion and lower post-learning completion time.
H2
Reliability. PUPPET yields more validation-ready trial records and fewer technician interventions, with consistency maintained across repeats.
H3
User experience. Users report lower workload, higher usability, and higher task confidence in VR than in 2D.

Results

Every trial in every condition completed the task, so task success alone could not tell the three access modes apart. The differences showed up in speed, in how much assistance was needed, and in the quality of the record each trial left behind.

27.7%

faster than 2D

Post-learning completion time was 68.7 s in VR vs. 95.0 s in 2D (p = .011), on par with in-person (72.1 s).

85.6%

validation-ready trials

VR trials passed every automated check far more often than 2D trials (66.0%, p = .012).

89.7%

intervention-free trials

VR trials needed technician help far less often than 2D (67.2%, p = .007), close to in-person (93.1%).

H1Task effectiveness

Bar chart of post-learning completion time: in-person 72.1 s, VR 68.7 s, 2D 95.0 s; VR vs 2D p = .011, in-person vs VR p = 1.000 — Mean post-learning (T2–T3) completion time per condition.

After the learning trial, participants finished the task in 68.66 s through PUPPET, 94.96 s through 2D teleoperation, and 72.10 s in person: a 27.7% reduction from 2D to VR. Twenty of the 30 participants were faster in VR than in 2D, and the difference was significant (W = 109.00, p = .011, r = −.531). Completion time also differed across the three conditions overall (χ²(2) = 6.10, p = .047), and condition order had no detectable effect (p = .639).

Learning looked different too. On T1, participants were already faster through PUPPET (147.7 s) than through 2D (181.2 s) or in person (174.3 s). All conditions improved sharply after that, but VR times were stable between T2 and T3 (71.7 s → 65.6 s) while 2D was still improving (104.2 s → 85.7 s). H1 was supported.

H2Reliability of remote experimentation

Three charts: (a) validation outcomes, Pass 79% in-person, 66% 2D, 86% VR; (b) data issues by condition; (c) intervention-free trials 93.1% in-person, 67.2% 2D, 89.7% VR — (a) Validation outcomes, (b) system and data issues (non-exclusive; top categories only), and (c) intervention-free execution trials.

85.6% of VR trials were validation-ready against 66.0% of 2D trials, a significant difference at the participant level (W = 21.00, p = .012, r = .691). No trial in any condition was assigned Fail. Every flagged trial landed in Review, so the problems we saw were partial or implausible data, not unrecoverable loss.

The issue profile shows why trials were flagged. Near-flat object tracking affected all conditions at similar rates (6.9% in-person, 5.6% 2D, 6.7% VR), which points to the physical sensing rather than the access mode. Large timestamp gaps appeared only in 2D (3.3%). Missing object-state data affected 10.3% of in-person and 6.7% of 2D trials and never occurred in VR. Our explanation is that the tracked state is always visible inside the virtual lab: when an object freezes or disappears, users notice it during the trial. 89.7% of VR trials also ran without technician help, against 67.2% in 2D and 93.1% in person (W = 24.50, p = .007, r = −.680).

Top-down object trajectories for execution trials; the VR median path is nearly straight while 2D and in-person median paths bow out by about one meter — Task-aligned object trajectories during execution trials (T2–T3). Light lines are individual executions; bold lines are each condition's median.

Object trajectories show that embodied control changed how people did the task. The VR median path was clearly more direct: median peak lateral deviation was 0.713 m in VR, 0.925 m in 2D, and 1.028 m in person, although the overall condition effect did not reach significance (χ²(2) = 5.15, p = .076).

That freedom did not cost consistency. T2–T3 repeatability did not differ between VR and 2D for RMS distance (p = .956), dynamic-time-warping distance (p = .956), or path-length difference (p = .312). These comparisons show only that no difference was detected. H2 was supported.

H3User experience

Grouped bar chart of mean scores: workload 1.96 in-person, 2.81 2D, 2.26 VR; usability 3.62, 2.43, 3.78; task confidence 4.43, 3.50, 4.43 — Workload, usability, and task confidence on a common 0–5 scale. SUS is normalized for display; statistics use standard 0–100 SUS scores.

VR showed lower workload than 2D (2.26 vs. 2.81), higher usability (3.78 vs. 2.43), and higher task confidence (4.43 vs. 3.50). In-person scores were 1.96, 3.62, and 4.43. 18 of 30 participants reported lower NASA-TLX workload in VR (W = 73.00, p = .009, r = −.584), and 21 of 30 reported higher confidence (W = 31.50, p < .001, r = .806).

VR produced lower workload than 2D even though it is the more complex system, and task confidence in VR matched in-person operation. The cost of remote operation seems to come from having to rebuild spatial relationships from a few camera views, not from system complexity.

Participants also felt in control of the real robot. Sense of agency in VR was well above the neutral midpoint of 3 (W = 0.00, p < .001, r = 1.00). H3 was supported.

Agency measure (VR, 1–5)	Mean	SD	Median
Perceived control over the robot	4.33	0.55	4.00
Robot motion reflected user actions	4.27	0.69	4.00
Robot responded as expected	4.33	0.61	4.00
Overall agency	4.31	0.53	4.33

What we learned

Takeaway: a remotely controlled trial can succeed as a task and still fail as a recorded experiment. Task success did not separate the three access modes, but the experimental record did. Through PUPPET, trials were validation-ready and intervention-free far more often than through 2D teleoperation, users settled into the interface faster, and the virtual lab worked as a continuous check on the data being recorded. Data quality deserves to be treated as an outcome measure in its own right.

Limitations and future work

We compared complete interaction methods, so the effects of spatial visualization and embodied control cannot yet be separated. The study used one robot, one structured table-to-shelf task, and one lab. The virtual lab is modeled by hand and relies on external motion capture, which made marker loss the main source of VR data issues (6.7% tracking anomalies). Next steps are markerless environment synchronization, representing people inside the virtual lab, and validating PUPPET across tasks, platforms, and distributed sites.

PUPPET: Physical-Unified Platform for Presence, Embodiment, and Teleoperation