← Back to the top

HOW TO USE

How to use

Placing, moving and touching work the same on every device. Only how you point and how you confirm differ.

On this page

  1. Six steps
  2. A robot with arms can do a little more
  3. Record, then replay in the same place
  4. How to start on your device
  5. Show the AR view on a PC or a big screen
  6. Serve the overlaid picture as a camera of its own
  7. Make a recording from USD or from a ROS 2 log (MCAP)

1. Six steps
A blue ring on the floor
A blue ring is an action on the floor

Where to place, where to move. It appears while you point at the floor.

An orange ring at a robot’s feet
An orange ring is an action on a robot

It appears at the feet of the selected robot. A dotted ring is its destination.

  1. 1Point at the floor

    Point at where you want to place a robot, and a blue ring appears on the floor. It sits on the real floor even where there are steps or a slope.

  2. 2Choose a model

    When you confirm, the menu “Place here” appears. Choose “Place” and this list appears. You can choose from seven models: humanoid, quadruped, arm and AMR.

  3. 3Place it

    It stands there at real size. You can step close and look up, or walk around it.

  4. 4Point at where to go

    Select a robot you placed and an orange ring appears at its feet. Point at the floor and a dotted ring shows the destination. Point within 8 m of the robot.

  5. 5Move it

    When you confirm, it turns that way and walks. A robot with legs moves its legs to match its stride.

  6. 6Operate or remove it

    Select a robot and a menu appears. “Operate” (robots with arms only) moves the arms, “Re-place” puts it somewhere else, and “Delete” removes it. Point at ⟲ ⟳ at its feet and confirm to turn it on the spot. You can place as many as you like.

2. A robot with arms can do a little more
Operating an arm
Hold the hand and move it

With a VR headset, grip and move your hand and the robot’s hand follows. Move your hand 10 cm and its hand moves 10 cm.

Raising and lowering the torso
Extend and retract the torso

A robot with a lifting torso (TIAGo++) changes its height with the up and down arrows that appear overhead.

3. Record, then replay in the same place
  • In the AR menu “Record”, put two marks on the floor and choose “Save recording”. The robots you placed and the paths you moved them along are saved to the server under your account.
  • Saved recordings are listed under “Recordings” on the app screen.
  • Choose a recording and start AR, put the marks back at the same two places, then choose “Replay” → “Replay recording” in the menu. You see the same movement in the same place.
  • Recordings are saved to the server under your account, so the same recordings appear on any device you sign in on. If the connection is lost while saving, the recording waits on the device with a red “Not saved to the server” mark and is sent once the connection is back.
  • To hand one to somebody else, write it out as .json with “Save as a file” and they read it in with “Open a file”.
  • If you set an MQTT broker in the settings, the devices in a recording can exchange messages over MQTT.

4. How to start on your device

VR headset

A WebXR-capable model (Meta Quest 3 / 3S and others). Use it indoors.

  1. On a phone or PC, sign in with Google first
  2. In the headset’s browser, open physicalsandbox.com and press “VR headset sign-in” at the top right (a six-digit code appears)
  3. On the phone or PC, open “Connect a VR headset” from the icon at the top right of the app, enter the six digits shown in the headset, and press “Allow”
  4. Press “Try it on site (AR)”
  5. Point with the right controller and confirm with the trigger
  6. To close a menu or drop a selection, close your hand into a fist (the grip does the same)
  7. To end AR, point at the floor, confirm, and choose “Exit AR” in the menu

Without controllers, you can still do it all with your hands alone (pinch = confirm, fist = cancel).
Place, move, arms, torso: all

Android

An ARCore-capable device and Chrome. It works outdoors too.

  1. Open physicalsandbox.com in Chrome and sign in with Google
  2. Press “Try it on site (AR)”
  3. Point the device at the floor and a blue ring appears on the floor at the centre of the screen
  4. Confirm with “Confirm” at the bottom of the screen
  5. Select a robot with arms and press “Operate”. Sticks appear at the bottom of the screen. When you finish moving it, press “End”

Tapping the screen directly does not confirm. This is so that a touch alone does not place a robot.
Place, move, arms, torso: all

iOS

Safari does not support WebXR, so you need WebXR Viewer (free).

  1. Install WebXR Viewer from the App Store
  2. Open physicalsandbox.com in that app and sign in with Google
  3. Press “Try it on site (AR)”
  4. Point the device at the floor and a blue ring appears on the floor at the centre of the screen
  5. Tap the screen to confirm at the blue ring. In menus, tap an item directly
  6. Select a robot with arms and press “Operate”. Sticks appear at the bottom of the screen (you can move both at once). When you finish moving it, press “End”

Place, move, arms, torso: all

5. Show the AR view on a PC or a big screen

To show it to people around you or to record it, use the device’s own feature. What you see is shown as it is, and Physical Sandbox needs no setting.

VR headset

Meta Quest’s “Cast”. Connect the headset and the PC to the same Wi-Fi.

  1. On the PC, open oculus.com/casting in Chrome or Edge and sign in with the same Meta account as the headset
  2. In the headset, open the menu and choose Camera → Cast
  3. Choose Computer as where to show it

You can also show it in the Meta Horizon app on a phone.

iOS

With a Mac, you can show it in QuickTime Player and record it as it is.

  1. Connect the iPhone or iPad to the Mac with a cable
  2. On the Mac, open QuickTime Player and choose File → New Movie Recording
  3. From the ∨ next to the record button, choose the connected device

Without a cable, use Screen Mirroring in Control Center to show it on a Mac or an Apple TV.

Android

To a TV or a big screen, use “Cast screen”. The name differs by model.

  1. Swipe down from the top of the screen to open Quick Settings
  2. Choose Cast screen (also called Screen Cast, Smart View and so on)
  3. Choose where to show it (Chromecast, Google TV and so on)

To show it on a PC, there is a free tool, scrcpy, that works over USB.

6. Serve the overlaid picture as a camera of its own

The picture on the PC's "overlay on a video" screen — the camera's picture with the robots over it — can be served the way a network camera serves one. A Node-RED dashboard, a recorder, a robot-learning toolkit (LeRobot, say) reads it as an ordinary camera. The picture stays on your computer and never reaches Physical Sandbox's servers.

What you need

A relay that takes the picture, on the same computer the page is open on.

  1. A PC and a webcam
  2. Chrome or Edge
  3. Node-RED, running on that PC (http://localhost:1880)
  4. The ps camera node for Node-RED (nodes/node-red-contrib-ps-camera in the open build)

Without Node-RED, run tools/camera_relay.py from the open build instead (http://localhost:8790).

Steps

It is served only while switched on, and the screen says "Serving" throughout.

  1. Put one ps camera node in a Node-RED flow, name the camera (cam-1), and deploy
  2. "Try it on site (AR)" → "Overlay on a video on this PC" → "Use a camera"
  3. Match the grid to the desk or the floor, confirm, and place a robot
  4. Press Serve this picture in the bar at the top
  5. If the browser asks whether this site may connect to devices on your local network, allow it

Only the camera's picture and the robots are in it: no grid, no menus, no labels.

Reading it

Anything that can open a camera takes the address.

  1. As it comes: http://localhost:1880/ps-camera/cam-1.mjpg
  2. One picture: http://localhost:1880/ps-camera/cam-1.jpg
  3. Inside Node-RED, each picture (a JPEG) comes out of the node

It can be read from the same computer only. A camera that has sent nothing for five seconds is not served. Not tried on Safari.

7. Make a recording from USD or from a ROS 2 log (MCAP)

A robot's motion made in Blender or elsewhere (USD), or a drive logged with ROS 2 (MCAP), is turned into a recording (.json) first and then opened with "Open a file". Physical Sandbox opens neither USD nor MCAP as it is. The turning is done by a small tool that ships with the open build, on your own computer: the scene never reaches Physical Sandbox's servers.

What you need

The converter under tools/ of the open build (physical-sandbox), run with the PC's Python.

  1. The open build's repository (GitHub) on the PC; run tools/dev-serve.sh once (it makes the robot data)
  2. Python 3 and, for USD, pip install usd-core; for MCAP, pip install mcap mcap-ros2-support
  3. Blender 5.3 or later if you use it (it reads a Gaussian splat directly). Steps in the open build's examples/usd/blender/README.md

The converter translates: the two floor markers become the origin and +x, the robot's joints become angles, time becomes 30 samples a second. Export only what should go and no narrowing option is needed.

USD (Blender and others)

The scene needs the two markers and the robot's model name.

  1. In Blender, put empties Marker1 and Marker2 under an empty named PhysicalSandbox, where the two floor markers will be
  2. Stand the robot with the open build's examples/usd/blender/ps_robot_import.py (it sets the model name ps_model). Turn the joint empties and keyframe them
  3. File → Export → USD with Animation and Custom Properties on, Selection Only
  4. python3 tools/usd_to_record.py scene.usda --out scene.recording.json --site desk
  5. In Physical Sandbox, "Open a file" → scene.recording.json, then "Select" → "Try it on site (AR)" → place the two markers → "Replay"

Units and the up axis are read from the USD. For a whole stage from another tool, point at the robot and the markers with --model /World/arm=so101 and --marker1 x,y,z --marker2 x,y,z.

A ROS 2 log (MCAP)

A bag holds everything that was published, so you say which topic is which robot.

  1. A rosbag2 in sqlite3 form is converted first: ros2 bag convert
  2. python3 tools/mcap_to_record.py run.mcap --out run.recording.json --site yard --device scout=scout,odom=/odom --device arm=so101,joints=/joint_states
  3. The two markers, in the map frame: --marker1 x,y,z --marker2 x,y,z. Without them the origin and +x stand in
  4. "Open a file" → run.recording.json

Read: odometry (nav_msgs/Odometry), one parent–child pair on /tf, joint states (sensor_msgs/JointState). No ROS needed.

Try it