Vision Cameras

Vision Cameras allow you to capture RGB(A), BGR, Depth or Point Cloud frames. These frames may be processed locally by a script, saved to the filesystem, passed to ONNX Runtime for inference or published over ROS.

Creating a Vision Camera

Before you can capture any frames, you must create a VisionCamera instance:

const camera = VisionCamera.create(world, {
    width: 640,
    height: 480,
    format: VisionBufferFormat.RGBA8,
    convention: VisionCameraConvention.ROS
});

If you intend to capture many frames, you should persist the vision camera instance rather than repeatedly creating it.

Once you no longer need the vision camera, you can dispose it:

camera.dispose();

Vision camera instances are automatically disposed when the simulation is reset.

Options

There are various options that can be passed when creating the vision camera:

• Width: The width for the frames (in pixels). Defaults to 1280px.
• Height: The height for the frames (in pixels). Defaults to 720px.
• Intrinsics: The camera intrinsics. Defaults to the standard centered 60° vertical field of view.
• Near: The near clipping distance (in meters).
• Far: The far clipping distance (in meters).
• Tonemapping: The tone mapping method.
• Format: The format for the frame. Defaults to RGBA.
• Convention: The pose convention. Defaults to World.

Intrinsics

The default intrinsics use a centered 60° vertical field of view:

const camera = VisionCamera.create(world, {
    intrinsics: CameraIntrinsics.fromVerticalFov(Util.radians(60))
});

You can instead provide a principal point (cx, cy) and focal lengths (fx, fy):

const camera = VisionCamera.create(world, {
    width: 1920,
    height: 1080,
    intrinsics: CameraIntrinsics.fromFocalLength(960, 540, 935.3074, 935.3074);
});

Vision cameras use a pinhole perspective projection and do not simulate lens distortion.

Format

The frame-buffer format determines the layout and interpretation of VisionFrame.view.

Note that depth is the point’s z-coordinate in camera space.

Point clouds use optical coordinates, where the points are in the local space of the vision camera. The native point-cloud buffer contains one point for every image pixel. Points without a valid depth contain NaN coordinates.

When point clouds are encoded as PCD, invalid points are removed. The remaining points are stored in camera-local optical space and the PCD VIEWPOINT is the identity transform.

Convention

The convention determines how the position and rotation supplied to the capture functions are interpreted.

For example, use the Camera convention when capturing from the perspective of the scene camera, which looks down the negative z-axis:

const camera = VisionCamera.create(world, {
    convention: VisionCameraConvention.Camera
});

const frame = await camera.requestSnapshotAsync(
    world.camera.worldPosition,
    world.camera.worldRotation
);

Capturing Frames

There are multiple ways to capture a frame once you have created a vision camera. In all cases, you must pass the position and rotation from which you want to capture the frame.

Request Frame

The most efficient method for capturing frames is to use the requestFrame() function. This function will capture the most recently rendered frame. This is highly efficient, since the frame has already been rendered and so calling the function will not wait for a frame to be rendered. Its intended use is for capturing frames at high frame rates.

camera.requestFrame(worldPosition, worldRotation, (frame: VisionFrame) => {
    // The simulation time at which the frame was captured.
    const time = frame.time;

    // View into native (WASM) memory!
    // Very fast, but can only be accessed inside this callback function.
    const data = frame.view;

    // Copy to a JS-buffer.
    // The copied buffer can be accessed outside of this callback function.
    const copy = frame.copy();
});

The requestFrame() function returns null when the vision camera is still processing a previous request. In such cases, the passed callback function won’t be called. For continuous capture, it’s fine to skip that frame and try again in the next timestep.

Request Snapshot

In some cases, you want to ensure that the obtained frame was captured after you requested it. For this, you can use the requestSnapshot() function. This function will return immediately, and the passed callback will be executed when the frame is ready. Note that this method is less efficient than requestFrame(), since the simulation cannot progress to the next timestep until the frame has been rendered and the callback function has finished executing.

camera.requestSnapshot(worldPosition, worldRotation, (frame: VisionFrame) => {
    // ...
});

Note that this method must not be called from standalone scripts, since waiting for the result will cause the simulation to deadlock and hang/crash unrecoverably. It is safe to call from scripted components and tools. For standalone scripts, the asynchronous method should be used.

The requestSnapshot function also returns null when the vision camera is still processing a previous request.

Asynchronous Snapshots

The asynchronous snapshot method is intended to be used by standalone scripts. It is highly inefficient, since it waits until the next frame has been rendered, stalling the simulation.

const frame = await camera.requestSnapshotAsync(worldPosition, worldRotation);
await frame.save();

The returned frame data remains valid after the promise resolves. Asynchronous snapshots are intended for occasional captures rather than high frame rate streaming.

Frame Data

Every VisionFrame provides:

• Width: Frame width in pixels.
• Height: Frame height in pixels.
• Encoding: The frame-buffer format.
• Time: The simulation time at which the frame was captured.
• View: A view into the native WASM frame data.

For example, a 32-bit depth frame can be viewed as floating-point values:

const depth = new Float32Array(frame.view.buffer, frame.view.byteOffset, frame.width * frame.height);
const firstDepthMetres = depth[0];

An XYZ point cloud contains three floating-point values per pixel:

const points = new Float32Array(frame.view.buffer, frame.view.byteOffset, frame.width * frame.height * 3);

const x = points[0];
const y = points[1];
const z = points[2];

Always copy transient frame data before accessing it from outside of the callback function:

camera.requestFrame(position, rotation, (frame: VisionFrame) => {
    processLater(frame.copy());
});

Saving Frames

The VisionFrame.save() function selects a file format based on the frame encoding:

• Point cloud frames are saved as binary PCD files.
• RGB(A) and BGR are saved as BMP files.
• Depth frames are saved as greyscale BMP files.

When using ProtoTwin Connect, you can write the frame directly to the local filesystem from inside the callback function.

camera.requestFrame(position, rotation, (frame) => {
    FileSystem.writeFile("C:/dir/capture.bmp", frame.bmp());
});

You can also save to the WebP format:

camera.requestFrame(position, rotation, (frame) => {
    const quality = 0.9;
    void frame.webp(quality).then((data) => {
        FileSystem.writeFile("C:/dir/capture.webp", data);
    });
});

For saving depth images, specify the distance that maps to white:

camera.requestFrame(position, rotation, (frame) => {
    FileSystem.writeFile("C:/dir/depth.bmp", frame.bmp(20));
});

ProtoTwin Simulate cannot directly access the local filesystem. Instead, you must defer to the browser’s API for saving the file. This will trigger a dialog asking you to save the file to a location on your computer.

camera.requestFrame(position, rotation, (frame) => {
    void frame.save();
});

For this reason, it is recommended that you use ProtoTwin Connect if you intend to continuously save frames to disk.

Coordinate Conversion

Vision cameras provide utility functions for converting between image and world coordinates.

Convert a screen coordinate and forward-axis depth into a world-space point:

const worldPoint = camera.screenToWorldPoint(screenPoint, depth, cameraWorldPosition, cameraWorldRotation);

Create a world-space ray through a screen coordinate:

const ray = camera.screenToWorldRay(screenPoint, cameraWorldPosition, cameraWorldRotation);

Project a world-space point into the camera image:

const screenPoint = camera.worldToScreenPoint(worldPoint, cameraWorldPosition, cameraWorldRotation); // (px, py, depth)

These functions all use the camera’s configured intrinsics and pose convention.