Technical docs
Have your agent get the data
Read https://shop.build.ai/docs.md, show me the license, then download the starter set and show me one clip with its hand pose and captions.- A clip
- Three minutes of head-camera video of one person at work, with the head IMU recorded alongside. Mono clips are 180 s; stereo clips are 6,018 frames per eye, about 180.5 s.
- Layers
- Head pose, 3D hand pose, and text are computed per clip and delivered as separate files, so the video can be bought alone or with any of them.
- Mono or stereo
- Two devices. Their files differ, so most tables below have a column for each.
Get the data
The sample clips can be downloaded: sign up once, then fetch a list of links. A person does it on the shop page; an agent can do all of it from here.
- License
- The sample files offered for download on shop.build.ai are released by Build AI, Inc. under the Apache License, Version 2.0. What you may do: Use, copy, change and share the samples, for research or for commercial work, including training models. The license gives no right to the Build AI name or marks. What it asks: When you share copies, changed or not: include the license, keep the notices that come with the files, and mark the files you changed. No warranty: The samples are provided as they are, with no warranty of any kind, and Build AI is not liable for damage arising from their use. Full text: https://www.apache.org/licenses/LICENSE-2.0 (the summary as a page: https://shop.build.ai/download/terms). An agent shows this summary to its user and waits for a yes before signing up for them.
- Sign up
- POST https://shop.build.ai/api/shop/download with the JSON body below and the header
Origin: https://shop.build.ai. The answer holds a key. One sign-up, one key: keep it and reuse it. - Sets
starter(one bundle per camera with its annotations, about 2 GB),mono,stereo, one camera and kind of work such asstereo-packing_boxing, orall. GET https://shop.build.ai/api/shop/download lists every set with its clips, bytes and files.- Lists
- GET https://shop.build.ai/download/<key>/<set>.txt returns one link per line.
- Files
<camera>-<kind of work>-<NNNN>.video.tarholds one<clip id>.mp4per clip, about 1 GB a bundle. The.layers.tar.gzof the same name holds<clip id>/<layer>.jsonfor the same clips: the files described under Sample files. Names are unique, so one folder is enough. A few clips lack a layer.- Downloading
- Follow redirects. Downloads can be resumed. Keep to 4 at once; an answer of 429 means wait a few seconds and carry on.
curl -s https://shop.build.ai/api/shop/download \
-H 'Content-Type: application/json' -H 'Origin: https://shop.build.ai' \
-d '{"email": "<the user email>", "terms": true, "contact": "", "ms": 3000}'
# {"key": "…"} 422 names the fields to fix; 429: wait a minute and try again
# email: the user's own work or university address, one that reaches them: Build AI may write to it to hear what they
# think of the samples. Never a made-up, placeholder (example.com, example.edu) or no-reply address. An agent names the
# address to its user before sending it (see If you are an agent).
# Optional fields: "organisation", and "use", what the user is working on in their own words, up to 600 characters.
# terms: true once the user accepts the license.
# contact: always "". ms: milliseconds the user took to give these details, at least 2500; an agent sends 3000.
# To ask for a quote in the same call, add "quote": true, the selection and an optional note, for example
# "selection": {"camera": "stereo", "layers": ["head_pose", "hand_pose", "text_dense"], "volume": "100k"}, "notes": "…"
# camera: mono or stereo. layers: any of head_pose, hand_pose, text, text_dense. volume: 100k, 500k, 1m or 5m (hours).
# A quote without the samples: "samples": false, and terms is not needed. We reply by email; the key in that answer is not needed.
# A selection the price list does not know is answered 422 with the field selection.# About 2 GB into the current folder, 4 files at once; run it again to resume. curl --retry 8 -fsSL https://shop.build.ai/download/<key>/starter.txt | xargs -n 1 -P 4 curl --retry 8 -fL -C - -O # The same with wget: wget -c --retry-on-http-error=429 -i https://shop.build.ai/download/<key>/starter.txt for f in *.tar *.tar.gz; do tar -xf "$f"; done # the archives can then be deleted; unpacked, the set is as large again # Any other set: put its name in place of starter. The set all is terabytes in thousands of files: # read its bytes from GET https://shop.build.ai/api/shop/download and check the disk first.
draw_clip.py)# draw_clip.py: one frame of a sample clip with its hands and its text drawn on it.
# Needs ffmpeg and Pillow 10.1 or later. python draw_clip.py <folder> <clip id> out.png [seconds]
import json, os, subprocess, sys
from PIL import Image, ImageDraw
folder, clip, out = sys.argv[1:4]
load = lambda name: json.load(open(f"{folder}/{clip}/{name}", encoding="utf-8"))
stereo = os.path.exists(f"{folder}/{clip}/hands_2d_left.json")
hands = load("hands_2d_left.json" if stereo else "hands_2d.json") # stereo: the left eye
tasks, steps = load("tasks.json"), load("hand_captions.json")
fps, w, h = hands["fps"], hands["src"]["w"], hands["src"]["h"]
whole = lambda side, i: hands[side]["kp"][i] and all(hands[side]["kp"][i]) # null: the hand, or a joint, was not found
seen = lambda i: [side for side in ("left", "right") if whole(side, i)]
frames = range(hands["frames"])
good = [i for i in frames if len(seen(i)) == 2] or [i for i in frames if seen(i)] or [0] # both hands if any frame has them
i = good[len(good) // 2] if len(sys.argv) < 5 else max(0, min(round(float(sys.argv[4]) * fps), hands["frames"] - 1))
t = i / fps # frame i is at i / fps seconds
# Frame i of the video. A stereo video holds both eyes side by side: the crop keeps the left one.
seek = f"{max(0, (i - 0.5) / fps):.4f}"
subprocess.run(["ffmpeg", "-v", "error", "-y", "-ss", seek, "-i", f"{folder}/{clip}.mp4",
"-frames:v", "1", "-vf", f"crop={w}:{h}:0:0", out], check=True)
img = Image.new("RGB", (w, h + 180)) # the frame, with room for four lines under it
img.paste(Image.open(out).convert("RGB"))
draw = ImageDraw.Draw(img)
FINGERS = [[0, 1, 2, 3, 4], [0, 5, 6, 7, 8], [0, 9, 10, 11, 12], [0, 13, 14, 15, 16], [0, 17, 18, 19, 20]]
for side, colour in (("left", (80, 160, 255)), ("right", (255, 150, 70))):
if not whole(side, i):
continue
points = [(x * w, y * h) for x, y in hands[side]["kp"][i]] # fractions of the width and height
for finger in FINGERS: # wrist to fingertip
draw.line([points[j] for j in finger], fill=colour, width=4)
at = lambda spans: next((s for s in spans if s["start_s"] <= t < s["end_s"]), None) # a time can fall in a gap
task, left, right = at(tasks["subtasks"]), at(steps["left"]), at(steps["right"])
lines = [f"{tasks['task_name']} (frame {i}, {t:.1f} s)", f"Task: {task['name'] if task else '-'}",
f"Left hand: {left['text'] if left else '-'}", f"Right hand: {right['text'] if right else '-'}"]
for n, line in enumerate(lines):
draw.text((20, h + 10 + 40 * n), line, fill=(255, 255, 255), font_size=28)
img.save(out)
print(out, *lines, sep="\n")Cameras
Both devices are worn on the head and look down at the hands and the work.
| Mono | Stereo | |
|---|---|---|
| Cameras | 1, worn on the head | 2, worn on the head, about 70 mm apart |
| Resolution | 1920 × 1080 | 1920 × 1080 per eye |
| Frame rate | 30 fps | 33.3 fps |
| Video | HEVC, one file per clip | HEVC, one file per eye |
| Lens | Fisheye. About 150° horizontal, 88° vertical, 164° diagonal. | Fisheye. About 157° horizontal, 97° vertical. |
| Sync | n/a | Hardware-synchronised: frame N of the left eye and frame N of the right eye were exposed together. |
IMU
The inertial sensor sits in the device with the camera and is recorded for the whole clip.
| Mono | Stereo | |
|---|---|---|
| Rate | One sample per video frame (30 Hz) | 800 Hz |
| Gyroscope | 3 axes, rad/s | 3 axes, rad/s |
| Accelerometer | 3 axes, m/s², gravity included | 3 axes, m/s², gravity included |
| Clock | Each sample carries its time in microseconds. | Video and IMU share one device clock. The camera-to-IMU time offset is in the calibration file and is well under 1 ms. |
The samples on this site carry the IMU at 30 Hz for both devices.
Calibration
Stereo devices are calibrated one by one at the factory; mono uses one calibration for the device model.
| Mono | Stereo | |
|---|---|---|
| Intrinsics | Fisheye (Kannala-Brandt, 4 coefficients): camera matrix and distortion. Nominal for the device model, not measured per unit. | Per device, for each eye: the factory model (unified omnidirectional with radial-tangential distortion) and a fisheye fit (Kannala-Brandt, 4 coefficients), with the fit error. |
| Stereo | n/a | Rotation and translation from the left eye to the right eye. |
| Camera to IMU | Rotation, for each way the camera is mounted. | Full transform for each eye, in metres. |
fx 699.194 fy 699.604 cx 976.751 cy 565.791 k1 -0.0180332 k2 0.06173989 k3 -0.05266772 k4 0.01903308
Kannala-Brandt fit: fx 612.39 fy 610.69 cx 992.58 cy 553.68
k1 0.05787 k2 0.00387 k3 0.00277 k4 0.0000082
Factory model: xi 1.1143 fu 1294.76 fv 1291.47 cu 992.56 cv 552.81
k1 -0.05740 k2 0.07468 p1 0.00329 p2 0.0000488In the samples, each stereo clip's head_pose.json carries its own device's left camera under camera: both models above and T_cam_imu, the 4 × 4 transform between the camera and the IMU.
Head pose
The path of the camera through the clip: where the head is and where it points, frame by frame.
| Mono | Stereo | |
|---|---|---|
| Rate | One pose per video frame | One pose per video frame |
| Rotation | Quaternion [x, y, z, w] | Quaternion [x, y, z, w] |
| Translation | [x, y, z] | [x, y, z] in metres |
| Frame | Pose of the camera in the frame of the camera at the start of the clip (camera to that frame). Camera axes: x right, y down, z forward. | Pose of the left camera in a world frame level with gravity (camera to world): z up, origin at the left camera on the first frame, x along the camera's forward direction on that frame. |
| Scale | Estimated from one camera. Treat distances as approximate. | Metric, from the stereo baseline and the 800 Hz IMU. |
| Quality | n/a | Each track carries a health status with its reasons, coverage and the frames that were filled. |
Hand pose
Both hands of the wearer as 21 joints each, on every frame where they are in view.
- Joints
- 21 per hand.
- Joint order
- Wrist; thumb CMC, MCP, IP, tip; then index, middle, ring and little finger, each MCP, PIP, DIP, tip.
- 3D
- One position per joint, per hand, per video frame, in metres, in camera coordinates: x right, y down, z forward. Row i is video frame i.
- Mask
- In a delivery, a mask per joint and frame marks positions not to use: the hand was not found, or the joint falls behind the camera. Masked values are still finite numbers, so apply the mask first. The sample files have no mask: a missing hand or joint is
null. - Left and right
- The wearer's own left and right hands. Nothing is mirrored.
- Stereo
- One estimate per eye, each in that eye's camera coordinates.
- What it is
- An estimate fitted from the image, not a depth measurement. It can contain errors, most often when a hand is hidden.
- 2D
- The samples also carry the hands on the image: joints as fractions of the image width and height, with one confidence per hand per frame. A joint off the edge of the picture falls outside 0 to 1.
- On this site
- The viewer carries a hand across a short gap (up to half a second) when the hand stays nearby, drawn fainter. The files are not filled in: a frame without the hand stays
null.
Text
What is being done, written at three levels: the clip, its tasks, and each hand's steps.
- Clip
- A caption, the task name and its goal, the setting, and the tools, objects and materials in use.
- Tasks
- The clip cut into tasks, each with a name and its start and end in seconds, the main object and the state before and after.
- Steps
- What the left hand and the right hand each do, as short timed phrases, a second or a few apiece.
- Cycles
- Where the work repeats, each repetition with its start, end and outcome.
- Also
- Difficulty, and whether the work is repetitive.
- Times
- Seconds from the start of the clip.
- Two levels
- Captions + task segmentation is the clip level and the tasks. Dense captions adds each hand's steps.
Mono and stereo carry the same levels; a few field names differ between them.
Clip metadata
Facts about each clip for filtering and balancing a training set, as shown with the stereo samples.
- Where
- Country, the kind of venue and the scene.
- Task
- Name, job family, sector, skill group, difficulty and the actions seen.
- Who
- Gender and dominant hand of the wearer. Consent was collected from every wearer, and each clip says so. No names.
- Activity
- Seconds spent working, walking, idle and on a phone.
Sample files
The files behind each clip on this site, one JSON file per layer. Deliveries carry the same content in the layout further down.
tasks.json: the clip, its tasks, each hand's steps{
"task_name": "…", "task_goal": "…", "setting": "…",
"tools": ["…"], "objects": ["…"], "materials": ["…"],
"subtasks": [{ "name": "…", "start_s": 0.0, "end_s": 34.0, "main_object": "…",
"before": "…", "after": "…", "goal": "…", "work_area": "…", "posture": "…" }],
"steps": [{ "start_s": 0.0, "end_s": 0.7, "kind": "task", "left": "…" | null, "right": "…" | null }],
"cycles": [{ "start_s": 0.0, "end_s": 34.0, "outcome": "success" }],
"meta": { "participant": {…}, "where": {…}, "task": {…}, "activity": {…} } // stereo
}hand_captions.json: the steps as one lane per hand, for drawing on a timeline (a phrase repeated in a row is one span){
"left": [{ "start_s": 1.5, "end_s": 2.2, "text": "pick up a spray bottle from the conveyor belt" }],
"right": [{ "start_s": 0.0, "end_s": 0.7, "text": "…" }],
"tasks": [{ "start_s": 0.0, "end_s": 34.0, "name": "…" }]
}hands_3d.json: 21 joints per hand, per frame, in metres, in camera coordinates{
"fps": 30, "frames": 5400, "units": "m", "axes": "x-right y-down z-forward",
"left": [ null | [[x, y, z] × 21] ] × frames, // null: the hand was not found on that frame
"right": [ null | [[x, y, z] × 21] ] × frames // a single joint can be null too: check before use
// stereo adds "eye": "left": the positions are in the left camera's coordinates
}hands_2d.json (stereo: hands_2d_left.json and hands_2d_right.json, one per eye){
"fps": 30, "frames": 5400, "src": { "w": 1920, "h": 1080 },
"left": { "kp": [ null | [[x, y] × 21] ] × frames, "conf": [c × frames] }, // x, y as fractions of width and height
"right": { "kp": […], "conf": […] } // kp null and conf 0: the hand was not found on that frame
}head_pose.json: one pose per video frame{
"fps": 30, "frames": 5400, "units": "m", "axes": "x-right y-down z-forward",
"q": [[x, y, z, w] × frames], // rotation: camera coordinates to the clip's frame
"t": [[x, y, z] × frames] // the camera's position in the clip's frame
// stereo adds "world" (the frame, level with gravity) and "camera" (that device's calibration)
}imu.json: 30 samples a second{
"hz": 30,
"acc": [[x, y, z] × samples], // m/s², gravity included
"gyro": [[x, y, z] × samples] // rad/s
}Time: every file counts from the start of the clip. Row i of a per-frame file is video frame i, at i / fps seconds. Use the file's own fps, not the video container's: 30 for mono; 100/3 (33.33…) for stereo, 6,018 frames. IMU sample i is at i / 30 s on both; a stereo sample's IMU covers the first 180 s. Text times are seconds.
A span covers start_s up to, not including, end_s. Spans can leave gaps: a time with no step or no task has no text.
The clip-level text of a sample is task_name and task_goal in tasks.json (task_goal can be null); there is no separate caption field. "Captions" for a moment of the clip are the task covering it (subtasks) and each hand's step.
A step's kind is task, walking, idle or other. A cycle's outcome is success or failure.
Each stereo sample carries the left camera's calibration; the right camera's and the transform between the eyes come with a delivery.
The video of a stereo sample is one file with the two eyes side by side, 3840 × 1080: the left half is the left eye.
Getting the files: see Get the data. In a download, a clip's files sit under <clip id>/ in a layers bundle and its video in the video bundle of the same name.
Delivery
Clips arrive as one folder each, with a manifest of every clip at the root.
| Mono | Stereo | |
|---|---|---|
| Video | video.mp4 | left.mp4, right.mp4 |
| IMU | imu.txt | imu.bin |
| Calibration | camera.json | calibration.json |
| Hand pose | hand_pose.npz | hand_pose_left.npz, hand_pose_right.npz (one file per eye, both hands in each) |
| Head pose | camera_pose.npz | camera_pose.npz |
| Text | caption.json | caption.json |
| Metadata | metadata.json | metadata.json |
- Samples differ
- The samples on this site play as H.264 and carry the IMU at 30 Hz; a delivery is the camera's own HEVC, with the stereo IMU at 800 Hz and the stereo eyes as two files.
We have also delivered in MCAP and LeRobot layouts. Data is copied to your own cloud storage. For anything this page does not answer, ask us. We respond within 24 hours.