Scaling robotics datasets with video encoding

Over the past few years, text and image-based models have seen dramatic performance improvements, primarily on account of scaling up model weights and dataset sizes. While the web provides an intensive database of text and pictures for LLMs and image generation models, robotics lacks such an enormous and diverse qualitative data source and efficient data formats. Despite efforts like Open X, we’re still removed from achieving the size and variety seen with Large Language Models. Moreover, we lack the needed tools for this endeavor, akin to dataset formats which are lightweight, fast to load from, easy to share and visualize online. This gap is what 🤗 LeRobot goals to handle.

What’s a dataset in robotics?

Of their general form — at the least the one we’re occupied with inside an end-to-end learning framework — robotics datasets typically are available in two modalities: the visual modality and the robot’s proprioception / goal positions modality (state/motion vectors). Here’s what this could appear to be in practice:

Until now, the very best approach to store visual modality was PNG for individual frames. This could be very redundant as there’s numerous repetition among the many frames. Practitioners didn’t use videos since the loading times may very well be orders of magnitude above. These datasets are often released in various formats from academic papers (hdf5, zarr, pickle, tar, zip…). As of late, modern video codecs can achieve impressive compression ratios — meaning the dimensions of the encoded video in comparison with the unique uncompressed frames — while still preserving excellent quality. Because of this with a compression ratio of 1:20, or 5% as an illustration (which is well achievable), you get from a 20GB dataset right down to a single GB of knowledge. For this reason, we decided to make use of video encoding to store the visual modalities of our datasets.

Contribution

We propose a LeRobotDataset format that is easy, lightweight, easy to share (with native integration to the hub) and straightforward to visualise.
Our datasets are on average 14% the dimensions their original version (reaching as much as 0.2% in the very best case) while preserving full training capabilities on them by maintaining a excellent level of quality. Moreover, we observed decoding times of video frames to follow this pattern, depending on resolution:

• Within the nominal case where we’re decoding a single frame, our loading time is comparable to that of loading the frame from a compressed image (png).
• Within the advantageous case where we’re decoding multiple successive frames, our loading time is 25%-50% that of loading those frames from compressed images.

On top of this, we’re constructing tools to simply understand and browse these datasets.
You may explore a couple of examples yourself in the next Spaces using our visualization tool (click the photographs):

But what’s a codec? And what’s video encoding & decoding actually doing?

At its core, video encoding reduces the dimensions of videos by utilizing mainly 2 ideas:

• Spatial Compression: This is similar principle utilized in a compressed image like JPEG or PNG. Spatial compression uses the self-similarities of a picture to scale back its size. As an illustration, a single frame of a video showing a blue sky could have large areas of comparable color. Spatial compression takes advantage of this to compress these areas without losing much in quality.

• Temporal Compression: Moderately than storing each frame as is, which takes up numerous space, temporal compression calculates the differences between each frame and keeps only those differences (that are generally much smaller) within the encoded video stream. At decoding time, each frame is reconstructed by applying those differences back. After all, this approach requires at the least one frame of reference to begin computing these differences with. In practice though, we use multiple placed at regular intervals. There are several reasons for this, that are detailed in this text. These “reference frames” are called keyframes or I-frames (for Intra-coded frames).

Because of these 2 ideas, video encoding is in a position to reduce the dimensions of videos right down to something manageable. Knowing this, the encoding process roughly looks like this:

1. Keyframes are determined based on user’s specifications and scenes changes.
2. Those keyframes are compressed spatially.
3. The frames in-between are then compressed temporally as “differences” (also called P-frames or B-frames, more on these within the article linked above).
4. These differences themselves are then compressed spatially.
5. This compressed data from I-frames, P-frames and B-frames is encoded right into a bitstream.
6. That video bitstream is then packaged right into a container format (MP4, MKV, AVI…) together with potentially other bitstreams (audio, subtitles) and metadata.
7. At this point, additional processing could also be applied to scale back any visual distortions brought on by compression and to make sure the overall video quality meets desired standards.

Obviously, it is a high-level summary of what is happening and there are numerous moving parts and configuration selections to make on this process. Logically, we wanted to guage the easiest way of doing it given our needs and constraints, so we built a benchmark to evaluate this in response to plenty of criteria.

Criteria

While size was the initial reason we decided to go along with video encoding, we soon realized that there have been other facets to think about as well. After all, decoding time is a crucial one for machine learning applications as we wish to maximise the period of time spent training fairly than loading data. Quality must stays above a certain level as well in order to not degrade our policies’ performances. Lastly, one less obvious but equally essential aspect is the compatibility of our encoded videos in an effort to be easily decoded and played on the vast majority of media player, web browser, devices etc. Having the flexibility to simply and quickly visualize the content of any of our datasets was vital feature for us.

To summarize, these are the factors we desired to optimize:

• Size: Impacts storage disk space and download times.
• Decoding time: Impacts training time.
• Quality: Impacts training accuracy.
• Compatibility: Impacts the flexibility to simply decode the video and visualize it across devices and platforms.

Obviously, a few of these criteria are in direct contradiction: you possibly can hardly e.g. reduce the file size without degrading quality and vice versa. The goal was subsequently to search out the very best compromise overall.

Note that due to our specific use case and our needs, some encoding settings traditionally used for media consumption don’t really apply to us. A great example of that’s with GOP (Group of Pictures) size. More on that in a bit.

Metrics

Given those criteria, we selected metrics accordingly.

• Size compression ratio (lower is healthier): as mentioned, that is the dimensions of the encoded video over the dimensions of its set of original, unencoded frames.

• Load times ratio (lower is healthier): that is the time it take to decode a given frame from a video over the time it takes to load that frame from a person image.

For quality, we checked out 3 commonly used metrics:

• Average Mean Square Error (lower is healthier): the common mean square error between each decoded frame and its corresponding original image over all requested timestamps, and in addition divided by the variety of pixels within the image to be comparable across different image sizes.

• Average Peak Signal to Noise Ratio (higher is healthier): measures the ratio between the utmost possible power of a signal and the facility of corrupting noise that affects the fidelity of its representation. Higher PSNR indicates higher quality.

• Average Structural Similarity Index Measure (higher is healthier): evaluates the perceived quality of images by comparing luminance, contrast, and structure. SSIM values range from -1 to 1, where 1 indicates perfect similarity.

Moreover, we tried various levels of encoding quality to get a way of what these metrics translate to visually. Nevertheless, video encoding is designed to appeal to the human eye by making the most of several principles of how the human visual perception works, tricking our brains to keep up a level of perceived quality. This might need a unique impact on a neural net. Due to this fact, besides these metrics and a visible check, it was essential for us to also validate that the encoding didn’t degrade our policies performance by A/B testing it.

For compatibility, we do not have a metric per se, but it surely mainly boils right down to the video codec and the pixel format. For the video codec, the three that we selected (h264, h265 and AV1) are common and do not pose a difficulty. Nevertheless, the pixel format is very important as well and we found afterwards that on most browsers as an illustration, yuv444p will not be supported and the video cannot be decoded.

Variables

Image content & size

We do not expect the identical optimal settings for a dataset of images from a simulation, or from the true world in an apartment, or in a factory, or outdoor, or with a lot of moving objects within the scene, etc. Similarly, loading times won’t vary linearly with the image size (resolution).
For these reasons, we ran this benchmark on 4 representative datasets:

• lerobot/pusht_image: (96 x 96 pixels) simulation with easy geometric shapes, fixed camera.
• aliberts/aloha_mobile_shrimp_image: (480 x 640 pixels) real-world indoor, moving camera.
• aliberts/paris_street: (720 x 1280 pixels) real-world outdoor, moving camera.
• aliberts/kitchen: (1080 x 1920 pixels) real-world indoor, fixed camera.

Encoding parameters

We used FFmpeg for encoding our videos. Listed here are the important parameters we played with:

Video Codec (vcodec)

The codec (coder-decoder) is the algorithmic engine that is driving the video encoding. The codec defines a format used for encoding and decoding. Note that for a given codec, several implementations may exist. For instance for AV1: libaom (official implementation), libsvtav1 (faster, encoder only), libdav1d (decoder only).

Note that the remainder of the encoding parameters are interpreted in a different way depending on the video codec used. In other words, the identical crf value used with one codec doesn’t necessarily translate into the identical compression level with one other codec. The truth is, the default value (None) is not the identical amongst the various video codecs. Importantly, it’s also the case for a lot of other ffmpeg arguments like g which specifies the frequency of the important thing frames.

Pixel Format (pix_fmt)

Pixel format specifies each the color space (YUV, RGB, Grayscale) and, for YUV color space, the chroma subsampling which determines the way in which chrominance (color information) and luminance (brightness information) are literally stored within the resulting encoded bitstream. As an illustration, yuv420p indicates YUV color space with 4:2:0 chroma subsampling. That is probably the most common format for web video and standard playback. For RGB color space, this parameter specifies the variety of bits per pixel (e.g. rbg24 means RGB color space with 24 bits per pixel).

Group of Pictures size (g)

GOP (Group of Pictures) size determines how continuously keyframes are placed throughout the encoded bitstream. The lower that value is, the more continuously keyframes are placed. One key thing to know is that when requesting a frame at a given timestamp, unless that frame happens to be a keyframe itself, the decoder will search for the last previous keyframe before that timestamp and might want to decode each subsequent frame as much as the requested timestamp. Because of this increasing GOP size will increase the common decoding time of a frame as fewer keyframes can be found to begin from. For a typical online content akin to a video on Youtube or a movie on Netflix, a keyframe placed every 2 to 4 seconds of the video — 2s corresponding to a GOP size of 48 for a 24 fps video — will generally translate to a smooth viewer experience as this makes loading time acceptable for that use case (depending on hardware). For training a policy nevertheless, we’d like access to any frame as fast as possible meaning that we’ll probably need a much lower value of GOP.

Constant Rate Factor (crf)

The constant rate factor represent the quantity of lossy compression applied. A worth of 0 implies that no information is lost while a high value (around 50-60 depending on the codec used) could be very lossy.
Using this parameter fairly than specifying a goal bitrate is preferable because it allows to aim for a relentless visual quality level with a potentially variable bitrate fairly than the alternative.

This table summarizes the various values we tried for our study:

Decoding parameters

Decoder

We tested two video decoding backends from torchvision:

• pyav (default)
• video_reader

Timestamps scenarios

Given the way in which video decoding works, once a keyframe has been loaded, the decoding of subsequent frames is fast.
This in fact is affected by the -g parameter during encoding, which specifies the frequency of the keyframes. Given our typical use cases in robotics policies which could request a couple of timestamps in numerous random places, we wish to duplicate these use cases with the next scenarios:

• 1_frame: 1 frame,
• 2_frames: 2 consecutive frames (e.g. [t, t + 1 / fps]),
• 6_frames: 6 consecutive frames (e.g. [t + i / fps for i in range(6)])

Note that this differs significantly from a typical use case like watching a movie, wherein every frame is loaded sequentially from the start to the top and it’s acceptable to have big values for -g.

Moreover, because some policies might request single timestamps which are a couple of frames apart, we even have the next scenario:

• 2_frames_4_space: 2 frames with 4 consecutive frames of spacing in between (e.g [t, t + 5 / fps]),

Nevertheless, on account of how video decoding is implemented with pyav, we do not have access to an accurate seek so in practice this scenario is actually the identical as 6_frames since all 6 frames between t and t + 5 / fps might be decoded.

Results

After running this study, we switched to a unique encoding from v1.6 on.

We managed to realize more quality because of AV1 encoding while using the more compatible yuv420p pixel format.

Sizes

We achieved a mean compression ratio of about 14% across the full dataset sizes. Most of our datasets are reduced to under 40% of their original size, with some being lower than 1%. These variations might be attributed to the varied formats from which these datasets originate. Datasets with the best size reductions often contain uncompressed images, allowing the encoder’s temporal and spatial compression to drastically reduce their sizes. Then again, datasets where images were already stored using a type of spatial compression (akin to JPEG or PNG) have experienced less reduction in size. Other aspects, akin to image resolution, also affect the effectiveness of video compression.

Loading times

Because of video encoding, our loading times scale significantly better with the resolution. This is particularly true in advantageous scenarios where we decode multiple successive frames.

1 frame	2 frames	6 frames

Summary

The total results of our study can be found in this spreadsheet. The tables below show the averaged results for g=2 and crf=30, using backend=pyav and in all timestamps-modes (1_frame, 2_frames, 6_frames).

Policies

We validated that this recent format didn’t impact performance on trained policies by training a few of them on our format. The performances of those policies were on par with those trained on the image versions.

Figure 1: Training curves for Diffusion policy on pusht dataset

Figure 2: Training curves for ACT policy on an aloha dataset

Policies have also been trained and evaluated on AV1-encoded datasets and compared against our previous reference (h264):

• Diffusion on pusht:
• ACT on aloha_sim_transfer_cube_human:
• ACT on aloha_sim_insertion_scripted:

Future work

Video encoding/decoding is an enormous and complicated subject, and we’re only scratching the surface here. Listed here are among the things we left over on this experiment:

For the encoding, additional encoding parameters exist that aren’t included on this benchmark. Particularly:

• -preset which allows for choosing encoding presets. This represents a set of options that can provide a certain encoding speed to compression ratio. By leaving this parameter unspecified, it is taken into account to be medium for libx264 and libx265 and 8 for libsvtav1.
• -tune which allows to optimize the encoding for certain facets (e.g. film quality, live, etc.). Particularly, a fast decode option is on the market to optimise the encoded bit stream for faster decoding.
• two-pass encoding would even be interesting to take a look at because it increases quality, even though it is prone to increase encoding time significantly. Note that since we’re primarily occupied with decoding performance (as encoding is barely done once before uploading a dataset), we didn’t measure encoding times nor have any metrics regarding encoding. Using a 1-pass encoding didn’t pose any issue and it didn’t take a major period of time during this benchmark (with the condition of using libsvtav1 as a substitute of libaom for AV1 encoding).

The more detailed and comprehensive list of those parameters and others is on the market on the codecs documentations:

Similarly on the decoding side, other decoders exist but aren’t implemented in our current benchmark. To call a couple of:

• torchcodec
• torchaudio
• ffmpegio
• decord
• nvc

Finally, we didn’t look into video encoding with depth maps. Although we did port datasets that include depth maps images, we aren’t using that modality for now.