In machine vision, 3D vision systems, depth cameras, industrial cameras, and video processing pipelines, timestamp, frame index, and frame latency are critical technical metrics used to measure system synchronization, data order, and real-time performance. These concepts are especially important for developers building depth camera systems, ToF cameras, robotics vision platforms, autonomous driving perception systems, and real-time video processing pipelines.

In addition to these parameters, modern video encoding and streaming systems also use decoding time stamp (DTS), which determines when a video frame should be decoded during the processing pipeline.

This article explains the concepts of timestamp, frame index, frame latency, and decoding time stamp, including their working principles, roles in video pipelines, and optimization methods for improving real-time performance.

Image Processing Pipeline in Vision Systems

In modern machine vision systems and 3D depth camera platforms, a single image frame typically goes through multiple stages before it is delivered to an application.

The common pipeline includes:

1. Image exposure

2. Image signal processing (ISP processing)

3. Data packaging and transmission

4. SDK processing

5. Application data retrieval

Because each stage requires processing time, every frame experiences a certain amount of system latency. To accurately track frame timing and order, systems rely on timestamp and frame index mechanisms.

What Is a Timestamp?

A timestamp is a data field used to record the exact moment when a frame is captured or generated by the device.

When a camera device starts operating, it typically initializes an internal system timer. This timer begins counting from zero and continues to increase over time. Each time a frame is captured, the system assigns a timestamp value to that frame.

Typical characteristics of timestamps include:

• Generated by the device

• Measured in milliseconds (ms) or microseconds (μs)

• Transmitted together with image data

• Used to identify the capture time of a frame

Key Functions of Timestamp

Device synchronization

In multi-camera systems or multi-sensor platforms, timestamps allow:

• sensor synchronization

• cross-device data alignment

• robotics perception timing calibration

Latency measurement

By comparing the device timestamp with the host system time, developers can calculate the time delay between capture and processing.

Frame ordering

In video streams and image pipelines, timestamps help ensure frames are processed in the correct chronological order.

Decoding Time Stamp (DTS)

In video encoding and streaming systems, another important timing parameter is the decoding time stamp (DTS).

The decoding time stamp indicates the time at which a video frame should be decoded by the decoder.

In most video pipelines, two primary timestamps exist:

Presentation Time Stamp (PTS)

PTS specifies when a frame should be displayed or presented to the viewer.

Decoding Time Stamp (DTS)

DTS specifies when a frame should be decoded by the decoder.

In video formats that use B-frames (bidirectional predicted frames), the decoding order and display order are often different. Therefore, decoding time stamp values are required to ensure frames are decoded in the correct sequence.

For example:

Display order

Frame 1 → Frame 2 → Frame 3

Decoding order

Frame 1 → Frame 3 → Frame 2

In this scenario:

• DTS controls decoding order

• PTS controls display order

The decoding time stamp mechanism is widely used in H.264 video encoding, H.265 video encoding, IP camera streaming, RTSP video transmission, and surveillance systems.

Frame Index Explained

Frame index is another important mechanism used to identify and track frames within a video or image stream.

When frames are delivered to the host system, the driver or SDK typically assigns a sequential frame number to each frame.

For example:

Frame 0
Frame 1
Frame 2
Frame 3

Frame index values typically have the following properties:

• Sequential numbering

• Unique identifier for each frame

• Continuously increasing value

Key Uses of Frame Index

Detecting frame loss

If a sequence appears as:

Frame 10
Frame 11
Frame 13

This indicates that Frame 12 was dropped.

Frame drops can occur due to:

• high CPU load

• slow application processing speed

• SDK buffer overflow

Debugging system performance

Developers often analyze frame index values to determine:

• data stream stability

• whether frames are being processed fast enough

• system performance bottlenecks

Understanding Frame Latency

Frame latency refers to the time difference between when an image frame is captured and when it becomes available to the application.

Frame latency usually consists of several components.

Exposure Time

Exposure time is the duration required for the camera sensor to capture the image. Longer exposure times increase the total latency.

Image Processing Time

Image signal processors perform operations such as:

• noise reduction

• depth computation

• image enhancement

• distortion correction

These processes add processing time to each frame.

Data Transmission Time

Image data must be transmitted from the device to the host system. Transmission time depends on factors such as:

• interface bandwidth

• USB or Ethernet connection speed

• system network load

SDK Processing Time

SDKs may perform additional processing tasks, including:

• image decoding

• point cloud generation

• depth and RGB alignment

Application Processing Time

If the application reads frames too slowly, the system may experience:

• frame buffering

• frame drops

• increased latency

All these stages together determine the overall frame latency of the system.

Measuring Frame Latency

Developers typically measure frame latency using timestamp comparison methods.

A common formula is:

Frame Latency = Host Time − Device Timestamp

By measuring latency across multiple frames, developers can analyze:

• average latency

• maximum latency

• system stability

In many real-time vision systems, frame latency usually ranges from tens of milliseconds to over one hundred milliseconds, depending on hardware performance and processing complexity.

Methods to Reduce Frame Latency

If system latency becomes too high, developers can apply several optimization strategies.

Reduce Image Resolution

Higher resolution images generate larger data sizes and require more processing time. Reducing resolution can significantly lower latency.

Upgrade Hardware Performance

Using more powerful CPUs, GPUs, or memory systems can accelerate image processing and SDK operations.

Reduce Point Cloud Computation

Point cloud generation in 3D vision systems requires heavy computation. Developers can optimize performance by:

• limiting processing to a region of interest (ROI)

• reducing point cloud generation frequency

Optimize Multithreading

Designing efficient multithreaded image processing pipelines helps prevent data bottlenecks and improves system throughput.

Disable Unnecessary Image Filters

Some image filtering algorithms increase computational load. Disabling non-essential filters can reduce processing delay.

Relationship Between Timestamp, Frame Index, Frame Latency, and DTS

In a complete vision or video system, these parameters serve different roles.

Timestamp records the frame capture time.
Frame index identifies the sequence of frames.
Frame latency measures the system delay.
Decoding time stamp determines the correct decoding order in video streams.

Together, they form the core timing and synchronization framework of modern video processing and real-time vision systems.

Conclusion

Understanding timestamp, frame index, frame latency, and decoding time stamp is essential when developing machine vision systems, depth cameras, industrial imaging solutions, robotics perception platforms, and real-time video processing systems.

These mechanisms enable developers to achieve:

• accurate frame timing control

• multi-device synchronization

• proper video decoding order

• system latency analysis and optimization

As technologies such as autonomous driving, robotics, AR/VR, industrial automation, and intelligent surveillance systems continue to evolve, timing management and frame processing techniques will remain fundamental components of high-performance real-time vision systems.

Synexens Industrial Outdoor 4m TOF Sensor Depth 3D Camera Rangefinder_CS40p

SHOP NOW

After-sales Support:
Our professional technical team specializing in 3D camera ranging is ready to assist you at any time. Whether you encounter any issues with your TOF camera after purchase or need clarification on TOF technology, feel free to contact us anytime. We are committed to providing high-quality technical after-sales service and user experience, ensuring your peace of mind in both shopping and using our products.