Video and Real-Time Sensing

Video comprises the vast majority of bits that are generated daily, and is the primary signal driving current innovations in robotics, remote sensing, and wearable technology. Yet, the most powerful video understanding models are too expensive for the resource-constrained platforms used in these applications. One approach is to offload inference to the cloud; this gives access to GPUs capable of processing high-resolution videos in real time. But even with reliable, high-bandwidth communication channels, the combined latency of video encoding, model inference, and round-trip communication prohibits use for certain real-time applications. The alternative is to use fully local inference; but this places extreme constraints on computational and power costs, requiring smaller models and lower resolution, leading to degraded accuracy. To address these challenges, we propose DeDelayed, a real-time inference system that divides computation between a remote model operating on delayed video frames and a local model with access to the current frame. The remote model is trained to make predictions on anticipated future frames, which the local model incorporates into its prediction for the current frame. The local and remote models are jointly optimized with an autoencoder that limits the transmission bitrate required by the available downlink communication channel. We evaluate DeDelayed on the task of real-time streaming video segmentation using the BDD100K driving dataset. For a round trip delay of 100 ms, DeDelayed improves performance by 6.4 mIoU compared to fully local inference and 9.8 mIoU compared to remote inference—an equivalent improvement to using a model ten times larger. We release our training code, pretrained models, and python library at InterDigitalInc/dedelayed.

Introduction

Fig. 14 Overview of various inference setups, including conventional local, remote, and split inference. DeDelayed combines a small on-device image model and a heavier cloud-based temporally predictive video model to produce accurate and on-time predictions.

In soft real-time applications—such as cloud gaming or video conferencing (Isikdogan et al., 2020, Jain et al., 2025, Jiang et al., 2025)—late outputs may be diminished in value, but are still useful. In these applications, expensive Deep Neural Network (DNN)-based operations can be offloaded to powerful cloud GPUs to save on-device power. As long as the typical latency is low, the loss of utility from latency is outweighed by power savings and extended battery life.

In hard real-time applications—such as aerial robotic control or obstacle avoidance (Liu et al., 2022, Moon et al., 2021)—late outputs can be catastrophic, and the system must be designed with a guaranteed deadline. Due to the irreducible high-tail latency in wireless communication, hard real-time applications must be equipped with a fully functional local inference pipeline as a fallback in the case that the remote predictions fail to meet the deadline (Chen et al., 2025, Chen et al., 2024).

Modern hardware for capturing digital video can operate with extremely low power while ingesting super-human amounts of information at high spatial and temporal resolution (Engel and others, 2023). Using video as the primary or exclusive modality of perception provides an opportunity to significantly reduce size, weight, and power by replacing multiple specialized sensors, and is a major focus of ongoing research. Existing video understanding systems based on masked prediction and diffusion-based foundation models (Agarwal et al., 2025, Kim et al., 2024, Ravi et al., 2024) are able to directly produce useful predictions or actions from video input, but require too much computation to run directly on the sensor devices used in these applications. Various approaches to split computing and collaborative inference (Gao et al., 2025, Kang et al., 2017, Matsubara et al., 2022, Wang et al., 2024) have been proposed to offload computation of these expensive image and video models to the cloud. For real-time streaming video applications, existing approaches still fall into one or more of three common pitfalls. (1) They allocate all on-device power and computation to a single linear inference pipeline, leaving no resources for a local-only fallback. (2) They do not account for the impact of latency on prediction accuracy. (3) They operate on videos with significantly reduced spatiotemporal resolution to manage computational cost, leaving out rich visual details available from modern camera systems.

To address these limitations, we introduce DeDelayed (Fig. 14), a collaborative inference framework suitable for hard real-time applications. DeDelayed is designed to maximally utilize high-resolution, full-motion video understanding models on cloud GPUs, while avoiding the pitfalls of other split-computing systems:

1. Full integration with local-only fallback model. No wireless communication channel can offer perfect reliability. For real-time applications with critical deadlines, any remote inference procedure must be accompanied by a lightweight local fallback model. Instead of two redundant inference pipelines, DeDelayed uses a single path based on a local model that optionally incorporates side information from the remote model. We choose a simple method to incorporate this side information—element-wise addition of activation maps—resulting in negligible overhead and well-defined behavior in the absence of remote outputs.

2. Temporal prediction for latency mitigation. During supervised training of the remote model component, we simulate a delay of \(D\) frames. In other words, the remote model is trained to predict features useful in the future. A delay embedding—similar to a position embedding in text or vision transformers—allows the behavior of the remote model to adapt to changes in the channel. As shown in Fig. 15, temporally predictive training is able to capture motion dynamics, which can be used to compensate for latency.

3. Mixed-resolution inference. On-device AI video processing at capture resolution and frame rate is rarely feasible, even with lightweight models. DeDelayed enables mixed-resolution inference—the local model runs at a lower resolution, and the remote model processes high-resolution frames with a 3D transformer that understands motion. Thus, the remote model supplies delayed yet accurate, high-level semantic features, while the local model aligns and localizes them to the current scene, as shown in Fig. 16.

Our contributions are threefold:

1. We provide measurements demonstrating how higher degrees of latency hurt the accuracy of dense visual predictions for semantic segmentation of driving scenes.

2. We introduce DeDelayed, a co-inference framework that integrates the output of a future-predicting remote model with the current input to a local model.

3. Using DeDelayed, we create a video segmentation system for urban driving scenes that outperforms any existing local or remote inference solution, while avoiding the pitfalls that limit the practicality of previous approaches.

Fig. 15 To demonstrate the effect of temporally predictive training, we train a 3D transformer to predict the next frame with an MSE loss on pixels. (a) shows the original video frame. (b) shows the difference between (a) and a future frame, with objects such as the traffic sign and road markings in different locations. (c) shows the pixel predictions of the 3D transformer. (d) shows the difference from the true future frame. While the predictive model cannot predict high-frequency details, it is able to accurately model the motion of objects, signs, and road markings.

Fig. 16 Example of activation maps from local and remote model components. The remote server uses the higher level of video detail to accurately distinguish and classify objects. The local model provides exact position adjustments based on the current frame. When making predictions from the combined activation map, small details that would be impossible to make out at low resolution (e.g., the distant pedestrians, labeled red) are accurately classified and localized.

Background

In the human visual system, the optic nerve can only transmit a small fraction of the information received by the retina (Kelly, 1962). Barlow’s efficient coding hypothesis (Barlow and others, 1961) posits that compression is the primary role of early processing; once this compressed representation is received in deeper layers of the visual cortex, more metabolically intense processing can occur. In the predictive coding model (Rao and Ballard, 1999), this processing is driven by feedback mechanisms that minimize a temporally predictive error signal to create a perceptual model that is consistent with sensory inputs.

Machines equipped with digital video sensors—which are at the center of ongoing innovation in robotics (Kim et al., 2024, O'Neill et al., 2024), remote sensing (Khani et al., 2021, Szwarcman et al., 2024), and wearable technology (Grauman et al., 2022, Grauman et al., 2024)—share similar constraints. The throughput and power efficiency of ingesting pixels on the sensor device (e.g., a battery-powered robot) are extremely high—typically tens or hundreds of megapixels per watt-second (Engel and others, 2023). However, moderately sized DNNs can only process visual data at about one megapixel per watt-second (Cai et al., 2023). For more advanced video AI based on autoregressive modeling (Agarwal et al., 2025) or temporal prediction (Assran et al., 2025), the efficiency may be as low as 500 pixels per watt-second. Instead of on-device processing, power constraints can be circumvented by compressing and transmitting video streams to cloud GPU datacenters supported by a 100-megawatt power infrastructure (Goldberg and Kehoe, 2013).

Nevertheless, fully remote processing is challenging for certain real-time applications (e.g., collision avoidance) due to unreliability in network and cloud infrastructure (Chen et al., 2025, Chen et al., 2024). This has motivated split computing systems (Azizian and Bajić, 2022, Choi and Bajić, 2018, Choi and Bajić, 2018, Choi and Bajić, 2022, Kang et al., 2017, Teerapittayanon et al., 2016, ISO/IEC, 2025) which can reduce the bandwidth and latency of remote inference by leveraging both on-device and remote computation. Still, delivering predictions by a guaranteed deadline requires a fallback procedure independent of the remote server. In many systems (e.g., autonomous motor vehicles) the limited accuracy and reliability of lightweight local models warrant a human operator as the fallback (On-Road Automated Driving (ORAD) Committee, 2021), preventing full automation.

Related work

The challenges of real-time video perception have been approached from many angles. Previous research can be grouped into four major categories: (1) improving the accuracy-efficiency trade-off of inference, (2) offloading computation from the sensor device to a remote server, (3) temporally predictive video modeling, and (4) dynamically augmenting the on-device model via remote collaboration.

Improving the accuracy-efficiency trade-off. Over the decade since AlexNet (Krizhevsky et al., 2012), the trade-off between accuracy and computational efficiency of image-based DNNs has improved significantly. This has made local inference (shown in the first column of Fig. 14) the de facto standard for many computer vision and robotics applications, as the target perception accuracy is often achievable on-device without introducing the additional failure modes or network requirements. Still, video models that process dense 3D pixel volumes at high resolution using convolutional or transformer-based DNNs are typically avoided for real-time applications. Instead, it is common to apply a lightweight 2D image model to each frame. Using recent 2D models like EfficientViT (Cai et al., 2023), high resolution images can be segmented accurately at real-time throughput (greater than 30 fps) using moderately powered hardware (10–100 Watts). For more restricted power budgets (less than 10 Watts) the spatial or temporal resolution must be reduced by one or more orders of magnitude to meet real-time deadlines, significantly reducing accuracy.

Offloading computation. Numerous works have explored ways to offload computation of expensive image and video models to the cloud. Early exiting (Teerapittayanon et al., 2016), split computing (Kang et al., 2017), and feature coding for machines (FCM) (Azizian and Bajić, 2022, Choi and Bajić, 2018, Choi and Bajić, 2018, Choi and Bajić, 2022, ISO/IEC, 2025) focus on partitioning DNN layers into two components: one that runs on the sensor device and another that runs in the cloud. The overall workflow of these approaches is shown in the third column of Fig. 14. A limitation of these methods and many of their successors is that they allocate local computation towards a prediction pipeline that terminates in the cloud, rather than on the device. This may be desirable for applications where the predictions are only used in the cloud (e.g., remote sensing). However, if predicted outputs must be available on-device (e.g., autonomous navigation), the delay incurred by round-trip communication leads to stale predictions. Even worse, the exclusive allocation of local computational resources to an inference pipeline that necessarily requires a remote server leaves little room for a readily available local fallback mode in the event of late or dropped predicted outputs.

Temporally predictive video modeling. Regardless of DNN size or added network latency, some amount of inference delay is unavoidable. Instead of focusing exclusively on reducing the degree of delay, prior works have explored correcting the estimated world state based on the anticipated delay. These corrections require some mechanism for temporally predictive video modeling, and can use on-device optical flow estimates (Zhang et al., 2024) or readily available motion vectors from an existing video codec (Xiao et al., 2024). If video is streamed to a sufficiently powerful remote server, generative video world models based on masked prediction (Assran et al., 2025) or diffusion (Agarwal et al., 2025) foundation models can provide temporal prediction. In each of these cases, existing methods for temporal prediction are repurposed from another application, rather than specialized and optimized for the desired perception task. This prevents temporally predictive abilities or behaviors from being learned or optimized with the rest of the DNN-based layers.

Dynamically augmenting the on-device model via remote collaboration during streaming inference. This category of methods improve the on-device model’s inference capabilities by incorporating remote information at runtime (e.g., streaming weights or fusing remote features). They are underexplored but important for real-time applications on computationally limited devices. Adaptive Model Streaming (Khani et al., 2021) streams weight updates to keep the local model tuned to changing scene characteristics. Knowledge Boosting (Srinivas et al., 2024) addresses the problem from a different lens—instead of dynamically updating the model weights, it directly fuses in remote features into a small on-device model. It requires separate models trained for specific fixed delays, and evaluates on audio data in a controlled local environment with delays up to 48 ms within a local networking context. Our method, DeDelayed, is capable of adapting to arbitrary remote inference delays by conditioning the remote model on the measured delay at runtime, enabling dynamic specialization across varying latency. Methods within this category are orthogonal to and compatible with many methods from earlier categories, and with each other. Our work extends this rarely-explored direction by explicitly inputting dynamic system characteristics such as delay to improve on-device predictions under realistic, variable latency.

Local Inference with Delay-Conditioned Remote Assistance

Fig. 17 Time progresses left to right. The client-side camera produces video frames, which are sent across a communication network to the server. The server runs a heavyweight model using the latest video frame \(x_{t - \tau}\) that it receives, in addition to a context of previously received video frames \(x_{< t - \tau}\), as well as the measured delay \(\tau\). This produces an output \(z_{t - \tau}\) that the server sends to the client. The client pairs the latest received response \(z_{t - \tau}\) with a freshly produced video frame \(x_t\), and runs these inputs through a lightweight model. This finally produces a timely result \(\hat{y}_t\) that can be used in real-time delay-sensitive applications.

DeDelayed introduces a general framework that improves the accuracy and robustness of real-time inference on resource-constrained sensor devices. It does so by combining the strengths of both local inference and remote inference, while mitigating their weaknesses. The local model has access to the latest sensor data, and yet lacks the computational capability needed to produce accurate outputs. The remote model provides accurate outputs, and yet delivers them with delay. With careful combination of both subsystems, DeDelayed is able to provide bounded performance guarantees. When trained appropriately such that the local and remote subsystems deliver the maximum accuracies that they would conventionally—e.g., by freezing the local and remote image backbones during the final joint training of the entire system—it can be verified that DeDelayed is never worse than either local inference or remote inference independently. As we will demonstrate later, we are able to glue together the two subsystems in a way that is simple yet effective.

DeDelayed addresses the problem of stale predictions from powerful remote models by integrating them with a lightweight, on-device model. The core idea is to leverage the high-quality features from a heavyweight remote model, despite their inherent delay, by explicitly conditioning them on the measured latency and fusing them early with live information from a local model. This ensures that the final predictions are both accurate and timely.

DeDelayed can be formulated in simple mathematical terms as follows. Given a fresh input frame \(x_t\) at current time \(t\), the final prediction \(\hat{y}_t\) is computed using a lightweight local model, \(f_{\text{local}}\), which processes \(x_t\) along with time-delayed features \(z_{t-\tau}\) from a heavyweight remote model, \(f_{\text{remote}}\). To produce powerful predictive features, the remote model is conditioned on the delay \(\tau\), and processes a short clip of past frames \(x_{\leq t-\tau}\) ending at time \(t-\tau\). This is expressed by the following equations:

\[\begin{split}\begin{gather} z_{t - \tau} = f_{\text{remote}}(\tau,\, x_{\leq t - \tau}) \\ \hat{y}_t = f_\text{local}(x_t,\, z_{t - \tau}) \end{gather}\end{split}\]

For clarity, the notation is summarized in Table 9. Fig. 17 presents a system diagram that demonstrates the fundamental principle we describe, and shows how information propagates through the various subsystems as time progresses.

Table 9 Notation.

Symbol	Meaning
\(x_t\)	Input frame at current time \(t\)
\(x_{\leq t - \tau}\)	Input frames up to time \(t - \tau\)
\(\hat{y}_t\)	Prediction for time \(t\)
\(z_{t - \tau}\)	Features outputted by remote model run at time \(t - \tau\)
\(\tau\)	Delay in time between the old and current frame
\(D\)	Delay in frames between the old and current frame
\(f_\text{local}\)	Light local model run at time \(t\)
\(f_\text{remote}\)	Heavy remote model run at time \(t - \tau\)

The entire DeDelayed system is trained end-to-end to minimize a task-specific loss function, \(\mathcal{L}_{\text{task}}\), evaluated against the ground truth \(y_t\) for the current frame.

\[\mathcal{L}_{\text{task}} = \ell(\hat{y}_t, y_t)\]

For semantic segmentation, \(\ell\) is typically the cross-entropy loss. The objective is to produce predictions \(\hat{y}_t\) that are accurate at time \(t\) on the local device.

In the next section, we detail how we designed a specific implementation to test the DeDelayed framework in action.

Design and implementation

Fig. 18 DeDelayed workflow. The complementary properties of local and remote inference are exploited to produce real-time streaming video understanding that is both accurate and on time.

DeDelayed aims to use remote computation to improve the accuracy and robustness of real-time machine vision on resource constrained sensor devices. The design of DeDelayed is guided by three key insights. (1) Incorporating the outputs of the remote model as side information to the local model eliminates computational redundancy and provides a fallback suitable for hard real-time applications. (2) Temporally predictive training can mitigate the impact of latency due to processing and round-trip communication. (3) Decoupling inference into a high resolution full-motion video stream and a low-resolution still-image stream exploits the complementary properties of remote and on-device processing. Fig. 18 shows an overview of the design in the context of semantic segmentation of video frames.

System overview. Information propagates through our system as follows:

1. The local device transmits input frames to the remote via the uplink.

2. Each incoming frame is fed into a heavyweight remote model \(f_{\text{remote}}\). To accelerate inference, this model caches and maintains a context window of the \(K\) most recent features, which are computed from each incoming frame using a pretrained 2D ViT backbone.

3. The \(K\) per-frame features are concatenated along the temporal axis, and a learned delay embedding conditioned on the measured delay \(\tau\) is added.

4. A 3D ViT encoder followed by learned pooling (MLP–pool–MLP) produces delay-conditioned remote features \(z_{t-\tau}\), which are sent back to the device via the downlink.

5. The lightweight local model \(f_{\text{local}}\) runs on a fresh input \(x_t\), and fuses in the remote features \(z_{t-\tau}\).

6. The local model finishes decoding the fused representation and outputs labels \(\hat{y}_t\).

Video compression. Each frame captured on the sensor device is processed by a lossy image or video codec to allow transmission over a wireless channel. In our experiments, we choose the resolution, framerate, and degree of lossy compression to represent a video signal that is transmissible via 5G cellular uplink (30fps 720p content compressed at rates between 1–10 Mbps).

Latency-aware remote video model. The remote model operates on high resolution, compressed video frames, and consists of four learnable modules: (1) a 2D vision transformer (ViT2D), (2) a 3D video transformer (ViT3D), (3) task-specific MLP layers (R-MLP), and (4) a dimensionality-reducing autoencoder (DR-AE). During the initial training stages, only modules 1–3 (ViT2D, ViT3D, and R-MLP) are used, as shown in Fig. 19. During the final training stage, the R-MLP module is discarded and replaced with the DR-AE to allow joint training of the local and remote components. Due to round trip communication and networking, the predictions from the remote model will be delayed by \(D\) frames at runtime. During training, an artificial delay is applied to the input, but not to the target. Thus, the training objective is to predict the target of a future frame. During training, we sample a delay uniformly between 0 and 5 frames (up to 167 ms). Additionally, we add a learnable “delay embedding” to the input activation maps of the ViT3D—similar to the position or timestep embedding used in diffusion transformers (Peebles and Xie, 2023). In our experiments, we provide the remote model with four frames of context and aim for a target latency of 33 ms (a single frame at 30 fps) on a high power GPU testbed.

Fig. 19 Overview of the remote model component as used during pretraining. A video sequence \(v[n]\) is captured on the sensor device and transmitted to a remote server, incurring a delay of \(D\) frames. The remote model operates on a fixed context window of \(K\) frames. A 2D vision transformer (EfficientViT-L1, effective patch size of \(8\times8\) pixels) is applied independently to each of the \(K\) input frames. The outputs of the 2D transformer are concatenated along the temporal axis, but spatially merged into larger \(16\times16\) patches, thus maintaining a similar sequence length (when \(K=4\), the sequence length is identical). A learned delay embedding is added, allowing the remaining layers to behave differently based on the expected value of \(D\). Finally, 3D video transformer layers and task-specific (e.g., segmentation) layers are applied, generating a prediction corresponding to the current frame (\(D\) frames in the future after the latest remote input frame).

Local image model incorporating remote side information. The local model processes the most recently collected video frame on the sensor device at low resolution, and consists of three learnable modules, shown in the right half of Fig. 18: (1) pixel-processing 2D convolutional layers (CNN2D), (2) 2D feature-processing convolution and attention layers (CoAt2D), and task specific (e.g., classification or segmentation) MLP layers (L-MLP). If present, features from the remote model are added element-wise to the activation map between the CNN2D and the CoAt2D. Our local model can operate at a resolution up to 0.34 megapixels (\(704 \times 480\)) while meeting the same target latency of 33 ms on our CPU (rather than GPU) testbed.

Joint prediction with autoencoded resolution adapter. After pretraining the remote model for the desired prediction task, its task-specific layers are discarded and replaced with the DR-AE, consisting of adaptive spatial pooling and a channel bottleneck. The degree of adaptive spatial pooling is chosen to match the operating resolution of the local model. The sample mean is used as the pooling operator. The remainder of the DR-AE consists of MLP layers that terminate in a low-resolution, low-channel count activation map suitable for downlink transmission. The output of the DR-AE is added element-wise to the output of the pixel-processing CNN2D. Then, the local and remote models undergo a final joint training stage.

Delay embedding. We use a two-layer MLP which maps \(\mathbb{R} \to \mathbb{R}^C\) via layers of sizes 1024 and 256.

ViT3D. This consists of 12 blocks with 256 input/output channels, where each block contains a 3D attention followed by \(3 \times 3 \times 3\) MBConv3D.

Evaluation

Experimental setup. We evaluate DeDelayed on the task of real-time semantic segmentation of driving scenes using the BDD100K video dataset (Yu et al., 2020), containing video of driving scenes at 30 frames per second (fps). We use the standard 19 label Cityscapes taxonomy (Cordts et al., 2016) for semantic segmentation of urban scenes. Since the BDD100K dataset does not provide dense segmentation labels for all video frames, we generate pseudo-labels using two models. For the validation set, we use the pretrained DepthAnything Cityscapes segmentation model (Yang et al., 2024). To our knowledge, this is the most accurate publicly available semantic segmentation model for the Cityscapes taxonomy. For the training set, we use EoMT (Kerssies et al., 2025), which provides high accuracy, but is significantly faster for labeling 70k training images.

We evaluate the performance subject to known delays ranging from 0 to 5 frames, corresponding to 0 to 167 ms at 30 fps. At training time, the delay \(\tau\) is sampled per batch from a uniform distribution over this range.

Training details. We adopt a multi-stage training strategy, as detailed in Table 10. The remote and local models are first trained individually and then later combined. Each model is pretrained on the large-scale ImageNet dataset (Russakovsky et al., 2015) for classification, then on the image segmentation task on Cityscapes (Cordts et al., 2016), before being fine-tuned on the smaller BDD100K driving dataset. We train the remote model to have temporally predictive capability by supplying it with a delay-aware (DA) objective: to predict the labels of future frames conditioned on the degree of delay. During the final training stage, the task-specific layers of the MLP are replaced with an entropy bottleneck and resolution adapter that allow its outputs to be added to the intermediate activation map of the local model. Using this configuration, the local and remote models are trained jointly for the target scenario of local segmentation incorporating delayed predictions from the remote model. The training loss is per-pixel cross-entropy loss. We use the Adan (Xie et al., 2024) optimizer, a warmup-stable-decay learning rate schedule, gradient clipping, and selectively applying discriminative fine-tuning or layer-wise learning rate decay (LLRD) (Howard and Ruder, 2018).

Table 10 Local and remote model components and training setup.

Stage	Local layers	Remote layers	Data	Res.
Remote (video-predictive)
1	–	ViT2D	IN1K, CS	–
2	–	ViT2D, ViT3D	BDD	496
Local (image only)
3	CNN2D, CoAt2D	–	IN1K	224
4	CNN2D, CoAt2D	–	CS	336
5	CNN2D, CoAt2D	–	BDD	496
DeDelayed
6	CNN2D, CoAt2D	ViT2D, ViT3D	BDD	480/720

Data: IN1K = ImageNet-1K; CS = Cityscapes; BDD = Berkeley DeepDrive 100K.

Bounded Performance Under Variable Delay

We compare how various inference systems perform under the effect of communication network latency. Table 11 shows the segmentation performance (mIoU) for different configurations (local-only, remote-only, and the proposed local+remote system, DeDelayed). Each configuration serves as an ablation on the final design.

• Local image and Remote image inference setups process individual frames in the conventional way, though the remote is susceptible to communication network delay.

• Remote video has access to past frames of context, but only predicts labels for its present view, and thus fares no better than “remote image”.

• Remote predictive is fed a tunable delay and sustains accuracy by predicting the future.

• Local + remote predictive represents a DeDelayed system, and thus, is able to further sustain accuracy by merging the remote predictive features with fresh local features.

As illustrated in Fig. 20, the performance of conventional remote inference is significantly higher (roughly 5%pt. mIoU) than local inference if communication delay is low. However, the accuracy degrades rapidly with increasing network delay, reflecting the detriment of using stale predictions for dense prediction tasks involving motion. Beyond 2 frames (67 ms) of round-trip latency, remote inference becomes worse than local inference (8%pt. mIoU lower at 167 ms). This impact is significantly mitigated by adopting a temporal prediction training objective, which extends the utility of remote inference to 5 frames (167 ms). Finally, the proposed configuration (DeDelayed) incorporates temporally predictive features from the remote model as side information to the local model, and almost completely eliminates the drop-off in accuracy due to delay. At the maximum tested delay of 5 frames, DeDelayed provides a 6.7%pt. mIoU improvement over the local baseline—a difference that would typically require using a model roughly 10 times larger (Cai et al., 2023, Kerssies et al., 2025).

Fig. 20 Segmentation accuracy (mIoU) versus round-trip latency (milliseconds or frames).

Table 11 shows the accuracy of various inference systems under various local and remote subsystem delays. For local inference delays of \(\leq\)8 ms, the “local + remote predictive” method consistently delivers better accuracy across all network round-trip delays.

Table 11 Accuracy (mIoU) given various subsystem delays (ms).

Local delay	Remote delay
	0 ms	33 ms	67 ms	100 ms	133 ms	167 ms
Remote (video-predictive)
—	0.655	0.649	0.640	0.627	0.611	0.596
Local (image only)
0 ms	0.588	0.588	0.588	0.588	0.588	0.588
4 ms	0.585	0.585	0.585	0.585	0.585	0.585
8 ms	0.582	0.582	0.582	0.582	0.582	0.582
33 ms	0.562	0.562	0.562	0.562	0.562	0.562
DeDelayed
0 ms	0.661	0.660	0.655	0.652	0.646	0.641
4 ms	0.656	0.655	0.650	0.647	0.642	0.637
8 ms	0.652	0.650	0.646	0.643	0.637	0.632
33 ms	0.624	0.619	0.616	0.612	0.607	0.603

Sub-frame (4 ms, 8 ms) accuracies were lerped between 0 ms and 33 ms.

Additional experiments

Effect of delay jitter. We evaluate how our model performs under delay jitter, i.e., when the delay varies over time. Our training loss targets accuracy for a fixed, tunable delay input—we do not explicitly train it to be jitter-resilient. Nonetheless, temporal structure in the data helps maintain accuracy even when the delay input differs from the observed delay. Fig. 21 characterizes this, showing performance across observed delays when the model is fed a possibly incorrect delay as input. Although the model was not explicitly trained for mismatched delays or delays beyond 5 frames, it still performs well at these out-of-distribution delays. Unsurprisingly, accuracy peaks when the model’s delay input matches the observed delay. Notably, the accuracy drop is smaller when the delay input exceeds the observed delay than the reverse. Thus, when jitter is high, it is safer to use a larger delay input than expected, since underestimating the delay tends to make the remote features overconfident about localization. This matrix can be precomputed during evaluation. At runtime, the device may consult it to choose among the received remote feature tensors \(\{z_1, z_2, \ldots\}\) by looking up the accuracy for each tensor’s delay pair \((D_o, D_i)\) and selecting the best. After a dramatic scene change or excessive staleness, the device may omit remote features from the local model input and run the local model alone, yielding its baseline performance.

Fig. 21 Segmentation accuracy (mIoU) over observed delay and model delay input.

Local input resolution. We evaluate performance across local input resolutions by finetuning the fused model (trained at 480 px) for 10 additional epochs at 224, 320, and 480 px. The results are shown in Fig. 22. Our remote-assisted local model operates at far lower resolutions (e.g., 224 px) while achieving better accuracy under round-trip latency than other solutions.

Fig. 22 Segmentation accuracy (mIoU) versus round-trip latency (milliseconds or frames). Further finetuned and evaluated on various local input resolutions.

Delay-input ablation. We ablate the delay input by removing the delay conditioning from the remote model and finetuning for the same number of epochs. This causes an average drop of 0.015 mIoU for local 224 px, showing that explicit delay inputs are beneficial.

Table 12 Delay-input ablation.

Experiment	Remote delay (frames)
	0	1	2	3	4	5	Local
BDD 224px	0.652	0.650	0.640	0.635	0.621	0.613	—
BDD 224px (\(-\) delay)	0.622	0.628	0.627	0.625	0.615	0.603	—

Robustness to uplink compression. We evaluate performance under x264/x265 compression (veryfast, zerolatency, bframes=0) to assess robustness to realistic streaming configurations. Results are reported in Table 13. DeDelayed retains most of its accuracy down to 3 Mbps for both H.264 and H.265, with H.265 holding up better at 1 Mbps.

Table 13 Robustness to uplink compression.

Experiment	Remote delay (frames)
	0	1	2	3	4	5	Local
BDD H.265 5 Mbps	0.658	0.657	0.652	0.648	0.642	0.635	—
BDD H.264 5 Mbps	0.657	0.656	0.651	0.647	0.640	0.634	—
BDD H.265 3 Mbps	0.655	0.654	0.649	0.644	0.640	0.635	—
BDD H.264 3 Mbps	0.652	0.652	0.647	0.644	0.637	0.632	—
BDD H.265 1 Mbps	0.631	0.630	0.627	0.623	0.619	0.615	—
BDD H.264 1 Mbps	0.612	0.610	0.608	0.605	0.603	0.599	—

Compute and runtime breakdown. Table 14 summarizes the parameter count, multiply-accumulate operations, and per-component wall-clock latency on a high-power GPU (NVIDIA H100, fp16) and a low-power CPU (Apple M3 Pro, fp32). The local model component fits comfortably within a 33 ms budget on CPU, while the heavyweight remote ViT3D requires GPU acceleration.

Table 14 Compute and runtime breakdown by component.

Component	Params	MACs	GPU (ms)	CPU (ms)
Remote ViT2D	42.206M	199.683G	2.01	704.57
Delay embedding	0.264M	0.0005G	0.04	1.26
Remote ViT3D+head	11.964M	551.665G	24.38	34475.63
Local ViT2D+head	8.101M	1.455G	0.67	9.34
Additive fusion	0	0	0.04	0.01

GPU: NVIDIA H100 fp16 PyTorch compiled. CPU: Apple M3 Pro fp32 PyTorch 6 cores. Local: 224px.

Additional dataset evaluation. We train DeDelayed models on 35k clips from the first 229 sequences in the Nymeria dataset (Ma et al., 2024) and evaluate on 2k clips from 25 disjoint sequences with unseen environments and actions, using the same semantic segmentation task and ADE20K taxonomy.

Table 15 Generalizability to new environments and actions.

Experiment	Remote delay (frames)
	0	1	2	3	4	5	Local
Nymeria (Ma et al., 2024)	0.245	0.245	0.245	0.245	0.244	0.244	0.203

Occlusion. Fig. 23 shows a qualitative example of a newly unoccluded object missed by all methods except DeDelayed.

Fig. 23 Due to 167 ms RTT, the remote model has not yet seen any frames containing the recently unoccluded cyclist (highlighted in red), so methods whose final inference is not local fail to detect it.

Conclusion

DeDelayed addresses a central challenge in real-time systems that rely on remote computation: prediction staleness induced by network delay. It mitigates remote inference delay by elevating delay to a first-class variable, conditioning the remote model via a learnable delay embedding, and fusing remote features with fresh local features. Across realistic network conditions, DeDelayed surpasses strong local-only and remote-only baselines, with a particular advantage for longer latencies and high-motion content. As a foundational framework, DeDelayed applies to a wide range of real-time problem domains, enabling intelligent systems that are not only accurate but also truly timely and dependable in dynamic environments. Future work includes studying variable and stochastic delay distributions, high-motion data, lighter local models, and local future prediction.

References

[AAB+25] (1,2,3)

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and others. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.

[ABF+25] (1,2)

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, and others. V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.

[ABajic22] (1,2)

Bardia Azizian and Ivan V Bajić. Privacy-preserving feature coding for machines. In 2022 Picture Coding Symposium (PCS), 205–209. IEEE, 2022.

[B+61]

Horace B Barlow and others. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217–233, 1961.

[CLT+23] (1,2,3)

Han Cai, Junyan Li, Muyan Tian, Zhekai Hu, and Song Han. Efficientvit: lightweight multi-scale attention for high-resolution dense prediction. In CVPR. 2023.

[CTJ+25] (1,2)

Kaiyuan Chen, Nan Tian, Christian Juette, Tianshuang Qiu, Liu Ren, John Kubiatowicz, and Ken Goldberg. FogROS2-PLR: probabilistic latency-reliability for cloud robotics. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 16290–16297. IEEE, 2025.

[CWG+24] (1,2)

Kaiyuan Chen, Michael Wang, Marcus Gualtieri, Nan Tian, Christian Juette, Liu Ren, Jeffrey Ichnowski, John Kubiatowicz, and Ken Goldberg. FogROS2-LS: a location-independent fog robotics framework for latency sensitive ros2 applications. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 10581–10587. IEEE, 2024.

[CBajic18a] (1,2)

Hyomin Choi and Ivan V Bajić. Deep feature compression for collaborative object detection. In 2018 25th IEEE International Conference on Image Processing (ICIP), 3743–3747. IEEE, 2018.

[CBajic18b] (1,2)

Hyomin Choi and Ivan V Bajić. Near-lossless deep feature compression for collaborative intelligence. In 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), 1–6. IEEE, 2018.

[CBajic22] (1,2)

Hyomin Choi and Ivan V Bajić. Scalable image coding for humans and machines. IEEE Transactions on Image Processing, 31:2739–2754, 2022.

[COR+16] (1,2)

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.

[E+23] (1,2)

J. Engel and others. Project aria. arXiv preprint arXiv:2308.13561, 2023.

[GMC+25]

Changsheng Gao, Yifan Ma, Qiaoxi Chen, Yenan Xu, Dong Liu, and Weisi Lin. Feature coding in the era of large models: dataset, test conditions, and benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1068–1077. 2025.

[GK13]

Ken Goldberg and Ben Kehoe. Cloud robotics and automation: a survey of related work. EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2013-5, pages 13–5, 2013.

[GWB+22]

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, and others. Ego4d: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 18995–19012. 2022.

[GWT+24]

Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, and others. Ego-exo4d: understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19383–19400. 2024.

[HR18]

Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 328–339. 2018.

[IGM20]

Furkan Isikdogan, Timo Gerasimow, and Gilad Michael. Eye contact correction using deep neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3318–3326. 2020.

[JWZ+25]

Varun Jain, Zongwei Wu, Quan Zou, Louis Florentin, Henrik Turbell, Sandeep Siddhartha, Radu Timofte, Qifan Gao, Linyan Jiang, Qing Luo, and others. Ntire 2025 challenge on video quality enhancement for video conferencing: datasets, methods and results. In Proceedings of the Computer Vision and Pattern Recognition Conference, 1184–1194. 2025.

[JHT+25]

Shan Jiang, Zhenhua Han, Haisheng Tan, Xinyang Jiang, Yifan Yang, Xiaoxi Zhang, Hongqiu Ni, Yuqing Yang, and Xiang-Yang Li. Real-time neural-enhancement for online cloud gaming. arXiv preprint arXiv:2501.06880, 2025.

[KHG+17] (1,2,3)

Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45(1):615–629, 2017.

[Kel62]

D. H. Kelly. Information capacity of a single retinal channel. IRE Transactions on Information Theory, 8(3):221–226, 1962.

[KCH+25] (1,2)

Tommie Kerssies, Niccolò Cavagnero, Alexander Hermans, Narges Norouzi, Giuseppe Averta, Bastian Leibe, Gijs Dubbelman, and Daan de Geus. Your ViT is secretly an image segmentation model. In Proceedings of the Computer Vision and Pattern Recognition Conference, 25303–25313. 2025.

[KHNEA21] (1,2)

Mehrdad Khani, Pouya Hamadanian, Arash Nasr-Esfahany, and Mohammad Alizadeh. Real-time video inference on edge devices via adaptive model streaming. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4572–4582. 2021.

[KPK+24] (1,2)

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, and others. OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.

[KSH12]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012.

[LNC+22]

Xu Liu, Guilherme V Nardari, Fernando Cladera, Yuezhan Tao, Alex Zhou, Thomas Donnelly, Chao Qu, Steven W Chen, Roseli AF Romero, Camillo J Taylor, and others. Large-scale autonomous flight with real-time semantic slam under dense forest canopy. IEEE Robotics and Automation Letters, 7(2):5512–5519, 2022.

[MYH+24] (1,2)

Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, and Richard Newcombe. Nymeria: a massive collection of multimodal egocentric daily motion in the wild. In ECCV. 2024.

[MLR22]

Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. Split computing and early exiting for deep learning applications: survey and research challenges. ACM Computing Surveys, 55(5):1–30, 2022.

[MKY+21]

Seung Jae Moon, Jinsol Kim, Hongsik Yim, Yeeun Kim, and Hyouk Ryeol Choi. Real-time obstacle avoidance using dual-type proximity sensor for safe human-robot interaction. IEEE Robotics and Automation Letters, 6(4):8021–8028, 2021.

[ONeillRM+24]

Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, and others. Open X-Embodiment: robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 6892–6903. IEEE, 2024.

[PX23]

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 4195–4205. 2023.

[RB99]

Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79–87, 1999.

[RGH+24]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, and others. SAM 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.

[RDS+15]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.

[SIC+24]

Vidya Srinivas, Malek Itani, Tuochao Chen, Emre Sefik Eskimez, Takuya Yoshioka, and Shyamnath Gollakota. Knowledge boosting during low-latency inference. In Proc. Interspeech 2024, 4338–4342. 2024.

[SRF+24]

Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Thorsteinn Elí Gíslason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Oliveira, Joao Lucas de Sousa Almeida, Rocco Sedona, Yanghui Kang, and others. Prithvi-EO-2.0: a versatile multi-temporal foundation model for earth observation applications. arXiv preprint arXiv:2412.02732, 2024.

[TMK16] (1,2)

Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), 2464–2469. IEEE, 2016.

[WYL+24]

Yingchao Wang, Chen Yang, Shulin Lan, Liehuang Zhu, and Yan Zhang. End-edge-cloud collaborative computing for deep learning: a comprehensive survey. IEEE Communications Surveys & Tutorials, 26(4):2647–2683, 2024.

[XZY+24]

Xuedou Xiao, Yingying Zuo, Mingxuan Yan, Wei Wang, Jianhua He, and Qian Zhang. Task-oriented video compressive streaming for real-time semantic segmentation. IEEE Transactions on Mobile Computing, 2024.

[XZL+24]

Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

[YKH+24]

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: unleashing the power of large-scale unlabeled data. In CVPR. 2024.

[YCW+20]

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: a diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020.

[ZJS24]

Zhiyong Zhang, Huaizu Jiang, and Hanumant Singh. Neuflow: real-time, high-accuracy optical flow estimation on robots using edge devices. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5048–5055. IEEE, 2024.

[ISOIEC25] (1,2)

ISO/IEC. ISO/IEC 23888: MPEG Artificial Intelligence (MPEG-AI). 2025. Parts: Part 2: Video coding for machines (VCM), Part 3: Optimization of encoders and receiving systems for machine analysis of coded video content, Part 4: Feature coding for machines (FCM), Part 5: AI-based point cloud coding.

[OnRADORADCommittee21]

On-Road Automated Driving (ORAD) Committee. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE international, 2021.