Wireless Deep Video Semantic Transmission

Abstract

In this paper, we design a new class of high-efficiency deep joint source-channel coding methods to achieve end-to-end video transmission over wireless channels. The proposed methods exploit nonlinear transform and conditional coding architecture to adaptively extract semantic features across video frames, and transmit semantic feature domain representations over wireless channels via deep joint source-channel coding. Our framework is collected under the name deep video semantic transmission (DVST). In particular, benefiting from the strong temporal prior provided by the feature domain context, the learned nonlinear transform function becomes temporally adaptive, resulting in a richer and more accurate entropy model guiding the transmission of current frame. Accordingly, a novel rate adaptive transmission mechanism is developed to customize deep joint source-channel coding for video sources. It learns to allocate the limited channel bandwidth within and among video frames to maximize the overall transmission performance. The whole DVST design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under perceptual quality metrics or machine vision task performance metrics. Across standard video source test sequences and various communication scenarios, experiments show that our DVST can generally surpass traditional wireless video coded transmission schemes. The proposed DVST framework can well support future semantic communications due to its video content-aware and machine vision task integration abilities.

I Introduction

The task of video transmission in today’s wireless networks is largely separated into two steps: source coding and channel coding [1]. Source coding compresses the source video as sequences of bits, and channel coding represents sequences of bits as transmitted signals against impacts of imperfect wireless channels such as noise, fading, and interferences. This separation-based approach has been employed for a large variety of applications, as the binary representations of various source data can be seamlessly transmitted over arbitrary wireless channels by changing the underlying channel code. This paradigm has benefited a lot due to the independent optimization of each component.

However, the limits of the separation-based design begin to emerge with more demands on low-latency wireless video delivery applications such as virtual reality (VR). On the one hand, current wireless video transmission systems suffer from time-varying channel conditions, in which case the mismatch between communication rate and channel capacity leads to obvious cliff-effect, i.e., the performance breaks down when the channel capacity goes below communication rate. On the other hand, the widely-used entropy coding, which converts the source representation into sequences of bits, is quite sensitive to the variational estimate of the marginal distribution of the source latent representation. Small perturbations on this marginal can lead to catastrophic error propagation in entropy decoding [2]. In practice, the small perturbation can often be caused by floating point round-off error [3]. Unfortunately, this round-off operation depends heavily on hardware or software platforms, and in various data compression applications, the transceiver may well employ different platforms. This non-determinism issue in transmitter vs. receiver may finally lead to severe performance degradation.

To address this problem, it is very time to bridge source coding and channel coding together to boosting the end-to-end communication system capabilities. By this means, the channel transmission process can be aware of the source semantic features [4, 5, 6, 7, 8, 9, 10]. The paradigm aiming at the integrated design of source and channel processing was named joint source-channel coding (JSCC) [11], a classical topic in the information theory and coding theory. However, conventional JSCC schemes [11, 12, 13, 14] are based on explicit probabilistic models and handcrafted designs, whose optimization complexity is intractable for complex sources. In addition, they ignored the semantic aspects of source messages. As one modern version, recent deep learning methods for realizing JSCC have stimulated significant interest in both artificial intelligence (AI) and wireless communication communities [15, 16, 17, 18, 19]. By using artificial neural networks (ANNs), source data can be directly encoded as continuous-valued symbols to be transmitted over wireless channels. Deep JSCC can overcome the catastrophic degradation problem by using analog transmission without entropy coding. Current deep JSCC methods have shown end-to-end image transmission performance surpassing classical separation-based JPEG/JPEG2000/BPG source compression combined with ideal channel capacity-achieving code family, especially for sources of small dimensions, e.g., small CIFAR10 image data set [20].

However, one can observe that, in general, as the source dimension increases, e.g., large-scale images, the performance of deep JSCC degrades rapidly, which is even inferior to the classical separation-based coding schemes as demonstrated in [9]. Moreover, existing deep JSCC schemes cannot provide comparable coding gain as that of classical separated coding schemes, i.e., the slope of the performance curve slows down quickly with the increase of coding rate or the channel signal-to-noise ratio (SNR). This poor coding gain stems from the naive design of codec networks. Current deep JSCC works simply employ one highly-integrated ANN as the encoder function to achieve dimension reduction with respect to the raw source data. By adding the wireless channel as one non-trainable layer, the learned codec ANNs can also combat the impacts of imperfect wireless channels. Nevertheless, this light auto-encoder structured deep JSCC cannot provide sufficient model expression capability for large-scale source data, resulting in a prematurely saturated coding gain of deep JSCC. Compared to the image source, the video source further induces the time dimension. Above saturation phenomenon on the coding gain is more likely to appear on video sources that need higher dimensional representation. Thus, a naive application of deep JSCC for wireless video transmission cannot provide satisfactory performance.

Inspired by the emerging data compression methods adopted in computer vision (CV) communities, a high-dimensional source will first be converted as latent representations defined by variational latent-variable models. This procedure is referred to as nonlinear transform [21, 22, 23, 24, 25]. The richness of latent representations preserves almost all the source semantic features that can be used for either recovering source data or directly driving the downstream intelligent tasks. By training appropriately, many such nonlinear transform models successfully represent the source data quite compactly and may be called compression in a sense. For practical data compression tasks, the latent representation needs to be further compressed as binary sequences through entropy coding. However, because the employed ANNs in nonlinear transform are typically based on floating point math, and the transmission is over time-varying wireless channels, a direct combination of nonlinear transform with traditional source coding (entropy coding such as arithmetic coding [2]) and channel coding (such as low-density parity-check (LDPC) coding [26]) will be also vulnerable to catastrophic failures left by entropy decoding.

In this paper, to attain high-efficiency and robust end-to-end video transmission, we leverage the advantages of nonlinear transform and deep JSCC together to formulate a new powerful framework, named deep video semantic transmission (DVST). It is specifically targeted at video transmission over imperfect wireless channels and preventing catastrophic failures caused by sensitive entropy coding. By integrating the emerging conditional coding paradigm [27] with nonlinear transform and deep JSCC, the proposed DVST framework works on the principle: considering the strong temporal correlations among video frames, DVST encodes the current frame in an efficient manner to generate channel-input symbols through contextual nonlinear transform and contextual deep JSCC. The contextual semantic information is used as part of the input of both nonlinear transform and deep JSCC codec. Benefiting from the temporal prior provided by the semantic feature domain context, the learned nonlinear transform function becomes temporally adaptive, resulting in a richer and more accurate entropy model to indicate how to allocate channel bandwidth resources to transmit the current frame. Moreover, we leverage the context to carry rich information as prior to deep JSCC codec, which helps to reconstruct the semantic feature map for higher video quality or downstream task performance. The whole DVST design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion (RD) performance under perceptual quality metrics or machine vision task performance metrics.

Specifically, the contributions of this paper can be summarized as follows.

1. DVST Framework: We propose a new end-to-end learnable framework for wireless video transmission, i.e., DVST, which integrates the advantages of nonlinear transform and deep JSCC. To the best of authors’ knowledge, this is the first work that establishes a temporally adaptive entropy model to customize deep JSCC for video. The proposed DVST framework exploits both nonlinear transform and conditional coding architecture for video semantic feature extraction, which contributes to higher efficiency and more robust wireless video transmission than traditional video coded transmission schemes.

2. Context-Driven Semantic Feature Modeling: We exploit a simple yet efficient method using temporal context to enhance the entropy model in nonlinear transform as well as the deep JSCC codec. We design ANN architectures to realize each module of DVST, in which the definition, usage, and learning manner of contextual semantic features as conditions are all clearly given.

3. Rate-Adaptive Semantic Feature Transmission: In light of the temporally adaptive entropy model on semantic features, we develop a method to improve the coding gain of video deep JSCC. In particular, we introduce a variable-length transmission mechanism for each embedding vector in the latent representation. The resulting DVST model learns to allocate the limited channel bandwidth within and among video frames to maximize the overall performance.

4. Performance Validation: We verify the performance of our DVST system across standard video source sequences. We show that for wireless video transmission, our DVST can achieve much better coding gain and RD performance on various established metrics such as PSNR and MS-SSIM. Equivalently, achieving identical end-to-end wireless transmission performance, the proposed DVST method can save up to 50% channel bandwidth cost, compared to classical H.264/H.265 combined with LDPC and digital modulation schemes. For task-oriented machine-type semantic communications, experimental results verify the effectiveness of DVST, which can better support machine vision tasks, meanwhile holds higher perceptual fidelity for human vision.

The remainder of this paper is organized as follows. In the next section II, we first review the system model of wireless video transmission, and propose the DVST framework. Then, in section III, we propose ANN architectures for realizing DVST, as well as key methods to guide the optimization of the DVST model. Section IV provides a direct comparison of a number of methods to quantify the performance gain of the proposed method. Finally, section V concludes this paper.

Figure 1: The framework of DVST. The visualization of the latents with the highest entropy.

Notational Conventions: Throughout this paper, lowercase letters (e.g., $x$) denote scalars, bold lowercase letters (e.g., $𝐱$) denote vectors. In some cases, $x_i$ denotes the elements of $𝐱$, which may also represent a subvector of $𝐱$ as described in the context. Bold uppercase letters (e.g., $𝐗$) denote matrices, and ${𝐈}_m$ denotes an $m$-dimensional identity matrix. $\ln (\cdot )$ denotes the natural logarithm, and $\log (\cdot )$ denotes the logarithm to base $2$. $p_x$ denotes a probability density function (pdf) with respect to the continuous-valued random variable $x$, and $P_{\overline{x}}$ denotes a probability mass function (pmf) with respect to the discrete-valued random variable $\overline{x}$. In addition, $𝔼[\cdot ]$ denotes the statistical expectation operation, and $ℝ$ denotes the real number set. Finally, $𝒩(x|\mu ,{\sigma }^2)\triangleq {(2\pi {\sigma }^2)}^{-1/2}\exp (-{(x-\mu )}^2/(2{\sigma }^2))$ denotes a Gaussian function, $ℒ(x|\mu ,\sigma )\triangleq {(2\sigma )}^{-1}\exp (-|x-\mu |/\sigma )$ denotes a Laplace function, and $𝒰(a-u,a+u)$ stands for a uniform distribution centered on $a$ with the range from $a-u$ to $a+u$.

II The Proposed Method

In this section, we first present the system model of wireless video transmission. Then, we describe the whole framework of DVST. After that, we introduce the contextual entropy model for rate-adaptively transmit the latent representations, followed by the learning methods of the context. Finally, we derive the optimization goal of DVST system.

II-A System Model

Consider a wireless video transmission problem. Given a video sequence $𝒳=\{{𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_T\}$, where the frame at time step $t$ is modeled as a vector of pixel intensities ${𝐱}_t\in {ℝ}^m$. The transmitter encodes the video frame sequence $𝒳$ as a sequence of variable-length continuous-valued channel input symbols $𝒮=\{{𝐬}_1,{𝐬}_2,\mathrm{\dots },{𝐬}_T\}$, where ${𝐬}_t\in {ℝ}^{k_t}$ denote the $k_t$-dimensional channel input vector at time step $t$. We usually have , and $R=\frac{1}{T}{\sum }_{t=1}^T\frac{k_t}{m}$ is defined as the channel bandwidth ratio (CBR) [28] denoting the average coding rate of $𝒳$. Then, the sequence $\{{𝐬}_t\}$ is successively sent over the wireless channel. This channel introduces random corruptions denoted as a transfer function $W(\cdot ;𝝂)$, where $𝝂$ denotes the channel parameters. The received sequence is ${\widehat{𝐬}}_t=W({𝐬}_t;𝝂)$ with the transition probability $p_{{\widehat{𝐬}}_t|{𝐬}_t}({\widehat{𝐬}}_t|{𝐬}_t)$. In this paper, we mainly consider the widely used additive white Gaussian noise (AWGN) channel such that the transfer function is ${\widehat{𝐬}}_t=W({𝐬}_t;{\sigma }^2)={𝐬}_t+{𝐧}_t$ where each component of the noise vector ${𝐧}_t$ is independently sampled from a time-invariant multidimensional Gaussian distribution, i.e., ${𝐧}_t\sim 𝒩(0,{\sigma }^2{𝐈}_{k_t})$, where ${\sigma }^2$ is the average noise power. Other channel models can also be similarly incorporated by changing the channel transition function. The receiver comprises a series of inverse operation which aims to recover ${\widehat{𝐱}}_t$ from the corrupted signal ${\widehat{𝐬}}_t$ or executes the downstream intelligent task.

We consider video transmission over the noisy wireless channel in a low-latency manner, i.e., the video sequence is transmitted to and reconstructed in the receiver frame-by-frame. We encapsulate $N$ consecutive frames as one group-of-pictures (GOP). A typical video coding algorithm first divides $𝒳$ into a stack of GOPs. Each GOP begins with an intra-coded picture (I-frame or keyframe) as a reference, followed by $N-1$ predictive coded frames (P-frames), which contain the motion compensated difference information for bitrate saving. In this paper, we exploit the classical GOP structure for end-to-end transmission. Since the transmission of I-frame is equivalent to that of image, which has been well studied in [17, 28, 18, 19], we concentrate on the transmission of P-frame.

II-B The Framework of DVST

We propose the DVST as a new learnable model for end-to-end wireless video transmission, which integrates the advantages of nonlinear transform and deep JSCC. Our DVST framework is illustrated in Fig. 1. To encode the current frame ${𝐱}_t$ efficiently, the transmitter adopts contextual analysis transform and contextual deep JSCC encoder as two critical modules. The analysis transform converts the source frame in pixel domain to the latent representation in semantic feature domain. Guided by the variational entropy modeling on the latent representation, a rate control module is added to achieve variable-length encoding in deep JSCC. For video source, there exists temporal correlation. Thus, the above two modules also employ the semantic feature domain context and the deep JSCC codeword domain context as the temporal prior. That makes nonlinear transform and deep JSCC modules temporally adaptive, resulting in a higher efficient video transmission framework.

Figure 2: Illustration of a DVST codec architecture and the entropy model of the primary link.

As illustrated in Fig. 1 and Fig. 2, $g_a$ and $g_s$ functions in nonlinear transform of the current frame are conditioned on the contextual semantic features ${\check{𝐜}}_t$ and ${\widehat{𝐜}}_t$, respectively. $f_e$ and $f_d$ functions in deep JSCC are conditioned on the contextual codewords ${\check{𝐝}}_t$ and ${\widehat{𝐝}}_t$, respectively. The primary link of DVST system is formulated as

$${𝐱}_t\stackrel{g_a(\cdot |{\check{𝐜}}_t)}{\to }{𝐲}_t\stackrel{f_e(\cdot |{\check{𝐝}}_t)}{\to }{𝐬}_t\stackrel{W(\cdot |𝝂)}{\to }{\widehat{𝐬}}_t\stackrel{f_d(\cdot |{\widehat{𝐝}}_t)}{\to }{\widehat{𝐲}}_t\stackrel{g_s(\cdot |{\widehat{𝐜}}_t)}{\to }{\widehat{𝐱}}_t.$$ (1)

In our DVST design, we use ANN to realize each function in (1) except for the channel transfer function $W$. In terms of the context information, transmitter (Tx) contexts ${\check{𝐜}}_t$ and ${\check{𝐝}}_t$ are obtained from the reference frame ${\check{𝐱}}_{t-1}$ and the reference feature map ${\check{𝐲}}_{t-1}$, respectively. These two references are generated at the transmitter by simulating the DVST process without passing over the wireless channel, i.e.,

$${𝐬}_{t-1}\stackrel{f_d(\cdot |{\check{𝐝}}_{t-1})}{\to }{\check{𝐲}}_{t-1}\stackrel{g_s(\cdot |{\check{𝐜}}_{t-1})}{\to }{\check{𝐱}}_{t-1},$$ (2)

where the codeword ${𝐬}_{t-1}$ is obtained from (1) by substituting the time step $t-1$. Receiver (Rx) contexts ${\widehat{𝐜}}_t$ and ${\widehat{𝐝}}_t$ are obtained from the reference synthesized frame ${\widehat{𝐱}}_{t-1}$ and the reference decoded feature map ${\widehat{𝐲}}_{t-1}$, respectively. These two references are directly obtained by taking out records from the receiver buffer at the time step $t-1$. Details about how to use context as conditions to formulate ANN-based functions will be introduced in the next section.

Specifically, in the transmitter, for the current frame ${𝐱}_t$ at time step $t$, $g_a(\cdot )$ extracts the source semantic features as a lower-dimensional latent representation ${𝐲}_t$, $f_e(\cdot )$ operates on this latent space. Consider the inter frame correlation in video sources, the analysis transform $g_a(\cdot )$ is formulated as

$${𝐲}_t=g_a({𝐱}_t|{\check{𝐜}}_t)\text{with}{\check{𝐜}}_t={\phi }_a({\check{𝐱}}_{t-1}).$$ (3)

${\phi }_a(\cdot )$ denotes the function to generate context ${\check{𝐜}}_t$ for the analysis transform, $g_a(\cdot )$ is therefore referred to as the contextual analysis transform. After that, the latent representation ${𝐲}_t$ is fed into the contextual deep JSCC encoder $f_e(\cdot )$ to generate the channel-input sequence ${𝐬}_t$ as

$${𝐬}_t=f_e({𝐲}_t|{\check{𝐝}}_t)\text{with}{\check{𝐝}}_t={\gamma }_e({\check{𝐲}}_{t-1}).$$ (4)

${\gamma }_e(\cdot )$ denotes the function of generating context for deep JSCC encoder. To provide rich and more correlated information for encoding ${𝐱}_t$, the context ${\check{𝐜}}_t$ is in the semantic feature domain with higher dimensions, and the context ${\check{𝐝}}_t$ is in the deep JSCC codeword space.

Then, the analog codeword sequence $𝐬$ is directly sent over the wireless communication channel. As aforementioned, we consider the AWGN channel such that the received sequence is ${\widehat{𝐬}}_t={𝐬}_t+{𝐧}_t$ with ${𝐧}_t\sim 𝒩(0,{\sigma }^2{𝐈}_{k_t})$. The receiver comprises a contextual deep JSCC decoder $f_d(\cdot )$ to reconstruct the corrupted signal ${\widehat{𝐬}}_t$ as ${\widehat{𝐲}}_t$, i.e.,

$${\widehat{𝐲}}_t=f_d({\widehat{𝐬}}_t|{\widehat{𝐝}}_t)\text{with}{\widehat{𝐝}}_t={\gamma }_d({\widehat{𝐲}}_{t-1}).$$ (5)

${\gamma }_d(\cdot )$ denotes the function of generating context for deep JSCC decoder. The contextual synthesis transform function $g_s(\cdot )$ is then performed on ${\widehat{𝐲}}_t$ to recover the current frame, i.e.,

$${\widehat{𝐱}}_t=g_s({\widehat{𝐲}}_t|{\widehat{𝐜}}_t)\text{with}{\widehat{𝐜}}_t={\phi }_s({\widehat{𝐱}}_{t-1}).$$ (6)

${\phi }_s(\cdot )$ denotes the function to generate context ${\widehat{𝐜}}_t$ for the synthesis transform.

For the contextual analysis transform $g_a$, we use a network to automatically learn the correlation between ${𝐱}_t$ and ${\check{𝐜}}_t$, which does not remove the redundancy through handcrafted subtraction operation like traditional residual video coding [29]. Herein, the context ${\check{𝐜}}_t$ comes from the reference frame ${\check{𝐱}}_{t-1}$. In this way, the contextual analysis transform becomes adaptive which generates the latent representation by selectively extracting semantic features from ${𝐱}_t$ and ${\check{𝐱}}_{t-1}$ [27]. Due to the motion in video, for old contents in ${𝐱}_t$ that can find a good reference from ${\check{𝐱}}_{t-1}$, $g_a$ still forces to generate its patch embeddings from the residue. For new contents in ${𝐱}_t$ that cannot find a good reference from ${\check{𝐱}}_{t-1}$, $g_a$ tends to generate its patch embeddings from ${𝐱}_t$ itself. The contextual nonlinear transform inherently learns to adaptively utilize the condition for semantic extraction. In addition, the context ${\check{𝐜}}_t$ is not only used for generating the latent representation, but also utilized to construct the entropy model, which will be introduced in the subsequent subsection.

For the contextual deep JSCC encoder $f_e$, we use a network to automatically learn the correlation between ${𝐲}_t$ and ${\check{𝐝}}_t$. Note that the context ${\check{𝐝}}_t$ comes from the reconstructed reference feature map ${\check{𝐲}}_{t-1}$, thus, the contextual deep JSCC encoder also becomes adaptive to generate the channel-input codewords. If patch embeddings in ${𝐲}_t$ can find a good reference from ${\check{𝐲}}_{t-1}$, $f_e$ inclines to transmit these embeddings with smaller channel bandwidth. In contrast, for patch embeddings in ${𝐱}_t$ that cannot find a good reference from ${\check{𝐲}}_{t-1}$, $f_e$ tends to allocate more channel bandwidth to transmit these embeddings. In this way, the contextual deep JSCC codec learns to adaptively utilize the condition for high-efficiency transmission. Moreover, the context ${\check{𝐝}}_t$ is not only used for generating the channel-input codeword, but also utilized to learn a rate-allocation function that controls the scaling rule from entropy value to channel bandwidth cost. Details will be introduced in the subsequent subsection.

II-C Entropy Model for Rate-Adaptive Transmission

Figure 3: The framework of context learning. Dashed lines denote MV data flows simulated at the transmitter, which are used to generate the reference MV without passing over the wireless channel. Solid lines denote data flows at both transmitter and receiver, which will pass over the realistic wireless channel.

In order to improve the coding gain of DVST, a variable-length transmission mechanism should be developed for each embedding $y_{t,i}$ of the semantic feature map ${𝐲}_t$. To this end, we estimate the entropy distribution on ${𝐲}_t$, and the channel bandwidth cost $k_{t,i}$ for transmitting $y_{t,i}$ can be accordingly determined. Therefore, our target is to design an entropy model which can accurately estimate the probability distribution of the latent representation ${𝐲}_t$.

Our entropy model is illustrated in Fig. 2. Following the work of [27], the latent representation ${𝐲}_t$ is variationally modeled as the Laplace distribution, where each embedding $y_{t,i}$ is of varying distribution parameters. In this paper, the hyperprior entropy model learns both the hierarchical prior [23] and the spatial prior [24]. In addition, our entropy model fuses the temporal prior provided by the context ${\check{𝐜}}_t$. Specifically, the entropy of each embedding $y_{t,i}$ is computed as

(7)

where ${\overline{y}}_{t,i}=\lfloor y_{t,i}\rceil$ denotes the quantized version of $y_{t,i}$ by using the uniform scalar quantization $\lfloor \cdot \rceil$ function (rounding to integers). denotes a tensor consisting of quantized embeddings ${\overline{y}}_{t,l}$ with . The quantized hyperprior ${\overline{𝐳}}_t=\lfloor {𝐳}_t\rceil =\lfloor h_{\mathrm{hpe}}({𝐲}_t)\rceil$ is obtained by stacking the hyperprior encoder network $h_{\mathrm{hpe}}(\cdot )$ on ${𝐲}_t$.

In order to use the gradient descent methods to optimize the entropy model, Ballé et al. have proposed a relaxed method for addressing the zero gradient problem caused by quantization [21]. A proxy “uniformly-noised” representation ${\widetilde{𝐲}}_t$ is adopted to replace the quantized representation ${\overline{𝐲}}_t=\lfloor {𝐲}_t\rceil$ during model training, i.e., ${\widetilde{𝐲}}_t={𝐲}_t+𝐨$ with $o_i\sim 𝒰(-\frac{1}{2},\frac{1}{2})$. Each ${\tilde{y}}_{t,i}$ is variationally modeled as a Laplace distribution with learned parameters ${\tilde{\mu }}_{t,i}$ and ${\tilde{\sigma }}_{t,i}$ such that

(8)

$={\displaystyle \prod _i}\left(ℒ({\tilde{\mu }}_{t,i},{\tilde{\sigma }}_{t,i})\ast 𝒰(-{\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{2}})\right)({\tilde{y}}_{t,i})$

where “$\ast$” is the convolutional operation, $i\in \{1,2,\mathrm{\dots },I\}$ denotes the path embedding index, and the proxy hyperprior ${\widetilde{𝐳}}_t$ is obtained by performing the hyperprior encoder network $h_{\mathrm{hpe}}(\cdot )$ on ${𝐲}_t$ and adding the uniformly sampled random offset $𝐨$, i.e., ${\widetilde{𝐳}}_t=h_{\mathrm{hpe}}({𝐲}_t)+𝐨$. Since we do not have prior beliefs about the hyperprior ${\widetilde{𝐳}}_t$, it can be modeled as non-parametric fully factorized density [23], i.e.,

$$p_{{\widetilde{𝐳}}_t}({\widetilde{𝐳}}_t)=\prod _j\left(p_{z_{t,j}|{𝝍}^{(j)}}(z_{t,j}|{𝝍}^{(j)})\ast 𝒰(-\frac{1}{2},\frac{1}{2})\right)({\tilde{z}}_{t,j}),$$ (9)

where ${𝝍}^{(j)}$ encapsulates all the parameters of $p_{z_{t,j}|{𝝍}^{(j)}}$. During model testing, the entropy model $P_{{\overline{𝐳}}_t}$ is established by taking discrete values from the learned entropy model $p_{{\widetilde{𝐳}}_t}$ by substituting ${\widetilde{𝐳}}_t$ as ${\overline{𝐳}}_t$. $h_{\mathrm{hpd}}(\cdot )$ denotes the hyperprior decoder network to provide the hierarchical prior. $h_{\mathrm{ar}}(\cdot )$ represents the auto regressive network to provide the spatial prior. $h_{\mathrm{tpe}}(\cdot )$ denotes the temporal prior encoder network to provide additional side information. $h_{\mathrm{pf}}(\cdot )$ denotes the prior fusion network operating on the above three types of prior information. During model testing, the entropy model in (7) will be established by taking discrete values from the learned entropy model by substituting ${\widetilde{𝐲}}_t$ and ${\widetilde{𝐳}}_t$ as ${\overline{𝐲}}_t$ and ${\overline{𝐳}}_t$, respectively.

As stated in NTC [25], the probabilistic model of ${𝐲}_{t,i}$ can be conditioned on some other vector ${\widetilde{𝐳}}_t$ like [23] or its preceding dimensions as that in [24]. The former corresponds to forward adaptation (FA) of the density model, and the latter is backward adaptation (BA). In this paper, due to the auto-regressive computation in (8), our entropy model is established under the BA mode. One can also use the FA mode, where $h_{\mathrm{pf}}(\cdot )$ relies only on the hierarchical prior and the context, i.e., $({\tilde{\mu }}_{t,i},{\tilde{\sigma }}_{t,i})=h_{\mathrm{pf}}(h_{\mathrm{hpd}}({\widetilde{𝐳}}_t),h_{\mathrm{tpe}}({\check{𝐜}}_t))$. Herein, the FA mode in DVST however cannot harvest the gain of better codec parallelism while it incurs performance degradation. The reason is that the auto-regressive computations of BA in our DVST are only used in entropy modeling to estimate the probability. The following deep JSCC codec runs in parallel for each embedding $y_{t,i}$. In comparison, traditional codec relying on arithmetic coding also adopts regressive computations in the BA mode which leads to higher latency. Therefore, we advertise the BA mode in our DVST framework.

With the learned entropy model , the allocated channel bandwidth cost $k_{t,i}$ to transmit the embedding $y_{t,i}$ is formulated as

(10)

where the scaling factor ${\eta }_t$ denotes the proportion from the entropy of embedding $y_{t,i}$ to the number of channel symbols. In particular, the physical meaning of ${\eta }_t$ can be interpreted as ${\eta }_t={\eta }_t^{\prime }/C$, where $C$ is the channel capacity (bits per channel symbol), and ${\eta }_t^{\prime }$ denotes an efficiency factor representing the capability of deep JSCC codec. Accordingly, ${\eta }_t^{\prime }=1$ stands for the ideal JSCC codec that is of the same performance as entropy-achieving source coding combined with capacity-achieving channel coding.

Following the aforementioned entropy model, the channel bandwidth cost of the primary link for transmitting semantic features ${𝐲}_t$ is derived as

$k_t^{\mathrm{pl}}$ $={\displaystyle \sum _i}{k}_{t,i}$ (11)

II-D Motion Transmission and Context Learning

As for context learning functions ${\phi }_a(\cdot )$, ${\phi }_s(\cdot )$, ${\gamma }_e(\cdot )$, and ${\gamma }_d(\cdot )$, inspired by [27], we also adopt the idea of motion estimation and motion compensation (MEMC) to formulate specific forms of these functions. Different from conventional MEMC applied in source pixel domain, we perform MEMC in semantic feature domain to generate contexts ${\check{𝐜}}_t$, ${\widehat{𝐜}}_t$ and in deep JSCC codeword domain to generate contexts ${\check{𝐝}}_t$, ${\widehat{𝐝}}_t$. This paradigm utilizes rich information density in feature/codeword domain to fascinate high-efficiency video transmission over wireless channels with limited bandwidth.

As shown in Fig. 3, the motion vector (MV) ${𝐦}_t$ is generated by using the flow estimation network [30] performed between the current frame ${𝐱}_t$ and the reconstructed reference frame ${\check{𝐱}}_{t-1}$. This MV is then transmitted over the wireless channel, which is referred to as the motion link. The whole process copies from the primary link without using context, i.e.,

${𝐦}_t\stackrel{g_a^{\mathrm{mv}}(\cdot )}{\to }{𝐲}_t^{\mathrm{mv}}\stackrel{f_e^{\mathrm{mv}}(\cdot )}{\to }{𝐬}_t^{\mathrm{mv}}\stackrel{W(\cdot |𝝂)}{\to }{\widehat{𝐬}}_t^{\mathrm{mv}}$ (12)

$\stackrel{f_d^{\mathrm{mv}}(\cdot )}{\to }{\widehat{𝐲}}_t^{\mathrm{mv}}\stackrel{g_s^{\mathrm{mv}}(\cdot )}{\to }{\widehat{𝐦}}_t.$

In our DVST design, we use ANN to realize each function in (12) except for the channel transfer function $W$. The reference MV ${\check{𝐦}}_t$ used at the transmitter is generated by simulating the motion link without passing over the wireless channel, i.e.,

$${𝐬}_t^{\mathrm{mv}}\stackrel{f_d^{\mathrm{mv}}(\cdot )}{\to }{\check{𝐲}}_t^{\mathrm{mv}}\stackrel{g_s^{\mathrm{mv}}(\cdot )}{\to }{\check{𝐦}}_t.$$ (13)

where the codeword ${𝐬}_t^{\mathrm{mv}}$ is obtained from (12).

In analogue to the primary link, by using the learned entropy model on ${𝐲}_t^{\mathrm{mv}}$, the allocated channel bandwidth cost $k_{t,j}^{\mathrm{mv}}$ to transmit the embedding $y_{t,j}^{\mathrm{mv}}$ is formulated as

(14)

The channel bandwidth cost of the motion link for transmitting MV semantic features ${𝐲}_t^{\mathrm{mv}}$ is derived as

(15)

The hyperprior ${𝐳}_t^{\mathrm{mv}}$ is obtained as ${𝐳}_t^{\mathrm{mv}}=h_{\mathrm{hpe}}^{\mathrm{mv}}({𝐲}_t^{\mathrm{mv}})$, and the entropy model $P_{{\overline{𝐳}}_t^{\mathrm{mv}}}$ of the quantized hyperprior ${\overline{𝐳}}_t^{\mathrm{mv}}$ can be obtained similar to (9).

Figure 4: Network architectures of primary link. Conv $k\times C$ is a convolution with $C$ channels and $k\times k$ filters, and the followed $\uparrow 2/\downarrow 2$ indicates upscaling or downscaling with a stride of $2$. ChannelNorm refers to a normalization layer from [31]. GDN denotes the Generalised Divisive Normalization proposed in [32], and IGDN is the inverse GDN. FC denotes fully-connected layer.

As illustrated in Fig. 3, at the transmitting end, the feature domain context generation function ${\phi }_a$ is formulated as

$${\check{𝐜}}_t={\phi }_a({\check{𝐱}}_{t-1})={\phi }_{\mathrm{ref}}(\mathrm{warp}({\phi }_{\mathrm{fe}}({\check{𝐱}}_{t-1}),{\check{𝐦}}_t)),$$ (16)

where the reference frame ${\check{𝐱}}_{t-1}$ is obtained as (2). We use the feature extractor network ${\phi }_{\mathrm{fe}}(\cdot )$ to convert the reference frame ${\check{𝐱}}_{t-1}$ to its feature domain representation. The reference MV guides where to extract feature domain context by using the warping function $\mathrm{warp}(\cdot )$ [33]. The refinement network ${\phi }_{\mathrm{ref}}(\cdot )$ is used to restore the spatial discontinuity problem caused by warping operation.

As for the codeword context ${\check{𝐝}}_t$, its generation function ${\gamma }_e$ is formulated as

$${\check{𝐝}}_t={\gamma }_e({\check{𝐲}}_{t-1})={\gamma }_{\mathrm{ref}}(\mathrm{warp}({\gamma }_{\mathrm{pc}}({\check{𝐲}}_{t-1}),{\check{𝐦}}_t)).$$ (17)

The precoding network ${\gamma }_{\mathrm{pc}}(\cdot )$ operates on the reference semantic feature map ${\check{𝐲}}_{t-1}$ which preprocesses the data before warping it. ${\gamma }_{\mathrm{ref}}(\cdot )$ denotes the refinement network.

At the receiving end, the context learning procedures are similar to (16) and (17), where the realistic received MV ${\widehat{𝐦}}_t$ is used. In addition, ${\check{𝐱}}_{t-1}$ and ${\check{𝐲}}_{t-1}$ are substituted by ${\widehat{𝐱}}_{t-1}$ and ${\widehat{𝐲}}_{t-1}$, respectively.

II-E Optimization Goal

The optimization goal of our DVST system is to use the least channel bandwidth cost to get the best video reconstruction quality or downstream task performance. Given the reference frame, the loss function at the current time step $t$ is formulated as a rate-distortion (RD) form, i.e.,

$$L_t=\lambda \cdot k_t+D_t=\lambda (k_t^{\mathrm{pl}}+k_t^{\mathrm{ml}})+D_t,$$ (18)

where $\lambda$ controls the trade-off between the total channel bandwidth cost $k_t$ and the distortion $D_t$.

Due to the conditional coding architecture, the performance of a previous frame will affect its subsequent frame. Therefore, during the training phase of DVST, we take into account the correlations of frames within a GOP. The DVST model can thus learn to allocate channel bandwidth resources within one frame and among various frames. Thus, the overall training loss function is formulated as

$$L=\frac{1}{N}\sum _{t=1}^N\left(\lambda \cdot k_t+D_t\right)=\frac{1}{N}\sum _{t=1}^N\left(\lambda (k_t^{\mathrm{pl}}+k_t^{\mathrm{ml}})+D_t\right).$$ (19)

The training procedure details will be introduced in the subsequent subsection.

For machine-type semantic communications, our DVST can directly drive the downstream machine vision tasks while preserving the advantages of signal level reconstruction. Different from the functional transmission mode adopted in [19], this paper aims to transmit videos friendly to both human vision and machine analytics [34]. Therefore, we incorporate the low-level signal distortion and the loss of high-level tasks, thus, the distortion term is reformulated as $D_t=D_{t,\mathrm{rec}}+\beta D_{t,\mathrm{task}}$, where $D_{t,\mathrm{rec}}$ denotes the reconstruction loss and $D_{t,\mathrm{task}}$ denotes the loss of downstream task.

III Architectures and Implementations

In this section, we present details of the adopted network architectures to implement our DVST. Then, we introduce the progressive training strategy to enable a stable model learning.

III-A Network Architectures

We illustrate ANN implementation details of the primary link in Fig. 4, including the contextual nonlinear transform modules and the contextual deep JSCC modules. For brevity, we do not repeat the structure of the motion link since it has almost the same architecture as the primary link except that the MV is of two channels, and the contextual operations are removed.

${\ddot{L}}_t^{\mathrm{mv}}$ (20)

$\text{with}{\widetilde{𝐲}}_t^{\mathrm{mv}}={𝐲}_t^{\mathrm{mv}}+𝐨,{\widetilde{𝐳}}_t^{\mathrm{mv}}={𝐳}_t^{\mathrm{mv}}+𝐨,{𝐲}_t^{\mathrm{mv}}=g_a^{\mathrm{mv}}({𝐦}_t),{𝐳}_t^{\mathrm{mv}}=h_{\mathrm{hpe}}^{\mathrm{mv}}({𝐲}_t^{\mathrm{mv}}),{\ddot{𝐦}}_t=g_s^{\mathrm{mv}}({\widetilde{𝐲}}_t^{\mathrm{mv}}).$

${\ddot{L}}_t$ (21)

$\text{with}{\widetilde{𝐲}}_t={𝐲}_t+𝐨,{\widetilde{𝐳}}_t={𝐳}_t+𝐨,{𝐲}_t=g_a({𝐱}_t|{\check{𝐜}}_t),{𝐳}_t=h_{\mathrm{hpe}}({𝐲}_t),{\ddot{𝐱}}_t=g_s({\widetilde{𝐲}}_t|{\widehat{𝐜}}_t),$

III-A1 Contextual Nonlinear Transform

Figure 5: Detailed structure of (a) semantic domain feature context generation networks, (b) deep JSCC codeword domain context generation networks.

The proposed DVST drives the current frame ${𝐱}_t$ in terms of the context extracted from MV rather than relying on a handcrafted subtraction operation. Fig. 5(a) illustrates the context generation network ${\phi }_a$ in semantic feature domain, which consists of feature extractor, warp, and refinement operation. After that, the analysis transform $g_a$ concatenates the Tx context ${\check{𝐜}}_t$ with the current frame ${𝐱}_t$ to learn a compact latent representation ${𝐲}_t$ in semantic feature domain. To estimate the spatial varying mean values and standard deviations of ${𝐲}_t$, we follow the hyperprior entropy model in [27], which fuses hierarchical prior, spatial prior, and temporal prior. The contextual synthesis transform $g_s$ has a symmetric architecture with $g_a$ except that it uses the Rx context ${\widehat{𝐜}}_t$ generated from ${\phi }_s$ which is of the same structure with ${\phi }_a$.

III-A2 Contextual Deep JSCC

Contextual deep JSCC module exploits the codeword contexts extracted from ${\check{𝐲}}_{t-1}$ and ${\widehat{𝐲}}_{t-1}$ to collaboratively transmit the current latent representation ${𝐲}_t$. It is of variable rates in accordance to the estimated entropy. In particular, the structure of codeword context generator ${\gamma }_e$ is shown in Fig. 5(b). Before the wrapping operation [33], we align the MV with the precoded feature by using an average pooling operation with stride 16. By using the learned entropy model, we pre-allocate the channel bandwidth cost of each spatial position $k_{t,i}$ as (11). Accordingly, the encoder $f_e$ fuses the ${𝐲}_t$ with the context ${\check{𝐝}}_t$ and partitions the fused feature map into patch embedding sequence $(y_{t,1},y_{t,2},\mathrm{\cdots },y_{t,I})$, where each embedding is an $M$-dimensional feature vector. After that, the practical channel bandwidth cost ${\overline{k}}_{t,i}$ for transmitting $y_{t,i}$ is determined as ${\overline{k}}_{t,i}=Q(k_{t,i})$. Herein, $Q$ denotes a scalar quantization whose range includes $2^q$ ($q=1,2,\mathrm{\dots }$) integers, and the quantization value set $𝒱=\{v_1,v_2,\mathrm{\dots },v_{2^q}\}$ is related to the scaling factor $\eta$ and the Lagrange multiplier $\lambda$ in the RD loss function. In this way, we inform the receiver which rate is allocated to each embedding $y_i$ by transmitting predetermined $q$ bits as extra side information.

Instead of naively training $2^q$ deep JSCC networks, we exploit the dynamic neural network structure to realize variable rate transmission. As shown in Fig. 4, $f_e$ consists of a powerful shared backbone to extract the contextual dependencies among $y_{t,i}$, and light FC layers to encode $y_{t,i}$ into the given dimension ${\overline{k}}_{t,i}$. In particular, we employ a group of $2^q$ FC layers with different output dimensions $\{v_1,v_2,\mathrm{\dots },v_{2^q}\}$, and each FC layer is invoked on demand. As a result, during the model forward pass, some FC layers may not be used while others may be used more than once. Additionally, to enhance the capacity of deep JSCC and extract the global and long-term correlations, we employ Swin Transformer blocks as the network backbone [35]. Inspired by the application of positional embeddings in vision Transformer, we develop a group of rate tokens $ℛ=\{r_{v_1},r_{v_2},\mathrm{\dots },r_{v_{2^{k_q}}}\}$ to indicate the CBR information. The rate tokens can be viewed as learnable parameters within Transformer. As shown in Fig. 4, each embedding $y_{t,i}$ will be added with its corresponding rate token $r_{{\overline{k}}_{t,i}}$ before fed into Transformer blocks. Hence, the output patch embeddings can get a better trade-off between fidelity and robustness. As a result, the following FC layer can efficiently rescale the dimensions of channel-input symbols $s_{t,i}$.

III-B Progressive Training Strategy

As aforementioned, the goal of DVST is to minimize a compromise between channel bandwidth cost (including primary link and motion link) and end-to-end distortion (reconstruction error or downstream task accuracy). Starting from a pretrained optical flow estimation network, the training procedure consists of the following steps:

1. Pretrain the nonlinear transform components of the motion link, including the motion estimation network, $g_a^{\mathrm{mv}}$, $g_s^{\mathrm{mv}}$, and the entropy model. In this step, we use the lossless previous frame as reference. The pretraining loss function ${\ddot{L}}_t^{\mathrm{mv}}$ of this step is formulated as (20), where $𝐨$ denotes uniformly sampled random quantization offsets.

2. Taking the wireless transmission error of the motion link into account, based on the previous step, deep JSCC codec $f_e^{\mathrm{mv}}$, $f_d^{\mathrm{mv}}$, and the rate adaptation module are jointly trained with nonlinear transform components to execute an MV transmission task. We also add the pretraining distortion terms ${\ddot{D}}_t^{\mathrm{mv}}$ to make the training process stable. The training loss function of this step is formulated as

   $$L_t^{\mathrm{ml}}={\ddot{D}}_t^{\mathrm{mv}}+D(\mathrm{warp}({𝐱}_{t-1},{\widehat{𝐦}}_t),{𝐱}_t)+\lambda k_t^{\mathrm{ml}},$$ (22)

   where the reconstructed MV ${\widehat{𝐦}}_t$ is obtained as (12) by passing over wireless channel, and the channel bandwidth cost of the motion link is obtained from (15).

3. Pretrain the nonlinear transform components of the primary link and context generation networks ${\phi }_a$ and ${\phi }_s$ meanwhile freezing the parameters of motion link. Similar to the pretraining process of motion link, we still employ the ideal lossless previous frame as reference, i.e., we manually set ${\check{𝐱}}_{t-1}={\widehat{𝐱}}_{t-1}={𝐱}_{t-1}$. In practice, following [27], we temporally remove the bitcost terms and readd them after several training epoches. This strategy helps model to generate useful contexts and exploit them, which actually accelerates the convergence. The training loss of this step is formulated as (21), where the generation of feature domain context ${\check{𝐜}}_t$ and ${\widehat{𝐜}}_t$ can refer to (16).

4. Based on the previous step, train the whole framework except the freezing motion link. The training loss of this step is formulated as

   $$L_t^{\mathrm{pl}}={\ddot{D}}_t+D({\widehat{𝐱}}_t,{𝐱}_t)+\lambda k_t^{\mathrm{pl}},$$ (23)

   where the reconstructed frame ${\widehat{𝐱}}_t$ is derived from (1), and the channel bandwidth cost of the primary link is obtained as (11). In this step, model learns to transmit current frame efficiently with the help of context ${𝐝}_t$ in JSCC codeword space. Also, for stable training, the end-to-end distortion ${\ddot{D}}_t$ is added.

5. Unfreeze the motion link and train the whole DVST model according to subsection II-E. The final training loss for a GOP is formulated as

   $$L=\frac{1}{N}\sum _{t=1}^N\left(\lambda (k_t^{\mathrm{pl}}+k_t^{\mathrm{mv}})+D({\widehat{𝐱}}_t,{𝐱}_t)+D({\ddot{𝐱}}_t,{𝐱}_t)\right),$$ (24)

   where the pretraining distortion $D({\ddot{𝐱}}_t,{𝐱}_t)$ serves as a regularization term to improve the training stability. In this step, the model indeed learns a bandwidth cost trade-off between primary and motion links. Moreover, due to the integrated training within a whole GOP, the rate allocation between frames has also been optimized, which will be shown in the ablation study of the subsequent section.

IV Experimental Results

IV-A Experimental Setup

IV-A1 Datasets

Our DVST model is trained with the Vimeo-90k dataset [36], which consists of 89800 video clips with a large variety of scenes and actions. During the model training, the chunks are randomly cropped to $256\times 256$ pixels. We use $N=7$ unrolled frames as a GOP in the last step of the training procedure and disallow the gradients passing from the I-frame reconstruction to the P-frame. We evaluate the performance of DVST using the HEVC test dataset [37] and the UVG dataset. As widely used standards to measure video-related algorithms’ performance, they contain sequences of various content, frame rate, and resolution. In particular, the HEVC dataset includes Class A ($2560\times 1600$), Class B ($1920\times 1080$), Class C ($832\times 480$), Class D ($416\times 240$), Class E ($1280\times 720$). And the UVG dataset consists of $7$ videos with the resolution of $1920\times 1080$. During model testing, we set the GOP size as $N=4$, which is identical to the end-to-end wireless video transmission scheme in [38]. As for I-frame coding, we adopt our previous work of image semantic transmission using nonlinear transform source-channel coding [9].

IV-A2 Implementation Details

In all experiments, the channel dimension $C$ in Fig. 4 is set to 96 for the primary link and 128 for the motion link. In addition, the channel dimension $C$ in Fig. 5 is 96. As mentioned before, we employ the Swin Transformer [35] as the backbone of the contextual deep JSCC codec, which greatly reduces the computation complexity of vision Transformers by conducting multi-head self-attention (MHSA) within local windows or shifted windows. In this paper, the number of Swin Transformer blocks is set to $N_e=N_d=4$, and we use 8 heads and $16\times 16$ window size in MHSA. In addition, the quantized channel bandwidth cost value set of primary link is chosen as ${𝒱}_{\mathrm{pl}}=\{0,2,4,6,8,10,15,20,26,32,40,48,56,64,80,96\}$, and the motion link ${𝒱}_{\mathrm{ml}}=\{0,1,2,4,8,16,32,48\}$. Hence, extra side information of total $q=7$ bits will be transmitted to inform the receiver of the CBR for each embedding. Since we adopt a large patch size of $32\times 32$ pixels, the side information cost is relatively trivial compared to video content. The composition of total CBR will be discussed in the ablation study.

For the reconstruction task, we optimize DVST in terms of the mean squared error (MSE) for the peak-signal-to-noise ratio (PSNR) metric, or multiscale structural similarity [39] (MS-SSIM) for perceptual quality. Multiple DVST models are trained with $\lambda \in \{1/4,1/8,1/16,1/32,1/64\}$ for MS-SSIM and $\lambda \in \{256,128,64,32,16\}$ for PSNR, thus achieving different RD tradeoffs. A smaller value of $\lambda$ leads to a larger CBR. We denote these models as “DVST (PSNR)” and “DVST (MS-SSIM)”, respectively. For each model, we use the Adam optimizer [40] with a learning rate of ${10}^{-4}$. We use a mini-batch size of 8, and it takes about one week to train the whole DVST model on single RTX 3090 GPU.

IV-A3 Comparison Schemes

Figure 6: PSNR performance versus the average channel bandwidth ratio (CBR) over the AWGN channel at $\text{SNR}=10$dB.

Following [38], we compare our DVST with classical video coded transmission schemes in current mainstream wireless communication systems. In particular, we employ the standard video codecs (H.264 [41] and H.265 [42]) for source coding combined with practical LDPC codes [26] or ideal capacity-achieving channel code family for channel coding. For brevity, we use “+” to concatenate the source coding and channel coding schemes, e.g., H.265 combined with capacity-achieving channel code is denoted as “H.265 + Capacity”. As we shall note, the ideal “H.264 + Capacity” or “H.265 + Capacity” scheme can be viewed as a performance upper bound on traditional separation-based source and channel coding schemes. The above simulations are implemented on the top of Sionna [43], an open-source library for the link-level simulation of digital communication systems. In addition, we refer to the configurations of H.264 and H.265 in [44], which adopt the typical ffmpeg settings for low-latency and veryfast mode. In practical implementation, to be aligned with previous works [17], we also convert two consecutive real symbols in $𝐬$ as one complex channel-input symbol and add complex Gaussian noise.

IV-B Reconstruction Task Results

Figure 7: MS-SSIM performance versus the average channel bandwidth ratio (CBR) over the AWGN channel at $\text{SNR}=10$dB. All MS-SSIM values are converted to dB ($-10{\log }_{10}(1-m)$, where $m$ is the MS-SSIM value in the range between 0 and 1).

Figure 8: PSNR and MS-SSIM performance versus the change of channel SNR over the AWGN channel. The CBR constraint for HEVC Class A sequence is set to $R=0.015$ and $R=0.025$ for Class D sequence.

Fig. 6 shows the RD results under the PSNR metric among various test sequences over the AWGN channel with channel $\text{SNR}=10$dB. For H.264 + LDPC and H.265 + LDPC, after traversing given combinations of LDPC coded modulation schemes, we exploit a $2/3$ rate $(4096,6144)$ LDPC code with 16-ary quadrature amplitude modulation (16QAM) to ensure reliable transmission and the highest efficiency [28]. We use the Gaussian capacity formula [1] to calculate the maximum transmission rate per channel symbol for the ideal H.264 + Capacity and H.265 + Capacity schemes. From Fig. 6, we can find that on most test sequences, the proposed DVST (PSNR) scheme can outperform the practical H.264 + LDPC scheme by a large margin for all CBRs, and the performance gap increases with the CBR which indicates a better coding gain achieved by our DVST method. Furthermore, the proposed DVST shows competitive performance to the H.265 + LDPC scheme and even performs close to H.265 + Capacity in some test sequences.

As for the coding gain that shows as the RD curve slope, by using the adaptive rate allocation and contextual transmission mechanism, our DVST model shows comparable coding gain as that of H.265/H.264 series in most cases. The coding gain generally increases with the resolution of the video sequence, which demonstrates the potential of DVST on transmitting higher resolution videos over wireless channels. However, we also note that DVST performs slightly worse than H.265 + LDPC on HEVC Class C and E. A possible reason is that many video sequences, e.g., BQMall, in these two Classes consist of complex foreground or various textures, which results in difficulty to the context generation in both semantic feature space and deep JSCC codeword space. As a comparison, our DVST performs better than H.265 + LDPC on HEVC Class A and D, which are of relatively flat foreground and simple textures.

Fig. 7 shows the RD performance in terms of MS-SSIM perceptual metric over the AWGN channel at $\text{channel SNR}=10$dB. Since MS-SSIM yields values between 0 (worst) and 1 (best), and most values are higher than 0.9, we converted the MS-SSIM values in dB to improve the legibility. For semantic communications, this perceptual metric aligns better with human feeling. Results indicate that the proposed DVST method can outperform classical schemes by a large margin, and it achieves a greater improvement on high-resolution images and high CBR regions. Compared to the PSNR results in Fig. 6, we can find that traditional video coded transmission series are inferior to the learning-based DVST because traditional video compression is designed to be optimized for squared error with hand-selected constraints.

Fig. 8 provides the PSNR and MS-SSIM results versus the change of channel SNR, where the CBR constraint for HEVC Class A sequence is $R=0.015$, and $R=0.025$ for Class D sequence. Since DVST learns an adaptive bandwidth allocation strategy depending on the video content and channel condition, it is difficult to strictly constrain the CBR to the predetermined value. In practice, based on the 10dB DVST models ($\lambda =64$ for PSNR, and $\lambda =1/32$ for MS-SSIM), we finetune ${\eta }_t$ to meet the CBR constraint in different SNRs. For comparison schemes, we evaluate the performance using all possible combinations of $(4096,8192)$, $(4096,6144)$, and $(2048,6144)$ LDPC codes with 4QAM, 16QAM, and 64QAM modulations. The solid blue line presents the envelope of the best performing configurations of H.265 + LDPC at each SNR. In Fig. 8, we use the solid red line to illustrate the performance of the DVST model trained with channel SNR at $-$2dB, 2dB, 6dB, 10dB, and 14dB, where the testing SNR equals to the training SNR. We also provide the performance of mismatched training and testing as the red dashed lines, where two models are trained under channel SNR 6dB and 10dB, respectively, but tested for various SNRs. We can find that the proposed DVST brings considerable performance gain. Comparing the three red lines, we observe that our DVST model also shows reasonable performance improvement with the increase of ${\text{SNR}}_{\text{test}}$ when ${\text{SNR}}_{\text{test}}>{\text{SNR}}_{\text{train}}$, and avoids catastrophic degradation when . In contrast, traditional separation-based video coded transmission schemes show significant cliff effect which are plotted as the PSNR-SNR curves of H.265 + LDPC in the blue dashdotted lines.

As for the slope of each curve in Fig. 8, our DVST shows better performance and comparable coding gain with that of H.264/H.265 series, especially on the high-resolution videos of HEVC Class A since it contains more high-frequency contents. Furthermore, we compare our DVST with the emerging neural video compression scheme DCVC [27] combined with LDPC codes for wireless transmission. For fair comparison, we use the same I-frame coding and GOP size and only compare the P-frame performance. Compared with DCVC + LDPC, we also achieve meaningful gain, which indicates our DVST can benefit from the good match between the learned deep JSCC and the nonlinear transform. Moreover, DVST does not rely on explicit entropy coding for compression and channel codes for error-correcting, which avoids the cliff effect and reduces the computational complexity, but DCVC + LDPC will also involve the cliff effect due to the use of entropy coding and left errors in LDPC decoding.

Figure 9: PSNR performance versus channel SNR over the Rayleigh fading channel. The CBR constraint for HEVC Class D sequence is set to $R=0.025$.

Figure 10: CBR and PSNR/MS-SSIM comparison between DVST and H.265 + LDPC. We use the last 12 frames of Kimono from HEVC Class B dataset as an example.

Figure 11: CBR savings for each video in all test sequences. We use the widely used Bjontegaard Delta (BD) rate reduction [45] algorithm to estimate the bandwidth savings. The rate saving value represents the percentage of channel bandwidth cost relative to H.264 + LDPC at the same PSNR (smaller is better), and this experiment is conducted at the AWGN channel with $\text{SNR}=10$dB. As the anchor, H.264 + LDPC scheme is always $100\%$ as plotted in the dashed horizontal line. In most cases ($21/25$), the proposed DVST can save more bandwidth compared to both H.264 + LDPC and H.265 + LDPC.

Original

H.264 + LDPC

H.265 + LDPC

DVST

(a) $R_{\text{GOP}}$ / PSNR (dB)

(b) 0.024 (0%) / 31.35

(c) 0.022 (–8.3%) / 31.93

(d) 0.015 (–37.5%) / 32.12

(e) $R_{\text{GOP}}$ / MS-SSIM (dB)

(f) 0.018 (0%) / 12.79

(g) 0.017 (–5.5%) / 13.02

(h) 0.016 (–11.1%) / 15.91

Figure 12: Examples of visual comparison. The first column shows the original frame. The second column shows the cropped patch in original frame. The third to the fifth column show the reconstructed frames by using different transmission schemes over the AWGN channel at $\text{SNR}=10$dB. Note that, the fifth column presents the cropped patch generated by DVST (PSNR) and DVST (MS-SSIM), respectively. $R_{\text{GOP}}$ denotes the average CBR of the current GOP. The blue number indicates the percentage of bandwidth cost saving compared to the baseline “H.264 + LDPC” scheme.

${\text{SNR}}_{\text{test}}=2\text{dB}$

${\text{SNR}}_{\text{test}}=6\text{dB}$

${\text{SNR}}_{\text{test}}=10\text{dB}$

${\text{SNR}}_{\text{test}}=14\text{dB}$

(a) DVST (${\text{SNR}}_{\text{train}}=2\text{dB}$)

(b) DVST (${\text{SNR}}_{\text{train}}=6\text{dB}$)

(c) DVST (${\text{SNR}}_{\text{train}}=10\text{dB}$)

(d) DVST (${\text{SNR}}_{\text{train}}=14\text{dB}$)

(e) Original

(f) H.265+1/2 LDPC+4QAM

(g) H.265+2/3 LDPC+4QAM

(h) H.265+2/3 LDPC+16QAM

(i) H.265+1/2 LDPC+64QAM

Figure 13: Examples of visual comparison. The first column shows the original frame and its cropped patch. The second to the fifth column shows the frames reconstructed by DVST (PSNR) or H.265 + LDPC over the AWGN channel at various channel SNRs, respectively. For the DVST (PSNR), we consider the condition when ${\text{SNR}}_{\text{test}}={\text{SNR}}_{\text{train}}$. For the H.265 + LDPC scheme, the rate of LDPC and the order of QAM modulation is marked below the reconstructed frame. The average CBR of GOP is limited to 0.012.

Next, we show the PSNR performance under the Rayleigh fading channel with CBR constraint $R=0.025$ in Fig. 9. In this case, we assume the Rayleigh fading channel gain vector ${𝐡}_t\sim 𝒞𝒩(\mathrm{𝟎},{𝐈}_{k_t})$, and it is known at the receiver with ideal channel estimation. Thus, the receiver first conducts channel equalization, such that the received signal can be equivalently written as ${\widehat{𝐬}}_t={𝐬}_t+{𝐧}_t/{𝐡}_t$, and then feed ${\widehat{𝐬}}_t$ into DVST decoder. In practice, our DVST models of the Rayleigh fading channel are finetuned from baseline models learned under the AWGN channel with the same SNR. Apparently, classical separation schemes (H.264/H.265 + LDPC + QAM) are still inferior to our DVST, especially in the low SNR region.

Fig. 10 shows a detailed comparison between DVST and H.265 + LDPC over the performance of a group of consecutive frames. It can be observed that the reconstruction quality of both schemes degrades with the increase of P-frame number within one GOP. In comparison, our DVST can spend fewer channel bandwidth costs while achieving much better reconstruction quality.

Furthermore, as shown in Fig. 11, we compute the BD rate reduction [45] relative to H.264 + LDPC for each video under AWGN channel at SNR = 10dB. Compared with H.264 + LDPC, the channel bandwidth cost of DVST is only 40% to 80% under the same reconstruction quality in terms of PSNR, which means a 60% to 20% the channel bandwidth can be saved. Compared with the results of H.265 + LDPC, DVST can still save more bandwidth in most videos ($21/25$).

Fig. 12 and Fig. 13 provide illustrative examples to demonstrate the performance of DVST intuitively. Specifically, as shown in Fig. 12, we visualize the specific reconstructed frames of Fig. 6 and Fig. 7 in the two rows, respectively. From the two groups of examples, we can observe that our DVST model generates high fidelity reconstructions with lower CBR costs. Fig. 13 presents the reconstruction results versus the change of channel SNR under a limited CBR budget. It can be seen that the results of H.265 + LDPC + QAM scheme have artifacts and block effects in the low SNR region, while DVST generates a clear text.

Figure 14: Ablation study results in terms of MS-SSIM over the AWGN channel at SNR = $10$dB. The hyperparameter $\lambda$ and the CBR percentage of motion link are also marked on the DVST curve.

(a) (a)

(b) (b)

Figure 15: Reconstruction and segmentation performance versus the average CBR over the AWGN channel at $\text{SNR}=10$dB.

Ground Truth

H.264 + LDPC

H.265 + LDPC

DVST

Figure 16: Examples of semantic segmentation results. The first two columns present the original frame and its label. The third to the fifth column shows the segmentation results from reconstructions of H.264 + LDPC or H.265 + LDPC or DVST (PSNR), respectively. The experiment is carried out over the AWGN channel with $\text{SNR}=10$dB and the average CBR of GOP is constrained to 0.009.

As for the ablation study, we verify the gains brought by proposed algorithms in Fig. 14, including the contextual model enhancement, rate-adaptive transmission, and GOP integrated training strategy. The bandwidth cost trade-offs between primary and motion links are also provided. When the $\lambda$ becomes smaller, the DVST model tends to allocate more channel bandwidth to reduce the distortion in the primary link while the percentage of channel bandwidth in the motion link decreases. This result means it is more efficient to focus on the primary link in the high CBR region since this link can preserve more details. We verify the effectiveness of the GOP training strategy, which considers the influence of temporal correlations among adjacent frames. Comparing the results of DVST with that of DVST (without (w/o) GOP training), about 0.2dB reconstruction quality improvement can be seen. To verify the advantages of contextual video transmission, we remove the context generation network of DVST and adopt the traditional residual coding structure [44] as an alternative, which exploits MV to warp the reference frame and transmit the residual between the wrapped frame and the input frame. This contextual model enhancement method in DVST improves the whole performance compared to traditional residual coding structure. Finally, we invalidate the rate adaption module in $f_e$, $f_e^{\mathrm{mv}}$, $f_d$, and $f_d^{\mathrm{mv}}$ by using a constant channel bandwidth cost for each patch embedding and deleting the additional rate tokens. Since the network can no longer learn a bandwidth cost trade-off between primary and motion links, the CBR proportion of the motion link is predetermined to $1/3$. Given the channel bandwidth cost for each patch embedding, the total CBR is then fixed, so the loss function of DVST (w/o rate allocation) only has the distortion term. As shown in Fig. 14, our DVST overpasses the DVST (w/o rate allocation) by a large margin, especially in the high CBR region, verifying the coding gain brought by the proposed rate-adaptive transmission mechanism. Furthermore, we provide the ablation study about the motion link. The DVST (w/o motion link) extracts the context semantics solely from previous reconstructions. In this case, the whole system architecture can be vastly simplified. However, without the guidance of optical flow, it will be difficult for the context generation module to extract valuable information to exploit the spatio-temporal dependencies, resulting in significant performance degradation.

To compare the computational complexity of different video transmission systems, we measured the average encoding time of DVST on a Linux server with an Intel Xeon Gold 6226R CPU and a RTX 3090 GPU. Following the complexity analysis from [44, 27], we transmit five videos in the 1080P HEVC Class B dataset and measure the encoding speed. As a result, our DVST model spends 280ms to encode a single P-frame as channel-input symbols, which is one times faster than DCVC + LDPC, mainly due to the savings in arithmetic coding time. It is worth mentioning that the coding speed can be further improved by employing the latest deep model acceleration techniques, which is beyond the scope of this paper. As a comparison, the scheme of H.265 + LDPC runs at the speed from 1.5fps (frames per second) to 25fps with different coding settings (the trade-off between the coding efficiency and encoding speed), and the encoding speed of H.264 + LDPC is 8fps to 150fps. Note that, both H.264 and H.265 are implemented using commercial softwares with highly parallel framework and advanced assembly optimization techniques, while their official reference software runs hundreds of times slower [46].

IV-C Downstream Machine Vision Task Results

To potentially support future machine-type communications [34], we further optimize DVST for driving the downstream machine vision tasks while preserving the advantages of signal level reconstruction. Specifically, an analytics model is concatenated after the transmission framework to complete high-level semantics-related tasks based on the reconstructed frames. The transmission framework can be either a JSCC scheme like DVST or classical separated system like H.265 + LDPC. Herein, we take semantic segmentation as an example visual analytic task, employ HyperSeg [47] as a powerful analytics model to generate segmentation prediction, and evaluate the capacity of our DVST on the popular CamVid benchmark [48, 49]. CamVid is a road scene understanding dataset, which offers four video clips of driving scenes, and part of frames are densely semantic annotated. Following the training protocol of [47], we use 468 annotated images as the training set and the other 233 ones as the test set. Each video and the labeled images are resized to $256\times 192$. To achieve a better rate-distortion-accuracy trade-off, we finetune the DVST (PSNR) model over the CamVid training set. The whole training loss extends (19) to achieve joint optimization of both video transmission and analysis. It is formulated as $L=\frac{1}{N}{\sum }_{t=1}^N\left(\lambda k_t+D_t+\beta D_{t,\mathrm{seg}}\right)$, where $D_{t,\mathrm{seg}}$ denotes the boot-strapped cross entropy loss [50] specialized for semantic segmentation. $\beta$ is the weighting parameter, we set $\beta =64$ to balance the importance among reconstruction and segmentation. During the evaluation on the test set, all frames are transmitted to the receiver to calculate signal level distortion in terms of PSNR, while only the labeled frame participates in the calculation of segmentation performance (the labeled frame evenly distributes in the second or last frame of GOP). We report the class mean intersection over union (mIoU) results, a standard evaluation metric for semantic image segmentation [47]. A higher mIoU score indicates a better match between prediction and ground truth, with a maximum value of 1.

The quantitative results are shown in Fig. 15. We observe that our DVST achieves better performance for both reconstruction and segmentation with various CBRs. It outperforms the two separated coding schemes and the gap increases with CBR. Hence, our DVST method can better support machine vision tasks and hold higher fidelity for human vision at the same time. In addition, we present a group of segmentation example in Fig. 16. The reconstruction of DVST preserves more semantic information for machine recognition, which leads to more accurate segmentation results.

V Conclusion

This paper has proposed a new class of high-efficiency deep JSCC methods to achieve end-to-end video transmission over wireless channels. It was collected under the name “DVST”. This DVST framework has exploited nonlinear transform and conditional coding architecture to adaptively extract semantic features across video frames, and transmit semantic features via a group of learned variable-length deep JSCC codecs and wireless channel. Benefiting from the strong temporal prior provided by the semantic feature domain context and the deep JSCC codeword domain context, the DVST framework works highly efficient and effective. The whole video transmission system design has been formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under established perceptual quality metrics or downstream task metrics, which well matches with the goal of end-to-end semantic communications. Extensive numerical results have shown that the proposed DVST method can generally surpass traditional wireless video coded transmission schemes. In a nutshell, this paper has proposed a promising method to attain a customized design of learning-based source-channel coding for video transmission in future semantic communications.