Latency-Aware Generative Semantic Communications with Pre-Trained Diffusion Models

Abstract

Generative foundation AI models have recently shown great success in synthesizing natural signals with high perceptual quality using only textual prompts and conditioning signals to guide the generation process. This enables semantic communications at extremely low data rates in future wireless networks. In this paper, we develop a latency-aware semantic communications framework with pre-trained generative models. The transmitter performs multi-modal semantic decomposition on the input signal and transmits each semantic stream with the appropriate coding and communication schemes based on the intent. For the prompt, we adopt a re-transmission-based scheme to ensure reliable transmission, and for the other semantic modalities we use an adaptive modulation/coding scheme to achieve robustness to the changing wireless channel. Furthermore, we design a semantic and latency-aware scheme to allocate transmission power to different semantic modalities based on their importance subjected to semantic quality constraints. At the receiver, a pre-trained generative model synthesizes a high fidelity signal using the received multi-stream semantics. Simulation results demonstrate ultra-low-rate, low-latency, and channel-adaptive semantic communications.

I Introduction

Semantic Communication (SemCom) is poised to play a pivotal role in shaping the landscape of future AI/ML-driven communication systems. SemCom systems aim to significantly reduce the transmission rate by extracting and transmitting only the semantic message of interest based on the communication intent. Generative AI (GenAI) models have recently proved to significantly enhance communication at the semantic-level [2, 3, 4]. GenAI models such as Diffusion [5, 6], Flow-based [7], and GAN [8] models, can learn the general distribution of natural signals through training and can generate new samples at the inference time. This generative process can be guided or conditioned to synthesize outputs with a desired semantic content. In Generative Semantic Communications (Gen SemCom), the semantics of interest are extracted at the transmitter, communicated over the channel, and then used at the receiver to guide a generative model to synthesize a semantically consistent signal with high fidelity. GenAI models are trained to maximize the perceptual quality and the fundamental bounds on Generative SemCom are governed by the rate-distortion-perception theory [9, 10], which determines the threefold trade-off between rate, distortion, and perceptual quality of the reconstructed signal.

The recent advent of powerful Generative Foundation Models provides ample opportunities to develop ultra-low-rate semantic communication systems. The ultra low rate transmission can be achieved by transmitting data semantics in compressed format as a textual message or prompt. For instance, the prompt “Teddy bear surfer rides the wave in the tropics” can be used to generate a short video with its semantic content matching with the prompt. The generative foundation models such as Sora [11], Lumiere [12], and DALL.E [13] are pre-trained on large amount of data and can synthesize various types of AI Generated Content (AIGC) with high quality. The pre-trained nature of such models and their applicability to a vast range of multi-media synthesis tasks, has the potential to revolutionize generative semantic communications enabling universal intent and channel-adaptive SemCom systems empowered by pre-trained foundation models.

In this paper, we develop a universal generative semantic communications framework with pre-trained foundation models. We claim two key benefits in comparison with the existing semantic communication frameworks. Firstly, the foundation models possess a vast general knowledge leveraging the intensive self-supervised training process on huge amount of data. This alleviates the need for a shared knowledge base/graph between the semantic transmitter and receiver, obviating the need for corresponding knowledge sharing overheads imposed in current SemCom frameworks [14, 16, 17, 15, 18]. This vast general knowledge also makes the proposed Generative SemCom framework applicable to various datasets and tasks thereby achieving universality. Secondly, the adoption of pre-trained models allows a separation-based SemCom architecture, alleviating the need for end-to-end joint training of the transmitter and receiver, which is required in many SemCom frameworks [14, 16, 17, 15]. Such a separation-based architecture offers better compatibility with the existing design of wireless communication networks in comparison with the end-to-end methods. The proposed framework is specifically suitable for scenarios which require communication of huge multi-modal data with stringent latency and reliability requirements, e.g. the wireless metaverse, extended/mixed reality (XR/MR), holographic teleportation, and the internet of senses. The contributions of this work are three-fold:

• We develop a semantic decomposition scheme at the transmitter which extracts the semantic content of the input signal in multiple semantic modalities. We extract the most important semantic contents as a compact textual message or prompt, along with multiple other modalities that act as conditioning signals to guide the synthesis process at the generative foundation model at the receiver.

• To achieve semantic-aware communication, we design a multi-stream scheme that transmits each extracted semantic modality with appropriate coding and communication techniques based on communication intent. Due to the importance of the textual prompt, a re-transmission scheme is applied to ensure reliable reception, while other modalities are transmitted with a modulation scheme adapted to the varying wireless channel.

• We design a semantic and latency-aware scheme to allocate the transmission power to different semantic streams based on semantic importance, and to adapt the modulation order to the varying wireless channels.

Notations: Boldface lower and upper-case symbols denote column vectors and matrices, respectively. $[K]$ denotes $\{1,2,\mathrm{\dots },K\}$. ${[𝐯]}_n$ and $\left|𝐯\right|$ denote the $n$-th element and the length of vector $𝐯$, respectively. Finally, $𝐯\cdot 𝐯$ is the dot production, and $Pr\{\cdot \}$ denotes probability of an event.

Figure 1: The Proposed Framework for Latency-aware Multi-stream Semantic Communication with Multi-Modal Generative Models.

II Generative Foundation AI-based Semantic Communication

Fig. 1 depicts our proposed framework for generative semantic communication with pre-trained foundation models. This framework includes multi-modal semantic decomposition/synthesis, semantic-aware multi-stream transmission, and latency-aware semantic power allocation.

II-A Multi-modal Semantic Decomposition and Synthesis

At the transmitter, a pre-trained textual transform encoder performs ultra-low-rate source-to-text transformation, extracting the textual message which acts as a prompt for the GenAI process at the receiver. Depending on the source signal (e.g. voice, image, video, point cloud, etc.), various large, medium, and small scale pre-trained AI models can be used for prompt extraction. The prompt contains the most important semantic contents of the source signal. At the same time, additional semantic information is extracted from the source signal based on communication intent. They are typically extracted in modalities other than text and provide additional conditioning signals to guide the GenAI process at the receiver. For example, if the source signal to be transmitted is a short video, the initial frame of that video can be a good conditioning signal to be transmitted along with the prompt to guide the generation process at the receiver. As another example, if the source signal is an image and the receiver is interested in the relative location of different objects in the image, an edge map can be a good conditioning signal, as discussed later in Section III.

The decomposed multi-modal semantics, i.e., prompt and conditioning signals, are then compressed, coded, and modulated, and transmitted over the wireless channel. At the receiver, after demodulation, decoding, and decompression, the received multi-modal semantics are simultaneously fed into a pre-trained generative model for high-fidelity synthesis of the source signal. We note that state-of-the-art generative models, e.g. stable diffusion (SD) [5], allow generation using only the prompt, which results in a relatively high distortion. On the other hand, some semantic information content of the source signal can be more efficiently communicated using other modalities. The conditioning signals improve the semantic fidelity as well as distortion at additional communication costs. Finally, the desired dimensions of the signal to be synthesized by GenAI are assumed to be fixed and known to the receiver.

II-B Semantic-aware Multi-Stream Transmission

The transmitter sends the extracted multi-modal semantics, i.e., prompt and conditioning signals, in multiple streams over orthogonal frequency channels to enable semantic-aware design of specific transmission mechanisms, e.g. coding rate, modulation order, for different semantic modalities. Since the prompt is a compact textual message and contains the most important semantic content of the source signal, we use lossless compression for prompt to obtain a bit-stream ${𝐯}_0$. However, for other semantic modalities we use state-of-the-art Deep Neural Network (DNN)-based techniques for lossy compression as will be discussed in Section III. The bit-stream of the $i$-th conditioning signal is denoted by ${𝐯}_i$ , $i\in [I]$.

We consider the Rayleigh fading channel and assume that the channels experience block fading within the transmit duration of each semantic modality. In particular, the channel gain of the $i$-th semantic modality is expressed as

$h_i=\sqrt{{ϵ}_o{\left(d\right)}^{-\phi }}\tilde{h},\forall i\in \{0\}\cup [I],$ (1)

where $i=0$ (resp. $i\in [I]$) denotes the prompt (resp. conditioning signals), $\tilde{h}$ is a random scattering element captured by zero-mean and unit-variance circularly symmetric complex Gaussian (CSCG) variables, ${ϵ}_o$ is the path loss at the reference distance $d_0=1$ m, $\phi$ is the path loss exponent, $d$ is the distance. Furthermore, the received signal-to-noise ratio (SNR) at the receiver for the $i$-th semantic modality is given by ${\gamma }_i(p_i)=\frac{p_i{|h_i|}^2}{B_i{N}_0},\forall i\in \{0\}\cup [I]$, where $p_i$ and $B_i$ denote the transmit power and the bandwidth allocated for the $i$-th modality, respectively, and $N_0$ is the noise power spectral density. For notational simplicity, we use ${\gamma }_i$ instead of ${\gamma }_i(p_i)$, $\forall i\in \{0\}\cup [I]$, in the sequel.

II-B1 Re-transmission-based Communication of the Textual Prompt

The textual prompt is a super compact message that is highly sensitive to errors. Even a single bit error can change a character/word causing a significant semantic error. Thereby, our framework uses re-transmissions if the prompt is received with any error. Based on this, we propose a cyclic redundancy check (CRC)-based re-transmission mechanism for the prompt to guarantee its reliable communication. Assuming the prompt is transmitted in packets of length $L$ bits, the packet error rate (PER) can be expressed as [19]

$$\text{PER}(p_0)\approx 1-L^{-k/{\gamma }_0}\exp \left(-\frac{b}{{\gamma }_0}\right),$$ (2)

where $k$ and $b$ are parameters determined by the channel coding scheme and the modulation order. We denote the coding rate and the modulation order as $r$ and $M_0$, respectively. Hence, the expected total transmission delay for the prompt is given by

$$T_0(p_0)={\eta }_P\cdot {\eta }_R\cdot {\eta }_T=\frac{\left|{𝐯}_0\right|}{rL}\cdot \frac{1}{1-\text{PER}(p_0)}\cdot \frac{L}{{\log }_2(M_0)B_0},$$ (3)

where ${\eta }_P=\frac{\left|{𝐯}_0\right|}{rL}$, ${\eta }_R=\frac{1}{1-\text{PER}(p_0)}$, ${\eta }_T=\frac{L}{{\log }_2(M_0)B_0}$ denote the number of packets, the number of re-transmissions, and the transmission delay of each packet, respectively.

II-B2 Adaptive Modulation and Coding for other Semantic Modalities

For the transmission of the other semantic modalities, we adopt an adaptive MQAM modulation and coding scheme to cope with varying wireless channel. Denote the bit error rate (BER) of the $i$-th, $i\in [I]$, conditioning signal as ${\text{BER}}_i=\frac{1}{|{𝐯}_i|}{\sum }_{n=1}^{|{𝐯}_i|}Pr\{{[{\widehat{𝐯}}_i]}_n\ne {[{𝐯}_i]}_n\}$, where ${\widehat{𝐯}}_i$ denotes the corresponding received data stream of the conditioning signal. With such a scheme, the achievable rate can be generally expressed as [20]

$$R_i(p_i,{\text{BER}}_i)=B_i{\log }_2\left(1+\frac{\alpha }{-\ln \left(\beta {\text{BER}}_i\right)}{\gamma }_i\right),i\in [I],$$ (4)

where $\alpha$ and $\beta$ are parameters determined by the coding schemes. Thereby, the transmission delay of $i$-th semantic stream is $T_i(p_i,{\text{BER}}_i)=\frac{\left|{𝐯}_i\right|}{R_i(p_i,{\text{BER}}_i)}$, $\forall i\in [I]$.

II-C Latency-aware Adaptive Semantic Communication

In this subsection, we propose a semantic and latency-aware approach for selecting the optimal communication parameters such as the transmission power, and modulation order, for various semantic modalities, tailored to the prevailing channel quality as well as the specified semantic quality requirements. We formulate the problem as minimizing the transmission delay under the power and semantic quality constraints given as

$\underset{\left\{p_0,p_i,i\in [I]\right\}}{\min }$ $\{\max \{T_0(p_0),T_i(p_i,{\text{BER}}_i),i\in [I]\}\}$ (5)

s.t. ${\displaystyle {\sum }_{i=0}^I}{p}_i\le P_{\text{T}},$

${\mathrm{\Phi }}_j\left({\text{BER}}_i,i\in [I]\right)\ge {\epsilon }_j,j\in [J],$

where $P_{\text{T}}$ is the maximum transmit power budget, ${\mathrm{\Phi }}_j$ and ${ϵ}_j$ denote the $j$-th semantic quality metric and its requirement, respectively. Since reliable transmission of the prompt is assumed via re-transmissions, ${\mathrm{\Phi }}_j$, $\forall j\in [J]$, is a multi-variable function of the BERs of the $I$ conditioning signals, denoted as ${\mathrm{\Phi }}_j\left({\text{BER}}_i,i\in [I]\right)$. The quality requirements can represent any semantic distortion or perception [9, 10] requirement of the signal synthesized at the receiver, and can be enforced by various metrics, e.g. MS-SSIM [21], LPIPS [22], FID [23], CLIP [24], etc. The semantic quality constraints represent epigraphs of the metrics, and in the most general case, convexity of problem (5) depends on the behaviour of the selected metrics in terms of ${\text{BER}}_i,i\in [I]$. In the typical scenario, where the prompt is transmitted along with only one additional modality, (5) is convex with a straightforward solution as given in Lemma 1.

Lemma 1.

For the case of one conditioning signal, i.e., $I=1$, if ${\mathrm{\Phi }}_j({\text{BER}}_1)$, $\forall j\in [J]$, are monotonically non-increasing with respect to ${\text{BER}}_1$, (5) is a convex problem. The optimal solution can be achieved if and only if $p_0+p_1=P_{\mathrm{T}}$, $T_0(p_0)=T_1(p_1,{\text{BER}}_1)$, and ${\text{BER}}_1=\min \{{\mathrm{\Phi }}_j^{-1}({\epsilon }_1),j\in [J]\}$, where ${\mathrm{\Phi }}_j^{-1}$ denotes the generalized inverse function of ${\mathrm{\Phi }}_j$, $\forall j\in [J]$.

Proof.

The epigraph of a single-variable function is convex iff the function is (weakly) monotonic. Hence, if ${\mathrm{\Phi }}_j({\text{BER}}_1)$, $\forall j\in [J]$, are monotonically non-increasing, then the intersection of their epigraphs determines a convex region with respect to ${\text{BER}}_1$, given by ${\text{BER}}_1\le \min \{{\mathrm{\Phi }}_j^{-1}({\epsilon }_1),j\in [J]\}$ where ${\mathrm{\Phi }}_j^{-1}$ exists because ${\mathrm{\Phi }}_j$ is (weakly) monotonic. Moreover, since ${\text{BER}}_1$ is a decreasing function of $p_1$ itself, the constraints collectively determine a convex feasible region in terms of $p_0,p_1$. Using (3) and (4), the latency functions $T_0,T_1$ are also convex in $p_0,p_1$, and thereby the optimization is convex with a global solution. Finally, it is straightforward to show that in the optimum solution, all constraints hold with equality and we have $T_0=T_1$. Hence, the optimum solution can be found by solving the non-linear system of equations $T_0(p_0)=T_1(p_1,{\text{BER}}_1)$, $p_0+p_1=P_{\mathrm{T}}$, and ${\text{BER}}_1=\min \{{\mathrm{\Phi }}_j^{-1}({\epsilon }_1),j\in [J]\}$ in $p_0,p_1$. The overall transmission delay $T$ is then $T=T_0(p_0)=T_1(p_1,{\text{BER}}_1)$. ∎

III Simulation Results

III-A System setup

We consider a generative image semantic communication setting, where the receiver is interested in the main objects inside the image, e.g. “car”, “building”, and its general structure, i.e. object shapes and locations. In this context, we consider a two-modal semantic decomposition framework, including a textual prompt to convey the main objects, and an edge map as the conditioning signal to convey the general image structure. To extract the prompt, we adopt the GPT-4 model [25] in this work. We also adopt the holistically-nested edge detection model [29], to extract a gray-scale edge map from the original image at the transmitter. The prompt is encoded to bits using UFT-8 [30], and the edge map is compressed to bits using learned nonlinear transform coding [31]. At the receiver, we adopt the state-of-the-art pre-trained SD model [5] for conditional image generation from the received prompt and the edge map. The wireless transmission parameters for the two semantic streams are listed in Table I.

TABLE I: Parameter settings

Parameters	Values
Packet length of prompt, $L$	$\left|{𝐯}_0\right|$ bits
Coding rate of prompt, $r$	1/2 (convolutional code)
Modulation order of prompt, $M_0$	4 (QPSK)
$\{k,b\}$ in (2)	$\{0.374,-0.31\}$
$\{\alpha ,\beta \}$ in (4)	$\{1.5,5\}$
Transmit power budget, $P_{\text{T}}$	10 mW
Path loss at $d_0=1$ m, ${ϵ}_o$	-30 dB
Path loss exponent, $\phi$	3.4
Noise power density, $N_0$	-174 dBm/Hz
Bandwidth, $B_0=B_1$	1 MHz

(a)

(b)

Figure 2: Normalized CLIP and MS-SSIM versus BER of the edge map: (a) Prompt generated by GPT-4; (b) Prompt generated by BLIP.

III-B Semantic quality metrics

Various reference-based or no-reference metrics can be adopted to evaluate the quality of the synthesized signal based on the semantic communication intent. In this work, we adopt two metrics, one to measure the semantic similarity, and the other to measure the structural similarity between the original and the synthesized images at the Gen SemCom receiver. In order to assess the quality of the most important semantic information conveyed by the prompt, we use the contrastive language-image pretraining (CLIP) model [24]. Specifically, we define the CLIP metric as the cosine similarity between the CLIP embedding of the original and synthesized images, i.e. $f(𝐱)$ and $f(\widehat{𝐱})$, given by $\text{CLIP}=(\frac{f(𝐱)\cdot f(\widehat{𝐱})}{{\Vert f(𝐱)\Vert }_2\cdot {\Vert f(\widehat{𝐱})\Vert }_2}+1)/2$. For structural similarity, we use the multi-scale structural similarity (MS-SSIM) [21] metric.

III-C Monotonicity of the quality metrics

The end-to-end performance of the proposed framework depends on behaviour of the pre-trained GenAI model at the receiver, i.e. its sensitivity to the BER of the conditioning signal, and the requirements of the semantic quality metrics. Hence, we investigate monotonicity of the CLIP and MS-SSIM metrics to apply Lemma 1 for latency-aware adaptive SemCom. Specifically, through intensive simulations in presence of random bit errors in the proposed framework, we demonstrate that the normalized CLIP/MS-SSIM metrics are monotonically non-increasing functions of the BER of the edge map, as depicted in Fig. 2. Note that the proposed framework works independent of the choice of the AI models adopted at the transmitter and receiver. For example, the GPT-4 model can be replaced by smaller image-to-text models, e.g. BLIP [26], Oscar [27], UNIMO [28], at the cost of a slight degradation of the semantic quality metrics. These smaller models can be implemented locally on-device. The absolute CLIP values achieved with GPT-4 and BLIP, under error-free transmission of the edge map, are $0.918$ and $0.896$, respectively, while the resulting MS-SSIM values are similar. The normalized CLIP and MS-SSIM are defined as $\frac{\text{CLIP}}{{\text{CLIP}}_0}$ and $\frac{\text{MS-SSIM}}{{\text{MS-SSIM}}_0}$, respectively, where CLIP₀ and MS-SSIM₀ are the reference values of these metrics for perfect transmission of the edge map. We note that “CLIP, Prompt + Edge Map” surpasses “CLIP, Prompt Only” when $\text{BER}\le {10}^{-4}$, indicating that transmitting the edge map enhances the semantic quality as interpreted by the SD model. On the contrary, when $\text{BER}>{10}^{-3}$, the inaccuracies in the received edge map due to the communication errors become detrimental, causing the SD model to misinterpret the prompt semantics. For example in Fig. 3, the “fence” is missing in the generated image due to the inaccuracies of the received edge map at $\text{BER}={10}^{-3}$, despite the word “fence” being part of the prompt. Moreover, transmitting the edge map alongside the prompt invariably boosts the MS-SSIM in comparison to relying solely on prompt.

Figure 3: Visualization of the semantic quality of our proposed framework. The absolute CLIP/MS-SSIM values are reported in this figure for comparison.

III-D Visual quality of the proposed framework

In Fig. 3, we provide illustrations to demonstrate the visual quality of our proposed framework on a sample natural image. These results demonstrate a good semantic quality for the proposed framework for ultra-low-rate transmission at bit per pixel (bpp) values as low as $0.0024$ and $0.017$ for the prompt and the edge map, respectively.

III-E Latency-aware Adaptive Semantic Communication

We evaluate the optimal semantic latency in terms of the average SNR, i.e. $\overline{\gamma }=\frac{p_{\text{T}}𝔼({|h_0|}^2)}{(B_0+B_1)N_0}$, as a channel quality indicator. With the investigation in subsection III-C, we apply Lemma 1 and solve the resulting nonlinear system of equations numerically. Fig. 4 quantifies the semantic quality-latency trade-off for the proposed Gen SemCom framework, and illustrates the optimal wireless parameters versus average SNR $\overline{\gamma }$ at various target semantic quality, i.e. normalized (CLIP, MS-SSIM), and BER values. It is evident that to achieve a higher target semantic quality (i.e. higher CLIP/MS-SSIM and lower BER) one should allocate more power to the transmission of the edge map, which in turn, increases the latency. Furthermore, when the channel SNR deteriorates, the sender should use a lower modulation order for the edge map to maintain the semantic quality at a target acceptable level. This will also increase the expected number of re-transmissions for the prompt.

For performance comparison, we adopt a semantic-unaware transmission benchmark, in which the prompt and edge map are treated equally and transmitted as one data stream regardless of the unequal importance of each semantic modality. For a fair comparison, the maximum power $p_{\mathrm{T}}$ and bandwidth $B_0+B_1$ are allocated for the transmission of this single stream with adaptive MQAM employed. The prompt BER becomes the bottleneck for single-stream transmission, and hence, the BER is kept at ${10}^{-7}$ to ensure reliable reception of the prompt. From Fig. 4(b), the proposed semantic-aware multi-stream transmission significantly reduces the transmission latency at the lower SNR region, i.e.,  dB. Finally, in Table II, we provide modulation adaptation guidelines to achieve various semantic qualities (CLIP, MS-SSIM) in varying channel qualities. Here, $M_1$ is the modulation order for the edge map, and $T$ is the transmission latency.

(a)

(b)

(c)

(d)

Figure 4: Optimal wireless parameters versus average SNR $\overline{\gamma }$ at various target BERs: (a) Percentage of power for prompt; (b) Transmission Latency; (c) Modulation order (bits per symbol) for the edge map; (d) Average numbers of prompt re-transmissions. The (CLIP, MS-SSIM) notation in the legend represents the normalized CLIP/MS-SSIM values achieved for each curve.

TABLE II: Modulation adaptation for the proposed framework.

	(0.999, 0.991)	(0.997, 0.982)	(0.978, 0.918)	$T$ (ms)
64	$30\ge \overline{\gamma }\ge 24.2$ ($\overline{{\gamma }_1}=27$)	$30\ge \overline{\gamma }\ge 23.4$ ($\overline{{\gamma }_1}=26$)	$30\ge \overline{\gamma }\ge 22.2$ ($\overline{{\gamma }_1}=25$)	1.31
16	$24\ge \overline{\gamma }\ge 18$ ($\overline{{\gamma }_1}=21$)	$23.2\ge \overline{\gamma }\ge 17.2$ ($\overline{{\gamma }_1}=20$)	$22\ge \overline{\gamma }\ge 16$ ($\overline{{\gamma }_1}=19$)	1.97
4	$17.8\ge \overline{\gamma }\ge 11.2$ ($\overline{{\gamma }_1}=14$)	$17\ge \overline{\gamma }\ge 10.4$ ($\overline{{\gamma }_1}=13$)	$15.8\ge \overline{\gamma }\ge 9.4$ ($\overline{{\gamma }_1}=12$)	3.93
0	$11\ge \overline{\gamma }$	$10.2\ge \overline{\gamma }$	$9.2\ge \overline{\gamma }$	/

III-F Computation latency

The computation latency depends on the choice of the AI models and the implementation scenario. For the on-device implementation scenario, smaller models like BLIP (55 GFLOPs) [26] and mobile diffusion (153 GFLOPs) [32] are used. Using a state-of-the-art A17 Pro chip (35 TFLOPS/s), the resulting computation latencies are 1.6 ms for BLIP and 4.6 ms for mobile diffusion. These are comparable to the transmission latencies as reported in Fig. 4(b) and TABLE II, emphasizing the need to minimize the transmission latency for reduced end-to-end latency.

IV Conclusions

This paper proposed a latency-aware and channel-adaptive semantic communication framework with pre-trained foundation generative AI models. In this framework, the transmitter extracts multi-modal semantic content of the input signal including a textual prompt and conditioning signals. The extracted semantics are then transmitted in multiple streams with semantic-aware appropriate coding and communication schemes and then input to a generative diffusion model at the receiver. Simulation results on natural images showcase the efficacy of the proposed framework in achieving ultra-low-rate, low-latency, and channel-adaptive semantic communications.