HyDiscGAN: A Hybrid Distributed cGAN for Audio-Visual Privacy Preservation in Multimodal Sentiment Analysis

Abstract

Multimodal Sentiment Analysis (MSA) aims to identify speakers’ sentiment tendencies in multimodal video content, raising serious concerns about privacy risks associated with multimodal data, such as voiceprints and facial images. Recent distributed collaborative learning has been verified as an effective paradigm for privacy preservation in multimodal tasks. However, they often overlook the privacy distinctions among different modalities, struggling to strike a balance between performance and privacy preservation. Consequently, it poses an intriguing question of maximizing multimodal utilization to improve performance while simultaneously protecting necessary modalities. This paper forms the first attempt at modality-specified (i.e., audio and visual) privacy preservation in MSA tasks. We propose a novel Hybrid Distributed cross-modality cGAN framework (HyDiscGAN), which learns multimodality alignment to generate fake audio and visual features conditioned on shareable de-identified textual data. The objective is to leverage the fake features to approximate real audio and visual content to guarantee privacy preservation while effectively enhancing performance. Extensive experiments show that compared with the state-of-the-art MSA model, HyDiscGAN can achieve superior or competitive performance while preserving privacy.

1 Introduction

With the growing prevalence of video content on social media, Multimodal Sentiment Analysis (MSA) is poised to provide new opportunities by leveraging multimodal data to enhance and go beyond traditional text-based sentiment analysis Zhao et al. (2023). MSA aims to predict the speaker’s sentiment by utilizing extra information available in audio and visual content instead of only textual content. The audio/visual modality extracts facial emotions and vocal expressions, enabling a more comprehensive sentiment understanding in wide-ranging applications.

(a)

(b)

Figure 1: Illustration of privacy preservation and MSA frameworks.

Notably, social video data contains massive private information, including Personally Identifiable Information (PII) and biometric data like face images and voiceprints Regulation (2016). Unfortunately, the misuse of personal information often causes a string of public security incidents, raising widespread concerns regarding personal privacy and security within society Nguyen et al. (2021); Yang et al. (2020). Upon closer examination of video data, we uncover a crucial yet often overlooked fact: different modalities carry varying requirements for privacy, as depicted in Figure 1(a). For instance, legislative efforts aimed at preserving privacy data have emphasized the privacy of personal audio or visual data over textual data Regulation (2016). In addition, techniques such as introducing noises or blurring faces (e.g., differential privacy Dwork (2006)) to de-identify audio and visual data can significantly impede the recognition of sentiment cues. In contrast, de-identifying textual data, such as removing sensitive words, can effectively protect privacy without altering the primary semantics Wang et al. (2023). The observations encourage us to contemplate an intriguing question: how to protect the privacy of specific modalities (i.e., audio and visual) when building MSA models?

Existing MSA approaches typically adopt a centralized paradigm where multimodal data is collected from personal devices and stored centrally for training, achieving excellent performance yet posing considerable challenges and risks in preserving personal privacy, as shown in Figure 1(b). Instead, increasing efforts have been made to apply Distributed Collaborative Learning (DCL) to multimodal tasks Yu et al. (2022); Chen and Zhang (2022). DCL frameworks Kairouz et al. (2021), such as Federated Learning (FL) Nguyen et al. (2021) and Split Learning (SL) Thapa et al. (2022), have gained prominence, offering privacy preservation by avoiding centralized data hosting and access. They rely on exchanging multimodal networks between a central server and clients that hold unshareable data for model training and testing, however, they struggle to strive a balance between performance and privacy preservation. Additionally, these endeavors primarily focus on scenarios where all multimodal content is isolated on separate clients, which does not align with the goal of modality-specific privacy preservation in practice. These insights prompt us to explore strategies for maximizing data utilization to improve performance while simultaneously protecting necessary modalities.

An intuitive solution to modality-specific privacy preservation is combining centralized and DCL frameworks to create a hybrid distributed learning paradigm, guaranteeing data utilization and protection, respectively. Accordingly, we can initiate a primary idea: train shareable modality data (text) centrally while private modality data (audio and visual) distributively. However, in the setting, where we have separate copies of shareable modality and private modality data on the server and clients respectively, we face a dilemma when it comes to performing model inference. On one hand, performing inference on the server requires access to the data representations of the private modality, which can increase communication costs and pose risks to privacy Thapa et al. (2022). On the other hand, performing inference on the client-side necessitates each client to train the entire MSA model to guarantee effective multimodal fusion, needing more client computational resources. Note that it is practically impossible for personal devices (clients) such as smartphones or laptops to have sufficient computing power to accommodate widespread large-scale MSA models. As a result, the hybrid distributed mode inevitably poses two primary challenges stemming from the misaligned treatment between modalities: 1) achieving effective multimodal alignment, and 2) ensuring efficient collaborative communication.

In light of the above discussion, we propose a novel hybrid distributed collaborative learning framework based on a cross-modality conditional Generative Adversarial Network (cGAN), termed HyDiscGAN. Specifically, we build an audio generator and a visual generator to generate fake features of private audio and visual data respectively in an autoregressive manner. The generators are placed in the server to approximate real features in the clients. On one hand, the generated features are sent to the corresponding audio and visual discriminators in the clients which are regulated by two customized contrastive losses and a cGAN loss. Generators and discriminators are based on Transformers to cater to the sequential audio and visual data. On the other hand, the features are fed into Transformer layers, followed by the gated attention unit to fuse multimodal features of text, visual, and audio, to perform the downstream sentiment analysis. Note that HyDiscGAN is trained in two stages: 1) the cross-modality cGAN is pre-trained to guarantee effective multimodal alignment, where global generators and local discriminators are distributively optimized in an alternating manner; 2) the MSA components are trained along with fine-tuning the generators under keeping the discriminators frozen. Its learning process simulates guessing audio and visual (semantic) features conditioned on text inputs, which is inspired by the empirical observation that individuals can envision the tone and facial expressions associated with a piece of text when it is narrated. Consequently, HyDiscGAN does not require any client-side computation during inference, reducing large collaborative costs to boost efficient communication.

Our key contributions can be summarized as follows:

• We propose a novel hybrid DCL framework HyDiscGAN for visual-audio privacy preservation in MSA. To the best of our knowledge, this forms the first endeavor to address modality-specified privacy preservation.

• We customize a cross-modality cGAN to achieve effective multimodal alignment and efficient collaborative communication in HyDiscGAN.

• Experiments on two MSA benchmarks show HyDiscGAN achieves desirable performance competent to SOTA baselines while preserving visual-audio privacy.

2 Related Work

Figure 2: The overall process of HyDiscGAN generates “sufficiently realistic” fake features for private modalities (audio and visual) through hybrid DCL between the server and clients and subsequently fuses them with the real features of the shareable modality (text) for MSA tasks.

2.1 Multimodal Sentiment Analysis

The current approaches to MSA can be broadly categorized: representation-based and fusion-based methods Yu et al. (2023); Lao et al. (2024). Representation-based methods aim to acquire effective representations for each modality, facilitating subsequent fusion processes. One perspective argues that effective sentiment representations should encompass both modality-invariant and modality-invariant features Hazarika et al. (2020); Yu et al. (2021); Lin and Hu (2022). Another viewpoint suggests that, in multimodal data, the text modality predominates, seeking to enhance text representation by integrating textual and non-textual modality information Wang et al. (2019); Yang et al. (2021); Guo et al. (2022); Su et al. (2023). Recently, Yang et al. Yang et al. (2023) further improved multimodal informative representation by incorporating contrastive learning and contrastive feature decomposition alongside representation learning. Regarding fusion-based methods, early research categorized them into early-fusion and late-fusion Yu et al. (2023). Early fusion emphasizes learning dependencies in multimodal sequence data Zadeh et al. (2018a), while late fusion initially learns independent unimodal representations, integrating them later for sentiment inference Zadeh et al. (2017). Recently, Zhao et al. Zhao et al. (2023) achieved SOTA performance in MSA by acquiring more enriched multimodal representations through data augmentation strategies on limited datasets.

2.2 Distributed Collaborative Learning

Algorithm 1 Training Cross-Modality cGAN.

Input: Multiple training clients; training epoch $T$; audio and visual generators $G^{\ast }$; audio and visual global discriminators $D^{\ast }$

for $i=1$ to $T$ do

$S$ training clients are randomly selected;

$⊳$ Send global discriminators $D^{\ast }$ to each client

for each client $C$ in $\left[S\right]$ in parallel do

$⊳$ Send textual data $t$ to the central server

Server Executes:

$X^t=\mathrm{BERT}\left(t\right)$;

$Z^{\ast }=G^{\ast }({\mu }_C^{\ast },X^t;{\theta }_{G^{\ast }}),\ast \in \{a,v\}$;

$⊳$ Send features $X^t$ and $Z^{\ast }$ to client $C$

Client Executes:

$X^a=\mathrm{COVAREP}\left(a\right)$, $X^v=\mathrm{FACET}\left(v\right)$;

${ℒ}_{D^{\ast }},{ℒ}_{\text{real}}^{\ast }=D^{\ast }(\left[X^{\ast }\right|\left|Z^{\ast }\right],X^t;{\theta }_{D^{\ast }}),\ast \in \{a,v\}$; Update local discriminators $D^{\ast }$ based on $\left(1-{\lambda }_D\right){ℒ}_{D^{\ast }}+{\lambda }_D{ℒ}_{\text{real}}^{\ast };$

${ℒ}_{G^{\ast }},{ℒ}_{\text{fake}}^{\ast }=D^{\ast }(Z^{\ast },X^t;{\theta }_{D^{\ast }}),\ast \in \{a,v\}$;

$⊳$ Send losses ${ℒ}_{G^{\ast }}$, ${ℒ}_{\text{fake}}^{\ast }$, and updated local discriminators $D^{\ast }$ to the central server

end for

Server Executes:

Update generators $G^{\ast }$ based on $\left(1-{\lambda }_G\right){ℒ}_{G^{\ast }}+{\lambda }_G{ℒ}_{\text{fake}}^{\ast }$; Update global discriminators $D^{\ast }$ by averaging the local discriminator parameters received from $S$ clients

end for

Output: Updated audio and visual generators $G^{\ast }$

DCL has gained significant attention in recent years due to its data protection capabilities Kairouz et al. (2021). The two most popular frameworks are federated learning McMahan et al. (2017) and split learning Gupta and Raskar (2018). For federated learning, the FedAvg was initially proposed by Brendan et al. McMahan et al. (2017). In FedAvg, a complete model is trained on each local client holding data, and the locally updated models are then sent to the server for aggregation, resulting in a global model. Subsequent researchers have made further improvements, such as introducing penalty terms to address non-convex problems Li et al. (2020) and incorporating momentum mechanisms to enhance its convergence speed and performance Hsu et al. (2019); Zhu et al. (2020). The advantages of federated learning lie in the parallelization of computations across multiple clients, while its drawback is its inapplicability to scenarios where client resources are limited. On the contrary, split learning Gupta and Raskar (2018) divides a model, such as a deep neural network, into multiple parts, and then performs computations on different devices. Thapa et al. Thapa et al. (2022) proposed the integration of federated learning and split learning, introducing federated split learning, which eliminates the inherent limitations of both frameworks.

2.3 Generative Adversarial Networks

Generative Adversarial Networks (GANs) were initially proposed by Goodfellow et al. Goodfellow et al. (2014). Subsequently, researchers have proposed various improvements and variations Radford et al. (2015); Almahairi et al. (2018); He et al. (2024). Importantly, conditional Generative Adversarial Network (cGAN) Mirza and Osindero (2014) introduces conditional information during the training process, enabling the generator to produce samples related to the given conditions. GANs were initially applied in computer vision to generate realistic images in a self-supervised manner and later spread rapidly to other fields Goodfellow et al. (2014), for natural language processing, like text generation Zhang et al. (2017), adversarial training Zhang et al. (2016), and data augmentation Zhao et al. (2023).

3 Methodology

3.1 Problem Statement

MSA is formulated as a binary/multi-classification or regression task for predicting sentiment labels. In contrast to all previous centralized models, our HyDiscGAN is implemented in a more realistic and secure scenario, encompassing a central server and numerous personal clients. Each client $C$ holds $N_C$ video clips as training or test samples. According to the Introduction, each sample comprises shareable modality data, i.e., text ($t$), as well as two private modality data, namely, audio ($a$) and visual ($v$). The raw data and extracted features of private modalities are securely maintained on their personal clients throughout the entire process.

For each sample, we obtain its real feature embedding sequences $X^m=[x_1^m,x_2^m,\mathrm{\dots },x_{L^m}^m]\in {ℝ}^{L^m\times d^m}$ from three modality data using BERT Kenton and Toutanova (2019), COVAREP Degottex et al. (2014), and FACET De la Torre and Cohn (2011), respectively. $L^m$ denotes the length of the sequence and $d^m$ is the feature dimension. $m\in \{t,\ast \}$ and $\ast \in \{a,v\}$ is the set of private modalities. Following BERT, we introduce <CLS> tag features at the end of audio and visual feature sequences to represent the comprehensive semantics of the sequence. is initialized through the average pooling of all features in the sequence.

It is worth noting that our primary motivation is to generate fake feature sequences , which approximate the real features $X^{\ast }$ extracted from the raw audio and visual data, rather than the raw data itself. This not only reduces sentiment-irrelevant redundant computations but also applies gradient truncation to prevent adversaries from reconstructing the raw data from gradients Thapa et al. (2022). Subsequently, $Z^{\ast }$ and $X^t$ are utilized for the training or testing of MSA models.

3.2 Hybrid Distributed Collaborative Learning

The training pipeline of HyDiscGAN for MSA is shown in Figure 2. Specifically, its training consists of two steps:

(1) Training Cross-Modality cGAN involves hybrid distributed collaborative learning among numerous clients and a central server to ensure effective multimodal alignment of generators on the server. As shown in Algorithm 1, at the beginning of each training epoch, the central server receives textual data from a group of clients. Guided by the text semantics, the generators $G^{\ast }$ on the server generate fake features for the audio and visual modalities, subsequently transmitted to their respective clients. Each client computes losses ${ℒ}_{D^{\ast }}$, ${ℒ}_{G^{\ast }}$, ${ℒ}_{\text{real}}^{\ast }$, and ${ℒ}_{\text{fake}}^{\ast }$ using its own real features and the received fake features, and then sends generator’s losses and the local discriminator’s parameters back to the server. The central server updates generators $G^{\ast }$ based on the received losses and updates discriminators $D^{\ast }$ by averaging the local parameters received from multiple clients.

(2) Training MSA Component begins with specific Transformer Layers to further encode the private modality fake features from the generators $G^{\ast }$. Subsequently, the Fusion Module is employed to combine these private modality fake features with the real features of the shareable modality, utilized for computing the MSA task loss ${ℒ}_{\text{task}}$. During this stage, the generators $G^{\ast }$ are fine-tuned as part of the MSA Component, while the global discriminators remain frozen.

3.3 Cross-Modality cGAN

cGAN Mirza and Osindero (2014) is a variant of the Generative Adversarial Network Goodfellow et al. (2014) designed to enable targeted sample generation based on given conditions. It comprises a generator and a discriminator, where the generator produces samples satisfying specific conditions, while the discriminator is used to determine whether the input samples are generated by the generator or are real.

In our conception, we aim to generate the fake features of private modalities that semantically align with corresponding shareable modality features. To achieve this, we use text information, i.e., the feature sequence $X^t$ encoded by BERT, as the conditional input. Additionally, since private modality features are sequential data, to maintain the contextual correlation in generated fake feature sequences $Z^{\ast }$, we adopt an autoregressive manner to generate features at each temporal position. The generation process can be formalized as:

$$z_i^{\ast }=G^{\ast }(z_{0:i-1}^{\ast },X^t;{\theta }_{G^{\ast }})$$ (1)

where ${\theta }_{G^{\ast }}$ is the set of trainable parameters. Especially, $z_0^{\ast }={\mu }^{\ast }$ and ${\mu }^{\ast }\sim 𝒩(0,1)$ is a random feature vector sampled from a Gaussian distribution.

3.3.1 Transformer Layer

In our framework, apart from the classifier and Fusion Module, all other components utilize the Transformer Vaswani et al. (2017) as the backbone, with distinctions solely in input and output. Transformer is an efficient neural architecture for modeling sequential data. The core computation is the Scaled Dot-Product Attention, which is defined as:

$$\mathrm{Att}(Q,K,V)=\mathrm{softmax}(\frac{Q{K}^T}{\sqrt{d_K}})V$$ (2)

where $Q$, $K$, and $V$ are obtained by linear mapping of the input feature sequence. Furthermore, the “Multi-head” operation Vaswani et al. (2017) is used to jointly focus on different parts of the input sequence on multiple subspaces, enhancing the ability to capture information.

3.3.2 Transformer-based Autoregressive Generator

Inspired by the neural machine translation model Vaswani et al. (2017), we construct a Transformer-based Autoregressive Generator. It is a simple variant of the basic Transformer Layer. Specifically, it comprises two attention structures serving different purposes: (1) Intra-modality Multi-Head Attention is employed to capture contextual relationships within the unimodal feature sequence, and corresponding $Q_{\text{ra}}^{\ast }$, $K_{\text{ra}}^{\ast }$, and $V_{\text{ra}}^{\ast }$ are all derived from the mapping of $Z_{0:i-1}^{\ast }$:

$$[Q_{\text{ra}}^{\ast },K_{\text{ra}}^{\ast },V_{\text{ra}}^{\ast }]=[z_{0:i-1}^{\ast }{W}_{Q_{\text{ra}}}^{\ast },z_{0:i-1}^{\ast }{W}_{K_{\text{ra}}}^{\ast },z_{0:i-1}^{\ast }{W}_{V_{\text{ra}}}^{\ast }]$$

where $W_{Q_{\text{ra}}}^{\ast }$, $W_{K_{\text{ra}}}^{\ast }$, and $W_{V_{\text{ra}}}^{\ast }$ are parameter matrices; (2) Inter-modality Multi-Head Attention layer is used to capture cross-modality alignment information attentive to the shareable modality feature sequence $X^t$. Hence, $Q_{\text{er}}^{\ast }$, $K_{\text{er}}^{\ast }$, and $V_{\text{er}}^{\ast }$ are obtained as follows:

$$[Q_{\text{er}}^{\ast },K_{\text{er}}^{\ast },V_{\text{er}}^{\ast }]=[z_{0:i-1}^{\ast }{W}_{Q_{\text{er}}}^{\ast },X^t{W}_{K_{\text{er}}}^{\ast },X^t{W}_{V_{\text{er}}}^{\ast }]$$

where $W_{Q_{\text{er}}}^{\ast }$, $W_{K_{\text{er}}}^{\ast }$, and $W_{V_{\text{er}}}^{\ast }$ are parameter matrices.

3.3.3 Transformer-based Discriminator

For discriminators $D^{\ast }$, their structure is identical to generators $G^{\ast }$, except for the exclusion of autoregressive iteration steps. The inputs includes fake feature sequences $Z^{\ast }$ generated by $G^{\ast }$ and corresponding real feature sequences $X^{\ast }$. An additional binary classifier is added to the output layer to discriminate between the generated and real features. The discriminator plays a crucial role by providing feedback to the generator to enhance its ability to generate “sufficiently realistic” fake features.

3.4 MSA Component

We further introduce two Transformer Layers for learning deep semantic representations of non-textual modality features. Specifically, the generated fake audio and visual feature sequences $Z^{\ast }$ are encoded through corresponding Transformer Layers before being fed to the Fusion Module.

3.4.1 Fusion Module

This module is used to fuse <CLS> tag features of different modalities and regulate the influence of each modality feature in the final sentiment prediction via the gated attention unit Dhingra et al. (2017). The operation of the gated attention unit is formulated for each modality as follows:

$$h_{\text{output}}^m=\mathrm{GAtt}(h_{\text{input}}^m;{\theta }_{\mathrm{GAtt}}^m)\odot h_{\text{input}}^m$$ (3)

where the gated attention function $\mathrm{GAtt}$ is a fully connected linear layer with sigmoid activation, and its output dimension is equal to the input dimension. ${\theta }_{\mathrm{GAtt}}$ is the set of trainable parameters. The symbol $\odot$ denotes the Hadamard product Horn (1990). Specifically, for the text modality and for the audio and visual modalities. Finally, the tensor $$hfinal= [$h_{\text{output}}^v:h_{\text{output}}^t:h_{\text{output}}^a$], connecting features from three modalities, is utilized for speaker’s sentiment prediction.

3.5 Learning Objectives

Our framework contains three learning objectives: cGAN Losses, customized Contrastive Losses, and MSA task Loss.

3.5.1 cGAN Losses

For a training sample that has private modality real feature sequences $X^{\ast }$ and generated fake feature sequences $Z^{\ast }$, cGAN losses are defined as:

$${ℒ}_{G^{\ast }}=\frac{1}{L^{\ast }+1}\sum _{i=1}^{L^{\ast }+1}[\log (1-D^{\ast }(G^{\ast }(z_{0:i-1}^{\ast },X^t)))]$$ (4)

(5)

where ${ℒ}_{G^{\ast }}$ and ${ℒ}_{D^{\ast }}$ represent the losses of the generator and discriminator, respectively. $L^{\ast }$ are lengths of feature sequences. Specifically, the <CLS> tag feature is also utilized for computation. In the training of Cross-modality cGAN, ${ℒ}_{G^{\ast }}$ and ${ℒ}_{D^{\ast }}$ are alternately minimized.

3.5.2 Contrastive Losses

We design two sample separation loss terms based on NT-Xent contrastive loss Chen et al. (2020), which are used to further regularize the learning process for both the discriminator and the generator. Specifically, for a sample in training client $C$, its real and fake <CLS> tag features are and , respectively. (1) the Real-Real contrastive loss ${ℒ}_{\text{real}}^{\ast }$ is employed to regulate the discriminator:

(6)

where $\mathrm{sim}$ is Cosine similarity function and $\tau$ is the temperature parameter. denotes the feature set of samples in client $C$ with a different sentiment polarity from the sample corresponding to . Conversely, is the feature of a sample with the same sentiment polarity, randomly sampled from client $C$.

(2) the Real-Fake contrastive loss ${ℒ}_{\text{fake}}^{\ast }$ is introduced to regulate the generator:

(7)

where is the feature set of samples in client $C$, excluding the sample corresponding to .

3.5.3 MSA Loss

Let $y$ and $\widehat{y}$ denote the true and predicted sentiment labels of a sample, respectively. The MSA task loss ${ℒ}_{\text{task}}$ is defined:

(8)

where $N_B$ is the batch size. $\widehat{y}$ is obtained through classification or regression predictions on $h_{\text{final}}$.

4 Experiments

4.1 Datasets and Distributed Settings

Model	MOSI					MOSEI
	Acc-2 $\uparrow$	F1-Score $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc-2 $\uparrow$	F1-score $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
(G) TFN Zadeh et al. (2017)	- / 80.8	- / 80.7	34.9	0.901	0.698	- / 82.5	- / 82.1	51.6	0.593	0.700
(G) LMF Liu et al. (2018)	- / 82.4	- / 82.4	33.2	0.917	0.695	78.5 / 81.9	79.0 / 81.7	51.6	0.573	0.714
(G) MFN Zadeh et al. (2018a)	77.4 / -	77.3 / -	34.1	0.965	0.632	79.0 / 82.9	79.6 / 82.9	51.3	0.573	0.718
(G) MulT Tsai et al. (2019)	- / 83.0	- / 82.8	40.0	0.871	0.698	- / 82.5	- / 82.3	52.8	0.580	0.703
(B) MISA Hazarika et al. (2020)	81.8 / 83.4	81.7 / 83.6	42.3	0.783	0.761	83.6 / 85.5	83.8 / 85.3	52.2	0.555	0.756
(B) MTAG Yang et al. (2021)	- / 82.3	- / 82.1	-	0.866	0.722	-	-	-	-	-
(B) Self-MM Yu et al. (2021)	83.4 / 85.5	83.4 / 85.4	46.7	0.708	0.796	83.8 / 85.2	83.8 / 84.9	53.9	0.531	0.765
(B) TMMDA Zhao et al. (2023)	- / 86.9	- / 86.9	-	0.703	0.801	-	-	-	-	-
(B) ConFEDE Yang et al. (2023)	84.2 / 85.5	84.1 / 85.5	42.3	0.742	0.784	81.7 / 85.8	82.2 / 85.8	54.9	0.522	0.780
(B) HyDiscGAN (ours)	84.1 / 86.7	83.7 / 86.3	43.2	0.749	0.782	81.9 / 86.3	82.1 / 86.2	54.4	0.533	0.761

Table 1: Predicted results of different MSA models on MOSI and MOSEI datasets. “$\uparrow$” indicates that larger values represent better results and “$\downarrow$” signifies the opposite. (G) and (B) represent using Glove and BERT as text feature extractors, respectively. In Acc-2 and F1 score columns, the number on the left side of “/” corresponds to “negative/non-negative” and the number on the right side corresponds to “negative/positive”. Bold values represent optimal performance and underlined values indicate suboptimal performance.

Two popular MSA benchmark datasets, MOSI Zadeh et al. (2016) and MOSEI Zadeh et al. (2018b), are utilized to evaluate the performance of our HyDiscGAN. Detailed descriptions of each dataset and their corresponding distributed settings are provided in Appendix A.

4.2 Baselines

Term	ConFEDE	-FL	-SL	-SFL	HyDiscGAN
Privacy preservation	✗	✓	✓	✓	✓
Distributed computing	✗	✓	✓	✓	✓
Generative capacity	✗	✗	✗	✗	✓
No computations on testing clients	✓	✗	✗	✗	✓
Client-side training	-	Parallel	Sequential	Parallel	Parallel
Scale of parameters (per client)	-	109.5M	23.9M	23.9M	77.8K
Scale of communication parameters (one epoch)	-	109.5M$\times$2$S$	23.9M$\times$2$S$	23.9M$\times$2$S$	77.8K$\times$2$S$

Table 2: Comparison of key attributes and training costs (on the MOSI dataset) of foundational DCL frameworks, including Federated Learning (-FL), Split Learning (-SL), Federated Split Learning (-SFL), and our HyDiscGAN. $S$ is the count of training clients in one epoch.

To validate the performance of the features generated by HyDiscGAN in MSA tasks, we conduct a comparison with several advanced and SOTA MSA models Zhao et al. (2023); Yang et al. (2023). These baseline models can be broadly categorized based on their backbone networks: (1) LSTM-based models, denoted as TFN Zadeh et al. (2017), LMF Liu et al. (2018), MFN Zadeh et al. (2018a), MISA Hazarika et al. (2020), and Self-MM Yu et al. (2021); (2) Transformer-based models, denoted as MulT Tsai et al. (2019), TMMDA Zhao et al. (2023), and ConFEDE Yang et al. (2023). Additionally, there is a model MTAG Yang et al. (2021) based on GNN. Note that only our HyDiscGAN utilizes generated private modality fake features for MSA, while all other baseline models neglect the preservation of the speaker’s privacy.

To assess the MSA performance and communication costs of HyDiscGAN in distributed training, we deployed the latest MSA model ConFEDE across three widely used DCL frameworks: Federated Learning (-FL) McMahan et al. (2017), Split Learning (-SL) Gupta and Raskar (2018), and Federated Split Learning (-SFL) Thapa et al. (2022). ConFEDE and its three variants are implemented based on the codes provided by the authors. In -FL, each client trains a complete ConFEDE model using local data and then aggregates them in the central server. In -SL and -SFL, we adhere to the minimum split principle, aiming to perform as much computation as possible on the central server.

4.3 Evaluation Criteria

Following previous works Yang et al. (2023), we evaluated the performance of our models on four metrics: Sentiment Binary Classification Accuracy (Acc-2), F1-Score, Mean Absolute Error (MAE), and Correlation Coefficient (Corr). Our results in both classification and regression experiments are reported as the averages of five different random seed runs. Detailed hyperparameter settings are included in Appendix B, and our source codes and processed datasets will be made publicly available upon acceptance.

4.4 Performance Analysis

4.4.1 Comparison with Advanced MSA Models

Table 1 presents the comparative results between our proposed HyDiscGAN and other MSA models on MOSI and MOSEI datasets. In detail, HyDiscGAN achieves suboptimal performance across all classification metrics on the MOSI dataset. In the fundamental binary sentiment classification metrics (Acc-2 and F1-Score), the results of HyDiscGAN are, on average, only 0.325% lower than the SOTA performance. Note that in the “negative/positive” binary classification, HyDiscGAN ranks second, closely trailing behind the SOTA model TMMDA which incorporates data augmentation techniques. Moreover, on the MOSEI dataset, HyDiscGAN significantly outperforms ConFEDE and achieves SOTA performance in this task, exhibiting an average improvement of 0.45% higher than the suboptimal model ConFEDE. Furthermore, in other metrics, HyDiscGAN also demonstrates competitiveness. This indicates that the private modality fake features generated by HyDiscGAN contain high-quality sentiment cues, comparable to real features.

HyDiscGAN does not exhibit the same level of performance in regression tasks, i.e., MAE and Corr, as in classification tasks. One possible reason is that the Real-Real contrastive loss can only separate samples with different sentiment polarities, lacking regularization for samples with the same sentiment polarity but differing only in intensity during the feature generation process.

4.4.2 Comparison with Existing DCL Frameworks

Table 2 presents a comparison of key attributes and training costs between our HyDiscGAN and three existing DCL frameworks that deploy the SOTA MSA model ConFEDE. Overall, the training costs on the client side are significantly reduced with HyDiscGAN (reduced by 99.93% compared to -FL, and 99.68% compared to -SL and -SFL). Simultaneously, HyDiscGAN possesses the capability to generate fake features for private modalities, resulting in zero costs on the client side during testing. In contrast, other DCL frameworks require the same costs during testing as in training, making HyDiscGAN more suitable for testing scenarios with completely limited resources.

The results of comparative MSA methods are outlined in Table 3. While HyDiscGAN may not surpass the centralized model ConFEDE on specific evaluation metrics, it demonstrates notable superiority over ConFEDE across all metrics when ConFEDE is deployed in existing DCL frameworks. The influence arises from the label distribution skew Zhang et al. (2022) in client data within existing DCL frameworks for ConFEDE. When applied to MSA tasks, HyDiscGAN follows the two-stage training approach and employs the hybrid DCL strategy exclusively during the stage of learning the private modality real feature distribution on the client. This stage involves self-supervised learning and is therefore not influenced by the distribution of sentiment labels.

4.5 Ablation Study

Model	Acc-2 $\uparrow$	F1-Score $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
ConFEDE	84.2 / 85.5	84.1 / 85.5	42.3	0.742	0.784
-FL	81.4 / 81.7	81.3 / 81.5	40.7	0.803	0.721
-SL	83.5 / 84.2	83.1 / 83.9	41.6	0.765	0.767
-SFL	82.8 / 83.2	82.7 / 83.0	41.3	0.811	0.734
HyDiscGAN	84.1 / 86.7	83.7 / 86.3	43.2	0.749	0.782

Table 3: Predicted results of different DCL frameworks on the MOSI dataset, including Federated Learning (-FL), Split Learning (-SL), Federated Split Learning (-SFL), and our HyDiscGAN.

Variant	Acc-2 $\uparrow$	F1-Score $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Real feature (Only Audio)†	58.2	57.0	1.150	0.144
Fake feature (Only Audio)	65.2	61.6	1.147	0.162
Real feature (Only Visual)†	57.4	57.0	1.160	0.143
Fake feature (Only Visual)	65.3	65.1	1.139	0.168
cGAN loss (Only)	85.3	84.9	0.751	0.778
w/o ${ℒ}_{\text{real}}^{\ast }$	85.4	85.2	0.752	0.774
w/o ${ℒ}_{\text{fake}}^{\ast }$	86.0	85.7	0.750	0.779
HyDiscGAN	86.7	86.3	0.749	0.782

Table 4: Ablation results of real/fake features for private modalities (audio and visual). $†$ indicates the results from the baseline TMMDA. “w/o” denotes “without”.

4.5.1 Effects of Generated Fake Features

Table 4 upper section displays the performance of predicting sentiment tendencies using only visual or audio modality features. One observation is that in both modalities, the fake features we generated show significant performance improvements across all metrics compared to real features. This is attributed to the Cross-Modality cGAN we constructed, which generates non-textual modality features from text features. Since the text modality contains more sentiment cues, the generated features carry more sentiment information. Refer to Appendix C for a comprehensive analysis of privacy and performance compatibility experiments.

4.5.2 Effects of Customized Contrastive Losses

Table 4 lower section demonstrates the impact of two customized contrastive losses ${ℒ}_{\text{real}}^{\ast }$ and ${ℒ}_{\text{fake}}^{\ast }$, developed by us, on the performance of MSA tasks. We observed a performance enhancement with both losses, especially with ${ℒ}_{\text{real}}^{\ast }$. This is because ${ℒ}_{\text{real}}^{\ast }$ is based on the regularization term between real features with different sentiment polarities. It promotes the aggregation of samples with the same polarity in the feature space while encouraging the separation of samples with different polarities, leading to a more distinct representation of sentiment information.

4.6 Convergence Analysis

When training Cross-modality cGAN, there is a mutual game between the generator and discriminator, which may lead to training instability Radford et al. (2015). We present the convergence curves of losses during the training of Cross-Modality cGAN in HyDiscGAN on two datasets in Figure 3. It can be observed that the losses of generators and discriminators eventually converge to low values. This indicates that HyDiscGAN is capable of generating “sufficiently realistic” fake features.

(a)

(b)

Figure 3: Convergence visualization of training the cross-modality cGAN in HyDiscGAN on MOSI and MOSEI datasets, respectively.

4.7 Visualization

(a)

(b)

Figure 4: Visualization of the gated attention weights in the Fusion Module for visual-audio features on the test set of MOSI. Brighter regions imply higher unimodal information flow through the gates.

To further validate the effectiveness of generated fake features for MSA tasks, we qualitatively visualize the differences in their contributions to the final sentiment prediction compared to real features. As shown in Figure 4, the information in audio and visual fake features generated by HyDiscGAN is more retained in $h_{\text{final}}$, indicating their broader involvement in sentiment prediction and underscoring their effectiveness. More detailed visualizations are provided in Appendix D.

5 Conclusion

In this paper, we propose a novel hybrid DCL framework HyDiscGAN for audio-visual privacy preservation in MSA. HyDiscGAN conducts training through direct communication between the server and clients, aiming to avoid constructing centralized datasets that expose personal privacy. Compared to other DCL frameworks, HyDiscGAN achieves a better balance between performance and privacy preservation. Additionally, it demonstrates significantly superior training efficiency on the client side, making it more suitable for scenarios with limited client resources. Extensive experiments verify that, while preserving privacy, HyDiscGAN competes comparably with the SOTA models in MSA tasks.

HyDiscGAN: A Hybrid Distributed cGAN for Audio-Visual Privacy Preservation in Multimodal Sentiment Analysis

Appendix

Appendix A Dataset and Distributed Settings Details

In this section, we provide a detailed description of the datasets used in the experiments and their corresponding distributed settings. We assume the textual data within these two datasets has been de-identified and is no longer subject to special processing. The statistical details are shown in Table 5.

Dateset	Train		Valid		Test
	#S	#Sp	#S	#Sp	#S	#Sp
MOSI	1,284	52	229	10	686	31
MOSEI	16,326	150	1,871	50	4,659	100

Table 5: Statistics of the MOSI and MOSEI Datasets. #S represents the number of video clips, i.e., the number of samples. #Sp indicates the number of distinct speakers, i.e., the number of clients.

MOSI collects 2,199 video clips from YouTube with each clip representing a monologue by a speaker. These clips are contributed by 93 distinct speakers. The builder also considers the division according to speakers when dividing training, validation, and testing portions. The training portion comprises 52 speakers, while the validation and testing portions include 10 and 31 speakers, respectively. We naturally treat each speaker as an independent personal client with varying numbers of video clips containing sentiment scores ranging in [-3, +3]. +3 indicates the strongest positive sentiment, while -3 represents the strongest negative sentiment.

MOSEI collects 22,856 video clips with sentiment scores ranging in [-3, 3]. However, the builder did not provide speaker tags. To ensure fairness, we follow previous work and further divide the training portion comprising 16,326 video clips, into 150 personal clients. The validation and testing portions consist of 50 and 100 clients, respectively. Each client has an equal number of samples to simulate the i.i.d. scenario Klaassen and Magnus [2001].

Appendix B Hyperparameters

All our models are based on Python 3.8.18 and PyTorch 2.0.0, while training and testing were conducted on a single Tesla V100 PCIe 32GB GPU. We utilize the Adam algorithm to optimize the objective losses in both stages of training (Training Cross-Modality cGAN and Training MSA Component). The key hyperparameters used in the experiments are detailed shown in Table 6.

Hyperparameters	MOSI	MOSEI
Num of Transformer Layers / heads
-Visual Generator $G^v$	2 / 2	4 / 2
-Audio Generator $G^a$	1 / 1	5 / 3
-Visual Discriminator $D^v$	2 / 2	4 / 2
-Audio Discriminator $D^a$	1 / 1	5 / 3
-Visual Transformer Layer	2 / 2	2 / 2
-Audio Transformer Layer	1 / 1	2 / 3
Feature Dimension
-Text $x^t$	768	768
-Visual $x^v$	20	35
-Audio $x^a$	5	74
Learning Rate
-Generator $G^{\ast }$	2e-4	2e-4
-Discriminator $D^{\ast }$	1e-4	1e-4
-MSA task	1e-4	1e-4
-${\lambda }_D$	0.1	0.1
-${\lambda }_G$	0.1	0.1
epoch $T$ in training Cross-Modality cGAN	100	100
Randomly select the number of training clients $S$	10	5
Batch size $N_B$ in training MSA Component	32	32

Table 6: Hyperparameters of HyDiscGAN applied to different datasets.

Appendix C Compatibility in Privacy and Performance

Scene	Acc-2 $\uparrow$	F1-Score $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
-All shareable	83.2 / 84.7	83.2 / 84.9	42.4	0.753	0.775
-Audio privacy	83.1 / 85.2	82.8 / 85.2	42.2	0.751	0.777
-Visual privacy	83.6 / 86.4	83.3 / 86.1	42.9	0.749	0.778
-Audio-Visual privacy	84.1 / 86.7	83.7 / 86.3	43.2	0.749	0.782

Table 7: Prediction results of HyDiscGAN under diverse modality-specified privacy preservation scenarios on the MOSI dataset.

(a)

(b)

Figure 5: t-SNE Visualization of private modality real/fake <CLS> tag features (i.e. and ) for all test samples on the MOSI dataset, with samples from different clients labeled in different colors. “Epoch=0, 30, and 100” represent the fake features at different epochs during the training of Cross-Modality cGAN.

HyDiscGAN aims to generate fake features of privacy modalities from shareable de-identified textual data, replacing real features for downstream tasks. In practical applications, more flexible modality-specified privacy preservation scenarios may arise. Table 7 presents a performance comparison in four different modality-specified privacy preservation scenarios for MSA tasks. Specifically, “-All shareable” denotes a scenario where privacy is not considered, and real features of all three modalities are directly input into the MSA Component. “-Audio privacy” and “-Visual privacy” indicate scenarios where the audio or visual modality is individually specified as the private modality, respectively. “-Audio-Visual privacy” represents the scenario where both modalities are private. In these scenarios, HyDiscGAN generates privacy modality features, while the features of other modalities use real features. Initially, it is observed that “-Audio-Visual privacy” achieves optimal performance across all scenarios. Furthermore, the performance in “-Audio privacy” and “-Visual privacy” scenarios is generally superior to that in the “-All shareable” scenario. This is attributed to the joint optimization of cGAN losses and customized contrastive losses by HyDiscGAN, enabling HyDiscGAN to generate “sufficiently realistic” fake features and obtain clearer sentiment tendencies in generated fake features. By customizing constraints in the learning process, we can acquire fake features better suited for the target task than real features, achieving dual benefits in privacy and performance. In addition, HyDiscGAN achieves competent performance under different scenarios, indicating the applicability and flexibility of HyDiscGAN in various modality-specified privacy preservation scenarios of MSA tasks.

Appendix D More visualization

D.1 Global Distribution of Real/Fake Features

Figure 5 showcases the distribution of real/fake features for private modalities in all test samples from the MOSI dataset. An important observation is that, for the visual modality, real features naturally exhibit a clear clustered distribution consistent with the division of clients (i.e., speakers). With the continuous iterations of training, the distribution of fake features generated by HyDiscGAN also shows this trend. Moreover, when real features do not exhibit a clear clustered distribution, such as the audio modality, HyDiscGAN can also effectively capture the distribution of real features and preserve this characteristic. Consequently, HyDiscGAN demonstrates an excellent ability to learn the global distribution of features across different modalities.

D.2 Local Distribution of Real/Fake Features

Figure 6 illustrates the distribution of real/fake features for private modalities in test samples from one client on the MOSI or MOSEI dataset. An observation reveals that the audio and visual fake features of different samples exhibit clustering tendencies based on the samples’ sentiment polarity. This trend arises from the customized contrastive losses applied to various samples during the training of Cross-Modality cGAN. This observation also explains the enhanced performance of HyDiscGAN in sentiment classification.

(a)

(b)

(c)

(d)

Figure 6: t-SNE Visualization of private modality real/fake <CLS> tag features for test samples from one client on MOSI or MOSEI dataset, where positive (POS) and negative (NEG) samples are marked in red and blue, respectively.

Appendix E Impact of hyperparameters ${\lambda }_{D/G}$

The hyperparameters ${\lambda }_D$ and ${\lambda }_G$ are used to adjust the ratio of cGAN losses and customized contrastive losses during the training stage of Cross-modality cGAN. Figure 7 depicts their influence on the performance of the final MSA tasks. The results show the same trend in both classification and regression tasks, i.e. an initial performance increase followed by a subsequent decrease in performance. Specifically, when ${\lambda }_D$ and ${\lambda }_G$ are set to 0.1, the performance of both MSA tasks reaches optimal. Moreover, it is evident that with an increase of ${\lambda }_D$, the performance of the regression task drops sharply. This aligns with the explanation provided in the Performance Analysis, that is, Real-Real contrastive loss ${\lambda }_D$ has a limited predictive effect on sentiment intensity. Therefore, by setting both ${\lambda }_D$ and ${\lambda }_G$ to 0.1, HyDiscGAN achieves a balance between learning the distribution of real features and acquiring a clearer representation of sentiment features.

(a)

(b)

Figure 7: Prediction results of MSA tasks when the hyperparameters ${\lambda }_D$ and ${\lambda }_G$ vary within the range of 0.1 to 0.4. The classification task in (a) refers to the ”negative/positive” binary classification.

Appendix F Baseline Details

F.1 MSA Models

• TFN Zadeh et al. [2017] proposes a Tensor Fusion Network for learning inter-modality and intra-modality dynamics.

• LMF Liu et al. [2018], known as Low-rank Multimodal Fusion network, stands as an advanced variant of the TFN. It effectively reduces the computational complexity of multimodal tensors.

• MFN Zadeh et al. [2018a] is a Memory Fusion Network, that constitutes a multi-view sequential learning architecture employing attention mechanisms to achieve cross-modality interaction learning.

• MulT Tsai et al. [2019] utilizes a cross-modality transformer to achieve the translation from the source modality to the target modality, thereby comprehending the deep semantics of different modalities.

• MISA Hazarika et al. [2020] maps features from different modalities to distinct feature spaces, facilitating the learning of modality-invariant and modality-specific representations, thereby enhancing the capture of commonalities and differences across various modalities.

• MTAG Yang et al. [2021] stands as the sole method employing graph neural networks to model multimodal data. It transforms multimodal data into a graph structure, capturing rich semantics across modalities and time through information aggregation on the graph.

• Self-MM Yu et al. [2021] devises a unimodal label generation network based on self-supervised learning strategies to acquire unimodal label information. Subsequently, it jointly trains unimodal and multimodal sentiment analysis, employing an adaptive weight adjustment strategy to balance progress across different tasks.

• TMMDA Zhao et al. [2023] proposes a token mixup technique for multimodal data augmentation, aiming to acquire efficient multimodal representations on limited labeled datasets.

• ConFEDE Yang et al. [2023] proposes a unified learning framework for contrastive representation learning and contrastive feature decomposition, aiming to acquire well-rounded multimodal information representations, including modality-invariant and modality-specific components.

F.2 DCL Frameworks

• FL McMahan et al. [2017] introduces Federated Learning (FL) and the Federated Averaging (FedAvg) Algorithm. It trains complete models on individual clients with their data and aggregates the updates on the server to learn a global model.

• SL Gupta and Raskar [2018] (Split Learning) divides the AI model and trains partial models on both the server and clients with data. In contrast to FL, it doesn’t need complete model training on clients, making it suitable for resource-limited scenarios. However, client-side training cannot be parallelized.

• SFL Thapa et al. [2022] (SplitFed Learning) combines FL and SL methods, eliminating their respective limitations.