Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data

Abstract

Federated learning achieves effective performance in modeling decentralized data. In practice, client data are not well-labeled, which makes it potential for federated unsupervised learning (FUSL) with non-IID data. However, the performance of existing FUSL methods suffers from insufficient representations, i.e., (1) representation collapse entanglement among local and global models, and (2) inconsistent representation spaces among local models. The former indicates that representation collapse in local model will subsequently impact the global model and other local models. The latter means that clients model data representation with inconsistent parameters due to the deficiency of supervision signals. In this work, we propose Fed${\text{𝚄}}^{\mathrm{2}}$ which enhances generating uniform and unified representation in FUSL with non-IID data. Specifically, Fed${\text{𝚄}}^{\mathrm{2}}$ consists of flexible uniform regularizer (FUR) and efficient unified aggregator (EUA). FUR in each client avoids representation collapse via dispersing samples uniformly, and EUA in server promotes unified representation by constraining consistent client model updating. To extensively validate the performance of Fed${\text{𝚄}}^{\mathrm{2}}$, we conduct both cross-device and cross-silo evaluation experiments on two benchmark datasets, i.e., CIFAR10 and CIFAR100.

1 Introduction

To meet the demands of privacy regulation, federated learning (FL) [31] is boosting to model decentralized data in both academia and industry. This is because FL enables the collaboration of clients with decentralized data, aiming to develop a high-performing global model without the need for data transfer. However, conventional FL work mostly assumes that client data is well-labeled, which is less practical in real-world applications. In this work, we consider the problem of federated unsupervised learning (FUSL) with non-IID data [43, 14], i.e., modeling unified representation among imbalanced, unlabeled, and decentralized data.

Utilizing existing centralized unsupervised methods cannot adapt to FUSL which has non-IID data [44]. To mitigate it, one of the popular categories is to train self-supervised learning models, e.g., BYOL [8], SimCLR [3], and Simsiam [5], in clients, and aggregate models via accounting extremely divergent model [44, 45], knowledge distillation [9], and combining with clustering [30]. However, two coupling challenges of FUSL, i.e., CH1: Mitigating representation collapse entanglement, and CH2: Obtaining unified representation spaces, are not well considered.

The first challenge is that representation collapse [13] in the client subsequently exacerbates the representation of global and other local models. Motivated by regularizing Frobenius norm of representation in centralized self-supervised models [15, 21], FedDecorr [36] tackles representation collapse with the global supervision signals in federated supervised learning. But directly applying these methods to FUSL has three aspects of limitations. Firstly, it relies on large data batch size [30] to capture reliable distribution statistics, e.g., representation variance. Besides, regularizing the norm of high-dimensional representations inevitably causes inactivated neurons and suppresses meaningful features [18]. Moreover, clients cannot eliminate representation collapse entanglement by decorrelating representations for FUSL problem, once clients represent data in different representation spaces.

The second challenge refers to optimizing inconsistent client model parameters toward discrepant parameter spaces, bringing less unified representations among local models. Most of the existing FUSL methods aggregate participating models with the ratio of samples, i.e., FedAvg [31]. This not only fails to tackle the client shift from global optimum to local optimum, but also brings sub-optimal results [22, 33]. To mitigate this, FUSL methods maintain consistency by (1) abandoning extremely divergent clients by threshold [45, 44], (2) obtaining global supervised signal via clustering client sub-clusters [30, 6], and (3) scaling angular divergence among client models in a layer-wise way [33]. These methods either forget to adjust clients updated with inconsistent directions, or break down the performance coherence among different layers of the whole model, failing to capture unified representations.

To fill this gap, we propose a framework, i.e., Fed${\text{𝚄}}^{\mathrm{2}}$, to enhance Uniform and Unified representation in FUSL with non-IID data. To tackle CH1, we initially devise a flexible uniform regularizer (FUR) to prevent the sample representation collapse with no regard to data distribution and client discrepancies. In each client, FUR minimizes unbalanced optimal transport divergence between client data and uniform random samples, i.e., samples from the same spherical Gaussian distribution among clients. Thus it not only flexibly disperses local data representations toward ideal uniform distribution, but also avoids the representation collapse entanglement among clients without leaking privacy. To mitigate CH 2, we propose efficient unified aggregator (EUA) to aggregate a global model that maintains model consistency among global optimization and different local optimizations. Specifically, EUA formulates model aggregation as a multiple-objective optimization based on the model deviation change rates of clients. EUA reduces computation by searching exact solutions in the dual formulation with alternating direction methods of multipliers. Compared with conventional aggregation methods, we equivalently maintain consistent model updating based on client model deviation change, enhancing unified representations.

Summarily, we aim to enhance the representation in FUSL by mitigating representation collapse and unifying representation generalization. (1) We enhance uniform representation by approaching data samples to spherical Gaussian distribution, which mitigates representation collapse and its subsequent entangled impacts. (2) We enhance unified representation by constraining the consistent updating of different client models. (3) To reach the above goals, we propose Fed${\text{𝚄}}^{\mathrm{2}}$ with FUR and EUA, which is agnostic and orthogonal for the backbone of self-supervised models. (4) In our empirical studies, we conduct experiments on two benchmark datasets and two evaluation settings, which extensively validate the performance of Fed${\text{𝚄}}^{\mathrm{2}}$.

2 Related Work

2.1 Federated Unsupervised Learning

To enhance FUSL with non-IID data [24], there are two categories of efforts, i.e., (1) generating global supervised signals, and (2) enhancing unified representation. The former targets at generating global supervised signals via local-global clustering [6], and sharing data representation among clients [41, 43]. But these methods either suffer from randomness in obtaining global supervision[30], or take the risk of leaking privacy [44]. The latter enhances unified representation by adapting existing unsupervised representation methods, and tackling non-IID modeling with divergence-aware model aggregation [44, 45, 33, 17, 30]. Both FedU [44] and FedEMA [45] enhance the awareness of heterogeneity in federated self-supervised learning by divergence-aware predictor update rule, and adaptive global knowledge interpolation, respectively. However, this kind of work overlooks representation collapse in non-IID clients. Orchestra [30] utilizes local-global clustering derived from K-Fed [6] to guide self-supervised learning. This brings additional cost for clustering and is fragile to random initialization. Moreover, FedX [9] devises local relational loss to distill the invariance of data samples, and global relational loss to maintain client inconsistencies. Recently, L-DAWA [33] corrects the FUSL optimization trajectory by measuring and scaling angular divergence among client models in a layer-wise way. However, it is hard to guarantee that the newly aggregated global model is still compatible and performant as a consistent model. Differently, the proposed Fed${\text{𝚄}}^{\mathrm{2}}$ enhances uniform and unified representation without the prior knowledge of unsupervised models, data distribution, and federated settings.

2.2 Representation Collapse

Representation collapse [15, 21] means representation vectors are highly correlated and simply span a lower-dimensional subspace, which is widely studied in metric learning [34], i.e., self-supervised learning [15], and supervised federated learning [36]. In federated supervised learning, FedDecorr [36] finds the dimensional collapse entanglement among server and client models, and decorrelates representations via regularizing the Frobenius norm of batch samples. However, FedDecorr relies on large batch size and deactivates lots of neuron parameters, degrading performance once the scale of clients increases [18]. To avoid representation collapse entanglement in FUSL, the proposed FUR in Fed${\text{𝚄}}^{\mathrm{2}}$ will regularize data representations to a uniform distribution that is the same among clients. In this way, decorrelating representation is not affected by the data sampling. Meanwhile, the data is uniformly dispersed into the same random distribution space, avoiding intriguing collapse impacts of client collaboration.

3 Method

Figure 1: Framework of Fed${\text{𝚄}}^{\mathrm{2}}$. For clients with agnostic self-supervised framework, FUR expands non-IID data uniformly to avoid representation collapse for FUSL. EUA in server maintains a balanced aggregation for all client models, bringing unified representations.

3.1 Federated Unsupervised Learning Formulation

We introduce the FUSL problem formulation and related assumptions in the following. Empirically, we assume a dataset decentralizes among $K$ clients, i.e., $𝒟={\cup }_{k\in [K]}{𝒟}_k$. Data distributions of different clients, i.e., ${𝒟}_k={\{{𝒙}_{k,i}\}}_{i=1}^{N_k}$, are unlabeled and non-IID in practice. FUSL can be formulated as a global objective that seeks a collaborative aggregation among clients, i.e.,

$${\mathrm{argmin}}_{𝜽}ℒ(𝜽;𝒑)={\mathrm{\Sigma }}_{k=1}^K{p}_k{𝔼}_{𝒙\sim {𝒟}_k}[{ℒ}_k(𝜽;𝒙)],$$ (1)

where ${ℒ}_k(\cdot )$ is the unsupervised model loss at client $k$, $𝒑=[p_1,\mathrm{\dots },p_K]$, and $p_k$ represents its weight ratio. The common aggregation approach is to assign the ratio of sample amount in client $k$ as the weight ratio, e.g., FedAvg [31]. Nevertheless, the client with a large amount of data will dominate in aggregating, deteriorating the optimization of other clients with inconsistent local optimums [33]. Due to privacy constraints, directly aligning client local optimums with representations is forbidden [44, 45]. Therefore, it is necessary to account for a multi-objective optimal combination, i.e., restraining the consistency between global and local model parameters.

3.2 Fed${\text{𝚄}}^{\mathrm{2}}$ Overview

Figure 2: Example of FUR. Firstly, data representations collapse in part of the spherical space. Then FUR flexibly maps data towards spherical Gaussian distribution with unbalanced optimal transport (UOT), dispersing data uniformly.

To address FUSL with non-IID data, i.e., Eq. (1), we propose Fed${\text{𝚄}}^{\mathrm{2}}$, whose framework overview is depicted in Fig. 1. There are one server and $K$ clients in Fed${\text{𝚄}}^{\mathrm{2}}$, which share the same self-supervised model, e.g., Simsiam [5], SimCLR [3], and BYOL [8]. Additionally, Fed${\text{𝚄}}^{\mathrm{2}}$ contains extra flexible uniform regularizer (FUR) module, which mitigates representation collapse without requiring prior knowledge for FUSL. We introduce the unsupervised local modeling at each client $k$, and then illustrate the communication between clients and server. For a batch of image data $𝑿$ at client $k$, we augment them with two transformations, i.e., ${𝑿}^v=T^v(𝑿)$ for $T^v\sim 𝒯$ with $𝒯$ denoting transformation set and $v\in \{1,2\}$. And feature extractor represents two views of augmented data with $d$-dimensional $l_2-$normalized representations, i.e., ${𝒁}^v={ℱ}_{{𝜽}_k}({𝑿}^v)$. Then for each view of sample representations, we maximize its representation space via approximating uniform Gaussian distribution in FUR. Meanwhile, we align two views of feature representations by predictor module (if available) and alignment module.

In one communication round, every participating client $k$ uploads its model parameters ${𝜽}_k$ to server. Next, server with efficient unified aggregator (EUA) module, first formulates the client model aggregation as a multi-objective optimization based on different client model deviation change rates, and searches for a balanced model combination. Server further restrains client parameters in a consistent space, which not only enhances the consistency between global optimum and local optimums, but also captures unified representation for data of the same class but different clients. This communication between server and clients iterates until the performance of Fed${\text{𝚄}}^{\mathrm{2}}$ converges.

3.3 FUR for Mitigating Representation Collapse

Representation collapse is a long-standing issue due to its intriguing phenomenon, e.g., constant collapse and partial/full dimensional collapse [15, 8]. In federated learning, representation collapse not only degrades the performance of local clients, but also intricately affects the representation of global and local models [36]. Besides, lacking labels, clients represent data samples to the space around local optimums, where decorrelating sample representations of limited client data suppresses capturing useful features [18].

Without ground truth labels, self-supervised learning not only keeps the invariance of the same sample with different augmentations, but also expands the uniformity of different representations to avoid representation collapse [40]. Given a batch of $B$ representations, i.e., ${𝒁}_B^1={\{{𝒛}_i^1\}}_{i\in [B]}$ and ${𝒁}_B^2={\{{𝒛}_i^2\}}_{i\in [B]}$, we train the self-supervised model by minimizing the total objective as below:

$$\begin{array}{c}ℒ={𝔼}_{T^1,T^2\sim 𝒯}{\mathrm{\ell }}_a({𝒁}_B^1,{𝒁}_B^2)+{\lambda }_U\left({\mathrm{\ell }}_u\left({𝒁}_B^1\right)+{\mathrm{\ell }}_u\left({𝒁}_B^2\right)\right),\end{array}$$ (2)

where ${\mathrm{\ell }}_a$ and ${\mathrm{\ell }}_u$ are alignment term and uniformity term, respectively. ${\lambda }_U>0$ is a hyperparameter that balances the two terms. The alignment term keeps data samples of the same class to be clustered, while others are separable, i.e., ${\mathrm{\ell }}_a({𝒁}_B^1,{𝒁}_B^2):=\frac{1}{B}{\sum }_{i\in [B]}{\Vert {𝒛}_i^1-{𝒛}_i^2\Vert }_2^2.$

The crucial of mitigating representation collapse is to enhance the representation uniformity [40]. To relieve reliance on prior knowledge of client data, we regularize local sample representations to a random distribution with high entropy. Specifically, we select samples following the spherical Gaussian distribution, i.e., $𝒔\sim 𝒩(0,1),s.t.,\parallel 𝒔\parallel =1$, as the prior. In this way, mitigating representation collapse in FUSL not only avoids leaking privacy, but also disentangles the collapse impacts among clients. Then FUR regularizes the divergence between the data representations ${𝒁}_B^v$ and a set of random samples following spherical Gaussian distribution ${𝑺}_B={\{{𝒔}_i\}}_{i\in [B]}$:

$${\mathrm{\ell }}_u\left({𝒁}_B^v\right):=\mathrm{Div}({𝒁}_B^v,{𝑺}_B).$$ (3)

Thus the uniform term ${\mathrm{\ell }}_u$ disperses uniformly to avoid representation collapse without repulsing in instance-based contrastive learning.

Since the client data is non-IID, it will break the class separation when strictively constraining sample representation to approach random instances [39]. A more flexible method is to match data samples and random Gaussian samples with arbitrary or proportional masses, i.e., leaving the sampling coupling with lower uncertainties unmatched. And unbalanced Optimal Transport (UOT) [1, 35] is one of the effective resolutions. UOT computes the transport mapping [26, 29, 28] between two sample masses of different distributions under the soft marginal constraints, e.g., $l_2-$normalization between the predicted margin and the ground truth margin. Given marginal constraints $𝒂$ and $𝒃$ for data and Gaussian distribution respectively, we formulate a UOT problem that searches a coupling matrix $𝝅$ with minimal distribution divergence:

$\underset{{\pi }_{i,j}\ge 0}{\min }{\mathrm{\ell }}_u\left({𝒁}^v\right)$ $=\mathrm{vec}{\left(𝑪\right)}^{\top }\mathrm{vec}\left(𝝅\right)+{\displaystyle \frac{{\tau }_a}{2}}\parallel {𝚽}_{\mathrm{r}}\mathrm{vec}\left(𝝅\right)-𝒂{\parallel }_2^2$ (4)

$+{\displaystyle \frac{{\tau }_b}{2}}{\Vert {𝚽}_{\mathrm{c}}\mathrm{vec}\left(𝝅\right)-𝒃\Vert }_2^2,$

where cost matrix ${𝑪}_{ij}={\Vert {𝒁}_i^v-{𝑺}_j\Vert }^2$, and ${𝚽}_{\mathrm{r}}={𝑰}_N\otimes {\mathrm{𝟙}}_N^{\top }$ ( ${𝚽}_{\mathrm{c}}={\mathrm{𝟙}}_M^{\top }\otimes {𝑰}_M$) are indicators for row-wise (column-wise) Kronecker multiplication with $𝑰$ denoting identity matrix. Optimization. Denoting ${\tau }_a{𝚽}_{\mathrm{p}}^{\top }{𝚽}_{\mathrm{r}}+{\tau }_b{𝚽}_c^{\top }{𝚽}_c=𝑸$ and $\mathrm{vec}(𝑪)-{\tau }_a{𝚽}_{\mathrm{r}}^{\top }𝒂-{\tau }_b{𝚽}_c^{\top }𝒃=𝒘$, we rewrite Eq. (4) as a positive definite quadratic form:

$$\underset{{\pi }_{i,j}\ge 0}{\min }{\mathrm{\ell }}_u\left({𝒁}^v\right)=\frac{1}{2}\mathrm{vec}{\left(𝝅\right)}^{\top }𝑸\mathrm{vec}\left(𝝅\right)+{𝒘}^{\top }\mathrm{vec}\left(𝝅\right)+𝛀,$$ (5)

where constant $𝛀=\frac{1}{2}({\tau }_a{𝒂}^{\top }𝒂+{\tau }_b{𝒃}^{\top }𝒃)$. Next we optimize $𝝅$ via steepest gradient descent as bellow:

$$\mathrm{vec}\left({𝝅}^{\left(\text{new}\right)}\right)=\max (0,\mathrm{vec}\left({𝝅}^{\text{(old})}\right)-{\eta }^{*}\frac{\partial {\mathrm{\ell }}_u\left({𝒁}^v\right)}{\partial \mathrm{vec}\left({𝝅}^{\left(\text{old}\right)}\right)}),$$ (6)

with ${\eta }^{*}=\frac{{\left({𝑸}_{\mathrm{vec}}\left({𝝅}^{(\text{old})}\right)+𝒘\right)}^{\top }\left({𝑸}_{\mathrm{vec}}\left({𝝅}^{(\text{old})}\right)+𝒘\right)}{{\left({𝑸}_{\mathrm{vec}}\left({𝝅}^{(\text{old})}\right)+𝒘\right)}^{\top }𝑸\left(𝑸\mathrm{vec}\left({𝝅}^{(\text{old})}\right)+𝒘\right)}$. Finally, we obtain the uniform UOT divergence by taking the optimal ${𝝅}^{*}$ back to Eq. (4). FUR minimizes UOT divergence to regularize data samples approaching the spherical Gaussian distribution. Note that the spherical Gaussian distribution maximizes its entropy and distributes its samples uniformly. The mapped data representations enjoy the above nice properties of spherical Gaussian distribution and further mitigate the representation collapse entanglement.

3.4 EUA for Generalizing Unified Representation

Due to non-IID client data, clients optimize to their local optimums with inconsistent model parameters, causing inconsistent even conflicting model deviations from server to clients. Without the guidance of supervision signals, i.e., data labels, this problem further exacerbates in representing data of the same class but different clients, towards inconsistent spaces. Thus it is vital to constrain the consistency among client models in parameter spaces, which further guarantees unified representations.

In round $t$, the impact of global aggregation on $k$-th local optimization can be measured with the model deviation change rate, i.e.,

$c_k(\eta ,{𝒅}^t)={\displaystyle \frac{u_k\left({𝜽}_g^t\right)-u_k\left({𝜽}_g^{t+1}\right)}{u_k\left({𝜽}_g^t\right)}}\approx {\eta }_g\nabla \log u_k\left({𝜽}_g^t\right){𝒅}^t,$ (7)

where $u_k({𝜽}_g^t)={\Vert {𝜽}_g^t-{𝜽}_k^t\Vert }^2$ is the model deviation from server to client $k$ [16, 12], and global model optimization is ${𝜽}_g^{t+1}={𝜽}_g^t-\eta {𝒅}^t$ with updating direction ${𝒅}^t$ and step size $\eta$. Overlooking inconsistent model deviations, global aggregated model inevitably gets close to a subset of clients while deviating from others. It corresponds that clients get close to the global model increase the model deviation change rate, and clients away decrease it [12, 32]. Motivated by this, we seek the clients with the worst model deviation change rate, and correct the global optimization with a direction maximizing the overall worst model deviation change rate. This can be formulated as a multi-objective optimization, which benefits for mitigating the inconsistencies and conflicts among clients [12, 42], i.e.,

$\underset{{𝒅}^t}{\max }\underset{𝒑\in {𝕊}^K}{\min }{\eta }_g{\displaystyle \sum _{k=1}^K}{p}_k\nabla \log u_k\left({𝜽}_g^t\right){𝒅}^t,$ (8)

$s.t.{\Vert {𝒅}^t\Vert }^2\le 1,{𝒑}^{\top }\mathrm{𝟏}=1,p_k\ge 0,$

where $𝒑$ denotes the weights for different clients.

Optimization. Adding the constraints as Lagrange multipliers, Eq. (8) can be rewritten as:

(9)

Differentiating Eq. (9) with regarding to ${𝒅}^t$, we have ${𝒅}^{*}=\frac{{\eta }_g}{\varphi }{(\nabla \log 𝒖)}^{\top }𝒑$, where $𝒖=(u_1({𝜽}_g^t),\mathrm{\dots },u_K({𝜽}_g^t))$. Taking back to Eq. (9), we can obtain its strong dual form with dual variable $𝒑$:

$$𝑱=\underset{𝒑}{\min }\frac{{\eta }_g^2}{2\varphi }{\Vert {\left(\nabla \log 𝒖\right)}^{\top }𝒑\Vert }^2=\underset{𝒑}{\min }\frac{{\eta }_g^2}{2\varphi }{𝒑}^{\top }𝑮𝒑,$$ (10)

where $𝑮={(\nabla \log 𝒖)}^{\top }(\nabla \log 𝒖)$. Then we can rewrite it as an augmented Lagrangian form,

$𝑱=\underset{𝒑}{\min }{\displaystyle \frac{{\eta }_g^2}{2\varphi }}{𝒑}^{\top }𝑮𝒑+\mu \left({𝒑}^{\top }\mathrm{𝟏}-1\right)+{\displaystyle \frac{\rho }{2}}{\Vert {𝒑}^{\top }\mathrm{𝟏}-1\Vert }^2,$ (11)

where $\mu$ denotes the Lagrange multipliers. This can be iteratively solved by alternating direction method of multipliers (ADMM) algorithm [7, 27], i.e., fixing $\mu$ to optimize $𝒑$, and vice versa:

(12)

The ADMM iteration guarantees exact solution in minimal computation complexity, then the global model updates towards ${𝒅}^{*}$ with step size $\eta$.

Theorem 1 (Optimization consistency of model deviations).

Rethinking the Lagrangian of dual form in Eq. (10),

$$𝑱=\underset{𝒑}{\min }\frac{{\eta }_g^2}{2\varphi }{\Vert {(\nabla \log 𝒖)}^{\top }𝒑\Vert }^2+\lambda {𝒑}^{\top }\mathrm{𝟏},$$ (13)

it holds $\mathrm{\nabla }\log \mathit{}\mathrm{(}{u}_i\mathit{}\mathrm{\left(}{𝛉}_g^t\mathrm{\right)}\mathrm{)}\mathrm{=}\mathrm{\nabla }\log \mathit{}\mathrm{(}{u}_j\mathit{}\mathrm{\left(}{𝛉}_g^t\mathrm{\right)}\mathrm{)}$, $\mathrm{\forall }i\mathrm{\ne }j\mathrm{\in }\mathrm{[}K\mathrm{]}$.

Proof.

We provide the proof details in Appendix A.1. ∎

After the convergence of global and local optimization, EUA balances the model deviation change rate among all clients, making the global aggregation improves all model equivalently. Therefore, all models optimize towards a consistent parameter spaces, obtaining unified representation.

3.5 Overall Algorithm and Convergence Analysis

We describe the overall algorithm of Fed${\text{𝚄}}^{\mathrm{2}}$ in Algo. 1. In detail, the server collaborates with clients in steps 1:10. After collecting participating client models in step 8, server uses EUA to reach a consistent model updating and obtain unified representations. The client executes self-supervised modeling in steps 11:21, where FUR enhances uniform representations to avoid collapse entanglement in step 17.

Convergence Analysis. In the following, we take four mild assumptions [23], and provide the generalization bounds of model divergence and overall convergence error.

Assumption 1.

Let $F_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}$ be the expected model objective for client $k$, and assume $F_{\mathrm{1}}\mathrm{,}\mathrm{\cdots }\mathrm{,}{F}_K$ are all L-smooth, i.e., for all ${𝛉}_k$, $F_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{\le }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}{\mathrm{(}{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{)}}^{\mathrm{\top }}\mathit{}\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}\frac{L}{\mathrm{2}}\mathit{}{\mathrm{\Vert }{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{\Vert }}_{\mathrm{2}}^{\mathrm{2}}$.

Assumption 2.

Let $F_{\mathrm{1}}\mathrm{,}\mathrm{\cdots }\mathrm{,}{F}_N$ are all $\mu$-strongly convex: for all ${𝛉}_k$, $F_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{\ge }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}{\mathrm{(}{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{)}}^{\mathrm{\top }}\mathit{}\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}\frac{\mu }{\mathrm{2}}\mathit{}{\mathrm{\Vert }{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{\Vert }}_{\mathrm{2}}^{\mathrm{2}}$.

Assumption 3.

Let ${\xi }_k^t$ be sampled from the $k$-th client’s local data uniformly at random. The variance of stochastic gradients in each client is bounded: $𝔼\mathit{}{\mathrm{\Vert }\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k^t\mathrm{,}{\xi }_k^t\mathrm{)}\mathrm{-}\mathrm{\nabla }{F}_k\mathit{}\mathrm{\left(}{𝛉}_k^t\mathrm{\right)}\mathrm{\Vert }}^{\mathrm{2}}\mathrm{\le }{\sigma }_k^{\mathrm{2}}$.

Assumption 4.

The expected squared norm of stochastic gradients is uniformly bounded, i.e., $𝔼\mathit{}{\mathrm{\Vert }\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k^t\mathrm{,}{\xi }_k^t\mathrm{)}\mathrm{\Vert }}^{\mathrm{2}}\mathrm{\le }{V}^{\mathrm{2}}$ for all $k\mathrm{=}\mathrm{1}\mathrm{,}\mathrm{\cdots }\mathrm{,}N$ and $t\mathrm{=}\mathrm{1}\mathrm{,}\mathrm{\cdots }\mathrm{,}T\mathrm{-}\mathrm{1}$

Lemma 1 (Bound of Client Model Divergence).

With assumption 4, ${\eta }_t$ is non-increasing and (learning rate of t-th round and E-th epoch) for all $t\mathrm{\ge }\mathrm{0}$, there exists $t_{\mathrm{0}}\mathrm{\le }t$, such that $t\mathrm{-}{t}_{\mathrm{0}}\mathrm{\le }E\mathrm{-}\mathrm{1}$ and ${𝛉}_k^{t_{\mathrm{0}}}\mathrm{=}{𝛉}^{t_{\mathrm{0}}}$ for all $k\mathrm{\in }\mathrm{[}N\mathrm{]}$. It follows that

$$𝔼\left[\sum _k^K{p}_k{\Vert {𝜽}^t-{𝜽}_k^t\Vert }^2\right]\le 4{\eta }_t^2{(E-1)}^2{V}^2.$$ (14)

Proof.

We provide the proof details in Appendix A.2. ∎

Theorem 2 (Convergence Error Bound).

Let assumptions 1-4 hold, and $L\mathrm{,}\mu \mathrm{,}{\sigma }_k\mathrm{,}V$ be defined therein. Let $\kappa \mathrm{=}\frac{L}{\mu }\mathrm{,}\gamma \mathrm{=}\max \mathit{}\mathrm{\{}\mathrm{8}\mathit{}\kappa \mathrm{,}E\mathrm{\}}$ and the learning rate ${\eta }_t\mathrm{=}\frac{\mathrm{2}}{\mu \mathit{}\mathrm{(}\gamma \mathrm{+}t\mathrm{)}}$. The Fed${\text{𝚄}}^{\mathrm{2}}$ with full client participation satisfies

$$𝔼\left[F\left({\overline{𝜽}}^t\right)\right]-F^{*}\le \frac{\kappa }{\gamma +t}\left(\frac{2B}{\mu }+\frac{\mu (\gamma +1)}{2}{\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2\right),$$

where $B\mathrm{=}\mathrm{4}\mathit{}{\mathrm{(}E\mathrm{-}\mathrm{1}\mathrm{)}}^{\mathrm{2}}\mathit{}{V}^{\mathrm{2}}\mathrm{+}K\mathrm{+}\mathrm{2}\mathit{}\mathrm{\Gamma }$.

Proof.

We provide the proof details in Appendix A.3. ∎

Algorithm 1 Training procedure of Fed${\text{𝚄}}^{\mathrm{2}}$

Input: Batch size $B$, communication rounds $T$, number of clients $K$, local steps $E$, dataset $𝒟={\cup }_{k\in [K]}{𝒟}_k$
Output: Global model ${𝜽}^T$

1: Server executes():

2: Initialize ${𝜽}^0$ with random distribution

3: for $t=0,1,\mathrm{\dots },T-1$ do

4: for $k=1,2,\mathrm{\dots },K$ in parallel do

5: Send ${𝜽}^t$ to client $k$

6: ${𝜽}_k^{t+1}\leftarrow$ Client executes($k$, ${𝜽}^t$)

7: end for

8: EUA optimize Eq. (10) for ${𝒑}^{*}$ and update global model ${𝜽}^{t+1}$ with optimal direction ${𝒅}^t$ in Eq. (9)

9: end for

10: return ${𝜽}^T$

11: Client executes($k$, ${𝜽}^t$):

12: Assign global model to the local model ${𝜽}_k^t\leftarrow {𝜽}^t$

13: for each local epoch $e=1,2,\mathrm{\dots },E$ do

14: for batch of samples ${𝑿}_{k,B}\in {𝒟}_k$ do

15: Augment samples ${𝑿}_{k,B}^v=T_{v\in \{1,2\}}^v({𝑿}_{k,B})$

16: Feature extraction ${𝒁}_{k,B}^v\leftarrow {ℱ}_{{𝜽}_k^e}({𝑿}_{k,B}^v)$

17: FUR enhances the uniformity of ${𝒁}_{k,B}^v$ by Eq. (4)

18: Compute total loss in Eq. (2) and update ${𝜽}_k^e$

19: end for

20: end for

21: return ${𝜽}_k^E$ to server

4 Experiments

4.1 Experimental Setups

Datasets.We adopt two benchmark datasets, i.e., CIFAR10 and CIFAR100 [20], to evaluate Fed${\text{𝚄}}^{\mathrm{2}}$. Both datasets have 50,000 training samples and 10,000 test samples, but differ in the number of classes. Following FedEMA [45], we simulate non-IID data distribution in $K$ clients by assuming class priors follow the Dirichlet distribution parameterized with non-IID degree, i.e., $\alpha$ [11]. The smaller $\alpha$ simulates the more non-IID federated setting. We conduct extensive experiments in both cross-silo ($K=10$) and cross-device ($K=100$) settings to validate performance generalization.

Comparison Methods. We compare Fed${\text{𝚄}}^{\mathrm{2}}$ with three categories of approaches , i.e., (1) combining the existing centralized self-supervised model with FedAvg [31]: FedSimsiam, FedSimCLR, and FedBYOL, (2) the state-of-the-art FUSL methods: FedU [44], FedEMA [45], FedX [9], Orchestra [30], and L-DAWA [33], and (3) adapting existing federated supervised learning models solving representation collapse to FUSL: FedDecorr [36]. Firstly, we evaluate the representation performance of the above methods with their best-performing models on both cross-silo and cross-device settings on CIFAR10 and CIFAR100 ($\alpha =0.1$). Secondly, we study the effectiveness of different methods with the same model, i.e., BYOL, for different non-IID degrees [25], i.e., $\alpha =\{0.1,0.5,5\}$. We evaluate the pre-trained encoder model via the accuracy of KNN [4], standard linear probing [2], and semi-supervised methods (i.e., fine-tuning 1% and 10% labeled data). All of the above metrics illustrate better performance when the values are higher.

Implemental Details. We conduct image augmentation for SimCLR, BYOL, and SimSiam, following their original papers. We adopt ResNet18 [10] as an encoder module, choose Projector/Predictor architectures like original papers, and optimize each model 5 local epochs per communication round until converging. We set all datasets with batch size as 128 and embedding dimension as 512. To obtain fair comparisons, we conduct every experiment for each method with its best hyper-parameters, and report the average result of 3 repetitions. We choose Adam [19] as the optimizer for each local model, and SGD [37] for updating the global model. We set the uniformity effect ${\lambda }_U=0.1$, the soft margin constraints ${\tau }_a={\tau }_b=0.8$, the coefficient of constraints $\varphi =0.1$, and the coefficient in ADMM $\rho =1$.

Table 1: Accuracy (%) of linear probing (LP), fine-tuning (FT) 1%, and 10% labeled data on CIFAR10 and CIFAR100 ($\alpha =0.1$).

Dataset	CIFAR10						CIFAR100
Setting $\alpha =0.1$	Cross-Device (K=100)			Cross-Silo (K=10)			Cross-Device (K=100)			Cross-Silo (K=10)
Method \Evaluation	LP	FT 1%	FT 10%	LP	FT 1%	FT 10%	LP	FT 1%	FT 10%	LP	FT 1%	FT 10%
FedSimsiam	60.49	44.45	70.46	70.61	57.60	69.88	31.91	12.58	37.33	49.81	21.64	43.08
FedDecorr-Simsiam	43.18	35.15	58.68	74.56	65.21	80.10	17.09	5.36	20.60	47.93	20.53	45.21
Fed${\text{𝚄}}^{\mathrm{2}}$-Simsiam	68.50	56.43	75.33	84.92	77.11	85.21	35.59	13.08	38.22	56.55	31.42	48.75
FedSimCLR	65.76	51.18	68.33	75.65	62.86	76.15	37.09	11.73	31.97	51.62	19.47	41.60
L-DAWA-SimCLR	65.63	49.66	69.89	75.48	63.4	78.66	41.28	13.52	36.56	51.11	21.07	45.02
FedX-SimCLR	67.33	49.96	70.18	78.29	65.03	79.43	38.11	11.18	33.96	51.67	19.65	42.38
Fed${\text{𝚄}}^{\mathrm{2}}$-SimCLR	66.49	51.77	70.76	82.37	69.84	82.39	41.56	14.32	36.90	56.56	26.11	47.99
FedBYOL	61.46	54.36	74.01	83.29	74.04	81.40	28.27	10.43	34.90	48.78	19.79	42.82
FedU-BYOL	60.15	53.53	74.62	82.33	69.24	83.37	28.09	10.46	36.06	58.02	28.38	48.12
FedEMA-BYOL	62.27	54.91	74.76	82.17	71.37	83.78	28.40	10.63	35.62	57.25	30.03	50.33
Orchestra	38.66	41.62	62.97	83.53	78.44	85.40	17.91	6.96	23.40	51.31	26.36	48.85
Fed${\text{𝚄}}^{\mathrm{2}}$-BYOL	67.62	54.74	74.93	85.58	78.64	86.24	38.09	13.16	36.87	59.71	34.83	53.87

Table 2: KNN accuracy (%) of different $\alpha$ on CIFAR10 and CIFAR100 for cross-silo settings.

Dataset	CIFAR10			CIFAR100
Method \$\alpha$	0.1	0.5	5	0.1	0.5	5
FedBYOL	76.12	77.23	82.71	38.13	43.93	45.30
FedU-BYOL	79.09	79.68	82.75	51.31	51.81	52.05
FedEMA-BYOL	80.32	82.01	82.80	53.18	53.18	53.28
FedDecorr-BYOL	76.76	79.66	81.09	49.87	49.54	52.07
L-DAWA-BYOL	65.40	66.17	82.82	23.09	51.10	52.25
FedX-BYOL	50.94	40.96	41.05	15.83	16.35	16.89
Orchestra	79.25	76.78	76.30	38.52	37.17	34.93
Fed${\text{𝚄}}^{\mathrm{2}}$-FUR-BYOL	81.04	82.18	83.45	53.45	53.86	54.34
Fed${\text{𝚄}}^{\mathrm{2}}$-EUA-BYOL	80.93	82.14	84.01	53.20	53.72	54.61
Fed${\text{𝚄}}^{\mathrm{2}}$-BYOL	81.39	82.21	84.79	53.87	54.07	55.06

4.2 Experimental Results

Representation Performance Comparison. Firstly, we follow the existing FUSL methods [45, 30, 33], to evaluate the performance of pre-trained models learned in Tab. 1. We group the state-of-the-art methods in terms of their best-performing models, i.e., Simsiam, SimCLR, and BYOL. For the first group, we can observe that FedDecorr performs the worst, especially on CIFAR100 cross-device task. It indicates that directly avoiding collapse via decorrelating a batch of data representations is unsuitable for FUSL with limited data and severely heterogeneous data distribution. In terms of the second group, compared with FedX, L-DAWA performs better on CIFAR100, while is less competitive on CIFAR10. We can conclude that: (1) L-DAWA can better control model divergence when clients have inconsistent optimums, and (2) L-DAWA fails to obtain discriminative representations since it takes no action to representation collapse. On mentioned the third group, Orchestra captures global supervision signals to guide data representation, whose effectiveness suffers from randomness. In general, directly combining existing self-supervised model with FedAvg cannot tackle FUSL with non-IID data well. Cross-device simulation on CIFAR100 is so challenging that some existing methods fail dramatically. Moreover, Fed${\text{𝚄}}^{\mathrm{2}}$ is agnostic to self-supervised model and performs better than existing work, which validates the superiority of enhancing uniform and unified representations.

Effect of Heterogeneity on Generalization. Next, we report the KNN-accuracy of cross-silo methods on CIAFR10 and CIFAR100 in Tab. 2, for validating the performance generalization. We choose the same model, i.e., BYOL, to be comparable among all FUSL methods. We can discover that: (1) Most FUSL methods increase their performance when the non-IID degree $\alpha$ increases, and the performance variances among different methods increase with the decreasing of $\alpha$. (2) FedEMA-BYOL is not sensitive to the non-IID degrees, while Orchestra behaves on opposite. This states that capturing global supervision signals to guide local representation suffers from clustering randomness. (3) Fed${\text{𝚄}}^{\mathrm{2}}$ performs the best among all tasks, even in $\alpha =0.1$, illustrating its performance generalization.

Ablation Studies. In Tab. 2, we also consider two variants of Fed${\text{𝚄}}^{\mathrm{2}}$: (1) Fed${\text{𝚄}}^{\mathrm{2}}$ removes FUR, i.e., Fed${\text{𝚄}}^{\mathrm{2}}$-FUR, (2) Fed${\text{𝚄}}^{\mathrm{2}}$ removes EUA, i.e., Fed${\text{𝚄}}^{\mathrm{2}}$-EUA, to study the effect of each module. From Tab. 2, we can see that either applying FUR or EUA can enhance representations, since they have better performance than the existing FUSL methods. Compared with Fed${\text{𝚄}}^{\mathrm{2}}$, Fed${\text{𝚄}}^{\mathrm{2}}$-FUR and Fed${\text{𝚄}}^{\mathrm{2}}$-EUA drop KNN accuracy slightly, validating the effectiveness of tackle two challenges, i.e., representation collapse entanglement, and generating unified representations. Fed${\text{𝚄}}^{\mathrm{2}}$-FUR performs better than Fed${\text{𝚄}}^{\mathrm{2}}$-EUA when $\alpha =0.1$, while gets worse when $\alpha =5$.

Figure 3: Top k log singular values of the covariance matrix of global model (left) and local model (right) representations.

(a)

(b)

(c)

(d)

Figure 4: The representations collapse issue on the sphere using BYOL model (on CIFAR10 $\alpha =0.1$ Cross-silo). The more blank representation space means the more severe collapse issue is.

4.3 Representation Visualization

Analysis of Representation Collapse Entanglement. To study the representation collapse entanglement caused by non-IID data, we capture the representation covariance matrices of $N$ test data points on CIFAR10 from both the global and local BYOL models of FUSL methods, i.e., FedDecorr, L-DAWA, FedBYOL, and Fed${\text{𝚄}}^{\mathrm{2}}$. And we utilize the singular value decomposition on each of the representation covariance matrices, and visualize the top-100 singular values in Fig. 3. Both L-DAWA and FedBYOL suffer from severe representation collapse, because they have less singular values beyond 0 than FedDecorr and Fed${\text{𝚄}}^{\mathrm{2}}$. The representation collapses in global model and local model are consistent, proving that collapse impacts are entangled intricately. Compared with the singular values decomposed from the covariance matrix of Gaussian random samples, the singular values of FedDecorr and Fed${\text{𝚄}}^{\mathrm{2}}$are not similar. Because a fully uniform distribution breaks down the alignment effect and deteriorates clustering. Furthermore, in Fig. 4, we visualize the representation collapse on 3-D spherical space, where the existing FUSL methods leave evident blank space and suffer from collapse entanglements.

Analysis of Unified Representation. We also use t-SNE [38] to picture the 2-D representation of both global (circle) and local (cross) BYOL models in Fig. 5. There are three interesting conclusions: Firstly, Fed${\text{𝚄}}^{\mathrm{2}}$ has clearer cluster boundary than FedDecorr, validating that directly decomposing the Frobenius norm of representations deteriorates generalization. Secondly, compared with FeBYOL, the global and local representations of L-DAWA are looser, implying its ineffectiveness in controlling conflicting model deviations. Lastly, with the effect of EUA, Fed${\text{𝚄}}^{\mathrm{2}}$ achieves tighter distribution consistency between global and local representations, as well as more clear cluster bound.

(a)

(b)

(c)

(d)

(e)

Figure 5: The distributions of data representations using global and local BYOL model (on CIFAR10 $\alpha =0.1$ Cross-silo).

(a)

(b)

Figure 6: The effect of ${\lambda }_U$ (on CIFAR10 $\alpha =0.1$ Cross-silo).

Hyper-parameters sensitivity. We consider the sensitivity of highly relevant hyper-parameters, i.e., the effect of uniformity term ${\lambda }_U=\{0,0.01,0.05,0.1,0.2,0.5\}$ on Cifar10 Cross-silo ($\alpha =0.1)$, in Fig. 6. We set ${\lambda }_U=0.1$ in experiments since it reaches the highest performance. And we leave the number of clients $K=\{5,10,20,50,100\}$ and the local epochs $E=\{5,10,20,50\}$ in Appendix B.

5 Conclusion

In this work, we propose a FUSL framework, i.e., Fed${\text{𝚄}}^{\mathrm{2}}$, to enhance Uniform and Unified representation. Fed${\text{𝚄}}^{\mathrm{2}}$ consists of flexible uniform regularizer (FUR) and efficient unified aggregator (EUA). FUR encourages data representations to uniformly distribute in a spherical Gaussian space, mitigating representation collapse and its subsequent entangled impacts. EUA further constrains the consistent optimization improvements among different client models, which is good for unified representation. In our empirical studies, we set both cross-silo and cross-device settings, and conduct experiments on CIFAR10 and CIFAR100 datasets, which extensively validate the superiority of Fed${\text{𝚄}}^{\mathrm{2}}$.

Acknowledgements. This work was supported by National Key R&D Program of China (2022YFB4501500, 2022YFB4501504), and the National Natural Science Foundation of China (No.72192823).

In the supplemental materials, we provide the theoretical analysis in Appendix A, and the additional experimental details in Appendix B.

Appendix A Theoretical Analysis

A.1 Optimization Consistency of Model Deviation

In the following, we provide the theorem related to optimization consistency of model deviations. When the aggregating weights, i.e., $𝒑$, achieve optimal, the model deviation rate is equally contributed to global updating.

Theorem 3 (Optimization consistency of model deviations).

Rethinking the Lagrangian of dual form in Eq. (10),

$$𝑱=\underset{𝒑}{\min }\frac{{\eta }_g^2}{2\varphi }{\Vert {(\nabla \log 𝒖)}^{\top }𝒑\Vert }^2+{\lambda }_E{𝒑}^{\top }\mathrm{𝟏},$$ (15)

it holds $\mathrm{\nabla }\log \mathit{}\mathrm{(}{u}_i\mathit{}\mathrm{\left(}{𝛉}_g^t\mathrm{\right)}\mathrm{)}\mathrm{=}\mathrm{\nabla }\log \mathit{}\mathrm{(}{u}_j\mathit{}\mathrm{\left(}{𝛉}_g^t\mathrm{\right)}\mathrm{)}$, $\mathrm{\forall }i\mathrm{\ne }j\mathrm{\in }\mathrm{[}K\mathrm{]}$.

Proof.

By deviating Eq. (15) with regarding to $𝒑$, we can obtain $(\nabla \log 𝒖)\nabla \log {(𝒖)}^{\top }{𝒑}^{*}=-\frac{\varphi {\lambda }_E}{{\eta }_g^2}𝑰$. Remind that ${𝒅}^{*}=\frac{{\eta }_g}{\varphi }{(\nabla \log 𝒖)}^{\top }{𝒑}^{*}$, for client $i\ne j\in [K]$ and ${\eta }_g\to 0$, we finally have consistent model deviation change rate as below:

$\underset{{\eta }_g\to 0}{lim}{c}_i({\eta }_g,{𝒅}^{*})=\nabla \log (u_i\left({𝜽}_g^{*}\right)){𝒅}^{*}$ (16)

$=\nabla \log (u_j\left({𝜽}_g^{*}\right)){𝒅}^{*}=\underset{{\eta }_g\to 0}{lim}{c}_j({\eta }_g,{𝒅}^{*}).$

∎

Therefore, the global model updates with a direction that balances all model deviation change rates, obtaining consistent parameters for server and client models.

A.2 Bound of Client Model Divergence

In this part, we first introduce mild and general assumptions [23], and induct the model updating divergence bound for each client.

Assumption 5.

Let $F_k\mathit{}\mathrm{(}𝛉\mathrm{)}$ be the expected model objective for client $k$, and assume $F_{\mathrm{1}}\mathrm{,}\mathrm{\cdots }\mathrm{,}{F}_K$ are all L-smooth, i.e., for all ${𝛉}_k$, $F_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{\le }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}{\mathrm{(}{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{)}}^{\mathrm{\top }}\mathit{}\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}\frac{L}{\mathrm{2}}\mathit{}{\mathrm{\Vert }{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{\Vert }}_{\mathrm{2}}^{\mathrm{2}}$.

Assumption 6.

Let $F_{\mathrm{1}}\mathrm{,}\mathrm{\cdots }\mathrm{,}{F}_N$ are all $\mu$-strongly convex: for all ${𝛉}_k$, $F_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{\ge }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}{\mathrm{(}{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{)}}^{\mathrm{\top }}\mathit{}\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k\mathrm{)}\mathrm{+}\frac{\mu }{\mathrm{2}}\mathit{}{\mathrm{\Vert }{𝛉}_k\mathrm{-}{𝛉}_k\mathrm{\Vert }}_{\mathrm{2}}^{\mathrm{2}}$.

Assumption 7.

Let ${\xi }_k^t$ be sampled from the $k$-th client’s local data uniformly at random. The variance of stochastic gradients in each client is bounded: $𝔼\mathit{}{\mathrm{\Vert }\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k^t\mathrm{,}{\xi }_k^t\mathrm{)}\mathrm{-}\mathrm{\nabla }{F}_k\mathit{}\mathrm{\left(}{𝛉}_k^t\mathrm{\right)}\mathrm{\Vert }}^{\mathrm{2}}\mathrm{\le }{\sigma }_k^{\mathrm{2}}$.

Assumption 8.

The expected squared norm of stochastic gradients is uniformly bounded, i.e., $𝔼\mathit{}{\mathrm{\Vert }\mathrm{\nabla }{F}_k\mathit{}\mathrm{(}{𝛉}_k^t\mathrm{,}{\xi }_k^t\mathrm{)}\mathrm{\Vert }}^{\mathrm{2}}\mathrm{\le }{V}^{\mathrm{2}}$ for all $k\mathrm{=}\mathrm{1}\mathrm{,}\mathrm{\cdots }\mathrm{,}N$ and $t\mathrm{=}\mathrm{1}\mathrm{,}\mathrm{\cdots }\mathrm{,}T\mathrm{-}\mathrm{1}$

Next, we introduce the lemma related to the bound of client model divergence.

Lemma 2 (Bound of Client Model Divergence).

With assumption 8, ${\eta }_t$ is non-increasing and (learning rate of t-th round and E-th epoch) for all $t\mathrm{\ge }\mathrm{0}$, there exists $t_{\mathrm{0}}\mathrm{\le }t$, such that $t\mathrm{-}{t}_{\mathrm{0}}\mathrm{\le }E\mathrm{-}\mathrm{1}$ and ${𝛉}_k^{t_{\mathrm{0}}}\mathrm{=}{𝛉}^{t_{\mathrm{0}}}$ for all $k\mathrm{\in }\mathrm{[}N\mathrm{]}$. It follows that

$$𝔼\left[\sum _k^K{p}_k{\Vert {𝜽}^t-{𝜽}_k^t\Vert }^2\right]\le 4{\eta }_t^2{(E-1)}^2{V}^2.$$ (17)

Proof.

Let $E$ be the maximal local epoch. For any round $t>0$, communication rounds from $t$ to $t_0$ exist . and the global model ${𝜽}^{t_0}$ and each local model ${𝜽}_k^{t_0}$ are same at round $t_0$.

$𝔼\left[{\displaystyle \sum _k^K}{p}_k{\Vert {𝜽}^t-{𝜽}_k^t\Vert }^2\right]$

$=𝔼\left[{\displaystyle \sum _k^K}{p}_k{\Vert \left({𝜽}_k^t-{𝜽}^{t_0}\right)-\left({𝜽}^t-{𝜽}^{t_0}\right)\Vert }^2\right]$ $\left(17a\right)$

$\le 𝔼{\displaystyle \sum _k^K}{p}_k{\Vert {𝜽}_k^t-{𝜽}^{t_0}\Vert }^2$ $\left(17b\right)$

$=𝔼{\displaystyle \sum _k^K}{p}_k{\Vert {\displaystyle \sum _{t=t0}^{t-1}}{\eta }_t\nabla F_k({𝜽}_k^t,{\xi }_k^t)\Vert }^2$ $\left(17c\right)$

$\le 𝔼{\displaystyle \sum _k^K}{p}_k\left(t-t_0\right){\displaystyle \sum _{t=t0}^{t-1}}{\eta }_{t_0}^2{\Vert \nabla F_k({𝜽}_t^k,{\xi }_k^t)\Vert }^2$ $\left(17d\right)$

$\le 4{\eta }_t^2{\left(E-1\right)}^2{V}^2,$ $\left(17e\right)$

where the Eq. (17b) holds since $𝔼({𝜽}_k^t-{𝜽}^{t_0})={𝜽}^t-{𝜽}^{t_0}$, and $𝔼\Vert X-𝔼(X)\Vert \le 𝔼\Vert X\Vert$, and Eq. (17d) derives from Jensen inequality. ∎

A.3 Convergence Error Bound

Definition 1 (Heterogeneity Quantification [23]).

Let $F^{*}$ and $F_k^{*}$ be the minimum values of $F$ and $F_k$, respectively. We use the term $\mathrm{\Gamma }=F^{*}-{\sum }_{k=1}^N{p}_k{F}_k^{*}$ for quantifying the degree of non-IID. If the data are IID, then $\mathrm{\Gamma }$ obviously goes to zero as the number of samples grows. If the data are non-IID, then $\mathrm{\Gamma }$ is nonzero, and its magnitude reflects the heterogeneity of the data distribution.

Theorem 4 (Convergence Error Bound).

Let assumptions 5-8 hold, and $L\mathrm{,}\mu \mathrm{,}{\sigma }_k\mathrm{,}V$ be defined therein. Let $\kappa \mathrm{=}\frac{L}{\mu }\mathrm{,}\gamma \mathrm{=}\max \mathit{}\mathrm{\{}\mathrm{8}\mathit{}\kappa \mathrm{,}E\mathrm{\}}$ and the learning rate ${\eta }_t\mathrm{=}\frac{\mathrm{2}}{\mu \mathit{}\mathrm{(}\gamma \mathrm{+}t\mathrm{)}}$. The Fed${\text{𝚄}}^{\mathrm{2}}$ with full client participation satisfies

$$𝔼\left[F\left({\overline{𝜽}}^t\right)\right]-F^{*}\le \frac{\kappa }{\gamma +t}\left(\frac{2B}{\mu }+\frac{\mu (\gamma +1)}{2}{\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2\right),$$

where $B\mathrm{=}\mathrm{4}\mathit{}{\mathrm{(}E\mathrm{-}\mathrm{1}\mathrm{)}}^{\mathrm{2}}\mathit{}{V}^{\mathrm{2}}\mathrm{+}K\mathrm{+}\mathrm{2}\mathit{}\mathrm{\Gamma }$.

Proof.

By L-smooth assumption 5, we can obtain:

$𝔼\left[F({𝜽}^t)-F({𝜽}^{*})\right]$ (18)

$\le 𝔼\left[{({𝜽}^t-{𝜽}^{*})}^{\top }\nabla F\left({𝜽}^{*}\right)+{\displaystyle \frac{L}{2}}{\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2\right]$

$=𝔼\left[{\displaystyle \frac{L}{2}}{\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2\right].$

Since the updating in EUA is ${𝜽}^{t+1}={𝜽}^t-{\eta }_t{𝒅}^t$ for ${𝒅}_t^{*}=\frac{{\eta }_t}{\varphi }{(\nabla \log {𝒖}^t)}^{\top }{𝒑}_t^{*}={\sum }_k^K{p}_k\nabla F_k({𝜽}_k^t)$, we can rewrite it as:

${\Vert {𝜽}^{t+1}-{𝜽}^{*}\Vert }^2$ (19)

$={\Vert {𝜽}^t-{\eta }_t{𝒅}_t-{𝜽}^{*}\Vert }^2$

Next, we induce the bound of the second term.

(20)

By Cauchy-Schwarz inequality and AM-GM inequality, we have inequality of the first term of Eq. (20):

(21)

By the $\mu -$strong convexity of $F_k(\cdot )$, we have

(22)

In Theorem 3, we get ${(\nabla \log 𝒖)}^{\top }\nabla \log (𝒖){𝒑}^{*}=-\frac{\varphi {\lambda }_E}{{\eta }_t^2}𝑰$, which indicates that:

${\Vert {𝒅}_t\Vert }^2$ $={\displaystyle \frac{{\eta }_t^2}{\varphi }}{\Vert {\left(\nabla \log {𝒖}^t\right)}^{\top }𝒑\Vert }^2$ (23)

$={\lambda }_E{\Vert {𝒑}^{\top }\Vert }^2$

$\le K,$

where the last inequation holds due to and $\Vert 𝒑\Vert \le K$. By combining Eq. (21)-(23) and Lemma 2, it follows that

${\Vert {𝜽}^{t+1}-{𝜽}^{*}\Vert }^2$ (24)

$={\Vert {𝜽}^t-{\eta }_t{𝒅}_t-{𝜽}^{*}\Vert }^2$

$\le {\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2+{\eta }_t{\displaystyle \sum _k^K}{p}_k\left({\displaystyle \frac{1}{{\eta }_t}}{\Vert {𝜽}^t-{𝜽}_k^t\Vert }^2+{\eta }_t{\Vert \nabla F_k\left({𝜽}_k^t\right)\Vert }^2\right)$

$+2{\eta }_t{\displaystyle \sum _k^K}{p}_k\left(-\left(F_k\left({𝜽}_k^t\right)-F_k\left({𝜽}^{*}\right)\right)-{\displaystyle \frac{\mu }{2}}{\Vert {𝜽}_k^t-{𝜽}^{*}\Vert }^2\right)+{\eta }_t^2{\Vert {𝒅}_t\Vert }^2$

$=\left(1-\mu {\eta }_t\right){\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2+{\displaystyle \sum _k^K}{p}_k{\Vert {𝜽}^t-{𝜽}_k^t\Vert }^2+{\eta }_t^{2}K+2{\eta }_t^2\mathrm{\Gamma }$

$\le \left(1-\mu {\eta }_t\right){\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2+4{\eta }_t^2{\left(E-1\right)}^2{V}^2+{\eta }_t^{2}K+2{\eta }_t^2\mathrm{\Gamma }.$

Lastly, let $D_t=𝔼{\Vert {𝜽}^t-{𝜽}^{*}\Vert }^2$, it follows that

$$D_{t+1}\le \left(1-{\eta }_t\mu \right)D_t+{\eta }_t^{2}B,$$ (25)

where $B=4{(E-1)}^2{V}^2+K+2\mathrm{\Gamma }$.

For a diminishing stepsize, ${\eta }_t=\frac{\beta }{t+\gamma }$ for some $\beta >\frac{1}{\mu }$ and $\gamma >0$ such that ${\eta }_1\le \min \{\frac{1}{\mu },\frac{1}{4L}\}=\frac{1}{4L}$ and ${\eta }_t\le 2{\eta }_{t+E}$. For $v=\max \{\frac{{\beta }^{2}B}{\beta \mu -1},(\gamma +1)D_1\}$, by definition, it holds $D_t\le \frac{v}{\gamma +t}$ for $t=1$. Assume $D_t\le \frac{v}{\gamma +t}$ holds, then we expand as below:

$D_{t+1}$ $\le \left(1-{\eta }_t\mu \right)D_t+{\eta }_t^{2}B$ (26)

$\le \left(1-{\displaystyle \frac{\beta \mu }{t+\gamma }}\right){\displaystyle \frac{v}{t+\gamma }}+{\displaystyle \frac{{\beta }^{2}B}{{\left(t+\gamma \right)}^2}}$

$={\displaystyle \frac{t+\gamma -1}{{\left(t+\gamma \right)}^2}}v+\left[{\displaystyle \frac{{\beta }^{2}B}{{\left(t+\gamma \right)}^2}}-{\displaystyle \frac{\beta \mu -1}{{\left(t+\gamma \right)}^2}}v\right]$

$\le {\displaystyle \frac{v}{t+\gamma +1}}.$

Recall Eq. (18), we finally catch:

$$𝔼\left[F\left({\theta }^t\right)-F\left({\theta }^{*}\right)\right]\le \frac{L}{2}{D}_t\le \frac{L}{2}\frac{v}{\gamma +t}.$$ (27)

Following the specific case of [23], we can choose $\beta =\frac{2}{\mu },\gamma =\max \{8\frac{L}{\mu },E\}-1$ and denote $\kappa =\frac{L}{\mu }$, then ${\eta }_t=\frac{2}{\mu }\frac{1}{\gamma +t}$. One can verify that the choice of ${\eta }_t$ satisfies ${\eta }_t\le 2{\eta }_{t+E}$ for $t\ge 1$. Then, we have

$v$ $=\max \{{\displaystyle \frac{{\beta }^{2}B}{\beta \mu -1}},\left(\gamma +1\right){\mathrm{\Delta }}_1\}$ (28)

$\le {\displaystyle \frac{{\beta }^{2}B}{\beta \mu -1}}+\left(\gamma +1\right){\mathrm{\Delta }}_1$

$\le {\displaystyle \frac{4B}{{\mu }^2}}+\left(\gamma +1\right)D_1$

and

$$𝔼\left[F\left({\overline{𝜽}}^t\right)\right]-F^{*}\le \frac{L}{2}\frac{v}{\gamma +t}\le \frac{\kappa }{\gamma +t}\left(\frac{2B}{\mu }+\frac{\mu \left(\gamma +1\right)}{2}{D}_1\right).$$ (29)

As we can see, Fed${\text{𝚄}}^{\mathrm{2}}$ similarly converges to a generalization error bound as the FedAvg-like FL model with non-IID data. Discriminatively, benefiting from the optimization of EUA, the communication round multiplies with a smaller $B$. ∎

Appendix B Experimental Supplementary

B.1 Hyper-parameter Sensitivity Analysis

In the following, we study the sensitivity of remaining highly relevant hyper-parameters, i.e., the effect of client numbers and local epochs. Specifically, we compare Fed${\text{𝚄}}^{\mathrm{2}}$-SimCLR and its runner-up method, i.e., FedX-SimCLR, on CIFAR10 $\alpha =0.1$, by varying the local epochs $E=\{5,10,20,50\}$ in Fig. 7 and the number of clients $K=\{5,10,20,50,100\}$ in Fig. 8. We train all models until converge to obtain fairly comparable results. As we can see: (1) With the increase of local epochs, each client of FedX-SimCLR obtains a better-performing model, while each client of Fed${\text{𝚄}}^{\mathrm{2}}$-SimCLR is insensitive. This states that Fed${\text{𝚄}}^{\mathrm{2}}$ balances the client model deviation change rate in EUA, bringing the benefits of quick convergence. (2) The performance of all methods decreases when the number of clients increases, but Fed${\text{𝚄}}^{\mathrm{2}}$-SimCLR consistently outperforms FedX-SimCLR. It validates that enhancing uniform and unified representations will make FUSL methods more generalizable to the cases of various participants amounts.

Figure 7: The effect of local epochs $E$ (on CIFAR10 $\alpha =0.1$).

Figure 8: The effect of client number $K$ (on CIFAR10 $\alpha =0.1$).

B.2 Enlarged Figures in Visualization

In our main paper, we depict the top-k singular values of covariance matrix representations in Fig. (3), the corresponding 3-D representation in Fig. (4), and the distribution of data representation in Fig. (5) between global model and randomly sampled local models. The purpose of the above figures is to illustrate the representation enhancement of Fed${\text{𝚄}}^{\mathrm{2}}$. In Fig. 9-11, we enlarge these figures to explore the detailed comparisons. In terms of Fig. 11, Fed${\text{𝚄}}^{\mathrm{2}}$ keeps the unified representation between global and local models as well as clearer decision boundary for each class.

Figure 9: Top k log singular values of the covariance matrix of global model (left) and local model (right) representations.

(a)

(b)

(c)

(d)

Figure 10: The representations collapse issue on the sphere using BYOL model (on CIFAR10 $\alpha =0.1$ Cross-silo). The more blank representation space, the more severe collapse issue is.

(a)

(b)

(c)

(d)

(e)

Figure 11: The distributions of data representations using global and local BYOL model (on CIFAR10 $\alpha =0.1$ Cross-silo).