Collaborative Graph Neural Networks for Attributed Network Embedding

Abstract

Graph neural networks (GNNs) have shown prominent performance on attributed network embedding. However, existing efforts mainly focus on exploiting network structures, while the exploitation of node attributes is rather limited as they only serve as node features at the initial layer. This simple strategy impedes the potential of node attributes in augmenting node connections, leading to limited receptive field for inactive nodes with few or even no neighbors. Furthermore, the training objectives (i.e., reconstructing network structures) of most GNNs also do not include node attributes, although studies have shown that reconstructing node attributes is beneficial. Thus, it is encouraging to deeply involve node attributes in the key components of GNNs, including graph convolution operations and training objectives. However, this is a nontrivial task since an appropriate way of integration is required to maintain the merits of GNNs. To bridge the gap, in this paper, we propose COllaborative graph Neural Networks–CONN, a tailored GNN architecture for attribute network embedding. It improves model capacity by 1) selectively diffusing messages from neighboring nodes and involved attribute categories, and 2) jointly reconstructing node-to-node and node-to-attribute-category interactions via cross-correlation. Experiments on real-world networks demonstrate that CONN excels state-of-the-art embedding algorithms with a great margin.

1 Introduction

Attributed networks [1, 2, 3, 4] are ubiquitous in a myriad of real-world information systems, such as academic networks and social medial systems. Unlike plain networks in which only node-to-node interactions are available, each node in attributed network is associated with a rich set of attributes, describing its distinctive characteristics. For example, in social networks, users connect with others as friends, share opinions, and post comments as attributes. In academic citation networks, different articles are connected via citation links, and each article has substantial text information, such as an abstract sentence to describe its own topic. Several studies [5, 6] in social science have revealed that attributes of nodes can reflect and affect their community structures [7] in practice. Thus, it is encouraging and important to study attributed networks. To this end, attributed network embedding [8, 1, 9], targeting at leveraging both network proximity and node attribute affinity to learn low-dimensional node representations, has attracted great attention in recent years, and many efforts have been devoted from both academia and industry [10, 11, 12, 13, 14, 15, 16].

Among them, embedding paradigm based on graph neural networks (GNNs) [17, 15, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 18] has achieved remarkable success over a variety of downstream graph analytical tasks, including node classification [31, 32, 33], graph classification [34, 35], link prediction [36, 37, 38, 39], node clustering [40, 41], and anomaly detection [42, 43, 44, 45]. The design recipes for GNNs based methods include two major components: 1) a GNN encoder that takes node attributes and node-to-node interaction network as input and outputs low-dimensional node representations; 2) the training objective, which is derived to reconstruct the input data (e.g., network structure), so as to train the model unsupervised. Thanks to the critical message propagation mechanism in GNNs, the GNN encoder is naturally applicable for attributed networks. Therefore, existing GNNs-based embedding efforts mainly focus on advancing model performance with more expressive message passing schema, such as adaptively aggregating messages from neighbors via the attention layer [46, 47, 48]

Despite their popularity and recent advances in refining GNNs architectures to effectively model the topological structures [49, 50, 51, 52, 53], little attention was paid to the node attributes. Prior GNNs studies focus on updating the representation of each node by aggregating the representations of itself and its neighbors [54] recursively. In this learning process, node attributes are merely employed as the representations of nodes in the initial layer [55]. They would be blocked from message propagation if the network structure is incomplete or missing, which is quite common in real-world applications where graphs exhibit long-tail node degree distribution [56, 57]. Besides, even when we design the training objectives of GNNs, the node attributes are seldom used. For instance, the common practice to train GNNs based embedding algorithm for attributed network embedding is to reconstruct the observed node interactions [15], by either employing a negative sampling based objective or directly recovering the whole input network structure [14]. As a summary, node attributes are not well exploited in the existing works.

Recently, there has been a revolution to rethink the value of node attributes in random-walk based embedding approach [58, 59]. The core idea is to redesign the crucial component–random-walk in an attribute aware fashion. Specifically, ANRL [60] refines the conditional probability that estimates the propensity score between the anchor node and its context by utilizing node attributes. FeatWalk [59] conducts an attribute-aware joint random walk to increase the diversity of generated walks. According to their empirical experiments, both of them have shown significant performance gains compared with their vanilla counterparts. Nevertheless, they are tailored for random-walk based approaches and cannot be directly applied to GNN models with trivial efforts. Motivated by this, in this paper, we propose to explore whether node attributes can be effectively employed to advance the essential building blocks of GNN architectures (i.e., message aggregation mechanism and training objective).

However, it is a non-trivial and challenging task to integrate node attributes into GNN architectures mainly because of two reasons: (i) GNNs already show promising results by integrating node attributes as the initial node representations. It is difficult to further incorporate node attributes into the key components of GNNs, while maintaining the existing benefits and prominent performance [54, 61]; (ii) real-world node attributes, such as comments of users, abstracts of papers, and descriptions of products, are distinct from the network topological structures and not in line with the graph convolutional operation in GNNs. Specifically, the values in node attributes are often multi-categorical or continuous variables, while these in the network structures are binary. They are not compatible with each other. Thus a tailored operation is required to learn from them jointly. For example, employing autoencoders as the training objective of GNNs would achieve sub-optimal performance [14, 62].

To address the aforementioned challenges, we propose a novel unsupervised representation learning model, dubbed COllaborative Graph Neural Network (CONN). It aims to develop a tailored GNN architecture for attributed networks, such that node attributes can be explicitly fused into the message aggregation process as well as the training objective. Specifically, we aim to investigate two important research questions. (i) How to leverage node attributes to explicitly guide the message propagation of vanilla GNN, and conduct a collaborative aggregation mechanism? (ii) In the training objective, how to effectively model the heterogeneous interactions, so as to jointly reconstruct the network structure and node attributes? We summarize our major contributions as follows.

• We focus on unsupervised representation learning on attributed networks and propose an effective GNN framework - CONN, to leverage node attributes in the two aforementioned key components of vanilla GNNs.

• By conducting a bipartite graph on node attributes, we develop a collaborative aggregation mechanism for node embedding. It not only helps to enrich or rebuild node connections through attribute category, but also provides a principled way to update node representation from both neighboring nodes and involved attribute categories.

• Based on the node and attribute category representations, we design a novel cross-correlation layer to effectively model the complex node or node attribute category interactions. It highlights the similarity of two anchor nodes from their multi-granularity features and significantly boosts the reconstruction capability of vanilla GNNs.

• We evaluate CONN on node classification and link prediction tasks. Empirical results on benchmark datasets show that CONN performs consistently better than other state-of-the-art embedding methods. Moreover, we also analyze the robustness and convergence speed of CONN in Section 5.8.

Figure 1: Instead of inferring the node-to-node links by using node attributes as side features, CONN targets at jointly learning the network structure and node attributes in a unified latent space under graph neural network framework.

2 Related Work

There are three types of related works, based on whether node attributes are modeled for network embedding or not. In this section, we briefly review some related works in these two fields. Please refer to [63, 64, 65, 66, 67] for comprehensive review.

2.1 Network embedding

The first class of approach can be tracked back to traditional graph machine learning problem, which aims to learn node embedding while preserving the local manifold structure, such as the LPP [68] and Laplacian Eigenmaps [69]. Nevertheless, these methods suffer from scalability challenges to large-scale networks, due to the time-expensive eigendecomposition operation on the adjacency matrix, whose time complexity is $O(n^3)$ with $n$ is the number of nodes. Inspired by the recent breakthrough of distributed representation learning [70] in natural language processing, a lot of scalable network embedding methods [58, 71, 72, 73, 74, 75, 76, 77] have been developed. For example, DeepWalk [58] and Node2vec [73] conduct truncated random walks on the network to generate node sequences, and then feed them into into the Skip-Gram algorithm [70] for node embedding. LINE [71] optimizes the first and second order neighborhood associations to learn representations. GraRep [72] extends LINE to capture high-order neighborhood relationships. SDNE [74] applies deep learning for node embedding and targets to capture the non-linear graph structure as well as preserve the global and local structures of the graph. [78] attempts to incorporate the community structure of the graph for representation learning. Some other studies aim to deal with large-scale graphs [79, 80, 81]. Despite their simplicity, the aforementioned methods may be limited in practice, since they cannot exploit side information, such as user profiles, worlds in posts, and contexts in photos on social media.

2.2 Attributed Network Embedding

Different from traditional network embedding, attributed network embedding [82, 31, 83, 84, 85, 86, 87, 88], which aims to learn node representations by leveraging both the network structure and node attributes, has attracted substantial attention in recent years. For instance, ANRL [60] incorporates the node attributes into the conditional probability function, which predicts the propensity scores between an anchor node and its context, of Skip-gram to capture structure correlation. MUSAE [13] advances Skip-gram model by considering multi-scale neighborhood relationships based on node attributes. FeatWalk [59] aims to conduct attribute-aware random walks to increase the diversity of generated walks, so as to boost the Skip-gram model. PANE [89] is another random walk-based method for scalable training. TADW [83] incorporates node attributes into DeepWalk under matrix factorization framework. PTE [90] employs different orders of world co-occurrence relationships and node label to generate predictive text representations. ProGAN [91] aims to preserve the underlying proximities in the hidden space based on a generative adversarial network. Although the aforementioned methods are capable of employing the network structure and node attributes for joint representation learning, they are limited in capturing the structure information.

More recently, graph neural networks [23] based attributed network embedding has attracted increasing attention, due to its ability in capturing structure information, incorporating node attributes, and modeling non-linear relationships. As a pioneering GNN work, GCN [23] formally suggests to update node presentation by recursively aggregating representations from its adjacent neighbors, and achieves significant performance improvement on semi-supervised classification task. To follow up, GAE [14] makes the first effort to extend GCN to an autoencoder framework to learn node representations unsupervisedly. Meanwhile, GraphSage [15] improves the scalability of GCN by sampling and aggregating features from a node’s local neighborhood. Their results show that GCN is naturally suitable for attributed networks, since it can directly utilize node attributes as initial node features in the first layer for training. Motivated by this, many follow-up studies [46, 50, 92] have been proposed to advance model capability by developing more expressive GNN architectures. Recently, some efforts have also been devoted to redefine the training objective of GCN to learn more effective node representations. CAN [62] and [93] advocate to jointly reconstruct the network structure and node attributes under variational autoencoder framework. DGI [94] and GIC [95] suggests to learn node representations by maximizing the mutual information between local node representation and the graph summary representation. GCA [96] and MVGRL [97] seek to update node representations by maximizing the agreement of representations between different views generated by data augmentation. However, they can only explore the node attributes implicitly in the message propagation process of GNN. In this paper, we propose a principled GNN variant for attributed network embedding, which allows node attributes to explicitly guide the message propagation and training objectives.

2.3 Attributed Network Embedding with Extra Knowledge

In addition to plain attributed networks, including node features and homogeneous graph structure, some studies also utilize extra knowledge, such as knowledge graph (KG) [98, 99, 100, 101],, to boost the model’s performance. For example, PGE [102] studies how to incorporate additional edge features. KGCN [103] and KGAT [104] investigate how to leverage external knowledge graphs (e.g., item knowledge graphs) to improve recommendation quality. However, these methods require additional efforts to generate high-quality information sources, such as edge features and knowledge graphs. In contrast, in this work, we focus on standard attributed network embedding, modeling on homogeneous graphs with pure node attributes, such as continuous features and categorical features.

3 Problem Statement

We assume that an attributed network denoted by $𝒢=(𝒱,𝐀,𝐗)$ is given. It has $n$ vertexes collected in set $𝒱$. These $n$ nodes are connected by an undirected network with its adjacency matrix denoted as $𝐀\in {ℝ}^{n\times n}$. If there is an edge between nodes $v_i$ and $v_j$, then ${𝐀}_{ij}=1$; otherwise, ${𝐀}_{ij}=0$. Besides the network $𝐀$, each node $v$ also has a descriptive feature vector ${𝐱}_v\in {ℝ}^m$, known as node attributes, where $m$ is the total number of attribute categories. We denote the neighbor set of node $v$ as ${𝒩}_v$, and represent the neighbor set of attribute category ${\delta }_j$ as ${𝒩}_{{\delta }_j}$, i.e., ${𝒩}_{{\delta }_j}=\{u|{𝐗}_{uj}>0,\text{for}u\in 𝒱\}$. We employ a diagonal matrix $𝐃=\text{diag}(d_1,\mathrm{\cdots },d_n)$ to denote the degree matrix, where $d_i={\sum }_j{𝐀}_{ij}$. The main symbols are listed in Table I. To study the GNNs in an unsupervised setting, we follow the literature [23, 105] and formally define the problem of attributed network embedding in Definition 1.

TABLE I: Summary of notations in this paper.

Notation	Description
$𝐀\in {ℝ}^{n\times n}$	adjacency matrix
$𝐗\in {ℝ}_{+}^{n\times m}$	a matrix collects all node attributes
$𝒱$	a set collects all $n$ nodes
$𝒰$	a set collects all $m$ attribute categories
${\delta }_j\in 𝒰$	the $j^{\text{th}}$ node attribute category
${𝒩}_v$	a set collects adjacent neighbors of $v$
$K$	the number of graph convolutional layers
$\widetilde{𝐏}\in {ℝ}^{(n+m)\times (n+m)}$	adjacency matrix of augmented network
${𝐡}_v^{(k)}\in {ℝ}^{1\times d}$	representation of node $v$ at the $k^{\text{th}}$ layer
${𝐡}_{{\delta }_j}^{(k)}\in {ℝ}^{1\times d}$	representation of ${\delta }_j$ at the $k^{\text{th}}$ layer

Definition 1

Attributed Network Embedding. Given an attributed network $𝒢=(𝒱,𝐀,𝐗)$, the goal is to learn a $d$-dimensional continuous vector ${𝐡}_v\in {ℝ}^d$ for each node $v\in 𝒱$, such that the topological structures in $𝐀$ and side information characterized by $𝐗$ could be preserved in the embedding representations $𝐇$. The performance of this learning task is evaluated by applying $𝐇$ to various downstream tasks such as node classification and link prediction.

To perform attributed network embedding, GNN models [23, 15] learn the embedding representation of each node $v$ by aggregating the representations of itself and its neighbors ${𝒩}_v$ in the previous layer. A typical neighborhood aggregation mechanism [23] is expressed as below,

${𝐡}_v^{(k)}$ $={\displaystyle \frac{1}{1+d_v}}{𝐡}_v^{(k-1)}+{\displaystyle {\sum }_{u\in {𝒩}_v}}{\displaystyle \frac{{𝐀}_{vu}}{\sqrt{(1+d_v)(1+d_u)}}}{𝐡}_u^{(k-1)},$ (1)

where ${𝐡}_v^{(k)}$ denotes the hidden representation of node $v$ at the $k^{\text{th}}$ layer of GNNs, and ${𝐡}_v^{(0)}={𝐱}_v$. The final output is ${𝐡}_v^{(K)}$, where $K$ is the maximum number of layers considered. The top subfigure in Figure 1 illustrates this traditional neighborhood aggregation by using a toy example. We could see that, by stacking two layers, the representations of two-hop neighbors $5$ and $6$ could be accessed by node $v=1$. To train this model in an unsupervised manner, a widely-adopted [15] graph-based loss function is defined as,

$ℒ=-{\displaystyle {\sum }_{v\in 𝒱}}{\displaystyle {\sum }_{u\in {𝒩}_v}}\log {\displaystyle \frac{\exp (\text{sim}({𝐡}_v^{(K)},{𝐡}_u^{(K)}))}{{\sum }_{u^{\prime }\in 𝒱}\exp (\text{sim}({𝐡}_v^{(K)},{𝐡}_{u^{\prime }}^{(K)}))}}.$ (2)

$\text{sim}(\cdot ,\cdot )$ is a similarity function, e.g., inner product. The goal is to make the representations of connected nodes similar to each other, while enforcing the representations of unconnected nodes to be distinct. We observe that, node attributes are employed only as the initial representations ${𝐡}_v^{(0)}$. They have not been further exploited, especially being integrated into the core mechanisms of GNNs.

4 Collaborative Graph Convolution

Node attributes are informative, and significantly correlated with and complementary to the network [60, 106, 107, 108, 109]. Since they have not been fully exploited in GNNs, we explore to deeply integrate node attributes into the core mechanisms of GNNs, and develop a novel framework named COllaborative graph Neural Network (CONN). Figure 1 depicts the two major components of CONN, i.e., collaborative neighborhood aggregation and collaborative training objective. First, we redefine the graph convolutions by considering node attribute categories $𝒰$ as another set of nodes. As illustrated in Figure 1, we augment the original network $𝐀$ to a new one $\widetilde{𝐏}$ with $n+m=6+4$ nodes. It preserves all edges in $𝐀$, and contains a link from nodes $v$ to ${\delta }_j$ if $v$ has a non-zero value in its node attributes ${𝐗}_{vj}$. Notice that ${𝐗}_{vj}$ can be both positive or negative, where negative value means the neighbor has negative impact towards the anchor node. Based on $\widetilde{𝐏}$, we perform neighborhood aggregation. For example, the first-order neighbors of node $1$ have been augmented from $\{2,3,4\}$ to $\{2,3,4,{\delta }_1,{\delta }_2\}$, while the second-order neighbors of node $1$ have been augmented from $\{1,5,6\}$ to $\{1,2,4,5,6,{\delta }_1,{\delta }_2,{\delta }_3\}$. We observe that our collaborative neighborhood aggregation could not only capture node-to-node interactions, but also node-to-attribute-category interactions. Second, to train our model, we design a collaborative loss. The goal is to collaboratively predict all links in the augmented network $\widetilde{𝐏}$, which incorporates $𝐀$ and X. Additionally, we design a novel cross correlation mechanism to model the complex interactions between any pair of nodes in $\widetilde{𝐏}$ (e.g., $1\in 𝒱$ and ${\delta }_1\in 𝒰$ in Figure 1). It employs not only the node representations in the last layer ${𝐡}_v^{(K)}$, but also all the remaining ones $\{{𝐡}_v^{(k)},\text{for}k=0,1,\mathrm{\dots },K-1\}$. We now introduce the details in the following subsections.

4.1 Collaborative Neighborhood Aggregation

Given an attributed graph $𝒢=(𝒱,𝐀,𝐗)$, existing GCN architectures mainly define the multiple-hop neighbors of node $v\in 𝒱$ purely based on the network structure $𝐀$. The first-order neighbors of node $v$ is represented by ${𝒩}_v=\{u|{𝐀}_{uv}=1\}$, while the second-order neighbors is denoted by ${𝒩}_v^{(2)}=\{u|{𝐀}_{uv}^2>0\}$, so on and so forth. ${𝐀}^k$ indicates the $k^{\text{th}}$ power of $𝐀$. As a result, other than serving as the initial node representations, i.e., ${𝐡}_v^{(0)}={𝐱}_v$, node attributes are excluded from the graph convolutions, i.e., the core operation of GNNs. As discussed in Introduction section, this could be suboptimal for inactive nodes with few or no neighbors in practice, since they don’t have sufficient neighborhood information for GNNs to effectively learn their embedding representations [56].

4.1.1 Augmented network

To tackle the aforementioned issue, we propose to leverage the geometrical property of node attributes. We regard each node attribute category ${\delta }_j\in 𝒰$ as a new node and node attributes $𝐗$ as a weighted bipartite graph. By adding it into the original network $𝐀$, we would get an augmented network, denoted as $𝐏$. Mathematically, its adjacency matrix is written as,

(3)

Eq.(9) is applicable for both categorical and continuous node attributes. For continuous features, however, it would generate a dense bipartite graph based on $𝐗$, which may significantly increase the computation costs when performing convolution on it. To save the computation, we empirically simplify the dense bipartite graph into a sparse one by only preserving the top-$N$ values in each row of $𝐗$. An appropriate $N$ value acts as a trade-off between the efficiency and accuracy, we analyze its impact in Section 5.8. In summary, we directly use the feature matrix $𝐗$ of categorical attributes to construct the augmented graph, while adopting top-$N$ values in each row of $𝐗$ to generate a sparse graph for continuous features.

4.1.2 Augmented multi-hop neighbors

By using $𝐏$, the first-order and second-order neighbors of node $v$ can be expanded as:

${𝒩}_v$ $=\{u|{𝐀}_{vu}=1\}+\{{\delta }_j|{𝐗}_{vj}\ne 0\},$ (4)

${𝒩}_v^{(2)}$ $=\{u|{𝐀}_{vu}^2>0\text{or}{({\mathrm{𝐗𝐗}}^{\top })}_{vu}>0\}+\{{\delta }_j|{(\mathrm{𝐀𝐗})}_{vj}>0\},$

where $({\mathrm{𝐗𝐗}}^{\top })\in {ℝ}^{n\times n}$ collects all node pairs that share at least one node attribute category, i.e., all $v\to {\delta }_j\to u$ paths. $(\mathrm{𝐀𝐗})\in {ℝ}^{n\times m}$ implies node-to-attribute-category interactions reflected by $v\to u\to {\delta }_j$ paths. Compared with traditional GNNs, we explicitly model the node-to-attribute-category interactions within $K$ hops. Original node-to-node interactions have been enriched by paths passing through ${\delta }_j\in 𝒰$. Similarly, the first and second-order neighbors of ${\delta }_j\in 𝒰$ could be computed as,

${𝒩}_{{\delta }_j}$ $=\{u|{𝐗}_{uj}\ne 0\},$ (5)

${𝒩}_{{\delta }_j}^{(2)}$ $=\{u|{(\mathrm{𝐀𝐗})}_{uj}>0\}+\{{\delta }_i|{({𝐗}^{\top }𝐗)}_{ji}>=1\},$

where $({𝐗}^{\top }𝐗)\in {ℝ}^{m\times m}$ denotes the attribute category correlations estimated by their common nodes. It collects ${\delta }_j\to u\to {\delta }_i$ paths. We enable nodes in $𝒱$ to propagate messages to attribute categories, and correlations among attribute categories to affect the graph convolution. Based on this, we can not only enrich node interactions using attribute-category as intermediate to improve model performance (see Table III and IV), but also enhance model robustness w.r.t. missing edges as shown in Section 5.8.

Another thing we want to remark is that different from previous efforts [110, 111] that define a node-to-node similarity network based on feature distance, we directly build a node-to-attribute-category bipartite graph on feature matrix by using attribute values as edge weights. As analyzed before, a bipartite graph between node and attribute-category can not only enrich or rebuild node interactions using attribute as intermedia, but also preserve feature information as much as possible.

4.1.3 Collaborative aggregation

We now illustrate how to learn node representations based on the augmented network $𝐏$. Given the recent advances in heterogeneous graph embedding [112, 112, 113], the intuitive solution is to apply these well-established heterogeneous GNNs to learn embeddings from $𝐏$ (it consists of two objects (node and attribute-category) and two relations (node-to-node and node-to-attribute-category)). However, our preliminary experiments show that these methods perform not good on our synthetic heterogeneous graph (See discussion in 5.8.4), since they may over-emphasize the heterogeneity between node and its attributes, making the learning process substantially complex. Since standard GNNs architectures [46, 50, 92, 62] could already achieve high performance on attributed networks by looking them as homogeneous graph, we propose to follow the tradition and regard the augmented network $𝐏$ as homogeneous resource with simple weight schema. We leave attributed network embedding from heterogeneous GNNs perspective as our future work.

Specifically, our essential idea is to treat node vertex and attribute-category vertex as identical vertices but do provide a weight hyperparameter $\alpha$ to control the information diffusion between the network $𝐀$ and node attributes $𝐗$. It is because the importance of node-to-node interactions and node attributes are not explicitly available. In our setting, the binary values in $𝐀$ might not be compatible with the feature values in $𝐗$ (e.g., continuous features), we define a refined transition probability matrix as follow.

(6)

where $\widetilde{𝐀}$ and $\widetilde{𝐗}$ denote the normalization of $(𝐀+𝐈)$ and $𝐗$ after applying ${\mathrm{\ell }}_1$ norm to normalize each row respectively. $\alpha \in [0,1]$ is a trade-off hyper-parameter to impose our inductive bias about the importance of network structure and node attributes. Specifically, when $\alpha =1$, it yields to a vanilla graph convolutional operation purely based on the network structure. As $\alpha$ increases, node representations would be more dependent on node attributes, and node-to-attribute-category interactions and attribute category correlations will be gradually incorporated into the graph convolution process.

Based on $\widetilde{𝐏}$, standard graph convolutional layers can be directly applied to update node representations. We follow a simple GNN model [54], and update the corresponding embedding matrix with a simple sparse matrix multiplication. We use ${𝐇}^{(k)}\in {ℝ}^{(n+m)\times d}$ to denote the intermediate representation of all $(n+m)$ nodes in the $k^{\text{th}}$ layer. Mathematically, it could be written as,

$${𝐇}^{(k)}\leftarrow {\widetilde{𝐏}}^K{𝐇}^{(0)}.$$ (7)

Our initial node representations ${𝐇}^{(0)}\in {ℝ}^{(n+m)\times d}$ are not based on $𝐗$. Instead, ${𝐇}^{(0)}$ is a trainable embedding matrix that is randomly initialized following common protocols [58, 114]. To further illustrate the correlation between ${𝐡}_v^{(k)}$ and $\{\alpha ,𝐀,𝐗\}$, we rewrite the corresponding update rule as follows,

${𝐡}_v^{(k)}=$ $\alpha {\widetilde{𝐀}}_{vv}{𝐡}_v^{(k-1)}+\alpha {\displaystyle {\sum }_{u\in {𝒩}_v}}{\widetilde{𝐀}}_{vu}{𝐡}_u^{(k-1)}$ (8)

$+(1-\alpha ){\displaystyle {\sum }_{{\delta }_j\in {𝒩}_v}}{\widetilde{𝐗}}_{vj}{𝐡}_{{\delta }_j}^{(k-1)}.$

Eq. (8) provides a principled solution to utilize and control node attributes. On the one hand, it can explicitly enrich or replenish node interactions by treating attribute-category as additional node. On the other hand, neighborhood information from node and attribute-category are selectively combined via trade-off parameter $\alpha$.

4.2 Collaborative Training Objective

Given the updated representations ${𝐇}^{(k)}$ for $k=0,1,\mathrm{\dots },K$, we need an unsupervised objective to train the GNN model. A widely-adopted approach [14, 15, 62] is to employ the node representations at the last layer ${𝐇}^{(K)}$ to reconstruct all edges. As illustrated in Eq. (2), it estimates the probability of two nodes $v$ and $u$ being connected based on the similarity between their vectors ${𝐡}_v^{(K)}$ and ${𝐡}_u^{(K)}$. This approach has been demonstrated to be effective in plain networks, but node attributes are often available in practice. Eq. (2) could not directly incorporate node attributes. Another intuitive solution [14] is to employ autoencoders to reconstruct both $𝐀$ and $𝐗$. It would achieve suboptimal performance because the topological structures $𝐗$ are heterogeneous with node attributes $𝐀$.

4.2.1 Cross correlation mechanism

To cope with aforementioned issues, we propose a novel cross correlation mechanism to model and predict the complex node-to-node and node-to-attribute-category interactions. It has two major steps. First, we remove all weights in $𝐏$ and convert it into a binary matrix. We target at integrating the network and node attribute into our training objective. To make them compatible with each other, we define a binary adjacency matrix as,

(9)

where ${𝐗}_{v{\delta }_j}^{\ast }=1$ if ${𝐗}_{v{\delta }_j}>0$. Our goal is to recover the node-to-node and node-to-attribute-category interactions in ${𝐏}^{\ast }$. Second, the node representations ${𝐇}^{(k)}$ for $k=0,1,\mathrm{\dots },K$ at all layers are learned based on different orders of neighbors. One merit of our model is that we have integrated node attributes $𝐗$ into the collaborative neighborhood aggregation and no longer need to employ $𝐗$ as the initial node representations ${𝐇}^{(0)}$. Since we are flexible to define ${𝐇}^{(0)}$, we could make the dimensions of all $\{{𝐇}^{(k)}\}$ the same. In such a way, we could easily take full advantage of them, and model the complex interaction from node $v$ to node $u$ or node $v$ to node attribute category ${\delta }_j$ as,

${𝐲}_{vu}=\text{MLP}(|{|}_{k=0}^K|{|}_{i=0}^K{𝐡}_v^{(k)}\odot {𝐡}_u^{(i)}),$ (10)

${𝐲}_{v{\delta }_j}=\text{MLP}(|{|}_{k=0}^K|{|}_{i=0}^K{𝐡}_v^{(k)}\odot {𝐡}_{{\delta }_j}^{(i)})$

where $||$ indicates the concatenation operation. ${𝐡}_v^{(k)}\odot {𝐡}_u^{(i)}\in {ℝ}^{n\times d}$ denotes the correlation feature between the under-${(k+1)}^{\text{th}}$-order neighborhood of node $v$ and the under-${(i+1)}^{\text{th}}$-order neighborhood of node $u$. $\odot$ represents the element-wise multiplication. ${𝐲}_{vu}\in ℝ$ and ${𝐲}_{v{\delta }_j}\in ℝ$ are the predicted scores for pairs $(v,u)$ and $(v,{\delta }_j)$, respectively. MLP denotes a three-layer multilayer perceptron with a Relu activation function.

It is worth noting that the element-wise operation between node representations has been explored in KGAT [104] and NGCF [115] for the recommendation. Our method differs in two ways. First, KGAT and NGCF utilize this technique to facilitate message propagation in each GNN layer (for node-level embedding). Yet, CONN applies it to generate edge representations of two end nodes from different granularities. Second, the proposed cross-correlation mechanism is far more element-wise. Its novelty lies in the proposal to obtain an informative edge representation of end nodes by integrating their cross-correlations from different combinations of the GNN layers. Therefore, the proposed cross-correlation layer is different from KGAT and NGCF because it aims to improve the quality of edge representation while the referenced methods work on boosting the representations of each node per GNN layer, from which they are orthogonal to us and can be incorporated as the base GNN backbone.

4.2.2 Analysis

The subfigure on the bottom of Figure 1 illustrates the key idea of the proposed cross correlation mechanism. Given any two entities (e.g., node $v=1$ and ${\delta }_1$ in Figure 1), we aim to capture the second-order correlations across all $K+1$ embedding representations of the two entities. So, in total, Eq. (10) has concatenated ${(K+1)}^2$ correlation features (e.g., $9$ in Figure 1), which is a small number. It should be noted that the element-wise multiplication [116] would highlight the shared patterns within the two input node representations, e.g., ${𝐡}_v^{(k)}$ and ${𝐡}_u^{(i)}$, while eliminating some inconsistent and noisy information.

4.2.3 Optimization and final representation

In our collaborative training objective, the goal is to reconstruct the node-to-node interactions in $𝐀$ by using $y_{vu}$ and the node-to-attribute-category interactions in ${𝐗}^{\ast }$ by using $y_{v{\delta }_j}$. The corresponding objective function is defined as follows.

$ℒ=$ $-{\displaystyle {\sum }_{v\in 𝒱}}{\displaystyle {\sum }_{z\in {𝒩}_v}}\log {\displaystyle \frac{\exp ({𝐲}_{vz})}{{\sum }_{z^{\prime }\in 𝒱\cup 𝒰}\exp ({𝐲}_{v{z}^{\prime }})}}$ (11)

$-{\displaystyle {\sum }_{{\delta }_j\in 𝒰}}{\displaystyle {\sum }_{u\in {𝒩}_{{\delta }_j}}}\log {\displaystyle \frac{\exp ({𝐲}_{{\delta }_{j}u})}{{\sum }_{u^{\prime }\in 𝒱\cup 𝒰}\exp ({𝐲}_{{\delta }_j{u}^{{}^{\prime }}})}}.$

Eq. (11) is usually intractable in practice, because the sum operation of the denominator is computationally prohibitive. Therefore, we employ the negative sampling [71, 15] strategy to accelerate the optimization.

After we have trained the model CONN by using the collaborative objective in Eq. (11), we need to define the final embedding representation of nodes to perform the downstream tasks. For link prediction task, we directly predict the probability of linking based on ${𝐲}_{vu}$ for a node pair $(v,u)$. For node classification, we adopt the output from the last layer ${𝐇}^{(K)}$ as node embedding, in which off-the-shelf classification algorithms could directly use it to classify.

4.3 Comparison with Prior Work

To the best of our knowledge, few efforts have been devoted to leverage node attributes to redefine the core components of embedding methods for graphs. We roughly divide them into two categories and analyze the difference below.

Random-walk based approach. Random-walk based methods focus on conducting truncated random walks to generate node sequences, and then applying Skip-gram algorithm to learn node representations based on the sequences. This approach is initially not applicable for node attributes. ANRL [60] addresses this issue by modifying the loss function of Skip-gram to depend on node attributes. Featwalk [59] suggests to conduct an attribute-aware random walks to inject node attributes in the random walk generation process. Compared with these methods, our model belongs to graph neural network approach that takes advantage of graph convolutional networks to explicitly model local graph structure.

Graph neural network based approach. This line of methods focuses on exploiting graph convolutional networks [23] to model the local subgraph of an anchor node for node representation. It is natural to cope with attributed graphs by using node attributes as initial node features in the first layer. However, such approach is rather limited in exploiting node attributes as they are excluded from the two crucial components of GCN, i.e., neighborhood message propagation and the training objective. CAN [62] attempts to alleviate this problem by jointly reconstructing node attributes and network structure under variational autoencoder. In contrast, our model focuses on exploiting node attributes to impact the two building blocks jointly.

Besides, we also want to remark that the utilization of node attributes in our work is different from skip connection trick [117]. Skip connection targets to skip some higher-order neighbors in deeper layers by looking back initial features, but our focus is to redefine the neighborhood set of nodes in different orders by regarding attribute categories as ”neighbor”. Namely, our model introduces additional node-to-attribute-category relations in the message aggregation process of each layer. In a summary, skip connection is complementary with our model and can be added to our GNN backbone to avoid over-smoothing.

Heterogeneous network embedding. Thanks to our proposal in constructing the augmented network between nodes and attribute categories in Section 4.1.1, we can also regard the resultant embedding task as a special heterogeneous network embedding (HNE) problem [118, 112, 119, 113], where we have two object types (node and attribute category) and two relation types (node-to-node and node-to-attribute-category). Under this principle, state-of-the-art heterogeneous models might be applied. However, we found that such approach may make the learning task unnecessarily complex, since HNE will over-emphasize the heterogeneity of attributed networks. This may be suboptimal because attributed networks are not truly ”heterogeneous” graphs. Moreover, it’s nontrivial to train the model well, because no features are available for the second object–attribute-category. Also, it’s hard to control the importance of two relations or node types in a mixed manner, such as mixed random-walk [112]. In contrast, by regarding the augmented network as homogeneous graph in our setting, the problem itself is substantially simplified and arbitrary standard GNNs architectures can be used in the plug-and-play fashion, equipping our proposal with broader applicability and practicability.

To summarize, we propose a new alternative approach for GNNs based attributed network embedding by regarding attribute category as additional nodes and converting attributed network into an augmented graph. The augmented graph offers flexible information diffusion between nodes and attribute categories without information loss. To effectively learn node representations from the new graph, we develop two tailored designs: collaborative aggregation and cross-correlation mechanism, where the former helps to explicitly control the information propagation between nodes and node attributes in convolution, while the latter improves model’s reconstruction capacity using multi-granularity features. Although our proposal seems general and is applicable for both homogeneous GNNs and heterogeneous GNNs, we empirically found that it’s nontrivial to learn node representations based on heterogeneous GNNs (See discussion in 5.8.4). Therefore, we focus on the simple case and leave attributed network embedding based on heterogeneous GNNs as the future work.

5 Experiments

We analyze the effectiveness of CONN on multiple real-world datasets with various scales and types. Specifically, our evaluation centers around three questions.

• Q1: Compared with the state-of-the-art embedding methods, can CONN achieve better performance in terms of node classification and link prediction tasks?

• Q2: There are three crucial components, i.e., graph mixing convolutional layer, graph correlation layer, and collaborative optimization, in CONN, how much does each component contribute?

• Q3: What are the impacts of hyperparameters: the trade-off parameter $\alpha$ and embedding dimension $d$, on CONN?

TABLE II: Statistics of the datasets.

	$|𝒱|$	# Edge	$|𝒰|$	$\mathrm{\#}Label$
Pubmed	$19$,$717$	$44$,$338$	$500$	$3$
ACM	$48$,$579$	$119$,$974$	$10$,$000$	$9$
BlogCatalog	$5$,$196$	$171$,$743$	$8$,$189$	$6$
ogbn-arxiv	169,343	1,166,243	128	40
Reddit	$232$,$965$	$11$,$606$,$919$	$602$	$41$
ogbl-collab	$235,868$	$1,285,465$	$128$	$-$

5.1 Datasets

We conduct experiments on six publicly available attributed networks of various scales and types. Their statistical information is summarized in Table II.

Pumbed [120]. It is the biggest benchmark citation network used in [23]. Nodes correspond to documents and edges correspond to citations. Each node has a bag-of-words feature vector according to the paper abstract. Labels are defined as the academic topics.

TABLE III: Node classification performance.

Method	Pubmed		ACM		BlogCatalog		Reddit		ogbn-arxiv
	F1-micro	F1-macro	F1-micro	F1-macro	F1-micro	F1-macro	F1-micro	F1-macro	F1-micro	F1-macro
GAE	$0.825$	$0.819$	$0.710$	$0.611$	$0.636$	$0.632$	$0.624$	$0.467$	$0.614$	$0.401$
ARGE	$0.837$	$0.833$	$0.736$	$0.674$	$0.606$	$0.601$	$0.645$	$0.601$	$0.633$	$0.423$
DSGC	$0.844$	$0.841$	$0.759$	$0.705$	$0.644$	$0.639$	$0.693$	$0.611$	$0.643$	$0.426$
DGI	$0.857$	$0.856$	$0.768$	$0.709$	$0.753$	$0.750$	$0.605$	$0.418$	$0.646$	$0.415$
GIC	$0.859$	$0.857$	$0.754$	$0.698$	$0.773$	$0.780$	$0.622$	$0.433$	$0.628$	$0.427$
GCA	$\mathbf{0.864}$	$\mathbf{0.860}$	$0.757$	$0.702$	$0.815$	$0.836$	$0.752$	$0.639$	$0.655$	$0.431$
FeatWalk	$0.843$	$0.843$	$0.760$	$0.703$	$0.935$	$0.934$	$0.663$	$0.503$	$0.637$	$0.424$
CAN	$0.841$	$0.835$	$0.721$	$0.657$	$0.652$	$0.648$	$0.820$	$0.726$	$0.650$	$0.428$
CONN	$\mathbf{0.866}$	$\mathbf{0.862}$	$\mathbf{0.775}$	$\mathbf{0.723}$	$\mathbf{0.945}$	$\mathbf{0.944}$	$\mathbf{0.913}$	$\mathbf{0.879}$	$\mathbf{0.674}$	$\mathbf{0.456}$

ACM [121]. It is a large-scale citation network consisting of 48,579 papers published in ACM. Words in the paper abstracts are adopted as node attributes based on the bag-of-words model. Citation links are treated as edges. Each paper is published under a specific area which serves as label for classification.

BlogCatalog [1]. It is a social network collected from a blog community. Nodes are web users and edges indicate the user interactions. Node attributes denote the keywords of their blogs. Each user could register his/her blogs into six different predefined classes, which are considered as class labels for node classification.

Reddit [15]. It is another social network dataset collected from the online discussion forum-Reddit. Nodes correspond to the Reddit posts and edges represent the co-comment relationships. The posts are preprocessed into 602-dimensional feature vectors via Glove CommonCrawl world embedding [122]. Hence, node attributes refer to the 602 latent dimensions. We use the communities or ‘subreddit’ that the post belongs to a target label.

ogbl-collab [123]. It is a challenging author collaboration network from KDD Cup 2021. Each node is an author and edges indicate the collaboration between authors. All nodes come with 128-dimensional features, obtained by averaging the word embeddings of papers that are published by the authors. It is widely used to conduct link prediction task.

ogbn-arxiv [123]. It is a large-scale paper citation network of arXiv papers from KDD Cup 2021. Each node is an arXiv paper and the edge indicates that one paper cites another one. Each paper is represented by a 128-dimensional feature vector obtained by averaging the embeddings of words in its title and abstract. The target is to predict 40 subject areas.

5.2 Baseline Methods

To validate the effectiveness of CONN, we include four categories of unsupervised baselines as follows. First, to study why we need tailored framework to incorporate node attributes into GCN-architectures, we compare with vanilla GCN methods GAE [14]. Second, to investigate how effective is CONN compared with other tailored solutions, we include two recent works, i.e., CAN [62] and FeatWalk [59]. Third, to have a comprehensive evaluation with state-of-the-art unsupervised models, we include three popular self-supervised learning based GNN methods, DGI [94], GIC [95], and GCA [96]. Note that, other non-GCN based embedding methods are not included, i.e., DeepWalk [58], LINE [71] and ANRL [60], since they are outperformed by CAN and FeatWalk in their experiments [62, 59]. Besides, other classical GNNs architectures, i.e., GAT [46], SGC [54], and APPNP [61], are not included for comparison, since they are initially dedicated for supervised learning while we focus on unsupervised representation learning.

• GAE [14]. It learns node embeddings by reconstructing the network structure under the autoencoder approach. Specifically, it employs graph convolutional network to encode a subgraph into latent space.

• ARGE [124].It is an adversarially regularized GAE. We do not consider the variational version since the two variants perform quick similar in most cases.

• DGI [94]. It learns node embeddings by maximizing the mutual information between the local patch representation and the global graph representation.

• GIC [95]. It updates DGI by leveraging cluster-level node representation for unsupervised representation learning.

• GCA [96]. It learns node representations by minimizing the contrastive loss between the original graph and its augmented forms.

• CAN [62]. It learns node embeddings by reconstructing both the network structure and attribute matrix under the variational autoencoder framework.

• FeatWalk [59]. It advances vanilla random-walk based methods via introducing an attribute-enhanced random walk strategy, which helps to generate diversified random walks for representation learning.

• DSGC [125]. It defines a $k$-NN graph based on node features and then uses it as an attribute-aware graph filter for network embedding.

Besides, we also introduce three variants to validate the effectiveness of core components in CONN.

• CONN-gcn. It replaces the graph mixing convolutional layer with vanilla graph convolutional layer, to verify the effectiveness of modeling mixed neighbors, i.e., nodes and attributes.

• CONN-inner. It excludes the graph correlation layer and utilizes the inner-product to estimate the similarity between two nodes based on their last layer representations similar to [15, 14]. We use it to verify the usefulness of the proposed correlation layer.

• CONN-ncoll. It only considers the node-to-node interactions in the objective function. This variant is used to certify the contribution of jointly optimizing the node-to-node and node-to-attribute interactions.

5.3 Experimental Settings

We follow the common protocol [14, 62] to evaluate the performance of CONN. The effectiveness of the learned latent representations is evaluated over two popular downstream tasks, i.e., link prediction and node classification. For link prediction task, we randomly split 85%, 10% and 5% edges in the network to form training, testing and validation sets similar to [14]. The link prediction task aims to estimate whether a missing edge in the network should be connected or not, based on its embedding representations. The performance is measured by two standard metrics, i.e., area under the ROC curve (AUC) and average precision (AP) scores.

For node classification task, it targets to classify a new instance into one or multiple categories, based on the obtained node representations and the trained classifier. Specifically, we apply 5-fold cross-validation on all datasets to construct the training and test sets. To perform classification, we build an SVM classifier based on scikit-learn package and train the classifier based on the nodes in the training group and corresponding labels. Then we apply the learned classifier to predict the labels of instances in the test groups. The averaged results of five-fold is reported.

We use the official released codes of baselines for experiments and use the validation set to tune their parameters. For our method, we train CONN for 100 epochs using Adam optimizer with learning rate 0.01 and early stopping with a patience of 20 epochs. If it is not specified, $d$ is set as 128, $K=2$, and $\alpha$ equals to 0.2 and 0.8 for node classification and link prediction tasks, respectively. For continuous datasets (Reddit, ogbn-arxiv, and ogbl-collab), we use top-$50$ values in each row of $𝐗$ to construct the bipartite graph by default. The source code of CONN is available at https://github.com/Qiaoyut/CONN.

TABLE IV: Link prediction results.

	Pubmed		ACM		BlogCatalog		Reddit		ogbl-collab
	AUC	AP	AUC	AP	AUC	AP	AUC	AP	AUC	AP
GAE	$0.920$	$0.911$	$0.957$	$0.956$	$0.824$	$0.822$	$0.578$	$0.565$	$0.821$	$0.737$
ARGE	$0.968$	$0.971$	$0.964$	$0.969$	$0.755$	$0.723$	$0.603$	$0.597$	$0.847$	$0.814$
DSGC	$0.964$	$0.970$	$0.961$	$0.965$	$0.788$	$0.774$	$0.593$	$0.586$	$0.842$	$0.808$
DGI	$0.942$	$0.927$	$0.582$	$0.644$	$0.733$	$0.737$	$0.594$	$0.586$	$0.818$	$0.728$
GIC	$0.937$	$0.935$	$0.674$	$0.775$	$0.748$	$0.745$	$0.693$	$0.677$	$0.892$	$0.804$
GCA	$0.955$	$0.956$	$0.756$	$0.820$	$0.826$	$0.808$	$0.717$	$0.741$	$0.886$	$0.833$
FeatWalk	$0.940$	$0.941$	$0.963$	$0.963$	$0.617$	$0.617$	$0.852$	$0.870$	$0.828$	$0.797$
CAN	$0.980$	$0.977$	$0.896$	$0.899$	$0.837$	$0.837$	$0.909$	$0.903$	$0.901$	$0.886$
CONN	$\mathbf{0.994}$	$\mathbf{0.993}$	$\mathbf{0.986}$	$\mathbf{0.987}$	$\mathbf{0.918}$	$\mathbf{0.906}$	$\mathbf{0.986}$	$\mathbf{0.985}$	$\mathbf{0.930}$	$\mathbf{0.912}$

TABLE V: Ablation study of CONN on node classification task.

		CONN-gcn	CONN-inner	CONN-ncoll	CONN
F1-micro	Pubmed	$0.795$	$0.701$	$0.851$	$\mathbf{0.866}$
	ACM	$0.704$	$0.643$	$0.756$	$\mathbf{0.775}$
	BlogCatalog	$0.745$	$0.794$	$0.900$	$\mathbf{0.945}$
	Reddit	$0.891$	$0.571$	$0.891$	$\mathbf{0.913}$
	ogbn-arxiv	$0.614$	$0.553$	$0.635$	$\mathbf{0.674}$
F1-macro	Pubmed	$0.795$	$0.708$	$0.846$	$\mathbf{0.862}$
	ACM	$0.633$	$0.551$	$0.694$	$\mathbf{0.723}$
	BlogCatalog	$0.740$	$0.787$	$0.898$	$\mathbf{0.944}$
	Reddit	$0.849$	$0.464$	$0.844$	$\mathbf{0.879}$
	ogbn-arxiv	$0.408$	$0.388$	$0.415$	$\mathbf{0.456}$

5.4 Node Classification

We start to evaluate the performance of CONN in node classification task (Q1). Table III summarizes the results on five datasets in terms of micro-average and macro-average scores.

From Table III, we observe that CONN performs consistently better than other baselines across two evaluation metrics on datasets with categorical attributes (PubMed, ACM, and BlogCatalog) and continuous feature embeddings (Reddit and ogbn-arxiv). It demonstrates the effectiveness of CONN. To be specific, compared with vanilla GCN variants, CONN improves 48.6% and 9.8% over GAE on BlogCatalog and ogbn-arxiv datasets in terms of micro-average score, respectively. CONN improves 46.7% and 4.8% over ARGE on BlogCatalog and ogbn-arxiv datasets under the macro-average score, respectively. This improvement validates the necessarity to design tailored GCN architecture for attributed networks. CAN is also proposed to model on node attributes, but it loses to CONN significantly on five scenarios. The major difference between them is that CAN targets to jointly reconstruct node attributes and network under auto-encoder framework, while CONN aims to revise GCN architecture by explicitly leveraging node attributes to guide message propagation. This comparison certifies that a tailored GCN solution for attributed networks is more promising and effective. FeatWalk incorporates node attributes to random-walk based models, but it is outperformed by CONN in all cases. Taking Reddit dataset for example, CONN improves FeatWalk over 37.7%. It is reasonable since our model can leverage structure information for node embedding while random-walk based methods fail to use that. Although DSGC also constructs an attribute-aware graph for network embedding, it loses to CONN in almost all cases. This is because DSGC requires to construct a $k$-NN graph based on node features, which may delete a lot of important attribute information.

Another promising observation is that CONN performs significantly better than state-of-the-art self-supervised competitors (DGI, GIC and GCA) in general. This results indicates the effectiveness of reconstructing original network structure for unsupervised representation learning. Besides, the performance gap between CONN and FeatWalk increases on continuous-value datasets (reddit and orgn-arxiv). This is mainly because FeatWalk reduces to random choice among all attribute categories as the sampling graph is fully-connected, which damages the quality of random walks.

​​​

(a) Trade-off parameter $\alpha$

(b) The number of layers $K$

(c) Embedding dimension $d$

​​​​​​ ​​​​​​

Figure 2: Hyper-parameter analysis of CONN.

5.5 Link Prediction

We now evaluate the performance of CONN in terms of link prediction task (Q1). Since DGI, GIC, GCA, and FeatWalk are not originally tested on this task, we delete the test edges from the adjacent matrix and use the resulting adjacent matrix to train them. Then, we estimate the similarity scores for test edges based on the inner product of corresponding node representations similar to [62]. Table IV reports the results on three median size (PubMed, ACM, and BlogCatalog) and two large-scale (Reddit and ogbl-collab) datasets in terms of AUC and AP scores.

From the table, we can see that CONN performs significantly better than other baselines. Specifically, it improves 11.4%, 21.5%, 16.5%, 25.2%, 22.7%, 11.1%, 48.8%, and 9.7% over GAE, ARGE, DSGC, DGI, GIC, GCA, FeatWalk, and CAN on BlogCatalog in terms of AUC value, respectively. CAN loses to FeatWalk on node classification task in most cases, but outperforms FeatWalk on predicting missing edges. It indicates the importance of capturing structure information for link prediction. Both CAN and CONN target to leverage node attributes for node embedding, but CONN outperforms CAN with a great margin in all cases. The main difference is that CAN focuses on using node attributes to enrich the objective function, with the hope to enhance GCN encoder optimization. In contrast, our model directly merges useful node attributes into the GCN building blocks, such that a more powerful GCN encoder could be explicitly achieved. Although DGI, GIC, and GCA are trained based on the advanced contrastive loss function, our model performs substantially better than them with a wide margin. These results indicate the insufficiency of existing GCN-based architectures in exploiting useful node attributes. Based on these observations, we believe that our proposed framework is more suitable for the link prediction task.

5.6 Ablation Study

We now investigate the second question (Q2), i.e., how much could CONN’s three major components, i.e., graph mixing convolutional layer, graph correlation layer, and collaborative optimization contribute? Three variants, i.e., CONN-gcn, CONN-inner, and CONN-ncoll, that are introduced at the beginning, are used for this ablation study. Table V records the node classification performance on five datasets in terms of micro-and macro-average scores.

Based on Table V, we have three major observations. First, without node attributes to guide message propagation in the graph convolution process, the performance of CONN-gcn decreases. It validates the effectiveness of explicitly incorporating node attributes into GCN architecture. Second, without the graph correlation layer, CONN-inner loses to CONN with great margin. For instance, CONN achieves 23.5% improvements than CONN-inner on Pubmed in terms of micro-average. The major difference between CONN-inner and CONN is that the former one adopts simple inner product to estimate edge similarity, while CONN devises deep correlation layer, which is capable of capturing the complex correlations between two nodes. This comparison verifies the effectiveness of the proposed graph correlation layer. Third, CONN performs slightly better than CONN-ncoll across five datasets. It indicates that jointly optimizing node-to-node and node-to-attribute interactions is beneficial. Given that CONN outperforms three variants, it verifies that we propose a principled framework to reinforce the reciprocal effects among the three components.

5.7 Parameter Sensitivity Analysis

We now study the impact of parameters $\alpha$ and embedding dimension $d$ over CONN on BlogCatalog (Q3). $\alpha$ controls the importance of node-to-node interaction and node-to-attribute interaction for message passing. We plot the performance of CONN when $\alpha$ varies from $0$ to $1$ with step size 0.1 on Figure 2-a. From the results, we observe that the performance of CONN increases as $\alpha$ increases from 0.0 to 0.2 on five datasets. CONN obtains the best results when $\alpha =0.2$ in general. Notice that, when $\alpha =0$, CONN only utilizes the node attribute interactions for graph convolution, and the network structure is excluded. When $\alpha =1$, node-to-attribute interactions are not considered. CONN reduces to the vanilla GCN, except that both node attribute and node interactions are jointly optimized.

$K$ is the number of layers for GCN backbones. Large $K$ means that high-order neighbors are included. Since FeatWalk has no such parameter, we omit it for comparison. We vary $K$ from 2 to 8, and the performance on BlogCatalog is shown in Figure 2-b. From the results, we observe that our model CONN consistently outperforms all baselines on the different number of layers. Another interesting observation is that when $K$ varies from 2 to 8, the performance of CAN, GAE, GIC, GCA, and DGI decreases while CONN performs stable. It validates the superiority of our model in capturing high-order dependencies. It is worth noting that although CONN achieves relatively stable results than standard GNN models in Figure 2 (b) when K is small, e.g., 8, we do observe an obvious performance drop when K is larger, i.e., 15. Therefore, by modeling attribute-aware relationships, our method can alleviate the over-smoothing problem when the GNN model is relatively shallow to some extent. Still, tailored efforts are required to thoroughly tackle the over-smoothing issues as done in [52].

In CONN, the dimension $d$ presents the output latent space for downstream applications, i.e., node classification and link prediction. We search it from $\{16,32,64,128,256\}$, and the performance of all methods are depicted in Figure 2-c. We observe that different methods have different optimal $d$, and our model CONN could achieve satisfactory performance when $d=128$. Similar observations are made on other datasets. To make it fair, we tune the best embedding dimension for all methods and report their best results.

​​​

(a) ogbl-collab

(b) Reddit

​​​​​​

Figure 3: CONN performance w.r.t. different top-$N$ values.

5.8 Further Analysis

With performance comparison with SOTA methods completed, we delve into our proposal for even more insights.

5.8.1 Constructing a sparse bipartite graph from continuous features is a good trade-off.

The first question we would like to answer is: what is the impact of top-$N$ values for graphs with continuous features? Figure 3 shows the results of our model w.r.t. different $N$ values on two graphs (Reddit and ogbl-collab) with embedding vectors (e.g., continuous values) as node attributes. From the figures, we can see that the performance of CONN increases with the increasing of $N$ values, and it achieves good results when $N$ equals the dimension of node attributes, i.e., 128 and 602 for collab and reddit, respectively.

This observation sheds light on the following insights. (i) Top-$N$ strategy is a good way to handle continuous feature vectors in our model, since $N=50$ can already achieve satisfactory results in both cases. (ii) Furthermore, it indicates that our model can be used in high-dimensional data, since we can first adopt classical dimensional reduction techniques to reduce the dimension.

To check the scalability of our model on high-dimensional data, we first adopt Deepwalk to obtain 128-d embedding vectors for nodes in ACM and BlogCataglog, and then use the resultant node features as input to train our model. We observe comparable results in two downstream tasks. Specifically, the AUC results for ACM and BlogCatalog on link prediction are 0.987 and 0.916; while the Micro-average results on node classification are 0.769 and 0.941.

5.8.2 Modeling node attributes as a bipartite graph enhances the robustness of GNNs for representation learning.

Next, we explore whether modeling node attributes as an augmented graph can improve model’s robustness towards noises, i.e., missing links. We analyze the robustness of our model and the vanilla GNNs competitor (GAE) towards edge perturbation, i.e., randomly masking some edges with ratios, ranging from 0.0 to 0.9 with step size 0.1. We show the results in terms of link prediction in Figure 4. Similar sensitive tendencies are observed in other datasets.

The results in two figures show that our model performs relatively stable when the masking ratio is less than 50% across two datasets, while the performance of GAE drops in general when the ratio increases. We believe this stable merit is attributed to the proposed augmented bipartite graph because it can replenish missing connections between nodes via using node categories as intermedia.

​​​

(a) BlogCatalog

(b) Reddit

​​​​​​

Figure 4: Robustness analysis of CONN vs. GAE with different edge perturbation ratios.

5.8.3 Jointly reconstructing node attributes and node interactions does not impact the convergence speed.

Moreover, we would like to examine the empirical convergence of our model. We show the training curves of CONN under different GNNs layers over two representative datasets (BlogCatalog and Reddit for categorical and continuous attributes, respectively) in Figure 5. We can observe that the training loss decreases very fast in the first 10 epochs, and our model tends to converge within 50 epochs in general.

TABLE VI: Performance of CONN vs. heterogeneous GNNs methods. ”NC” stands for node classification results in terms of F1-micro metric; ”LP” stands for link prediction results in terms of AUC.

		HetGNN	MAGNN	CONN
NC	Pubmed	$0.836$	$0.839$	$\mathbf{0.866}$
	ACM	$0.724$	$0.726$	$\mathbf{0.775}$
	BlogCatalog	$0.785$	$0.816$	$\mathbf{0.945}$
	Reddit	$0.646$	$0.659$	$\mathbf{0.913}$
	ogbn-arxiv	$0.625$	$0.637$	$\mathbf{0.674}$
LP	Pubmed	$0.936$	$0.942$	$\mathbf{0.994}$
	ACM	$0.869$	$0.877$	$\mathbf{0.986}$
	BlogCatalog	$0.808$	$0.812$	$\mathbf{0.918}$
	Reddit	$0.646$	$0.681$	$\mathbf{0.986}$
	ogbn-collab	$0.834$	$0.856$	$\mathbf{0.930}$

5.8.4 Regarding the augmented network as heterogeneous graph is suboptimal.

Finally, we investigate the generalization of our proposal in terms of heterogeneous GNNs. In particular, we want to explore whether the well-established heterogeneous GNNs efforts can be directly applied for attributed network embedding by using our proposed augmented network. To this end, we select two popular unsupervised heterogeneous GNN embedding methods as backbone: HetGNN [112] and MAGNN [126]. For a fair comparison, we initialize learnable embeddings for nodes and attribute categories similar to our model. Table VI reports the results on both node classification and link prediction tasks.

We observe a clear performance gap between our CONN and two heterogeneous GNNs methods (HetGNN and MAGNN) on all evaluation scenarios. Jointly considering the results in Table III and IV, the performance of two heterogeneous variants ranks in the middle against all baselines. The possible explanation is that regarding augmented networks as a heterogeneous graph may over-emphasize the heterogeneity between node interactions and node attributes. It is hard to control the importance between them via either random walk [112] or meta-path [126]. These comparisons demonstrate our motivation to treat the augmented network as a homogeneous graph. We believe nontrivial efforts are needed to unleash the power of heterogeneous GNNs on attributed network embedding.

​​​

(a) BlogCatalog

(b) Reddit

​​​​​​

Figure 5: Empirical training curves of CONN on two datasets with different GNN layers.

6 Conclusion

In this paper, we study the problem of unsupervised node representation learning on attributed graphs under graph neural networks (GNNs). Existing GNNs efforts mainly focus on exploiting topological structures, while only using node attributes as initial node presentations in the first layer. Here, we argue that such GNN architecture is suboptimal to modeling real-world attributed graphs, since node attributes are totally excluded from the key factors of GNNs, i.e., the message aggregation mechanism and the training objectives. To tackle this problem, we propose a novel collaborative graph neural network termed CONN. It allows node attributes to determine the message-passing process for neighborhood aggregation and enables the node-to-node and node-to-attribute-category interactions to be jointly recovered. Empirical results on node classification and link prediction tasks over social and citation graphs demonstrate the superiority of CONN against state-of-the-art embedding methods. Our future work is to explore its applicability for dynamic graphs and further improve the robustness of our model by integrating adversarial training. Moreover, we are interested in exploring similar ideas in recommendation scenarios [127], such as sequential recommendation [128, 129] and session recommendation [130, 131].

7 Acknowledgement

We thank the anomalous reviewers for the feedback. The work is, in part, supported by NSF (IIS-2224843 and IIS-1849085).