Graph Foundation Models

Abstract

Graph Foundation Model (GFM) is a new trending research topic in the graph domain, aiming to develop a graph model capable of generalizing across different graphs and tasks. However, a versatile GFM has not yet been achieved. The key challenge in building GFM is how to enable positive transfer across graphs with diverse structural patterns. Inspired by the existing foundation models in the CV and NLP domains, we propose a novel perspective for the GFM development by advocating for a “graph vocabulary”, in which the basic transferable units underlying graphs encode the invariance on graphs. We ground the graph vocabulary construction from essential aspects including network analysis, theoretical foundations, and stability. Such a vocabulary perspective can potentially advance the future GFM design following the neural scaling laws.

1 Introduction

Foundation models (Bommasani et al., 2021), which are pre-trained on massive data and can be adapted to tackle a wide range of downstream tasks, have achieved inimitable success in various domains, e.g., computer vision (CV) (Radford et al., 2021) and natural language processing (NLP) (Bubeck et al., 2023; Touvron et al., 2023). Typically, foundation models can effectively utilize both the prior knowledge obtained from the pre-training stage and the data from downstream tasks to achieve better performance (Han et al., 2021) and even deliver promising efficacy with few-shot task demonstrations (Dong et al., 2022).

Meanwhile, graphs are vital and distinctive data structures that encapsulate non-Euclidean and intricate object relationships. Given that various graphs embody unique relations, most graph learning approaches are tailored to train from scratch for a single task on a particular graph. Such an approach requires data collection and deployment for each individual graph and task. Consequently, an intriguing question emerges: Is it possible to devise a Graph Foundation Model (GFM) capable of generalizing across different graphs and tasks?

Despite the significant advancements in foundation models in other domains, the development of GFMs remains in the infant stage. Recent research has demonstrated the potential of GFMs in specialized areas, such as knowledge graphs (Galkin et al., 2023) and molecular structures (Beaini et al., 2023). However, these models often concentrate on limited scenarios: they may be applied across multiple datasets but restricted to a single task, span various tasks within a single domain, or be confined to a small set of datasets and tasks. To date, a versatile GFM that can generalize across the broad spectrum of graph-based applications has yet to be realized.

The key difficulty in achieving the GFM is how to achieve positive transfer across diverse graph data ranging from social networks to molecule graphs with countless structure patterns. The answer from the CV and NLP domains is a shared vocabulary. In the NLP foundation models, the text is first broken down into smaller units based on the vocabulary, which can be words, phrases, or symbols. In the CV foundation models, the image is mapped to a series of discrete image tokens (Yu et al., 2023a; Bai et al., 2023) based on the vision token vocabulary. The vocabulary defines the basic units in the particular domain, transferable across different tasks and datasets. Therefore, the key challenges in achieving the GFM narrow down to how we can find the graph vocabulary, the basic transferable units underlying graphs to encode the invariance on graphs.

However, finding a suitable graph vocabulary that works across diverse graphs is challenging, which is the primary focus of this paper.

Our contributions: In this paper, we present a vocabulary perspective to clearly state the position of the GFM. In particular, we attribute the existing success of primitive GFMs to the suitable vocabulary construction guided by the particular transferability principle on graphs in Section 2. The comprehensive review of the transferability principles on graphs is illustrated in Section 3, serving us the principle for future vocabulary construction and the GFM design.

In Section 4, we discuss the potential for building the GFM following neural scaling laws from the vocabulary perspective and review the current progress towards advanced GFM. Finally, we introduce more insights and open questions to inspire constructive discussions on the GFM in Section 5.

Comparison with past relevant literature Concurrent to our position paper, Jin et al. (2023a); Li et al. (2023b); Zhang et al. (2023c) reviews those methods adapting large language model (LLM) for graph, which haven’t shown transferring capabilities and thus diverge from our scope to build a graph-centric GFM. Liu et al. (2023e) further discusses existing graph pre-training and adaption techniques with a focus on their implementations. Instead of technical details, our work focuses more on the fundamental principles, e.g., geometric invariance across datasets. With principle guidance, we depict the promising and relatively elusive directions for the development of GFMs.

2 Existing primitive GFMs and the key reasons for the existing success

The GFM is formally defined in Liu et al. (2023e), which expects a GFM with the capability to transfer across all the graph tasks and datasets.

Nonetheless, there remains no such GFM satisfying the above definition yet. In contrast, we have observed that existing models (Galkin et al., 2023; Zheng et al., 2023a) have achieved the initial success, obtaining the desired capability of the GFM on certain aspects.

In this work, we name them as the “primitive GFM”, which can further be categorized into task-specific, domain-specific, and prototype GFMs based on their transferability. Details can be found in Section 2.1. The key reasons for their success is then further discussed in Section 2.2.

2.1 Existing primitive GFMs

Based on the model transferability across domains and tasks, we can roughly distinguish the existing primitive GFMs into three categories: task-specific, domain-specific, and prototype GFMs. We provide definitions and examples for each category, with additional illustrations in Appendix B.

A task-specific/domain-specific GFM should be transferable across the specific task/domain and thus adapt to diverse downstream datasets and domain-specific tasks. A notable example of a task-specific GFM is ULTRA (Galkin et al., 2023), achieving superior zero-shot knowledge graph completion (KGC) performance across datasets from various domains. A domain-specific GFM instance, DiG (Zheng et al., 2023a), learns universal representations across various chemical tasks by leveraging domain-specific knowledge.

A Prototype GFM exhibits the capability to generalize towards a limited number of datasets and tasks. A notable example is PRODIGY (Huang et al., 2023a) which can perform few-shot in-context learning over node classification and link prediction on text-attributed graphs. However, since Huang et al. (2023a) employs a shallow GNN as its core model, the scalability and flexibility to additional datasets and tasks, given its modest model size, remains uncertain.

Concurrently, there have been efforts (Li et al., 2023b) to harness LLMs for graph-related tasks. Although LLMs excel in the NLP realm, directly simply flattening graph structures and prompting LLM yields suboptimal results, evidenced by studies on both text-attributed (Chen et al., 2023b) and non-text-attributed graphs (Wang et al., 2023a). A noteworthy approach is GraphText (Zhao et al., 2023b) employs a tree-based prompt design that retains structural semantics, delivering satisfactory few-shot in-context learning performance. However, the LLM meets fundamental issues (Saparov & He, 2022; Dziri et al., 2023) on addressing graph data. We provide a further discussion on the usage of LLM in the graph domain in Section 5.

2.2 The key reasons for the existing GFM success.

Despite the empirical success achieved by the primitive GFMs, the underlying reasons remain unclear. In particular, we investigate the mechanism on the ULTRA (Galkin et al., 2023), which is a task-specific GFM focusing on the knowledge graph completion (KGC) task. THe KGC task aims to infer the missing triplet (edge), denoted as $(h,r,t)$, where $r$ is a query relationship, $h$ and $t$ are the head and tail entities, respectively. The KGC model aims to answer the query $(h,r,\mathrm{?})$ by predicting the tail entity $t$.

The first reason for its success is to utilize the NBFNet (Zhu et al., 2021b) backbone model which enables the inductive generalization to new graphs with an expressive relational vocabulary. The NBFNet proposes a conditional message passing that can learn the pairwise-node representation conditioned on a head entity node and a query relation.

Huang et al. (2023c) demonstrates that this conditional message passing, grounded in the relational Weisfeiler-Leman algorithm, theoretically offers greater expressiveness in KGC compared to standard, unconditional GNNs (Li et al., 2022). Such expressiveness helps to distinguish the difference between knowledge graphs with different structural features, leading to a suitable relational vocabulary. In contrast, Barcelo et al. (2022) indicates that those unconditional GNNs, e.g., R-GCN (Schlichtkrull et al., 2018) and CompGCN (Vashishth et al., 2019), map non-isomorphic node pairs into the same representation, leading to a contracted relational vocabulary. Such contracted vocabulary may lead to negative transfer with inappropriately generalizing knowledge across non-isomorphic node pairs with inherent differences.

However, such expressive relational vocabulary only considers the pre-defined relation types which cannot generalize to the scenario with new relation types during inference. To extend the existing relational vocabulary including new relationship type, Galkin et al. (2023) constructs a graph of relations that captures fundamental interactions independent from any graph-specific relation types, serving as the second reason for its success. The graph of relations is theoretically grounded (Gao et al., 2023) which aims to learn the double permutation-equivariant representations. Such representation is equivariant to permutations of both node entities and edge relation types. Such equivariance can be an analogy to a shared relational vocabulary. It connects the new unseen relationship types to the existing ones and maps the equivariant node pairs into the same representation despite different relation types, leading to the positive transfer.

In summary, we can conclude the key for ULTRA to achieve good transferability is finding a suitable vocabulary for KGC satisfying two principles: (1) The vocabulary should not be compacted, which causes distinct node pairs to share representations, leading to potential negative transfer. (2) The vocabulary should be sufficiently inclusive to map new, unknown relationships onto the existing vocabulary, potentially enabling positive transfer.

The effectiveness of finding a suitable vocabulary for building the GFM can also be found in other existing primitive GFMs with the following evidence. GraphGPT (Zhao et al., 2023c) constructs a dataset-specific vocabulary where each node corresponds to a unique node ID. Notably, GraphGPT requires specific pre-training and fine-tuning on each dataset. MoleBERT (Xia et al., 2023), the foundation model for molecule graphs, manually designs a vocabulary that transforms atom attributes into chemically meaningful codes. Similarly, OFA (Liu et al., 2023b) manually converts all the node features into meaningful text descriptions.

3 The underlying transferability principle

In the last section, we investigate the key reason for the existing success of building the GFM. The key principle for building the GFM is to construct a suitable graph vocabulary, which keeps the essential invariance across datasets and tasks. Despite the existing success,

more graph transferability principles can discover different invariances and serve as guidance for constructing new suitable graph vocabulary for future GFMs.

The following discussions are organized as follows:

We first provide a general introduction to the graph transferability principles in Section 3.1. Detailed task-specific principles on node classification, link prediction, and graph classification tasks can be found in Section 3.2, 3.3, and 3.4, respectively. We finally discuss the principles for task transferability in Section 3.5. Notably, the following discussions majorly concentrate on the transferability of the graph structure.

3.1 Graph transferability principles

In this subsection, we introduce principles that enable transferability on graphs, focusing on three key aspects: network analysis, expressiveness, and stability.

Network analysis provides a conventional understanding of the network system by identifying fundamental graph patterns, e.g., network motif (Menczer et al., 2020) and establishing the key principles, e.g., triadic closure principle (Huang et al., 2015) and homophily principle, which are generally valid across different domains. Those principles have been generally utilized to guide the design of advanced GNNs. For example, the state-of-the-art GNN for link prediction (Wang et al., 2023b) is a Neural Common Neighbor, inspired by the triadic closure principle. Despite its effectiveness, network analysis heavily relies on expert knowledge without a provable guarantee.

Expressiveness provides a theoretical background as to which functions graph neural architectures can model in general, e.g., a well-known connection that graph-level performance of GNNs is bounded by Weisfeiler-Leman tests (Xu et al., 2019; Morris et al., 2019, 2023). The most-expressive structural representation (Srinivasan & Ribeiro, 2019) is the key concept describing that the representation of two node sets should be invariant if and only if the node sets are symmetric with a permutation equivalence. Such most-expressive structural representation serves as an important principle to design a suitable graph vocabulary that perfectly distinguishes all non-isomorphic structural patterns in multi-ary prediction tasks.

Stability (Ruiz et al., 2023) assesses the representation sensitivity to graph perturbations. It aims to maintain a bounded gap in predictions for pairs under minor perturbations, rather than the expressiveness only distinguishing between isomorphic and non-isomorphic cases. The stability imposes a stricter constraint leading to better generalization. It can be an analogy to the constraint on the graph vocabulary where similar structure patterns should have similar representation.

Moreover, there are some other challenges that would affect the deeper GNN design to include more higher-order information. A deeper GNN may suffer from performance degradation and fail to capture higher-order structural information. There are majorly three issues: (i) The oversquashing problem (Topping et al., 2021) illustrates that the node representation is insensitive to information from important but distant nodes. (ii) The oversmoothing problem (Oono & Suzuki, 2019; Cai & Wang, 2020) illustrates that more aggregations lead to the node representations converging to a unique equilibrium, which loses the distinction between different nodes. (iii) The underreaching (Barceló et al., 2020) illustrates the failure to explore, cover, or affect all relevant nodes in the graph, leading to information loss. Various techniques are proposed to identify the root causes (Di Giovanni et al., 2023; Wu et al., 2023) and solve the expressiveness issues via new GNN (Yang et al., 2021) and graph transformer (Wu et al., 2022; Müller et al., 2023) architecture designs.

3.2 Transferability principles in node classification

Network analysis. Homophily (Khanam et al., 2020), which describes the phenomenon of linked nodes often sharing similar features (“birds of a feather flock together”), is a longstanding principle in social science. It serves as the principle guidance for methods ranging from conventional pagerank (Chien et al., 2021) and label propagation (Chawla & Karakoulas, 2005) to the recent advanced GNNs. Existing GNN architectures, often crafted based on the homophily principle, demonstrate strong performance on diverse homophilous graphs across various domains. This adherence to homophily not only enhances model effectiveness but also facilitates model transferability among homophilous graph datasets. Notably, successful transfers among such graphs are evidenced in Ying et al. (2018).

While homophily predominates in network analysis, it is not a universal rule. In many real-world scenarios, “opposites attract”, resulting in networks characterized by heterophily—where nodes are more likely to link with dissimilar nodes. GNNs built with the homophily principle often struggle with heterophilious networks, except in cases of “good heterophily” (Ma et al., 2021; Luan et al., 2021), where GNNs can identify and leverage consistent patterns in connections between dissimilar nodes. However, most heterophilious networks are complex and varied, posing challenges for GNNs due to their irregular and intricate interaction patterns (Luan et al., 2023; Wang et al., 2024; Mao et al., 2023a). Consequently, GNNs’ transferability, more assured in homophilous graphs, is facing significant challenges in heterophilous ones.

Stability. You et al. (2023) theoretically establishes the relationship between transferability and network stability, demonstrating that graph filters with enhanced spectral smoothness and a smaller maximum frequency response exhibit improved transferability in terms of node features and structure, respectively. In particular, spectral smoothness, characterized by the Lipschitz constant of the graph filter function of the corresponding GNN, indicates stability against edge perturbations. The maximum frequency response, reflecting the highest spectral frequency after applying a graph filter (essentially the largest eigenvalue of the Laplacian matrix), describes stability against feature perturbations.

3.3 Transferability principles in link prediction

Network Analysis. Important network analysis principles (Mao et al., 2023b) fall into three primary concepts including: (1) local structural proximity corresponding to the triadic closure principle (Huang et al., 2015), where friends of friends become friends themselves. It inspires well-known conventional methods including CN, RA, AA (Adamic & Adar, 2003). (2) global structural proximity corresponding to the decay factor principle, where two nodes with more short paths between them have a higher probability of being connected. It inspires well-known conventional methods e.g., Simrank and Katz (Katz, 1953; Jeh & Widom, 2002). (3) feature proximity corresponding to the homophily principle (Murase et al., 2019) where shared beliefs and thoughts can be found in connected individuals.

These principles guide the evolution of link prediction algorithms, from basic heuristics to sophisticated GNNs (Chamberlain et al., 2022; Li et al., 2023a). GNNs, inspired by these principles, perform well across diverse graphs in multiple domains. Moreover, Zheng et al. (2023b) provides empirical evidence supporting the beneficial transferability of these guiding principles.

Figure 1: In this graph, nodes $v_1$ and $v_4$ are isomorphic; links $(v_1,v_2)$ and $(v_2,v_4)$ are not isomorphic. However, vanilla GNN with the same node representations $v_1$ and $v_4$ gives the same prediction to links $(v_1,v_2)$ and $(v_2,v_4)$.

Expressiveness. A vanilla GNN, equipped with only single-node permutation equivalence, cannot achieve transferability for the link prediction task due to its lack of expressiveness. An example to showcase such failure is shown in Figure 1 with a featureless graph. $v_1$ and $v_4$ are represented identically by the vanilla GNN, as they possess identical neighborhood structures.

Therefore, the similarity between $v_1$ and $v_2$ will be the same as the one between $v_4$ and $v_2$, leading to identical representations and predictions for both links $(v_1,v_2)$ and $(v_2,v_4)$ However, according to the global structural proximity, $(v_1,v_2)$, with a shorter distance of 1, should be more likely to be connected. The vanilla GNN, computing $v_1$’s representation solely from its neighborhood, overlooks the structural dependence with $v_2$. As a result, this potentially leads to negative transfer, where the GNN might erroneously predict both or neither link to exist, whereas it’s more likely that only $(v_1,v_2)$ has a link.

To consider all the possible dependencies between node pairs, we aim for the most expressive structural representation for the link prediction. This representation should be invariant if and only if links are symmetric. Zhang & Chen (2018) achieves such structural representation by incorporating node labeling features that depend on both the source and target nodes in a link. Zhang et al. (2021) further highlights the key aspects of node labeling design, including: (1) target-nodes-distinguishing, where the source and target nodes have distinct labels compared to other nodes; and (2) permutation equivariance. Node labeling methods that fulfill these criteria, such as double radius node labeling (DRNL) and zero-one (ZO) labeling, can produce the most expressive structural representations. The expressiveness representation can find the complete set of distinct relations to differentiate all non-isomorphic node pairs, thereby mitigating the risk of negative transfer in standard GNNs. Huang et al. (2023c) extends the relational Weisfeiler-Leman framework (Barcelo et al., 2022) to link prediction and incorporate the concept of labeling tricks to multi-relational graphs.

Stability. For those equally expressive structural representations, there may still be a gap in terms of their stability. For example, empirical evidence (Zhang et al., 2021) shows that GNNs with DRNL labeling outperform those with ZO labeling. From the perspective of stability, it is crucial to maintain a bounded gap in predictions for pairs under minor perturbations. Wang et al. (2021) provides a theoretical analysis identifying key properties of stable positional encoding (GNNs should be rotation and permutation equivariant to positional encodings) that enhance generalization.

3.4 Transferability principle in graph classification

Network Analysis. Network motifs, typically composed of small and recurrent subgraphs, are often considered the building blocks of a graph (Milo et al., 2002; Benson et al., 2016). A proper selection of the motif set can cover most essential knowledge on the specific datasets. Graph kernels (Vishwanathan et al., 2010) are proposed to quantify motif counts or other pre-defined graph structural features and then utilize the extracted features to build a classifier such as SVM. Despite the essential motif sets from different domains being generally different, there could exist a uniform set of motifs shared across different domains. In such cases, the positive transfer can be found on the uniform sets, where Battiston et al. (2020) shows the positive transfer across neuronal connectivity networks, food webs, and electronic circuits. Therefore, we conjecture that the network motif could be the base unit for the vocabulary (a set of invariant elements) for the graph classification as it is both explainable and potentially shared across graphs.

Expressiveness. Zhang et al. (2024) proposes a unified framework to understand the ability of different GNNs to detect and count graph substructures (motif). More expressive GNN which could detect more diverse motifs and construct a richer graph vocabulary. In analogy with the uniform motif sets, we conjecture that it is more possible for the expressive GNN to find the uniform motif sets and achieve better transferability.

Stability. Huang et al. (2023d) proposes a provably stable position encoding that surpasses the expressive sign and invariant encoding (Kreuzer et al., 2021) and modeling (Lim et al., 2022), enabling minimal changes to positional encodings on the minor modifications to the Laplacian. The key innovation is to apply a weighted sum of eigenvectors instead of treating each eigensubspace independently. Satisfying performance can be observed on the out-of-distribution molecular graph prediction.

Figure 2: Unifying different task formulations: (a) Link view: Given the target node, node classification is converted to the link prediction between the target node and corresponding label nodes. (b) Subgraph view: Node classification (orange node) is converted to the (green) ego-graph classification. Link prediction (orange nodes) is converted to the (green) induced-subgraph classification.

3.5 The transferability across tasks

A unified task formulation is generally employed to facilitate transferability across various tasks. The significance of aligning task formulations is evident in the following example: Jin et al. (2020b) shows that using link prediction directly as a pretext task leads to negative transfer for node classification. However, by reformulating node classification into a link prediction problem (Sun et al., 2022; Huang et al., 2023a), where a node’s class membership is treated as the link likelihood between the node and label nodes, positive transfer is achieved.

Liu et al. (2023g); Sun et al. (2023) further propose a subgraph view to adapt the node classification as an ego graph classification, and link prediction as a binary classification on the induced subgraph of the target node pair.

Figure 2 provides illustrative examples for these two unified views. The unified task formulation enables (1) enlarging the dataset size via converting datasets for different downstream tasks as one and (2) utilizing one pre-training model to serve different tasks.

Surpassing the unified formulation, there are shared principles on different tasks listed as follows: (1) Node classification and link prediction tasks share the feature homophily as an important principle. (2) Liu et al. (2023d) indicates that the global structural proximity principle on the link prediction can improve the node classification performance on the non-homophilous graph. (3) The triadic closure in the link prediction is a particular network motif utilized in the graph classification. There are more shared motifs (Hibshman et al., 2021; Dong et al., 2017; AbuOda et al., 2020; Kriege et al., 2020) on both graph classification and link prediction tasks.

4 Towards GFM following neural scaling law

The success of the foundation model can be attributed to the validity of the neural scaling law (Kaplan et al., 2020) which shows performance enhancement with increasing model scale and data scale. In this section, we first discuss when the neural scaling law happens in Section 4.1 from a graph vocabulary perspective. We then discuss techniques towards successful data scaling and model scaling in Section 4.2 and 4.3, respectively. The pre-text task design is finally discussed in Section 4.4.

4.1 When neural scaling law happens

In section 3, we discuss the underlying transferable principles guiding future vocabulary construction. Such principle guidance has led to the successful scaling behavior in the material science domain (Shoghi et al., 2023; Zhang et al., 2023a; Batatia et al., 2023) with the help of the geometric prior. Nonetheless, we are still cautious about whether the existing success can be extended to the graph domain. The key concern is whether graphs can strictly follow those principles. Uncertainty can be found on the human-defined graph construction criteria (Brugere et al., 2018). For instance, the construction knowledge relying on expert knowledge may lead to uncertainty in edges (Ye et al., 2022). Chen et al. (2023b); Li et al. (2023c) observe that the mislabeled samples widely exist across datasets, where the popular CiteSeer dataset has more than 15 Despite the above uncertainty, different graph constructions with manual design can follow opposite principles. For instance, Ogbn-arxiv (Hu et al., 2020) and Arxiv-year (Lim et al., 2021) are two node classification datasets with identical graph information. The only difference lies in the label where Ogbn-arxiv employs paper categories, and Arxiv-year uses publication years as labels, resulting in conflicting homophily and heterophily properties (Mao et al., 2023a). Therefore, when uncertainties and opposite graph constructions exist, the scaling behavior may not happen as the data does not obey the graph transferable principles.

4.2 Data scaling

Chen et al. (2023a); Huang et al. (2023a) initially validate that GNNs trained in both supervised and self-supervised manners follow data scaling law on molecular property predictions, and node classification on text-attributed graphs. Xu et al. (2023) further exhibits that the similarity between pre-training data and downstream task data serves as a prerequisite for the data scaling on graphs. Specifically, Cao et al. (2023); Xu et al. (2023) provide concrete guidance on how to select the pre-training data via the graphon signal analysis and the essential network property, i.e. network entropy, respectively. The positive transfer can be found when the data exhibits similarity in the graphon space or with shared network properties.

A limitation in current research on data scaling is the small dataset size, particularly in contrast to the pre-training for LLMs (Raffel et al., 2019). This disparity may be attributed to the scarcity of high-quality data in the graph domain. NetworkRepository (Rossi & Ahmed, 2015), the largest graph database presently available, may support larger-scale pre-training, while burdensome pre-processing is required to clean those noisy and disordered data. Moreover, we illustrate a collection of useful datasets in Appendix A. We then show existing solutions as follows.

Tackling feature heterogeneity. Existing graph datasets cannot be uniformly utilized for pre-training due to the heterogeneous feature induced by missing features and different semantic spaces. Feature imputation techniques (Taguchi et al., 2021; Um et al., 2023; Gupta et al., 2023) are generally adapted to predict the missing attributes based on neighboring features. However, those techniques require each feature dimension to share the same semantic meaning. When features are not aligned in the same semantic space, Liu et al. (2023b) manually converts the original features with text descriptions and then encodes the embedding with LLMs. Moreover, feature misalignment can also be found in the inference stage between the pre-training model input and the test data. Jing et al. (2023) concatenates a learnable padding feature on the downstream task feature to align with the pre-trained GNN. However, such a technique cannot adapt to the case when the feature space is not aligned. Zhao et al. (2023b) directly abandons the original feature and utilizes the feature similarity as guidance.

Generating synthetic graphs is another path to enhance data scaling and enlarge the training data size. Traditional graph generative models (Albert & Barabási, 2002; Robins et al., 2007; Airoldi et al., 2008; Leskovec et al., 2010) are capable of generating graphs satisfying some certain statistical properties, which still plays an important role on node-level and link-level tasks.

Deep generative models on graph (Jin et al., 2020a; Luo et al., 2021; Jo et al., 2022; Vignac et al., 2023) have shown great success in generating high-quality synthetic graphs which helps graph-level tasks by providing a more comprehensive description of the graph distributions space.

Liu et al. (2023a) utilizes diffusion models to generate task-specific data argumentation to enhance the GNN on graph-level tasks with significant performance gain. With successful evidence of pre-training on synthetic data from other domains (Mishra et al., 2022; Trinh et al., 2024), we anticipate the potential to generate high-quality synthetic graphs for large-scale pre-training towards GFM.

4.3 Model scaling

Previous research in NLP indicates that apart from data, the backbone model constitutes a fundamental for scaling (Kaplan et al., 2020). Liu et al. (2024) primarily validates the neural scaling law on various graph tasks and model architectures under the supervised setting.

However, Kim et al. (2022) demonstrates that the GAT (Veličković et al., 2017) with a larger number of parameters underperforms on the graph regression tasks compared to the smaller-sized counterparts. As a comparison, geometric GNNs scale well to predict atomic potentials in material science (Shoghi et al., 2023; Zhang et al., 2023a; Batatia et al., 2023). The contradictory phenomenon indicates that the alignment between the geometric prior of GNNs and the specific task may influence model scaling.

Graph transformer is another popular choice for the model architecture, where geometric priors are explicitly modeled through either GNN or positional encoding (Müller et al., 2023). Masters et al. (2022); Lu et al. (2023) show that graph transformers show positive scaling capabilities for molecular data under a supervised setting. More recently, Zhao et al. (2023c) demonstrates vanilla transformer’s effectiveness in protein and molecular property prediction. Particularly, it views the graph as a sequence of tokens forming an Eulerian path (Edmonds & Johnson, 1973), which ensures the lossless serialization, and then adopts next-token prediction to pre-train transformers. After fine-tuning, it achieves promising results on the protein association prediction and molecular property prediction and shows that vanilla transformers also follow the model scaling law (Kaplan et al., 2020). Nonetheless, the effectiveness of such pure transformers on other tasks remains unclear.

4.4 Pretext task design

Given the scarcity of labeled data, a pretext task that can effectively utilize unsupervised data is the cornerstone for larger-scale neural scaling. We provide a brief review of the representative pretext designs.

Graph contrastive learning designs the pretext tasks (Sun et al., 2019; Veličković et al., 2018; Hassani & Khasahmadi, 2020; You et al., 2020) to obtain the equivalence via contrasting original and augmented views of the graph without materially changing the semantic content of the input. An initial unified understanding (Liu et al., 2022) on those pre-text tasks illustrates that existing pretext tasks focus on preserving the invariance with the low frequency on the graph spectrum. Nonetheless, different pretext tasks remain different where Zhu et al. (2021a) observes that satisfactory performance requires pretext tasks and downstream tasks share similar philosophies, such as homophily. To obtain a pre-training model that benefits different downstream tasks, Ju et al. (2023) adaptively combined pretext tasks with different philosophies via a multi-task learning framework.

The generative self-supervised learning designs the pre-text tasks (Hou et al., 2022; Hu et al., 2019; Kipf & Welling, 2016) to capture the shared data generation process among different tasks. Particularly, they attempt to predict the masked portions of the graph using the remaining structure and features. Liu et al. (2023f); Xia et al. (2023) further observe that task granularity also plays an important role in generative modeling. Specifically, employing node-level pretext tasks may lead the model to learn only low-level features (Liu et al., 2023f) while ignoring the global information essential for graph-level tasks. To address this issue, they adopt a GNN-based tokenizer to explicitly model high-level information in the pre-training stage and thus improve the downstream task performance.

More recently, the next token prediction (NTP) pre-text task (Zhao et al., 2023c) achieves initial success in the molecular graph. Notably, this is the first pretext task demonstrating empirical evidence of model scaling. The potential reason for its success may be (1) the construction of a fixed token set, narrowing down the problem space in a finite set to only predict a discrete token and (2) choosing transformers as the backbone model. However, it remains unclear whether the success can be easily extended to more tasks.

5 Insights & open questions

The usage of LLM in the graph domain. With the existing success of LLMs (Bubeck et al., 2023) in the NLP domain, a natural question is whether we can directly apply its original language-centric ability on the graph. Primary exploration of the LLM focuses on text-attributed graphs. Chen et al. (2023b); He et al. (2023) treat node classification as the text classification on the target node, illustrating promising results on the zero-shot setting even without any structural information. For graphs without text attributes, Fatemi et al. (2023); Wang et al. (2023a) feed the data into LLM via describing graph structure with natural language. Observations show that LLMs present limited capability in understanding the essential graph structures. The key reason is that the LLM is required to tackle problems sequentially greedily (McCoy et al., 2023), leading to a shortcut solution (Saparov & He, 2022; Dziri et al., 2023) rather than a formal analysis on the graph structure. Moreover, Yue et al. (2023) points out the efficiency issue of utilizing LLM on the recommendation, the downstream link prediction task.

Tang et al. (2023); Ye et al. (2023); Chai et al. (2023) thus utilizes the GNN to encoder the graph structure knowledge and then introduces a linear layer to transform the embeddings obtained from the GNN into text space as a prompt token. Although these models can perform well, they still have two shortcomings: (1) The ability to process structures is bounded by the capabilities of GNN; (2) The instruction tuning can be costly while the tuned model can only tackle the corresponding downstream task and is not transferable to other tasks and datasets, which makes their capabilities distant from a GFM. We agree that LLM illustrates superior performance on textual node feature understanding. Nonetheless, it remains unclear whether LLM should play a key role in building GFM or just serve as a better textual feature encoder.

A potential strategy for equipping LLMs to effectively handle graph structures could be inspired by the success of multi-modal foundation models (Liu et al., 2023c). The key idea (Yu et al., 2023b) is to map the graph structure to a series of discrete tokens with a pre-trained tokenizer, which may convert graphs into a format comprehensible by LLMs. More discussion can be found in Appendix C.

Efficiency issue in subgraph-based methods. Subgraph-based extraction is a widely adopted technique to achieve inductive inference (Zeng et al., 2021) and unify different task formulations (Sun et al., 2023). Nonetheless, the subgraph-based extraction leads to the following issues: (1) information loss in high-order neighborhoods; (2) duplicate subgraph information with excessive memory consumption, and (3) the time complexity of vanilla subgraph extraction grows exponentially with the number of hops. Those issues hinder the applicability of subgraph-based methods. Graph sampling techniques like (Zeng et al., 2019) and global state vectors (Fey et al., 2021) can help to alleviate these issues.

Potential redundancy on pre-text task and architecture design. There are mainly two approaches to achieving transferability: (1) designing GNNs with specific geometric properties for transfer, e.g., ULTRA (Galkin et al., 2023), and (2) creating pretext tasks to automatically learn these properties. (Jin et al., 2020b) suggests an overlap between these approaches, indicating that pretext tasks targeting local structural information might be unnecessary, given that GNNs often inherently encode this information. Investigating the strengths and limitations of these techniques, along with providing practical guidance for their selection, could be a valuable research direction. A hypothesis might be that model design methods are more suitable for data that strictly adheres to geometric priors, while pretext task designs are more effective in the opposite scenario.

The necessity of GFM. Graphs can be defined in different ways based on different criteria like similarity or influence between node pairs (Brugere et al., 2018). We can then categorize graphs based on the observability of the criteria. The observable graph is unambiguously known, e.g., whether one paper cites another paper in a citation graph. Text and images can also be viewed as a specific case of observable graphs. In contrast, the unobservable ones are manually conducted with ambiguous descriptions of the relationship, e.g., whether one gene regulates the expression of another in a gene-regulate graph. These graphs may not naturally exist in the world, leading to uncertainty with a lack of invariant principle. We are not sure whether such graphs should be considered in the GFM.

Moreover, there are concerns about the benefit of training a GFM on graphs that are neither from the same domain nor share the same downstream task. On the one hand, it seems that training on them simultaneously shows no positive transfer benefit while increasing the risk of the negative transfer. On the other hand, we are not aware of potential undiscovered transferability leading to success.

GFMs on more applications. In this paper, we majorly focus on the conventional graph tasks, e.g., node classification, link prediction, graph classification, and the principle for building GFM on them. However, graph formulation is the universal representation ability, which can easily extend to different domains, e.g., scene graph for CV (Zhai et al., 2023; Zhong et al., 2021), bipartite graph for linear programming (Chen et al., 2022), and physical graphs for understanding physical mechanisms (Shi et al., 2022). Each domain can have its principle requiring specific graph vocabulary construction. Existing success can be found in material science (Zhang et al., 2023a; Batatia et al., 2023). For instance, (Batatia et al., 2023) utilizes the Equivariant Graph Tensor Network which can encode many-body information of atomic geometry can conduct molecular dynamics simulation across different chemistries. A potential application of GFM is to aid the generation of scene graphs (Zhong et al., 2021), involving predicting relationships between localized objects. Knowledge graph foundation models (Galkin et al., 2023) which can generalize to unseen entities and relations, may effectively model relationships and thus generate high-quality scene graphs.

6 Conclusion

From the transferability principles of graphs, we review existing GFMs and ground their effectiveness from a vocabulary view to find a set of basic transferrable units across graphs and tasks. Our key perspectives can be summarized as follows: (1) Constructing a universal GFM is challenging, but domain/task-specific GFMs are approachable with the usual availability of a specific vocabulary. (2) One challenge is developing GFM following the neural scaling law, which requires principle-driven design in data engineering, architecture, and pretext tasks. This paper summarizes the current position and challenges toward the next step, which may be a blueprint for GFM to inspire relevant research.

7 Impact Statements

In this paper, we provide principle guidance for the development of graph foundation models, which can be a pivotal infrastructure empowering diverse applications like nature science and E-commerce. The graph foundation model may reduce the resource consumption associated with training numerous task-specific models. Moreover, it may substantially curtail the requirement for manual annotation, particularly in domains such as molecular property prediction. We anticipate that our contributions will advance the ongoing efforts aimed at developing next-generation graph foundation models with better versatility and fairness.

Appendix A A collection of datasets to support large-scale pre-training

In this section, we show a collection of large-scale graph datasets from various fields to support pre-training massive-scale graph foundation models.

Table 1: A collection of datasets together with their URL and descriptions to support larger-scale pre-training

Name	URL	Description
TU-Dataset (Morris et al., 2020)	https://chrsmrrs.github.io/datasets/	A collection of graph-level prediction datasets
NetworkRepository (Rossi & Ahmed, 2015)	https://networkrepository.com/	The largest graph datasets, with graphs coming from 30+ different domains
Open Graph Benchmark (Hu et al., 2020)	https://ogb.stanford.edu/	Contains a bunch of large-scale graph benchmarks
Pyg (Fey & Lenssen, 2019)	https://pytorch-geometric.readthedocs.io	Official datasets provided by PYG, containing popular datasets for benchmark
SNAP (Leskovec & Krevl, 2014)	https://snap.stanford.edu/data/	Mainly focus on social network
Aminer (Tang, 2016)	https://www.aminer.cn/data/	A collection of academic graphs
OAG (Zhang et al., 2023b)	https://www.aminer.cn/open-academic-graph	A large-scale academic graph
MalNet (Freitas et al., 2021)	https://www.mal-net.org/#home	A large-scale function calling graph for malware detection
ScholKG (Dessí et al., 2022)	https://scholkg.kmi.open.ac.uk/	A large-scale scholarly knowledge graph
Graphium (Beaini et al., 2023)	https://github.com/datamol-io/graphium	A massive dataset for molecular property prediction
Live Graph Lab (Zhang et al., 2023d)	https://livegraphlab.github.io/	A large-scale temporal graph for NFT transactions
Temporal Graph Benchmark (Huang et al., 2023b)	https://docs.tgb.complexdatalab.com/	A large-scale benchmark for temporal graph learning
MoleculeNet (Ramsundar et al., 2019)	https://moleculenet.org/	A benchmark for molecular machine learning
Recsys data (Project, )	https://cseweb.ucsd.edu/~jmcauley/datasets.html	A collection of datasets for recommender systems
LINKX (Lim et al., 2021)	https://github.com/CUAI/Non-Homophily-Large-Scale	A collection of large-scale non-homophilous graphs

Appendix B Existing GFMs

In this section, we demonstrate existing representative GFMs and categorize them into prototype GFM, domain-specific GFM, and task-specific GFM, as shown in Table 2.

Table 2: A collection of existing GFMs

	Name	Domain	Task
Prototype GFM	PRODIGY (Huang et al., 2023a)	Text-attributed graph, Knowledge graph	Node classification, Knowledge graph reasoning
	OneForAll (Liu et al., 2023b)	Text-attributed graph, Knowledge graph, Molecule	Node classification, Knowledge graph reasoning, Graph classification
	GraphText (Zhao et al., 2023b)	General graph	Node classification
Domain-specific GFM	DIG (Zheng et al., 2023a)	Molecule	Molecular sampling, Property-guided structure generation.
	MACE-MP-0 (Batatia et al., 2023)	Material Science	Property predictions of solids, liquids, gases, and chemical reactions.
	JMP-1 (Shoghi et al., 2023)	Material Science	Atomic property prediction
	DPA-2 (Zhang et al., 2023a)	Material Science	Molecular simulation
	MoleBERT (Xia et al., 2023)	Molecule	Molecule property prediction
Task-specific GFM	ULTRA (Galkin et al., 2023)	Knowledge graph	Knowledge graph reasoning

Appendix C A more comprehensive discussion on applying LLMs to graph

In this position paper, we mainly discuss how to build a graph-centric foundation model. Many people also understand the graph foundation model as the combination of graphs and other foundation models (Jin et al., 2023a; Li et al., 2023b; Liu et al., 2023e). In this part, we provide an introduction to those representative works. Specifically, the current integration of graph and foundational models follows two primary pathways. The first involves using graphs to augment the capabilities of other foundational models. The second employs foundational models to address challenges encountered in graph machine learning.

The first type of work focuses on enhancing the capabilities of foundation models by graphs. Yasunaga et al. (2022a, b); Jin et al. (2023b); Xie et al. (2023) further pre-train LLMs on text-attributed graphs with a structure-aware pretext task. For example, Yasunaga et al. (2022b) trains an LLM to predict masked edges, which is formalized as pair classification on two end nodes’ attributes. Structure-aware training can effectively enhance language models’ capability on those tasks requiring structure reasoning, such as multi-hop reasoning. These works still view the graph as a second-class citizen providing auxiliary information and put more emphasis on text-centric tasks like question answering (Yasunaga et al., 2022a, b). Tang et al. (2023); Ye et al. (2023); Chai et al. (2023) improves LLMs’ capabilities tackling graph-related problems with instruction tuning. They adopt a linear map to inject the embedding generated by GNNs as a prompt token, then tune LLMs according to downstream tasks. A notable drawback of these works lies in that the tuned models don’t present capabilities beyond GNNs while introducing computation overhead.

The second line of work adopts LLMs’ capabilities to solve challenges in the graph domain. Chen et al. (2023b); He et al. (2023) adopts LLMs as encoders to augment the node attributes and enhance GNNs’ performance on node classification. Chen et al. (2023c) aims at solving the cold-start problem in node classification, and shows that actively annotating nodes with LLMs can significantly improve GNNs’ zero-shot inference capability. Zhao et al. (2023a) unifies graph and text representations of molecules via instruction-tuning, which then achieves promising zero-shot performance on property prediction.

We then provide a more comprehensive discussion on the positive potential of graphs on LLM. As discussed in Section 2.1, the straightforward application of LLms to graph learning by natural language description is less than ideal. The primary issue is that LLMs tend to approach problems with a sequentially greedy mindset (McCoy et al., 2023), which often results in superficial, ’shortcut’ solutions (Saparov & He, 2022; Dziri et al., 2023) rather than in-depth structural analysis. Thus, the principal challenge in adapting LLMs to serve as GFMs is enabling them to comprehend graph structures effectively. One promising solution involves structurally formatted prompts that LLMs can process. For instance, GraphText (Zhao et al., 2023b) introduces a tree-based prompt format that may allow LLMs to perform graph reasoning while preserving structural semantics. It remains positive potential for LLM to serve as a GFM.