Knowledge-augmented Deep Learning and Its Applications: A Survey

Abstract

Deep learning models, though having achieved great success in many different fields over the past years, are usually data hungry, fail to perform well on unseen samples, and lack of interpretability. Various prior knowledge often exists in the target domain and their use can alleviate the deficiencies with deep learning. To better mimic the behavior of human brains, different advanced methods have been proposed to identify domain knowledge and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning, which we refer to as knowledge-augmented deep learning (KADL). In this survey, we define the concept of KADL, and introduce its three major tasks, i.e., knowledge identification, knowledge representation, and knowledge integration. Different from existing surveys that are focused on a specific type of knowledge, we provide a broad and complete taxonomy of domain knowledge and its representations. Based on our taxonomy, we provide a systematic review of existing techniques, different from existing works that survey integration approaches agnostic to taxonomy of knowledge. This survey subsumes existing works and offers a bird’s-eye view of research in the general area of knowledge-augmented deep learning. The thorough and critical reviews of numerous papers help not only understand current progresses but also identify future directions for the research on knowledge-augmented deep learning.

I Introduction

Despite the impressive performance that existing deep models have achieved in various fields, they suffer from several serious deficiencies, including high data dependency and poor generalization [1]. These deficiencies originate primarily from the models’ data-driven nature and their inability to effectively exploit the domain knowledge. To address these limitations, a knowledge-augmented deep learning paradigm begins to attract researchers’ attention, whereby domain knowledge and observable data work together synergistically to produce data-efficient, generalizable, and interpretable deep learning algorithms.

Real-world domain knowledge is rich. In the context of deep learning, the domain knowledge mainly originates from two sources: target knowledge and measurement knowledge. Target knowledge governs the behaviors and properties of the target variables we intend to predict, while measurement knowledge controls the underlying mechanism that produces the observed data of the target variables. Based on its representation, we propose to divide the domain knowledge that has been explored in deep learning into two categories: scientific knowledge and experiential knowledge. Scientific knowledge represents the well-established laws or theories in a domain that govern the properties or behaviors of target variables. In contrast, experiential knowledge refers to well-known facts or rules extracted from longtime observations and can also be inferred through humans’ reasoning. Knowledge can be represented and organized in various formats. Scientific knowledge is usually well represented rigorously with mathematical equations. Experiential knowledge, on the other hand, is usually represented less formally, such as through logic rules, knowledge graphs or probabilistic dependencies. Knowledge with different representations is integrated with data in a deep learning framework through different integration approaches.

Recognizing the deficiencies with current deep learning, there is a growing interest recently in capturing and encoding prior knowledge into the deep learning. Two mainstream techniques are neural-symbolic integration and physics informed deep learning. Neural-symbolic integration models focus on encoding experiential knowledge into the traditional symbolic AI models and integrating the symbolic models with deep learning models. The physics informed deep learning focuses on encoding various theoretical physical knowledge into different stages of deep learning. Current survey papers in this area are limited in scope as they focus on reviewing either neural-symbolic models or physics-informed machine learning methods, while ignoring many other related works. Specifically, the existing surveys on neural-symbolic models mainly consist of the discussions on logic rules or knowledge graphs, and their integration into deep models [2, 3]. Existing surveys on physics-informed machine learning, however, are limited to a specific scientific discipline, and the integration methodologies are usually task-specific, e.g., physics [4, 5], cyber-physical systems [6], geometry [7], and chemistry [8]. These surveys are hence focused on the methodologies for solving scientific problems under a lab environment, lack of discussions on real-world applications. To address this limitation, we present a comprehensive yet systematic review of existing works on knowledge-augmented deep learning. The contributions of our survey are three folds:

• This survey creates a novel taxonomy on domain knowledge, including both scientific knowledge and experiential knowledge. Our work subsumes existing works that focus on a subset of domain knowledge on specific disciplines [4, 5, 6, 7, 8].

• This survey covers a wide array of methodologies for knowledge representation and integration with a systematic categorization. It differs from existing surveys on general integration techniques, that are agnostic to the taxonomy of domain knowledge [9, 10, 11, 12, 13].

• This survey covers the methodologies that are not only for solving scientific problems under a lab environment, but, more importantly, for real-world application tasks. Not being limited to a specific application task, this survey involves the tasks ranging from computer vision to natural language processing. Our survey is hence of interest not only to deep learning researchers but also to deep learning practitioners in different fields.

We organize this survey as follows. We first introduce the concept of KADL in Section II, whereby we define the three basic tasks (i.e., knowledge identification, knowledge representation, and knowledge integration). We then review KADL methodologies, where we categorize different techniques based on their domain knowledge of focus: 1) deep learning with scientific knowledge in Section III and 2) deep learning with experiential knowledge in Section IV. In each category, we identify the domain knowledge, its representation formats, and the existing methods proposed for the integration of knowledge with data. An overview of existing methodologies in knowledge-augmented deep learning is included in Table I.

II Knowledge-augmented Deep Learning

Major tasks of knowledge-augmented deep learning include knowledge identification, knowledge representation, and knowledge integration into deep models. In the following sections, we introduce each of the major tasks in detail.

TABLE I: Overview of Knowledge-augmented Deep Learning

Knowledge Identification	Knowledge Representation	Knowledge Integration
		Data-level	Architecture-level	Training-level	Decision-level
Scientific Knowledge	Mathematical Equations	[14]	[15][16][17][18]	[19] [20][21][22]
			[23][24][25]	[26][27][28][29]
			[30] [31] [32][33]	[34][35] [36] [37]
			[38] [39][40][41]	[42][43][44][45]
			[46] [47] [48][49]	[50, 51, 52][53]
				[54] [55]
	Simulation Engines	[56][57][58][59]
		[60][61] [62][63]
Experiential Knowledge	Probabilistic Relationships	[14][64][65]	[66][67][68][69]	[70][71][72][73]	[64][14]
			[74]
	Logic Rules	[22]	[75]	[76][77]
	Knowledge Graphs		[78][79][80]	[81][82]	[83][84]

II-A Knowledge Identification

Knowledge identification involves identifying the domain knowledge for a specific task. For some tasks, domain knowledge is readily available and hence easy to identify, while in other tasks, the knowledge is less straightforward and requires domain experts’ efforts to identify. In this survey, we divide the domain knowledge into scientific knowledge and experiential knowledge. Scientific knowledge is prescriptive and mainly refers to well-formulated mathematical theories or physics laws. These laws are validated extensively through scientific experiments and are true in a universal setting (e.g., Newton’s laws). These laws explicitly define the deterministic relationships among different variables that are involved in the system. Recent work in physics-informed machine learning seeks to exploit various scientific knowledge from different disciplines for enhancing deep learning. Experiential knowledge is descriptive and mainly refers to well-known facts from daily life, indicating semantic properties of an entity or semantic relationships among multiple entities. Experiential knowledge usually is extracted from longtime observations but can also be derived from well-established studies or theories. The latter type of experiential knowledge is science-grounded and is focused on a semantic and abstract level of description. Experiential knowledge usually contains lots of fragmented information, and could be uncertain, imprecise, or ambiguous. Recent work in neural-symbolic models focuses on embedding experiential knowledge into deep learning.

II-B Knowledge Representation

Knowledge representation involves representing the identified domain knowledge in a well-organized and structured format. The appropriate representation depends on the type of domain knowledge. Scientific knowledge is usually expressed using equations. Besides, a simulation engine is also considered as an alternative representation of the scientific knowledge. Experiential knowledge is less formal compared to scientific knowledge. Experiential knowledge can be represented through probabilistic relationships, logic rules or knowledge graphs.

II-C Knowledge Integration

Knowledge integration entails integrating domain knowledge into deep models. Through the integration, a deep model can leverage both existing datasets and domain knowledge for certain tasks. Different integration methods can be employed depending on the types of knowledge, and can be divided into four levels: data-level, architecture-level, training-level, and decision-level, as shown in Figure 1.

Figure 1: Four levels of integration: data-level, architecture-level, training-level, and decision-level.

Data-level integration focuses on integrating the knowledge by augmenting the original training data with pseudo data, which is generated based on the knowledge. Architecture-level integration embeds knowledge by modifying neural network architecture. Training-level integration regularizes the training of a deep model via regularization terms or primary loss terms that are derived from the knowledge. Finally, decision-level integration combines a top-down prediction from a prior model with a bottom-up prediction from a deep model, whereby knowledge encoded in the prior model help refine the predictions of a deep learning pipeline.

Each type of integration could be beneficial from different aspects. Data-level integration can help alleviate data paucity issue that faced by many deep models [56, 57, 58, 59]. Additionally, data generation via automatic simulation is usually less expensive compared to human annotation. Architecture-level integration brings the bonus of making deep models interpretable and explainable, which are two crucial factors for trustworthy artificial intelligence [85]. Training-level integration can be regarded as the most common approach due to its straightforwardness. It can be flexibly applied to different deep model frameworks. Flexibility of training-level integration also promotes the quantification of knowledge uncertainties [53, 54, 45, 55]. Decision-level integration, different from previous three approaches, employs knowledge independent of the training of deep models, and is relatively less explored by existing works.

Selection of integration approach could be both task-dependent and knowledge-dependent. Architecture-level integration would be preferred if the knowledge is expected to be integrated in a hard way. The rest of three integration approaches (i.e., data-level, training-level, or decision-level) introduce knowledge into a deep learning pipeline in a soft way. Secondly, if the knowledge only involves target variables (i.e., predictions of a neural network), training-level integration would be preferred. To perform the other three types of integration, the knowledge is expected to involve measurements (e.g., intermediate variables or observations) and target variables. Lastly, if the identified knowledge consists of highly nonlinear and complex relationships, leveraging well-established engines or simulators for data-level integration would be the primary choice.

III Deep Learning with Scientific Knowledge

Deep learning models are gaining importance in advanced science and engineering areas that have traditionally been dominated by mechanistic (e.g., first principle) models. Such models produce particularly promising performances for scientific problems whose undergoing mechanisms are not well understood by experts or those problems for which exact solutions are computationally infeasible. Existing deep learning, however, requires a significant amount of annotated data, and it generalizes poorly to novel data or settings.

There is a growing consensus in the research community regarding combining conventional methodologies in science and engineering with existing data-driven deep models. Deep learning with scientific knowledge explores the continuum between classical mechanistic models and modern deep ones. There have been growing efforts in the machine learning community to incorporate scientific knowledge into deep learning (also referred to as physics-informed machine learning) to generate physically consistent and interpretable predictions and to reduce data dependency.

In the following, we first identify the types of scientific knowledge and their representations. We then introduce different methodologies on integrating scientific knowledge with deep models.

III-A Scientific Knowledge Identification

Scientific knowledge refers to well-formulated mathematical or physics equations that have been validated through extensive scientific experiments and are true in a universal setting (e.g., Newton’s laws). These laws explicitly define a deterministic and precise relationship among different objects that are involved in a system.

Current physics-informed deep learning seeks to explore the usage of classical mechanics models. For a dynamic system, the most widely considered scientific knowledge is Newtonian mechanics, which includes kinematics and dynamics. The former refers to observable motion (such as motion trajectories), which is typically represented with polynomial equations involving measurable properties (e.g., velocity, acceleration, or positions). Kinematics studies motion without regard for the cause. In contrast, dynamics studies the causes of a motion, whereby partial differential equations (PDEs) are used to capture the relationships between forces and measurable properties. Existing works explore the usage of dynamics in various physics systems (e.g., gas and fluid dynamics [86] and molecular dynamics of protein [87, 88]). With the understanding of dynamics, kinematics can be better predicted. Newtonian mechanics has hence been leveraged in real-world applications, such as human body behavior analysis [29, 89, 90]. Newtonian mechanics, unfortunately, can result in equations of motion that are intractable to solve, even for a seemingly simple system (e.g., double pendulum system). Lagrangian mechanics or Hamiltonian mechanics can instead be considered. As re-formulations to Newtonian mechanics, both Lagrangian mechanics and Hamiltonian mechanics leverage generalized coordinates, making them flexible with respect to which coordinates to use to understand a system. In Lagrangian mechanics, $L$ is defined as the difference between the kinetic energy, $T$, and potential energy, $U$, of a system ( i.e., $L=T-U$). The Hamiltonian $H$ resembles the Lagrangian $L$, and is defined as the summation of the kinetic energy, $T$, and potential energy, $U$, of a system (i.e., $H=T+U$). In Lagrangian mechanics, the time derivative of position $\dot{x}$ is considered as generalized momentum, whereas in Hamiltonian mechanics, momentum $p$ is considered. For simple particle systems, such differences are trivial, while in more complicated systems (e.g., magnetic fields), momentum can no longer be computed as simple a product of mass and velocity. The dynamic equations for both a Lagrangian system and a Hamiltonian system conserve energy over time with conservative forces.

Symmetries have also been widely explored in physics. Philip Anderson famously argued, “It is only slightly overstating the case to say that physics is the study of symmetry” [91]. Discovering symmetries has been proven to be important in both deepening the understanding of physics and enhancing machine learning algorithms. Equivariant or invariant functions preserve symmetries and have often been exploited for incorporating these symmetries into deep learning algorithms.

Optics, another type of physics knowledge, has also been considered. Optics studies the behavior and properties of light. Fermat’s principle [92] is the basic law in optics. In addition, the illumination models [93] and the rendering equation [94] capture the 3D object appearances with their image appearances. Existing works explore the usage of various illumination models for different computer vision tasks [95, 96] and computer graphics tasks [97, 98, 99].

Besides physics knowledge, mathematical theories such as theorems on existing algorithms (e.g., sorting or ranking), as well as continuous relaxation [100], have also been considered. Projective geometry theories [101] are widely applied to various computer vision tasks.

III-B Representation of Scientific Knowledge

Figure 2: Taxonomy of scientific knowledge and its representations.

Scientific knowledge is usually represented as equations, such as differential equations. Another important representation tool of scientific knowledge is simulation. Simulation mimics a real-world physics system and is treated as a surrogate representation of the physics principles that govern the real-world physics system. We visualize the taxonomy of scientific knowledge and its representations in Figure 2.

III-B1 Mathematical equation

Equations can include polynomial equations, differential equations, and integral equations, among which differential equations are widely explored by existing works. Dynamic laws are usually represented by PDEs, which indicate a deterministic relationship among different variables. In general, the PDEs are of the form

(1)

where $𝒙=\{x_1,x_2,\mathrm{\dots },x_N\}$ are $N$ variables involved in the system. ${𝒩}_{𝒙}$ is a general differential operator, and ${ℬ}_{𝒙}$ is the boundary condition operator. $D$ represents a physical domain, and $\mathrm{\Gamma }=\partial D$ represents the boundary of the domain. $𝝀(𝒙)$ represents the physics parameters involved in the PDEs. For simple systems, physical parameters are constants (i.e., $𝝀(𝒙)=𝝀$). $f(𝒙)$ is a forcing term, and $b(𝒙)$ specifies the boundary condition, e.g., Dirichlet boundary conditions for the Darcy flow problem [54]. $u(𝒙)$ is the solution of the differential equation given the specified boundary conditions. The equations become ordinary differential equations (ODEs) when only one variable is present. When terms (e.g., physics parameters $𝝀$) exist in differential equations that undergo stochastic processes, the equations become stochastic differential equations (SDEs). The general form of SDEs resembles standard differential equations, except for the random event $\omega$:

(2)

The physics parameters $𝝀(𝒙;\omega )$ and forcing terms $f(𝒙;\omega )$ are modeled as random processes, and thus the solution $u(𝒙;\omega )$ follows stochastic processes specified by $f$ and $𝝀$.

Differential equations describe a system’s evolution over time when we explicitly set one of the variables to correspond to time $t$, which commonly occurs in different dynamic systems. For example, the Euler-Lagrange equation defines the dynamics of a Lagrangian system,

$$\frac{\partial L}{\partial x}-\frac{d}{dt}\frac{\partial L}{\partial \dot{x}}=0$$ (3)

which connects the derivatives of the Lagrangian $L$ with respect to location in generalized coordinates $x$, time $t$, and generalized momentum $\dot{x}$. Differential equations have been widely explored by existing works, such as Newton’s second law [16, 34], Burgers’ equation occurring in gas and fluid dynamics [86], Hamilton’s equations in Hamiltonian dynamics [25], Euler-Lagrange equations for Lagrangian dynamics [102], and the Lorenz equations, which describe a nonlinear chaotic system used for atmospheric convection [103].

Laws of light are also expressed as equations. Fermat’s principle considers the integration over a light path. In [97], the transient $I$ in a transient imaging system is characterized through an integration as

$$I(\tau ;𝒗)={\int }_{𝒙\in \chi }f(𝒙;𝒗)\delta (\tau -\tau (𝒙;𝒗))𝑑A(p,q)$$ (4)

where $\tau$ is the traveled pathlength and $𝒗$ is the visible point. $A(p,q)$ measures the unit area of a surface, and the function $f$ absorbs reflectance and shading. The rendering equation represented as an integral equation has also been considered [104]. Equality algebraic equation has also been considered. According to the law of reflection, an image $I$ with reflections is a sum of the glass reflected back scene $\widehat{R}$ and the glass transmitted front scene $\widehat{T}$, i.e.,

$$I(x,y)=\widehat{T}(x,y)+\widehat{R}(x,y)$$ (5)

Such equality algebraic equation is demonstrated to be helpful in reflection removal tasks [98]. Another example is Malus’ law, expressed as an algebraic equation, which defines the effect of polarization [99].

III-B2 Simulation engines

Besides explicitly representing physics laws with equations, simulation through engines is another method of representation. Simulation mimics a real physical system governed by physics laws and is hence considered a surrogate representation of the knowledge. A physics engine mainly encodes the governing dynamic laws of a physics system, such as rigid-body, soft-body, and fluid. It computes the accelerations, velocity, and displacement of an object from forces, by solving the equation of motion. Specifically, a physics engine simulates observable kinematics given certain causes following governing dynamic rules. To simulate a sequence of motion of rigid-body objects given specific forces, a physics engine is considered [61, 105, 62, 106, 36, 107]. Most of these simulators are non-differentiable, making them prohibited to be employed in an end-to-end deep learning framework. There also exist simulation engines that encode kinematic laws for robotic manipulation. These inverse engines, however, estimate control actions using kinematic equations such that a desired position can be reached, agnostic to the underlying dynamic laws [35]. Graphic engines have also been explored. A graphic engine encodes the principled projection and illumination models, and renders realistic 2D observations by following the governing principles. For example, an engine, governed by laws of reflection, was proposed for the generation of faithful image rendering [98].

III-C Integration into Deep Models

To integrate domain-specific scientific knowledge into deep models, existing methodologies can be classified into three categories: data-level, architecture-level, and training-level integration as shown in Figure 1. Decision-level integration is rarely considered for scientific knowledge. Below we review methods to integrate scientific knowledge into deep models using these methods.

III-C1 Data-level Integration

One way to harness domain knowledge is to train deep models with data synthetized from conventional mechanistic models. Physics-based mechanistic models capturing the domain knowledge serve as simulators, and are used for generating synthetic data. The simulated data can either be combined with real data to jointly train the model or be used independently to pre-train the model through self-supervised learning.

Simulated data has been widely employed for entire training, where no additional real training data is required. In computer vision, Mottaghi et al. [62] proposed understanding the force acting on a query object by predicting its long-term motion in 3D space as the response to the force, given a static 2D image. An overview of its procedure appears in Figure 3.

Figure 3: Understanding physics from 2D images through simulation [62]. The game engine takes scene configurations as input and generates in total 66 Newtonian scenarios as an abstraction of the real physical world. The Newtonian neural network learns to understand the physics by classifying a static image into one of the 66 possible Newtonian scenarios.

The Blender game engine, consisting of a physics and a graphic engine, is employed for the simulation. Specifically, the physics engine takes a scene configuration as input (e.g., a ball sliding on a slide) and simulates the scene forward in time according to motion equations derived from Newton’s second law to produce different Newtonian scenarios. Physics parameters (e.g., force magnitude) are randomly sampled. For each Newtonian scenarios, the graphic engine renders 2D videos from the synthetic 3D world under different viewpoints in perspective projection. In total, 66 synthetic videos corresponding to 12 possible physical generated scenarios are used for training. Similarly, to better understand the human interaction with objects from 2D videos, Ehsani et al. [61] proposed a physical understanding of actions by inferring contact points together with forces from videos. A forward physics simulation is applied to supervise the force estimation, given observed videos, without requiring GT labels for forces. In particular, the 3D contact points of a moving object are estimated over a period given estimated forces, through physics simulation which is governed by Newton’s second law. A projection operator is then applied transforming the estimated 3D keypoints into a 2D space. By minimizing the difference between the estimated and observed 2D contact points over time, physically consistent forces are obtained. To perform differentiable physics simulations, a finite difference method is applied for gradient computation using the PyBullet simulator, which is focused on rigid-body simulation following Newton’s second law. In a similar vein, Tobin et al. [63] showed the effectiveness of synthetic samples generated by robotic simulation for training a deep model for object localization tasks, which is instrumental in robotic manipulation. Simulation is performed by employing the MuJoCo physics engine,, whereby simulated 2D images are generated based on its built-in graphic engine [108]. In particular, the MuJoCo physics engine is built based on Newtonian mechanics. The built-in graphic engine renders a 2D image given a selected camera in the 3D virtual environment through a perspective projection. To ensure sufficient simulation variabilities, a domain randomization strategy is proposed, where simulation parameters, such as the positions and orientation of objects, are all randomly specified during the simulation.

Deep model training often starts with a pre-training stage, followed by a fine-tuning. Existing studies also show that pre-training affects the final performance of deep models, mainly because poor pre-training can lead models to anchor in a local optimum. Pre-training through simulated data has shown to help improve parameters’ initialization. Jia et al. [56, 57] introduced a physics-guided recurrent neural network (PGRNN) to model lake temperature dynamics. PGRNN is pre-trained on synthetic data generated from a physics-based mechanistic simulator and is then fine-tuned with some observation data. The simulator models the lake temperature dynamics as a function of physics parameters (e.g., water clarity and wind sheltering) through PDEs. It was shown that, even with synthetic data generated with an imperfect set of physics parameters, PGRNN still achieves competitive performance. Such an idea has also been explored in engineering disciplines. In robotics, Bousmalis et al. [58] showed that the observation data required for accurate object grasping is significantly reduced (by a factor of 50) through physics-guided initialization. In autonomous driving, Shah et al. [59] pre-trained a driving algorithm with synthetic samples generated by a proposed simulator built on a game engine with physics laws embedded. In particular, the simulator includes a vehicle model and a physics engine. To describe a virtual 3D environment, physics parameters, such as gravity, air density, air pressure, and magnetic field, are specified manually. With these specified parameters, the physics engine predicts the kinematic states, given forces and torques estimated from a vehicle model by following motion equations derived from Newton’s second law. This work showed that the data needs of driving algorithms can be drastically lessened through pre-training with simulated samples. In addition to augmenting data through simulation with physics engines, synthetic data can be generated from mathematical equations [14].

III-C2 Architecture-level Integration

Domain knowledge can also be integrated through a customized design of neural network architectures. Architecture-level integration can be accomplished by 1) introducing specific physically meaningful variables or neural network parameters, 2) introducing layers derived from domain knowledge, and 3) introducing physics-inspired connectivity among neurons. We introduce each type of approaches at below.

Integrating through variables or parameters

One way to embed physical principles into the architecture of neural networks is to introduce physically meaningful variables in neural networks. The variable can be the output node of a neural network. Hamiltonian functionality that enforces energy conservation has attracted much attention [25, 24, 30, 31]. The Hamiltonian operator in physics is the primary tool for modeling dynamic systems with conserved quantities. In Hamiltonian mechanics, a classical physical system is described by a generalized coordinate, $q$, and a conjugate momenta, $p$. Hamiltonian $H$ then calculates the total energy of the system. The Hamiltonian equations defining the dynamics of a system are as follows:

$$\dot{q}=\frac{\partial H}{\partial p};\dot{p}=-\frac{\partial H}{\partial q}$$ (6)

Inspired by the Hamiltonian mechanics, a Hamiltonian neural network (HNN) has been proposed, where the output represents the Hamiltonian dynamics, through which energy conservation is explicitly enforced [24]. The differences between traditional NNs and HNNs can be understood easily, as shown in Figure 4, which shows that traditional NNs learn to predict particle trajectories, while HNNs learn a particle’s Hamiltonian, upon which trajectories can be predicted.

Figure 4: Comparison between a Hamiltonian neural network (HNN) and conventional neural network (NN). For dynamic modeling, instead of directly predicting trajectories, the HNN predicts the Hamiltonian $H$ of a system, through which the trajectory can be obtained under the energy conservation constraint.

Choudhary et al. [25] later showed that an HNN can better handle highly nonlinear dynamics in a chaos system by enforcing the conservation of the total energy. To demonstrate the practical importance of Hamiltonian formalism, Hamiltonian functionality was incorporated into a generative network, where Hamiltonian dynamics is learned from 2D observations without domain coordinate assumptions, such as images. The proposed Hamiltonian generative network has been applied to density estimation, leading to a neural Hamiltonian flow [24]. By leveraging Hamiltonian formalism, the total probability can be easily conserved, while the density modeling remains expressive. Sharing a similar idea with HNNs, a generalized energy conservation employing Lagrangian mechanics was explored in the Lagrangian neural network (LNN) [32, 33], where the output of the LNN is the Lagrangian dynamics. Nevertheless, evaluations of all these existing proposed models remain conceptual, without real applications in practice [38].

Variables can also be intermediate variables in a neural network. Jaques et al. [16] proposed a latent dynamics learning framework called the Newtonian Variational Autoencoder (NewtonianVAE). Inspired by Newton’s second law, a linear dynamic system in hidden space is defined, specifically by considering a rigid-body system with $D$ degrees of freedom and modeling the configuration of this system by a set of coordinates, $𝒙\in R^D$. Its dynamics are defined as

(7)

where $𝒖$ is the given actuation. To incorporate the above dynamic equations into a VAE, positions $𝒙$ are treated as random variables with velocities $𝒗$ being intermediate variables. Specifically, velocities are computed as ${𝒗}_t=\frac{𝒙-{𝒙}_{t-1}}{\mathrm{\Delta }t}$ with time interval $\mathrm{\Delta }t$. The conditional distribution of ${𝒙}_t$ given ${𝒖}_t$ now becomes

(8)

where $A$, $B$, and $C$ are estimated through a neural network, $f$, whose input consists of current system states (i.e., $\{{𝒙}_t,{𝒗}_t,{𝒖}_t$}). Newtonian VAE then outputs 2D images given the estimated $𝒙$. This is in contrast with the existing approaches that simply assume a Gaussian distribution over $p({𝒙}_t|{𝒙}_{t-1},{𝒖}_{t-1})$ without considering the underlying deterministic relationships among positions, forces, and velocities, as shown in Eq. (8). Similarly, to incorporate the physics behind the transport dynamics governed by advection-diffusion PDEs, Liu et al. [15] proposed a learning framework based on an autoencoder with the advection-diffusion equation explicitly incorporated. Two hidden variable outputs from the encoder are physically meaningful, representing velocity field and diffusion field, respectively, within the advection-diffusion equation. To model lake temperature, Daw et al. [17] introduced a physically meaningful intermediate variable for the proposed monotonicity-preserving long short term memory (LSTM) architecture. Specifically, the density value, as the intermediate variable of the LSTM, is enforced to monotonically increase as the depth increases, which is a crucial characteristic of lake temperature. A similar idea is applied in modeling drag forces acting on each particle in moving fluids [18]. Muralidhar et al. [18] proposed a PhyNet, where physics-constrained intermediate variables are introduced into a convolutional neural network (CNN) architecture. Specifically, two intermediate variables, characterizing velocity field and pressure field, respectively, are introduced into CNN for drag force prediction.

Besides introducing physically meaningful variables, another approach is to directly map some of the neural network parameters to physically meaningful parameters. These physics parameters can either be non-modifiable during training or be fine-tuned through learning from observations. In geophysics, neural networks have been considered for modeling the dynamic process of seismic waveform inversion [23]. To mimic seismic wave propagation, a theory-guided recurrent neural network (RNN) is proposed; an RNN is specially designed for solving the governing differential equations with some of the parameters assigned as the physics parameters in governing physics equations. In particular, given the wave equation discretized in the time domain, the wave field at the next time step (i.e., $u(t+\mathrm{\Delta }t)$) is calculated in terms of two previous time steps (i.e., $u(t)$ and $u(t-\mathrm{\Delta }t)$) as

where ${\nabla }^2$ is the spatial Laplacian operator, $r$ stands for position, and $s$ is the source function. The symbolic computation of $u(r,t+\mathrm{\Delta }t)$ given $u(r,t)$ and $u(r,t-\mathrm{\Delta }t)$ is directly implemented via a neural network, whose trainable parameters correspond to physics parameters $v^2\mathrm{\Delta }{t}^2$ in the wave equation Eq. (9).

Integrating through layers

The most representative type of knowledge that is usually integrated through neural network layers involves symmetries. Symmetry usually refers to a set of invertible transformations $g$, such as translation, rotation, or scaling.

Equivariance and invariance as representative types of symmetry have been widely considered. An invariant function is a mapping such that the output space is not affected by the symmetry transformations in the input space, and an equivariant function relaxes the invariant function. It states a mapping such that symmetries in the input space can be preserved in the output space. Mathematically, assume a symmetry transformation, $g$, and a function, $F$, mapping from $X$ to $Y$. $F$ is then said to be equivariant to $g$ if

$$F(g\cdot x)=g\cdot F(x)$$ (10)

The symmetry transformation $g$ on the input space $X$ is preserved on the output space $Y$. $F$ is said to be invariant if

$$F(g\cdot x)=F(x)$$ (11)

In other words, the output $y=F(x)$ is not affected by the symmetry transformation, $g$, acting upon the input space, $X$. Invariance can be a special case of equivariance, and vice versa. For example, transformation $g$ is an identify transformation with $g\cdot F(x)=F(x)$.

Equivariant or invariant neural network is designed to preserve symmetries. Illumination invariance features were explored [47] in computer vision, where a knowledge-guided convolutional layer is incorporated into existing deep models. Consider a day-night domain adaptation problem, and illumination changes from source domain to target domain cause a distribution shift. To tackle the distribution shift issue, features that are invariant to illumination are desired, which can be derived from Kubelka-Munk theory. The Kubelka-Munk theory [109] models material reflections by defining the spectrum of light reflected from an object in the viewing direction. The calculation of illumination invariant features defined by Kubelka-Munk theory is directly implemented through the proposed Color Invariant Convolution (CIConv) layer, as shown in Figure 5.

Figure 5: A physics layer (marked as red) for illumination invariant features [47]. The new layer extracts illumination invariant features given input images and is customized based on Kubelka-Munk theory, termed as Color Invariant Convolution (CIConv) layer.

Similarly, in turbulence modeling, rotational invariance states that the physics of a fluid flow does not depend on the orientation of the coordinate of an observer and is a fundamental physical principle. To embed the rotational invariance into a neural network, Ling et al. [41] defined a tensor basis neural network (TBNN), where the NN architecture is modified by adding a higher order multiplicative layer. Particularly, TBNN has one additional input layer accepting the tensor basis, and its last hidden layer performs a pair-wise multiplication using this tensor basis input layer to provide the output. The modified architecture ensures the prediction lie on a rotationally invariant tensor basis. By incorporating rotational invariance, TBNN achieves improved accuracy in predicting the normalized Reynolds stress anisotropy tensor. In the application to molecular dynamics, Anderson et al. [46] proposed a rotation invariant neural network, named Cormorant, whereby the behavior and properties of complex many-bodied physics systems are learned. Each neuron in Cormorant explicitly corresponds to a subset of atoms. Given specified neurons, the activation layer is ensured to be covariant to rotations, making the proposed Cormorant is guaranteed to be rotationally invariant.

Equivariance has also been explored. Wang et al. [39] showed that existing spatiotemporal deep models can achieve improved generalization ability by incorporating symmetries through equivariant functionality. More specifically, they consider four types of equivariance: time and space translation equivariance, rotation equivariance, uniform motion equivariance, and scale equivariance. These symmetries are incorporated into neural networks using customized equivariant layers. Through the composition of equivariant functions of layers, the network becomes equivariant.

In the real physical world, however, symmetries could be brittle. For example, a small perturbation can easily cause the discontinuous transition of a dynamic system or break the rotational symmetry of a pendulum system. Small perturbations often occur, and these can cause significant difference by accumulating over time. As a result, enforcing equivariance through customized layers as a hard inductive bias, could be problematic. To address this problem, Finzi et al. [40] recently proposed a soft way to impose equivariance constraints, whereby the proposed neural network architecture consists of a hybrid of restrictive layers and flexible layers. Restrictive layers are strictly constrained while flexible layers are unconstrained. Through a mixture of two types of layers, the equivariance is introduced as a flexible inductive bias.

Integrating through connectivity

Given physical dependencies among objects, connectivity among neural network neurons can be manually specified. To model dynamics with multiple objects involved, physically plausible interactions are employed to design NN connectivity. Neural Physics Engine (NPE), a differentiable physics simulator that combines symbolic structure with gradient-based learning, has been proposed [48]. Different from conventional physics engines based on mechanistic models, NPE is realized as a learning-based neural network while remaining generalizable across different scenarios. The development of NPE considers the fact that physics is Markovian, both temporally and spatially. Temporal Markovian allows the NPE to predict system states by only considering the states at the current step. Spatial Markovian allows the NPE to factorize the interaction dynamics into pairwise interactions. The NPE consists of a symbolic model structure mimicking pairwise interactions among objects, realized as a neural network. The NPE takes 2D observations as input and performs forward dynamics to predict the motions of objects in the future. A very similar work was independently developed for n-body interaction systems [49].

III-C3 Training-level Integration

One of the most common techniques to integrate scientific knowledge into deep models is through the training of deep models. In particular, constraints over the outputs of deep models are derived from scientific knowledge and are used as regularization terms for training a deep model. An augmented training objective is generally expressed as

$$ℒ={ℒ}_{TRN}(y^{GT},y^{Pred})+\lambda {ℒ}_{PHY}(y^{Pred};x)$$ (12)

${ℒ}_{TRN}(y^{GT},y^{Pred})$ represent standard training loss given predictions $y^{Pred}$ and ground truth labels $y^{GT}$. For classification task, ${ℒ}_{TRN}(y^{GT},y^{Pred})$ is usually defined based on the cross-entropy loss. The physics-based regularization term ${ℒ}_{PHY}(y^{Pred};x)$ corresponds to physical constraints, with an adjustable importance coefficient, $\lambda$. When the physics-based constraints are independent of inputs $x$, the regularization term is reduced to ${ℒ}_{PHY}(y^{Pred})$. Through ${ℒ}_{PHY}(y^{Pred};x)$, training is guided towards producing models with physically consistent outputs. The computation of physics-based regularization, ${ℒ}_{PHY}(y^{Pred};x)$, does not require annotations of observations and hence allows unlabeled data to be included in training, reducing dependencies on data. Physics-based regularization, ${ℒ}_{PHY}(y^{Pred};x)$, can also be employed directly for training deep models in a label-free manner.

${ℒ}_{PHY}(y^{Pred};x)$ can be explicit or implicit and can be flexibly employed in different deep learning frameworks. Explicit regularization is directly defined over the output of deep models based on domain knowledge, while implicit regularization is induced by the physics-based models that are embedded into a deep learning pipeline. In the following, we introduce the knowledge-guided model regularization under two distinguishable deep model frameworks: discriminative deep models and generative deep models.

Regularization with discriminative deep models

Model regularization with scientific knowledge is widely seen in discriminative deep models. In the context of climate modeling, constraints derived from the conservation laws that a physical system should satisfy are encoded as a regularization term. In particular, a NN, $f$ maps input $x$ to output $y$ (i.e., $y=f(x)$). Conservation constraints $𝒞$ are summarized as a linear system (i.e., $𝒞=\{C\left[\begin{array}{c}x\\ y\end{array}\right]=0\}$), where $C$ is a given constraint matrix. These physical constraints are then encoded as a regularization term on the NN outputs:

$${ℒ}_{PHY}(x,y)=\Vert C\left[\begin{array}{c}x\\ y\end{array}\right]\Vert$$ (13)

Evaluation results show that by adding this physics-guided regularization, prediction performance is improved for emulating cloud processes [19, 20]. Similarly, Zhang et al. [21] proposed parameterizing atomic energy for molecular dynamics through a NN, whose loss function considers the conservation of kinetic and potential energy. Regularization for a physics system with non-conservative forces has also been considered [22]. In a double pendulum system with friction, the total energy of a system is decreasing due to the existence of friction. Decreasing of energy is formulated as a constraint , where $E({𝒙}_t)$ and $E({𝒙}_{t+1})$ denote the total energy of the system at current time step and a future time step, respectively. Considering a neural network which takes the state at current time step (i.e., ${𝒙}_t$) as input and outputs the estimated state at the next time step (i.e., $𝒚:={\widehat{𝒙}}_{t+1}$), the constraint can be integrated into the neural network through the regularization term as

$${ℒ}_{PHY}({𝒙}_t,𝒚)=\mathrm{𝚁𝚎𝙻𝚄}(E(𝒚)-E({𝒙}_t))$$ (14)

where $E({𝒙}_t)$ and $E(𝒚)$ computes the energy of the system at the current and next time step, respectively. According to the decreasing of energy, $E(𝒚)$ is expected to be smaller than $E({𝒙}_t)$, leading to the constraint above. Similarly, a set of common physical properties of a dynamic system is considered in [26], where each of these physical properties is represented as an equality or inequality constraint. These physics-informed constraints are then incorporated into deep models as regularization via an augmented Lagrangian method.

PDEs (as described in Eq. (1)) have been widely considered as constraints and are integrated into deep models as regularization terms. A physics-informed neural network (PINN) [27] is proposed for solving PDEs by leveraging NNs. A PINN learns solutions, $u$, by using both the observed data and PDEs whereby PDEs serve as inductive bias. Consider the viscous Burgers’ equation as an example [28]:

$$\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}=v\frac{{\partial }^{2}u}{\partial x^2}$$ (15)

In a PINN, a feed forward NN predicts a PDE solution, $\tilde{u}$, by taking positions $x$ and time $t$ as input. The objective function consists of a data loss term and an PDE residual,

$$ℒ={ℒ}_{data}+\lambda {ℒ}_{PDE}$$ (16)

where ${ℒ}_{data}$ measures the difference between predicted PDE solution $\tilde{u}$ and given $u$ at certain positions and time step (i.e., ${ℒ}_{data}={\sum }_{i=1}^{N_{data}}{(\tilde{u}(x_i,t_i)-u_i)}^2$). ${ℒ}_{PDE}$ measures the PDE residual of predicted solution $\tilde{u}$ at position and time step:

$${ℒ}_{PDE}={\sum _{j=1}^{N_{PDE}}{(\frac{\partial \tilde{u}}{\partial t}+u\frac{\partial \tilde{u}}{\partial x}-v\frac{{\partial }^2\tilde{u}}{\partial x^2})}^2|}_{(x_j,t_j)}$$ (17)

Partial derivatives are calculated through numerical estimator given predicted solution $\tilde{u}$. Data points ${\{(x_i,t_i)\}}_{i=1}^{N_{data}}$ for data loss and ${\{(x_j,t_j)\}}_{j=1}^{N_{PDE}}$ for PDE residual are collected separately. $\lambda$ is the coefficient for the regularization term. In PINN, PDEs are directly encoded as a regularization term by measuring the solution residuals to constrain the model parameters. In the context of human body pose estimation, physics mechanics is employed to ensure physically plausible estimations, where the Euler-Lagrange equation represented as an ODE is derived, and is encoded as a soft constraint for model regularization. By integrating the Euler-Lagrange equation into data-driven deep models, the estimated 3D body pose is ensured to be physically plausible [29].

Physics-guided functions derived from domain knowledge have been used for training deep models in a label-free manner. Stewart and Ermon [34] proposed a label-free supervision of NNs with physics equations. The goal of this paper was to supervise NNs by specifying constraints $g$ that should hold over the output space $f$ instead of using labels. The loss function then becomes

$${\widehat{f}}^{\ast }=\arg \underset{f\in F}{\min }\sum _{i=1}^{n}g(x_i,f(x_i))+R(f)$$ (18)

where $R(f)$ refers to an additional regularization term penalizing model complexity. One example provided in the paper is to track an object performing free fall. The training of a regression network is formulated as a structured prediction problem operating on a sequence of $N$ images (i.e., ${({ℛ}^{H\times W\times 3})}^N\to {ℛ}^N$). Newton’s second law of gravity (free fall motion) is represented as the algebraic equation and is directly incorporated into the loss function for training. In particular, for an object in free fall, its height at $i$-th time step with time interval $\mathrm{\Delta }t$ is computed as $y_i=y_0+v_0(i\mathrm{\Delta }t)+a{(i\mathrm{\Delta }t)}^2$, with $y_0$ and $v_0$ being initial height and velocity. $a=-9.8m\backslash s^2$ is the fixed acceleration for an object performing free fall. Any predicted trajectory $f(𝒙)$ hence should fit such parabola with fixed curvature. A loss is then defined measuring the fitting residual,

(19)

where

(20)

and a = [$a\mathrm{△}{t}^2,a{(2\mathrm{△}t)}^2,\mathrm{\dots },a{(N\mathrm{△}t)}^2$] with $a=-9.8m\backslash s^2$. Besides, the algorithmic supervisions derived based on well-established algorithms have been used for training a neural network such that direct supervisions from ground truth annotations are no longer required [100].

For all the related works discussed above, the relative importance between generic knowledge and data information is pre-defined by the design of training objective, and is not adjustable after training. However, the relative importance could vary with different inputs. For example, given an unseen input, data-based prediction is less reliable and knowledge is playing a more important role in the final prediction. Seo et al. [22] proposed a framework where the relative importance of generic knowledge compared to data information is adjusted by a controlling parameter $\alpha$. The controlling parameter $\alpha$ is assumed to be a random variable following a pre-defined distribution $p(\alpha )$. Two sets of latent representations ${𝒛}_r$ and ${𝒛}_d$ extracted from input data $𝒙$ correspond to generic knowledge and data information, respectively. The final latent features are obtained as $𝒛=\alpha {𝒛}_r\oplus (1-\alpha ){𝒛}_d$ and are used for generating final predictions $\widehat{𝒚}$. Two sets of losses ${ℒ}_{rules}$ and ${ℒ}_{data}$ are defined based on generic knowledge and annotations from downstream tasks, respectively. ${ℒ}_{rules}$, being a function of inputs $𝒙$ and outputs $𝒚$, measures the violation of the rules derived based on prior knowledge specific to a target downstream task (e.g., Eq. 14). The final training loss is computed as the expected loss over $p(\alpha )$, i.e.,

$$ℒ={𝔼}_{\alpha \sim p(\alpha )}[\alpha {ℒ}_{rule}+\rho (1-\alpha ){ℒ}_{data}]$$ (21)

where $\rho$ is a scale parameter to balance the units of two loss terms. Through the proposed framework, $\alpha$ denoting the relative importance becomes a variable during testing.

The physics-guided regularization terms discussed above are all explicitly defined over the output space of deep models. Physics-guided regularization terms can also be implicit (i.e., induced by a physics-based model embedded as one intermediate primitive of a neural network pipeline) [35, 36, 37]. For example, Wu et al. [36] constructed a system for understanding physical scenes without human annotations. The core of the system is a physical world representation that is first recovered by a perception module and then utilized by a simulation engine. The perception module is a deep neural network that is self-supervised without annotation. The simulation engine, consisting of a physics engine and graphics engine, is aimed at generating physics predictions.

Figure 6: Physical scene understanding without human annotations [36]. A physics engine and graphic engine are concatenated to a perception model. By aligning the rendered RGB images with observed ones, the perception model captures meaningful object features without annotations. The knowledge represented by the physics and graphic engines is encoded into the loss function implicitly.

The proposed pipeline is evaluated on a synthetic billiard table experiment, as shown in Figure 6. Given the features extracted by the perception model, the physics engine predicts the future physics states of a system by following motion equations, and the graphic engine renders RGB images given the predicted physics states. The perception model is learned by maximizing the likelihood of estimated RGB images given observed sequences.

Regularization with generative deep models

Besides discriminative models, knowledge has also been applied to deep generative models. To generate realistic results, different generative models have explored the usage of domain knowledge. For example, to efficiently employ a generative adversarial network (GAN) for generating solutions of PDE-governed complex systems, Wu et al. [44] proposed a statistically constrained GAN, where a statistical regularization term is derived measuring the distance between covariance structures of training samples and generated samples, respectively,

$${ℒ}_c(D,G)=ℒ(D,G)+\lambda d(\mathrm{\Sigma }(p_{data}),\mathrm{\Sigma }(p_G))$$ (22)

where $D$ and $G$ represent standard discriminator and generator, respectively, and $L(D,G)$ represent the standard training loss for GAN. $\mathrm{\Sigma }(p_{data})$ and $\mathrm{\Sigma }(p_G)$ represents the covariance structures of the distributions of training data and generated samples, respectively. $d(\cdot )$ represents a distance measurement, such as the Frobenius norm. $\lambda$ denotes the coefficient of the regularization term. Introducing the statistical constraints on the lower dimensional manifold (i.e., covariance structure) helps reduce the searching space for finding desired solutions in high dimension. As a result, not only is the amount of data reduced, but the training time needed to converge on solutions is shorten as well, as demonstrated in solving turbulence PDEs. In the application to computational materials science, Shah et al. [43] proposed a deep generative model named InvNet, through which synthetic structured samples satisfying desired physical properties are generated. InvNet is an extension of conventional GAN, whereby an additional invariance checker is introduced, along with a traditional generator and discriminator. The invariance checker is introduced as an intermediate primitive, based on which an implicit knowledge-guided regularization term is defined. An invariance loss is defined based on the proposed invariance checker, measuring the violation of invariance. Through the invariance loss, the generated samples are encouraged to satisfy certain invariances (such as motif invariance, planting a predefined motif in all synthetic images at a fixed location). Besides knowledge represented as PDEs, physical connectivity and stability have been considered for realistic 3D shape generation. Mezghanni et al. [42] proposed incorporating physical constraints into a deep generative model, whereby the physical constraints capture both the connectivity of 3D components and the physical stability of the 3D shape. Fully differentiable physical loss terms are then defined for integrating physics constraints into neural networks. Specifically, a neural stability predictor, implemented as a neural network classifier and pre-trained with simulated data, is proposed for enforcing physical stability constraints. For each synthetic 3D shape in the simulated data, its stability is labeled by the Bullet physics engine. The stability constraint is encoded via the pre-trained stability classifier and is integrated into the deep model via the stability loss.

Variational AutoEncoder (VAE) has also been explored with physics integrated for robust and interpretable generative modeling [51, 52]. In particular, physics knowledge represented as PDEs is integrated into VAEs. Latent variables of VAEs are subject to the constraints defined by PDEs. Furthermore, in [50], instead of assuming a full access to the complete expression of PDEs, only part of the PDE is assumed to be known with latent variables of VAE being partially grounded with physics meaning for the known part of PDEs. The rest of unknown PDEs are modelled in a data-driven manner.

Leveraging probabilistic framework, uncertainty quantification has been considered in physics-informed deep models [53, 54, 45, 55]. Zhu et al. [54] considered a physics-informed CNN solving PDEs with uncertainty quantification. The uncertainty is originated from the randomness of the physics parameters $𝝀$, which is denoted as random vector $𝒙=\{{𝝀}_1,\mathrm{\dots }.,{𝝀}_{n_s}\}$, where $n_s$ is the total number of possible physics parameter settings and can be very high. Correspondingly, the solution of PDE $u$ with respect to each possible physics parameter setting becomes $𝒚=\{u_1,u_2,\mathrm{\dots },u_{n_s}\}$. The task is to model $p_{\theta }(𝒚|𝒙)$ given a set of observations ${𝒟}_{input}={\{{𝒙}^{(i)}\}}_{i=1}^N$ with ${𝒙}^{(i)}\sim p(𝒙)$. $𝜽$ denote the neural network parameters to be learned. In the meantime, uncertainty of PDE solutions is modeled via the variance $Var[𝒚]$. To train the NN, instead of using labeled data, an energy-based model is defined based solely on the PDE and its boundary conditions, from which we obtain the reference density. In particular, the reference density $p_{\beta }(𝒚|𝒙)$ follows a Boltzmann-Gibbs distribution:

$$p_{\beta }(𝒚|𝒙)=\frac{\exp (-\beta E(𝒚\mathbf{,}𝒙))}{Z_{\beta }(𝒙)}$$ (23)

with $\beta$ being a learnable parameter of the constructed energy-based model. Energy function $E(𝒚\mathbf{,}𝒙)=V_{PDE}(𝒚,𝒙)+\lambda V_{boundary}(𝒚)$ measures the violation of the PDE and boundary conditions. $\lambda$ is a tunable hyper-parameter. Physics equations are encoded into the energy-based probabilistic model. In the end, the NN is trained by minimizing the KL divergence between the estimated distribution and the reference distribution:

$$\underset{\beta ,𝜽}{\min }{D}_{KL}(p(𝒙)p_{𝜽}(𝒚|𝒙)||p(𝒙)p_{\beta }(𝒚|𝒙))$$ (24)

Through the training process, physics equations are integrated into the NN. Along the same line, Yang and Perdikaris [53] simplified the objective function by considering the lower bound of Eq. (24). Following a similar idea, Karumuri et al. [55] employed a deep residual network (ResNet) for solving elliptic stochastic PDEs in a label-free manner. In particular, the physics-informed loss function is defined as the expectation of PDE residuals over the probability distribution of stochastic variables. All three works discussed above focus on time-independent physics systems without evolution over time. Geneva and Zabaras [45] extended the idea to a dynamic system, where an auto regressive network is employed for predicting future physics states given a history of system states.

IV Deep Learning with Experiential Knowledge

Besides scientific knowledge, experiential knowledge has been widely considered, as the major source of knowledge for neural-symbolic models. Experiential knowledge refers to well-known facts from everyday life, describing semantic properties of an object or semantic relationships among multiple objects. It is generally intuitive and is derived through long time observations or well-established studies. Unlike the scientific knowledge, experiential knowledge, though widely available, is descriptive and imprecise. Experiential knowledge containing semantic information can serve as a strong prior knowledge for predictive tasks in deep learning (e.g., regression or classification tasks), especially in the small-data regime where training data alone is insufficient in capturing relationships among variables [65].

IV-A Experiential Knowledge Identification

Depending on the application domain, experiential knowledge may manifest in two types: entity properties and entity relationships. Entity relationships reveal semantic relationships among entities. They may be inferred from daily facts on relationships between named entities or be derived from well-established studies or theories, e.g., anatomy. For example, human anatomy has been considered widely in computer vision for human body and facial behavior analysis [14, 110, 111]. For facial behavior analysis, facial anatomy knowledge may provide information on relationships among the facial muscles to produce natural facial expression. Similarly, body anatomy may provide information on the relationships among the body joints to produce stable and physically plausible body pose and movement. The semantic relationships can be directly given or indirectly inferred from the existing ones. For example, from the facts that Helen Mirren acted in the Ink Heart and Helen Mirren wins the Best Actress award, one can infer that The Ink Heart is nominated for an award. Inferred facts, however, are susceptible to errors. Entity properties capture the knowledge about properties of entities. They can refer to ontological information describing hierarchical relations of concepts in the world perceived by humans [112]. For example, Rain is made of water, and Sea is a synonym for ocean. Linguistic knowledge, a major source of experiential knowledge, is analyzed in [113, 114]. A large language model has been considered as inductive bias for an abstract textual reasoning task [115]. Linguistic knowledge such as textual explanations has been explored for language model refinement [116].

IV-B Representation of Experiential Knowledge

Figure 7: Taxonomy of experiential knowledge and its representations.

Representations of experiential knowledge vary with domains. In general, the representations for experiential knowledge include probabilistic dependencies, logic rules and knowledge graphs. These representations capture both relationships and properties of entities revealed by the experiential knowledge. We illustrate the taxonomy of experiential knowledge and its representations in Figure 7.

IV-B1 Probabilistic Dependencies

Due to inherent uncertainty, semantic relationships among objects are widely represented through probabilistic dependencies. States of objects are modelled in a probabilistic way, whereby relationships among objects are captured via probabilistic dependencies. Relationships can be further divided into positive correlations and negative correlations. Let’s take facial action units (AUs) as an example. According to FACS [117], AUs represent facial muscles and one facial muscle can control one or multiple AUs. A binary AU can be on if the corresponding muscle is activated. AU1 (inner brow raiser) and AU2 (outer brow raiser) usually occur together because they are controlled by the same muscle frontalis. The pair AU15 (lip corner depressor) and AU24 (lip pressor) is another example of positive correlation, which is due to the fact that their underlying controlling muscles (i.e., depressor anguli oris and orbicularis oris, respectively) always move together. If two variables $X$ and $Y$ are positively correlated (e.g., $X$=“AU1” and $Y$=“AU2”) with $X=\{$0, 1$\}$ and $Y=\{$0, 1$\}$, then we have

(25)

Considering negative correlation, AU12 (lip corner puller) and AU15 (lip corner depressor) cannot show up together as their corresponding muscles (i.e., Zgomaticus major and depressor anguli oris, respectively) are unlikely to be activated simultaneously. Negative correlation can be represented in a similar way. If two variables $X$ and $Y$ are negatively correlated (e.g., $X$=“AU12” and $Y$=“AU15”) with $X=\{$0, 1$\}$ and $Y=\{$0, 1$\}$, then we have

(26)

IV-B2 First-order and Propositional Logic

Logic can be categorized into first-order logic and propositional logic. First-order logic (FOL) [118] employs logic rules to infer new experiential knowledge from existing experiential knowledge; it has been employed as an inference method to derive different types of knowledge, e.g., textual explanations [116]. The formula of FOL is as follows:

$$R_1(𝒙)\wedge R_2(𝒙)\wedge \mathrm{\cdots }\wedge R_n(𝒙)\Rightarrow H(𝒙)$$ (27)

where $R_1(𝒙),R_2(𝒙),\mathrm{\cdots }{R}_n(𝒙)$ represent logic atoms. Each atom captures a known object property or relationship through a predicate. The atoms are combined through connective (e.g. conjunctive) operators to form the condition part of the logic rule. $H(𝒙)$ is the implied result or conclusion of the logic rule. It represents the new experiential knowledge derived from the logic rule. The conditional part and the conclusion part of the rule are connected through the implication operator. For example, we have

$$\text{Smokes}(x)\Rightarrow \text{Cough}(x),\forall x$$ (28)

$\text{Smokes}(x)$ is an atom, with Smokes being a predicate and $x$ as a logic variable. It captures the entity property knowledge that the person represented by variable $x$ smokes or not. $\text{Cough}(x)$ is the implied result or derived knowledge, which captures the knowledge that the person coughs or not. The rule states that if the conditional part, $\text{Smokes}(x)$ is true, then $\text{Cough}(x)$ is also true.

IV-B3 Knowledge Graph

The knowledge graph is another symbolic representation of experiential knowledge, that is used primarily to capture the semantic relationships among objects, whereby semantic knowledge is expressed in triple format: (subject, predicate, object). The number of such triples is usually huge. In a knowledge graph, these triples are organized as a graph containing nodes and edges. Nodes represent subjects or objects, such as animals or places, as well as named entities, such as a person named Mary Kelley. Edges represent the predicate and connect pairs of nodes and describe the relationship between them. Besides, edges can also represent properties of an entity with nodes representing the attributes. Taking the triplet (cat, attribute, paw) as an example, nodes are cat and claw, and the relationship is attribute. This triple states a fact: “The attribute of a cat is paw”. Edges can be directed or undirected, for example, the food chain relationship between animals or the social relationship between people. Knowledge graphs can encode a large amount of commonsense, rules, and domain knowledge that capture semantic relationships and properties about entities. A knowledge graph hence is an important basic resource for obtaining experiential knowledge. For example, experiential knowledge of the semantic meanings of objects can be organized in a knowledge graph for image classification [119]. The Miscrosoft Concept Graph [120] is another example of a knowledge graph, where vertices in the Miscrosoft Concept Graph [120] could represent food such as fruit, mammals such as dog and cat, or facilities such as bus and gas station. The edges indicate the relationships between concepts based on daily facts, such as a cat is a mammal, where is reflects relationship between cat and mammal.

IV-C Integration into Deep Models

To integrate experiential knowledge into deep models, existing methodologies cover four types of integration: data-level, architecture-level, training-level, and decision-level as shown in Figure 1. We introduce each group of the approaches in the following subsections.

IV-C1 Data-level Integration

Pseudo training data is usually considered to incorporate experiential knowledge represented as probabilistic dependencies, and is employed to augment existing training data. For example, Teshima and Sugiyama [65] proposed to incorporate the conditional independence relationships among variables into predictive modeling. A set of conditional independence relationships among variables are firstly extracted from prior knowledge. Training data is then augmented by generating synthetic data that satisfies the extracted conditional independence relationships. Besides augmenting training data, pseudo data can also be employed for constructing a prior knowledge model. For facial AU recognition, Li et al. [64] proposed to leverage pseudo data generated based on the knowledge for constructing a data-free prior model that captures the prior distribution of the target variables for downstream tasks. Constraints on parameters and variables are firstly derived from generic AU knowledge. Effective sampling methods are then proposed for generating the pseudo data satisfying the variable and parameter constraints. A Bayesian network is then learned from pseudo data, serving as the data-free prior model. Similar idea has been exploited in upper body pose estimation task [14]. Four types of constraints (i.e., connectivity constraint, body length constraint, kinesiology constraint and symmetry constraint) are firstly derived from human anatomic knowledge. Synthetic data is then generated given these constraints, based on which a prior probabilistic model is learned.

Besides, synthetic data can be generated from generic knowledge represented as Boolean rules [22]. For instance, the probability of $j$-th class (i.e., ${\widehat{𝒚}}_j$) is higher when $k$-th input feature is bigger than a constant $a$ (i.e., ${𝒙}_k>a$). To incorporate this Boolean rule into a deep model, Seo et al. [22] proposed to augment each of the training data point $𝒙$ with a paired perturbed one ${𝒙}_p=𝒙+\delta 𝒙$. $\delta 𝒙$ is a small positive value for perturbation. The regularization is then be defined as

If and ${𝒙}_{p,k}>a$, according to the prior knowledge, we should have . If the constraint is satisfied, we have $ℒ(𝒙,{𝒙}_p,{\widehat{𝒚}}_j,{\widehat{𝒚}}_{p,j})=0$. Otherwise, $ℒ(𝒙,{𝒙}_p,{\widehat{𝒚}}_j,{\widehat{𝒚}}_{p,j})>0$.

IV-C2 Architecture-level Integraion

Domain knowledge describing relationships between variables can be integrated into deep models through architecture design. We discuss the architecture-level integration methods for experiential knowledge represented as probabilistic dependencies, logic rules, and knowledge graphs, respectively.

Architecture design to incorporate probabilistic dependencies

One representative line of architecture-level integration is focused on the experiential knowledge represented as probabilistic dependencies, whereby a probabilistic model, constructed from the knowledge, serves as a prior model and is embedded as one layer of a neural network. Semantic relationships among variables can hence be incorporated into neural network in a probabilistic way. Usually, a probabilistic model is concatenated to the last layer of a neural network. A conditional random field (CRF) is usually employed, which takes hidden features from neural networks as input and outputs final predictions that satisfy the knowledge encoded in the CRF (e.g., among AUs [66]). In [67], a fully-connected CRF is concatenated to the last layer of a CNN to jointly perform facial landmark detection. By leveraging the fully-connected CNN-CRF, probabilistic predictions of facial landmark locations are obtained capturing the structural dependencies among landmark points. For scene graph generation [68], structured relationships among entities and relations are firstly captured via an energy-based probabilistic model. The energy-based probabilistic model takes the output of a typical scene graph generation model as input and refines it by minimizing the energy. Prior knowledge captured via a probabilistic model can also be leveraged to define the adjacency matrix of a graph convolutional network [69]. For facial action unit (AU) density estimation task, a Bayesian network is employed to capture the inherent dependencies among AUs. A probabilistic graph convolution is then proposed whereby its adjacency matrix is defined by the structure of the Bayesian network. In addition, a probabilistic model can also be introduced as a learnable intermediate layer of a neural network. A CausalVAE [74] was proposed whereby a causal layer is introduced to the latent space of a variational auto-encoder (VAE). The causal layer essentially describes a structural causal model (SCM). Through the causal layer, independent exogenous factors are transformed to causal endogenous factors for causal representation learning.

Architecture design to incorporate logic rules

Integration through the architecture of neural network is a conventional neural-symbolic approach to integrate symbolic logic rules into deep models. Logic rules are integrated into neural network architectures by introducing logic variables or parameters. Such an approach can be traced back to the 1990s, when knowledge-based artificial neural network (KBANN) [121] and connectionist inductive learning and logic programming (CILP) [122] methods were introduced. More recently, logical neural network (LNN) [123] is proposed where each neuron represents an element in a logic formula, which can either be a concept (e.g., cat) or a logical connective (e.g., AND, OR). These works, however, are focused on leveraging neural networks for differentiable and scalable logic reasoning.

Few works are proposed to improve deep models via customizing their architectures via logic rules. To leverage logic rules for improved deep model performance, logic rules are encoded into a Markov Logic Network (MLN), and the constructed MLN, serving as a prior model, is embedded into a neural network as an output layer for improved knowledge graph completion task [75]. Particularly, four types of logic rules are firstly identified for capturing the knowledge in a Knowledge graph: (1) composition rules: a predicate $R_k$ is composed of two predicates $R_i$ and $R_j$ if for any three variables $Y_1,Y_2,Y_3$ we have $R_i(Y_1,Y_2)\wedge R_j(Y_2,Y_2)\Rightarrow R_k(Y_1,Y_3)$; (2) Inverse rules: a predicate $R_i$ is an inverse of $R_j$ if for any two variables $Y_1$ and $Y_2$ we have $R_i(Y_1,Y_2)\Rightarrow R_j(Y_2,Y_1)$; (3) Symmetric rules: a predicate $R$ is symmetric if for any two variables $Y_1$ and $Y_2$ we have $R(Y_1,Y_2)\Rightarrow R(Y_2,Y_1)$; (4) Subset rules: a predicate $R_j$ is a subset of $R_i$ if for any two variables $Y_1$ and $Y_2$ we have $R_j(Y_1,Y_2)\Rightarrow R_i(Y_2,Y_1)$. Given a set of identified logic rules $l\in ℒ$, a Markov logic network (MLN) defines a joint distribution of target variables $y$ of a triple as

$$p(y)\propto \exp (\sum _{l\in ℒ}{w}_l\sum _y{\varphi }_l(y_l))$$ (30)

where ${\varphi }_l$ is the potential function and is computed given the observed triples. Such an MLN is then concatenated to the last layer of a deep model whereby the confidence scores $w_l$ are learned via the deep model given the observed triples. By introducing the MLN, the task of predicting the missing triplets is re-formulated as inferring the posterior distribution of unseen configurations following the encoded logic rules.

Architecture design to incorporate knowledge graphs

The knowledge graph can also be integrated into the architecture of neural networks as one layer. Liang et al. [78] proposed a graph convolution with symbolic reasoning. Prior knowledge from the knowledge graph is specifically integrated into the neural network through a proposed symbolic graph reasoning (SGR) layer, as shown in Figure 8.

Figure 8: Image is adopted from [78]. Nodes in the current convolutional layer are first mapped to symbolic nodes. Connectivities among the symbolic nodes are defined based on a knowledge graph. Through a graph reasoning based on the knowledge graph, evolved symbolic nodes are finally mapped to the nodes in the next convolutional layer.

To make the proposed SGR cooperate with a convolutional layer, local hidden features from the current convolutional layer are firstly transferred to features of corresponding symbolic nodes in SGR. Connectivities among the symbolic nodes in SGR are defined based on prior knowledge from the knowledge graph. Guided by prior knowledge, SGR then performs graph reasoning and updates features. Finally, the updated features are mapped onto local features in the next convolutional layer. To generate a medical report in an unsupervised manner in a cross visual domain and textual domain, a knowledge-driven encoder-decoder model is proposed to leverage knowledge graphs [79]. The knowledge graph is encoded into the knowledge-driven attention module within the encoder. Taking an image and a medical report as inputs, the encoder first obtains the image embedding and the report embedding through standard deep models, respectively. An attention mechanism is then introduced whereby the embeddings are the queries and a knowledge graph is used to define the lookup matrix. The learned representations from the attention module bridge the vision and the textual domain by leveraging the knowledge graph. During training, the proposed model is learned by minimizing reconstruction error between generated and observed medical reports in the textual domain. During testing, the knowledge-driven encoder-decoder model can generate medical reports from medical images by leveraging experiential knowledge in knowledge graphs that is applicable to both visual and textual domains.

CRF has been leveraged to capture the experiential knowledge in a knowledge graph and is integrated as one layer of a neural network. Luo et al. [80] proposed a context-aware zero-shot recognition (CA-ZSL) method. Prior inter-object relation is extracted from knowledge graphs and encoded using a conditional random field (CRF). For an image containing $N$ objects, an image region and a class assignment of each object are denoted as $B_i$ and $c_i$, respectively, with $i=1,2,\mathrm{\dots },N$. The CRF model is then defined as

(31)

where the unary potential $\theta (c_i|B_i)$ is estimated given extracted features of each object correspondingly. Pairwise potential $\varphi (c_i,c_j|B_i,B_j)$ is estimated using both the extracted features and the knowledge graph. Semantic relations extracted from the knowledge graph are encoded in the pairwise potential functions, where $\gamma$ is a tunable hyper-parameter. Neural networks are trained by maximizing the log-likelihood. During testing, labels of unseen objects are inferred in a context-aware manner through a maximum-a-posteriori (MAP) inference in the learned CRF model.

IV-C3 Training-level Integration

Experiential knowledge is treated as prior bias that guides the training of a deep model. Constraints are obtained based on knowledge and are integrated as regularization terms into a deep model [77, 124, 125, 112]. Regularization can be derived from probabilistic dependencies, logic rules, or knowledge graphs, and we discuss further in the paragraphs below.

Regularization with probabilistic dependencies

Probabilistic dependencies derived from knowledge about semantic relationships are commonly integrated into deep models through regularization. Srinivas Kancheti et al. [70] considered causal domain priors for regularizing neural networks during training, whereby the learned causal effects in NNs are enforced to match the prior knowledge on causal relationships through regularization. Consider a neural network $f$ with $d$ inputs and $C$ outputs, and for $j$th input, $\delta G^j$ is a $C\times d$ matrix containing prior causal knowledge (in terms of gradients). To enforce $f$ to be consistent with the prior knowledge, a regularization is defined

$$R=\frac{1}{N}\sum _{j=1}^N\max \{0,{\Vert {\nabla }_{j}f\odot M-\delta G^j\Vert }_1-ϵ\}$$ (32)

where $M$ is a $C\times d$ binary matrix indicating the availability of prior knowledge and $N$ is the total number of training samples. ${\nabla }_{j}f$ is the $C\times d$ Jacobian of $f$ w.r.t. $j$th input. $ϵ$ indicates an acceptable error margin, and $\odot$ is the element-wise product. Similarly, Rieger et al. [71] proposed to penalize model explanations that did not align with prior knowledge through an explanation loss. For AU detection task, probabilistic relationships among AUs are derived from facial anatomic knowledge. Each of these probabilistic relationships is formulated as a constraint. In [72], a loss function measuring the satisfaction of each of these constraints is correspondingly defined and is employed for learning an AU detector. Differently, Cui et al. [73] proposed to learn a Bayesian network (BN) to compactly capture a large set of constraints on AU relationships. The BN is then used to construct the expected cross-entropy loss to train a deep neural network for AU detection.

Regularization with logic rules

Logical knowledge is encoded as constraints for model regularization. Through regularization, a deep model is penalized if its output violates the constraints derived from logical rules. Xu et al. [76] proposed to combine the automatic reasoning technology of propositional logic with existing deep learning models. Propositional logic was encoded in the loss function through the proposed semantic loss. A sentence $\alpha$ in propositional logic is defined over variables $𝑿=\{X_1,\mathrm{\dots },X_n\}$. The sentence is the semantic constraint to be imposed on the outputs of a neural network. Suppose that $p$ is a vector of probabilities, where each element $p_i$ denotes the predicted probability of variable $X_i$ and corresponds to a single output of the neural net. The semantic loss $L^s(\alpha ,p)$ measures the violation of $\alpha$ given $p$ as

$$L^s(\alpha ,p)\propto -\log \sum _{𝒙\vDash \alpha }\prod _{i:𝒙\vDash X_i}{p}_i\prod _{i:𝒙\vDash \neg X_i}(1-p_i)$$ (33)

$𝒙\vDash \alpha$ represents that the state $x$ satisfies the sentence $\alpha$. The larger the probability of the state satisfying the sentence, the smaller the semantic loss. The proposed semantic loss bridges neural network regularization with logic reasoning. It is effective for different applications, such as classification and preference ranking.

For relation prediction task, a logic embedding network with semantic regularization (LENSR) was proposed [77], where a propositional logic is integrated into a relation detection model. For a given image, the probability distribution of the relation predicate is first estimated using a standard visual relation detection model. Another probability distribution of the relation predicate is then proposed based on the propositional logic formula pre-defined given the input image. Finally, a semantic regularization is defined to align these two probability distributions by minimizing their difference.

Regularization with knowledge graph

The knowledge graph, a graphical representation of experiential knowledge, has also been employed for model regularization. Fang et al. [81] proposed to extract semantic consistency constraints from knowledge graph which are then used as regularization terms. Particularly, a consistency score between a pair of object $O$ and subject $S$ is calculated through random walk with restart

$${𝒞}_{O,S}=\underset{T\to \mathrm{\infty }}{lim}p(Y_T=Y_S|Y_0=Y_O;\alpha )$$ (34)

where $𝒞\in R^{|O|\times |S|}$. $|O|$ and $|S|$ represent the total number of target objects and target subjects. $T$ is the total moving steps and $\alpha$ is the restart probability, meaning at each moving step, there is a probability of $\alpha$ to restart from the starting node instead of moving to one of the neighbors of node. The calculated matrix $𝒞$ is employed as constraints on semantic consistency and is used for regularizing the neural network for an object detection task. Similarly, Gu et al. [82] proposed extracting external knowledge from KGs and applying image reconstruction to improve scene graph generation, especially when the dataset is biased, or annotations are noisy or missing. Object relationships are retrieved from ConceptNet as the external domain knowledge and are applied to refine object features through an object-to-image generation branch. The object-to-image generation branch reconstructs images based on object features and prior relational knowledge about the objects. Semantically meaningful object features can be learned by minimizing reconstruction error.

IV-C4 Decision-level Integration

Predictions from deep models and from the prior knowledge can be directly combined through the joint top-down and bottom-up prediction strategy. Through integrating two sets of predictions, the final prediction can be more accurate and robust.

For open-domain knowledge-based visual question answering, Marino et al. [83] combined implicit and symbolic knowledge. Implicit knowledge refers to knowledge learned from data (e.g., raw texts). Symbolic knowledge refers to graph-based knowledge encoded in existing knowledge graphs (e.g., DBpedia [126] and ConceptNet [127]). The proposed KRISP model contains two sub-modules: implicit knowledge reasoning and explicit knowledge reasoning. Two sources of knowledge are then combined to generate the final output by using a late-fusion strategy. Through the late-fusion strategy, predictions from data and symbolic knowledge are directly combined, independent of the training of deep models.

Two sets of predictions can be combined in a probabilistic way by following Bayes’ rule. To obtain probabilistic knowledge-based predictions, a PGM model capturing prior knowledge is employed as a prior model and its prediction is obtained via a probabilistic inference. For AU recognition task, Li et al. [64] considered a top-down and bottom-up integration, where a Bayesian network learned from generic knowledge serves as the top-down model and a data-driven model serves as the bottom-up model. Predictions from two models are then combined using Bayes’ rule to produce the final predictions. Probabilistic knowledge-based predictions can also be directly defined based on the knowledge. For a knowledge graph completion task, Cui et al. [84] derived a prior distribution on relations based on type information. The prior distribution is then combined with bottom-up predictions from existing embedding-based models through Bayes’ rule for final prediction.

V Discussion and Future Direction

In this survey, we reviewed traditional and prevailing technologies for knowledge-augmented deep learning, including knowledge identification, knowledge representation, and integration. We divided knowledge into two categories: scientific knowledge and experiential knowledge. Within each category, we introduced knowledge identification, representations, and integration into deep learning. As we have discussed, much work has been done to improve deep learning with related prior knowledge to yield data-efficient, generalizable, and interpretable deep learning models. To help readers better understand and apply the KADL to their work, we provide a prescriptive tree based on the summary of existing works (Table I). The prescriptive tree as shown in Figure 9 serves as a recipe that includes different pathways to incorporate a particular type of prior knowledge into deep models. Each pathway consists of a specific knowledge type, a knowledge representation format, and a knowledge integration method, accompanied with the references to related works.

Figure 9: Prescriptive tree on knowledge-augmented deep learning

Despite these efforts, existing methods suffer from several shortcomings. In the following paragraphs, we discuss existing techniques and highlight promising directions to pursue in the future.

Diverse types of knowledge

Existing knowledge-augmented deep models explore different types of domain knowledge, including scientific knowledge and experiential knowledge. However, most of the knowledge explored are scientific knowledge in physics and symbolically represented experiential knowledge. Injecting well-established algorithmic knowledge into deep models has already began to attract researchers’ attention whereby deep models are trained through algorithmic supervisions instead of ground truth annotations [100]. Furthermore, existing approaches are usually limited to one specific type of knowledge. For a certain application task, both scientific knowledge and experiential knowledge could exists from multiple sources. Hence, diverse types of knowledge can be combined jointly for improved deep model performance.

Effective knowledge integration

Existing integration methods utilize synthetic data, model architecture design, regularization function, or prediction refinement. Among them, the majority of the integration methods are performed during training. Existing integration techniques hence heavily depend on the specific training procedure, through which deep models are trained by considering two sources of information jointly, without explicitly differentiating data from knowledge. The problem can be addressed through decision-level fusion. Decision-level knowledge integration scheme using a prior model to capture the domain knowledge attracts relatively less attention. Knowledge integration with a prior model to combine top-down prediction from the knowledge and bottom-up estimation from the data could be beneficial from several aspects. Firstly, the construction of a prior model is independent of deep models whereby deep models are initialized through observable data. Since the prior model and the deep model are built independent of the integration process, top-down and bottom-up integration process can be flexibly applied to any deep model and prior model. Secondly, knowledge integration is performed by following Bayes’ rule in a principled manner. Refinement of data-based prediction based on knowledge becomes tractable and interpretable.

Hybrid integration methods

Existing methods tend to integrate scientific knowledge and experiential knowledge separately. In addition, they tend to employ one specific approach to perform the knowledge integration. For certain application domains, both types of knowledge may exist at the same time. Hence, they should be integrated jointly to further improve the performance of the deep models. In addition, users always need to choose an integration approach. There is no universal integration scheme that applies to all types of knowledge, and it remains an open question on how to automatically integrate knowledge with data in an optimal way. Hence, given the complementary nature of different integration methods, it may be beneficial to simultaneously employ different integration methods to exploit their respective strengths.

Knowledge integration with uncertainty

Existing works have explored the encoding of experiential knowledge in a probabilistic way such as capturing the uncertain relationships using a probabilistic graphical model. But in general, existing knowledge integration methods are deterministic methods, ignoring the underlying knowledge uncertainty and their impacts on deep model learning and inference. Uncertainty not only exists with experiential knowledge but also with scientific knowledge. For example, in physics, uncertainty arises from random physics parameters or unknown physics parameters or incomplete observations. Existing works along this line aim to measure the quality of PDE solutions governing a physics system and thus are subject to specific domain assumptions. Probabilistic tools, such as probabilistic graphical models (PGMs), are powerful in capturing uncertainties for experiential knowledge. However, few works explored the usage of PGMs in modeling uncertainties in scientific laws [128, 129]. It remains an open question to deep learning community on how to effectively and systematically model uncertainties in scientific knowledge for real-world application tasks.

Zijun Cui received the B.S. degree from the Department of Physics, University of Science and Technology of China, Hefei, China, in 2015, and received the M.S. degree from the School of Engineering, Brown University, Rhode Island, USA, in 2017. She is currently pursuing the Ph.D. degree with the Rensselaer Polytechnic Institute, NY, USA. She has broad experiences with deep learning and probabilistic graphical models. Her current research interests include knowledge-augmented deep learning, learning and inference on probabilistic graphical models, and their applications to computer vision and natural language processing.

Tian Gao received the Ph.D and B.S degrees both from the Department of Electrical, Computer, and System Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA. He is currently a research staff member of IBM Research AI at T. J. Watson Research Center. His research focuses on machine learning and its application in computer vision and natural language processing. He has worked on probabilistic graphical models, knowledge extraction, causal discovery, and other aspects of machine learning. From 2010 to 2012, he was a National Science Foundation Triple Helix Program Fellow.

Kartik Talamadupula is a Senior Research Scientist and Research Manager at IBM Research AI. His background is in automated planning and sequential decision making. He has applied decision-making techniques to human-in-the-loop AI, data-driven dialog, and human-agent collaboration.

Qiang Ji received his Ph.D degree in Electrical Engineering from the University of Washington. He is currently a Professor with the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute (RPI). From 2009 to 2010, he served as a program director at the National Science Foundation (NSF), Arlington, VA, USA, where he managed NSF’s computer vision and machine learning programs. He also held teaching and research positions with the Beckman Institute at University of Illinois at Urbana-Champaign, Urbana, IL, USA; the Robotics Institute at Carnegie Mellon University, Pittsburgh, PA, USA; the Dept. of Computer Science at University of Nevada, Reno, Nevada, USA; and the Air Force Research Laboratory, Rome, NY, USA. Prof. Ji currently serves as the director of the Intelligent Systems Laboratory (ISL) at RPI. Prof. Ji’s research interests are in human-centered computer vision, probabilistic graphical models, probabilistic deep learning, and their applications in various fields. He has published over 300 papers in peer-reviewed journals and conferences, and has received multiple awards for his work. Prof. Ji is has served as an editor on several related IEEE and international journals and as a general chair, program chair, technical area chair, and program committee member for numerous international conferences/workshops. Prof. Ji is a fellow of the IEEE and the IAPR.