2.1 Problem

Physics-Informed Neural Networks (PINNs) offer a definite advantage over conventional neural networks by explicitly integrating known governing abnormal or partial differential equations (ODEs/PDEs) of physical processes. The enforcement of those governing equations in PINNs relies on a set of points often called residual points. These points are strategically chosen throughout the simulation domain, and the corresponding network outputs are substituted into the governing equations to judge the residuals. The residuals indicate the extent to which the network outputs align with the underlying physical processes, thereby serving as an important physical loss term that guides the neural network training process.

It is clear that the distribution of those residual points plays a pivotal role in influencing the accuracy and efficiency of PINNs during training. Nevertheless, the prevailing approach often involves easy uniform sampling, which leaves ample room for improvement.

Consequently, a pressing query arises: How can we optimize the distribution of residual points to boost the accuracy and training efficiency of PINNs?

2.2 Solution

Promising ways of distributing the residual points are by adopting the adaptive strategy and the refinement strategy:

1. The adaptive strategy signifies that after every certain number of coaching iterations, a latest batch of residual points will be generated to interchange the previous residual points;
2. The refinement strategy signifies that extra residual points will be added to the prevailing ones, thus “refining” the residual points.

Based on those two foundational strategies, the paper proposed two novel sampling methods: Residual-based Adaptive Distribution (RAD) and Residual-based Adaptive Refinement with Distribution (RAR-D):

1. RAD: Residual-based Adaptive Distribution

The important thing idea is to attract latest residual samples based on a customized probability density function over the spatial domain x. The probability density function P(x) is designed such that it’s proportional to the PDE residual ε(x) at x:

Here, k and c are two hyperparameters, and the expectation term within the denominator will be approximated by e.g., Monte Carlo integration.

In total, there are three hyperparameters for RAD approach: k, c, and the period of resampling N. Although the optimal hyperparameter values are problem-dependent, the suggested default values are 1, 1, and 2000.

2. RAR-D: Residual-based Adaptive Refinement with Distribution

Essentially, RAR-D adds the element of refinement on top of the proposed RAD approach: after certain training iterations, as an alternative of replacing entirely the old residual points with latest ones, RAR-D keeps the old residual points and draws latest residual points in line with the custom probability density function displayed above.

For RAR-D, the suggested default values for k and c are 2 and 0, respectively.

2.3 Why the answer might work

The important thing lies within the designed sampling probability density function: this density function tends to position more points in regions where the PDE residuals are large and fewer points in regions where the residuals are small. This strategic distribution of points enables a more detailed evaluation of the PDE in regions where the residuals are higher, potentially resulting in enhanced accuracy in PINN predictions. Moreover, the optimized distribution allows for more efficient use of computational resources, thus reducing the full variety of points required for accurate resolution of the governing PDE.

2.4 Benchmark

The paper benchmarked the performance of the 2 proposed approaches together with 8 other sampling strategies, when it comes to addressing forward and inverse problems. The considered physical equations include:

• Diffusion-reaction equation (inverse problem, calibrating response rate k(x))

• Korteweg-de Vries equation (inverse problem, calibrating λ₁ and λ₂)

The comparison studies yielded that:

1. RAD all the time performed the very best, thus making it a very good default strategy;
2. If computational cost is a priority, RAR-D could possibly be a robust alternative, because it tends to supply adequate accuracy and it’s inexpensive than RAD;
3. RAD & RAR-D are especially effective for classy PDEs;
4. The advantage of RAD & RAR-D shrinks if the simulated PDEs have smooth solutions.

2.5 Strength and Weakness

👍Strength

• dynamically improves the distribution of residual points based on the PDE residuals during training;
• results in a rise in PINN accuracy;
• achieves comparable accuracy to existing methods with fewer residual points.

👎Weakness

• will be more computationally expensive than other non-adaptive uniform sampling methods. Nevertheless, that is the value to pay for the next accuracy;
• for PDEs with smooth solutions, e.g., diffusion equation, diffusion-reaction equation, some easy uniform sampling methods may produce sufficiently low errors, making the proposed solution potentially less suitable in those cases;
• introduced two latest hyperparameters k and c that should be tuned as their optimal values are problem-dependent.

2.6 Alternatives

Other approaches have been proposed prior to the present paper:

Amongst those methods, two of them heavily influenced the approaches proposed in the present paper:

1. Residual-based adaptive refinement (Lu et al.), which is a special case of the proposed RAR-D with a big value of k;
2. Importance sampling (Nabian et al.), which is a special case of RAD by setting k=1 and c=0.