2.1 Problem

Physics-Informed Neural Networks (PINNs) offer a definite advantage over conventional neural networks by explicitly integrating known governing atypical or partial differential equations (ODEs/PDEs) of physical processes. The enforcement of those governing equations in PINNs relies on a set of points generally known as residual points. These points are strategically chosen inside the simulation domain, and the corresponding network outputs are substituted into the governing equations to guage the residuals. The residuals indicate the extent to which the network outputs align with the underlying physical processes, thereby serving as a vital physical loss term that guides the neural network training process.

It is obvious that the distribution of those residual points plays a pivotal role in influencing the accuracy and efficiency of PINNs during training. Nonetheless, the prevailing approach often involves easy uniform sampling, which leaves ample room for improvement.

Consequently, a pressing query arises: How can we optimize the distribution of residual points to reinforce the accuracy and training efficiency of PINNs?

2.2 Solution

Promising ways of distributing the residual points are by adopting the and the

1. The adaptive strategy implies that after every certain number of coaching iterations, a recent batch of residual points might be generated to switch the previous residual points;
2. The refinement strategy implies that extra residual points might be added to the prevailing ones, thus “refining” the residual points.

Based on those two foundational strategies, the paper proposed two novel sampling methods: Residual-based Adaptive Distribution (RAD) and Residual-based Adaptive Refinement with Distribution (RAR-D):

1. RAD: esidual-based daptive istribution

The important thing idea is to attract recent residual samples based on a customized probability density function over the spatial domain .The probability density function P() is designed such that it’s proportional to the PDE residual ε() at :

Here, k and c are two hyperparameters, and the expectation term within the denominator might be approximated by e.g., Monte Carlo integration.

In total, there are three hyperparameters for RAD approach: k, c, and the period of resampling N. Although the optimal hyperparameter values are problem-dependent, the suggested default values are 1, 1, and 2000.

2. RAR-D: esidual-baseddaptive efinement with istribution

Essentially, RAR-D adds the element of refinement on top of the proposed RAD approach: after certain training iterations, as an alternative of replacing entirely the old residual points with recent ones, RAR-D keeps the old residual points and draws recent residual points in accordance with the custom probability density function displayed above.

For RAR-D, the suggested default values for k and c are 2 and 0, respectively.

2.3 Why the answer might work

The important thing lies within the designed sampling probability density function: this density function tends to put more points in regions where the PDE residuals are large and fewer points in regions where the residuals are small. This strategic distribution of points enables a more detailed evaluation of the PDE in regions where the residuals are higher, potentially resulting in enhanced accuracy in PINN predictions. Moreover, the optimized distribution allows for more efficient use of computational resources, thus reducing the full variety of points required for accurate resolution of the governing PDE.

2.4 Benchmark

The paper benchmarked the performance of the 2 proposed approaches together with 8 other sampling strategies, when it comes to addressing forward and inverse problems. The considered physical equations include:

• Diffusion-reaction equation (inverse problem, calibrating response rate k(x))

• Korteweg-de Vries equation (inverse problem, calibrating λ₁ and λ₂)

The comparison studies yielded that:

1. RAD at all times performed the perfect, thus making it default strategy;
2. If computational cost is a priority, RAR-D could possibly be a powerful alternative, because it tends to offer adequate accuracy and it’s cheaper than RAD;
3. RAD & RAR-D are especially effective for classy PDEs;
4. The advantage of RAD & RAR-D shrinks if the simulated PDEs have smooth solutions.

2.5 Strength and Weakness

👍

• dynamically improves the distribution of residual points based on the PDE residuals during training;
• results in a rise in PINN accuracy;
• achieves comparable accuracy to existing methods with fewer residual points.

👎

• might be more computationally expensive than other non-adaptive uniform sampling methods. Nonetheless, that is the value to pay for a better accuracy;
• for PDEs with smooth solutions, e.g., diffusion equation, diffusion-reaction equation, some easy uniform sampling methods may produce sufficiently low errors, making the proposed solution potentially less suitable in those cases;
• introduced two recent hyperparameters k and c that should be tuned as their optimal values are problem-dependent.

2.6 Alternatives

Other approaches have been proposed prior to the present paper:

Amongst those methods, two of them heavily influenced the approaches proposed in the present paper:

1. Residual-based adaptive refinement (Lu et al.), which is a special case of the proposed RAR-D with a big value of k;
2. Importance sampling (Nabian et al.), which is a special case of RAD by setting k=1 and c=0.