Many real-world phenomena involve multiple interacting systems governed by their own physical laws. Accurate simulation of such systems requires detailed modeling of the individual components and capturing the intricate interactions of these components. This work proposes Physics Informed Networks with Guided Supermasks (PINGS) to solve multiple Partial Differential Equations (PDE) with the same model without losing performance on any single problem, marking a confluence of Physics Informed Neural Networks (PINN) and continual learning algorithms. PINGS leverages the expressive power of over-parametrized networks to encapsulate the solutions of multiple PDEs within its subnetworks, which are derived from the original model by retaining only a subset of its weights. Unlike previous approaches for continual learning, PINGS is less susceptible to catastrophic forgetting because the bulk of its weights are untrained. Three versions of PINGS with different levels of fine-tuning are considered and evaluated on multiple heat problems and wave problems. With fine-tuning applied at the input and output layers, PINGS comes close in performance to using separate models for solving each problem while offering a computationally efficient alternative to the latter.