Solving Partial Differential Equations (PDEs) is very important in many areas. Since PDE solvers take very long time for numerous applications of interest, we need efficient parallel implementations. An attractive parallel computing platform that is widely used at present is the Graphics Processing Unit (GPU). In this paper we present an efficient technique that uses the red-black Gauss-Seidel method to solve PDEs. This technique allows the efficient use of the relatively larger register file available in each Streaming Multiprocessor (SM), as well as the shared memory. It also allows the communication between the threads of a block. We employ the red-black Gauss-Seidel method, in this paper, to solve the 2D steady state heat conduction problem on two different GPUs. An overall speedup of 484 relative to the CPU sequential implementation is achieved. A speedup of about 2.6 relative to Foster's GPU implementation on the same GPUs is also achieved.