Rendezvous of multiple autonomous agents has been of active interest in the last decade. Considerable work has been done on this problem with constraints on sensing and shape of the environment. However, not much is known on rendezvous amidst obstacles. When obstacles are introduced into the setting, it becomes natural to explore strategies for rendezvous that optimize some parameter (such as distance, time etc.). Our objective in this paper is to compute a location (which we refer to as the Time Optimal Rendezvous Point (TORP)) that minimizes the time for rendezvous amidst obstacles. We discuss challenges in finding TORP and develop efficient algorithms to compute TORP for r agents moving amidst m static polygonal obstacles. We then extend the analysis to handle dynamic obstacles. Experimental results are presented to validate the theory.