This paper presents a special nonlinear switched Griffiths-Jim beamformer (SGJBF) structure. The main objective of this paper is to reduce the background noise from an acquired speech signal. The interference we considered here is non-stationary in nature and can arrive from a variety of potential sources; for example, competing talkers, radio, TV and so on. In this paper, we propose an adaptive Time Delay Neural Network (TDNN) based nonlinear noise canceller. The proposed structure consists of a three-layer feedforward network with partially connected layers to achieve real-time processing. The error backpropagation learning algorithm is used here to train the TDNN. This system is tested with different types of interference signals from the Noise-X database. A comparison analysis of the proposed structure and the traditional linear adaptive beamformer is presented here. The nonlinear approach investigated here show remarkable improvements over the previous linear based beamforming approach.