3D object detection based on point clouds has important theoretical and application values. As an innova-tive algorithm, VoteNet takes raw point clouds as input and uses a Hough vote module to shift the points scattering on surfaces of potential objects towards their centers as vote points, then achieves efficient 3D object detection. However, in-depth analysis has revealed that in the proposal prediction stage, vote points from walls, grounds, and adjacent objects are inevitable noise that seriously affects the object detection performance. To solve this problem, we propose a two-step proposal prediction to explicitly denoise the negative vote points. First, step I predicts an initial 3D bounding box (3D-BBox) for each proposal. According to the initial 3D- BBox, we define a mask that identifies negative vote points. Then, step II uses the mask as a filter to eliminate the negative vote points for each proposal, thereby reducing the effect of the negative vote points in proposal prediction. Experiments show that our proposed optimization scheme can effectively improve the object detection performance, achieving a significant performance improvement on the public SUN RGB-D dataset, (58.2% in terms of mAP@0.25 and 33.7% in terms of mAP@0.5).