Breast cancer is one of the most prevalent diseases among female populations worldwide. Because the cause and prevention remain unknown, early detection is considered the only way to increase the survival rate. X-ray imaging is currently the most reliable technique for detecting abnormalities in the breast and it is highly recommended for middle-aged women (40- 60) who are at higher risk of developing the disease. However, the efficiency of X-ray is dependent on radiologists' expertise as image interpretation is challenging and can lead to false positives and false negatives. Image processing algorithms are highly useful for providing reliable diagnosis that can help radiologists make a fast, reliable medical interpretation. Several techniques for isolating tumors from digital mammograms have been proposed, including histogram-based methods, region-based algorithms, and edge detection approaches. However, these techniques are not effective, as they use fixed threshold levels to isolate suspicious areas from the non-uniform image backgrounds. In this paper, we propose an approach based on Invasive Weed optimization (IWO) and Smallest Univalue Segment Assimilating Nucleus (SUSAN). The IWO algorithm determines the optimal threshold for the extraction of the suspicious regions in mammograms. The selected threshold is then used for the detection of dense abnormalities using the SUSAN algorithm. The results show that this technique is more accurate in detecting suspicious areas in breasts, and particularly dense breasts, than the existing techniques.