We propose an efficient algorithm to directly optimize class average precision (AP) with a Maximal Figure-of-Merit (MFoM) learning scheme. AP is considered as a staircase function with respect to each individual sample score after rank ordering is applied to all samples. A combination of sigmoid functions is then used to approximate AP as a continuously differentiable function of the classified parameters used to compute the sample scores. Compared to pair-wise ranking comparisons, the computational complexity of the proposed MFoM-AP learning algorithm can be substantially reduced when estimating classifier parameters with a gradient descent algorithm. Experiments on the TRECVID 2005 high-level feature extraction task showed that the proposed algorithm can effectively improve the mean average precision (MAP) over 39 concepts from a baseline performance of 0.4039 with MFoM maximizing F1 to 0.4274 with MFoM-AP, while showing significant impromvements for 12 concepts as more than 10%.