In this paper, a novel collective network of binary classifiers (CNBC) framework is presented for content-based audio classification. The topic has been studied in several publications before, but in many cases the number of different classification categories is quite limited and needed to be fixed a priori. We focus our efforts to increase both the classification accuracy and the number of classes, as well as to create a scalable network design, which allows introducing new audio classes incrementally. The approach is based on dividing a major classification problem into several networks of binary classifiers (NBCs), where each NBC adapts its internal topology according to the classification problem at hand, by using evolutionary Artificial Neural Networks (ANNs). In the current work, feed-forward ANNs, or the so-called Multilayer Perceptrons (MLPs), are evolved within an architecture space, where a stochastic optimization is applied to seek for the optimal classifier configuration and parameters. The performance evaluations of the proposed framework over an 8-class benchmark audio database demonstrate its scalability and notable potential, as classification error rates of less than 9% are achieved.