We propose a novel platform FACTORY, which fully automates the comprehensive performance evaluation of ICT system components. Our architecture requires only two data elements: a parameter definition of an ICT system component and a user-friendly template of evaluation environments for evaluating the component. FACTORY automatically generates patterns of evaluations and executes the rest of the evaluation processes: designing and deploying of evaluation environments, generating and executing evaluation programs, and recording evaluation results. Evaluation experiments show that FACTORY can shorten the evaluation period and reduce workload compared with manual operation of engineers.