The rapid increase in scale and complexity of malware attacks has made traditional signature-based defense approaches less effective due to the inability to detect new forms of malware. Therefore, there is a need for more advanced malware classification methods, which can identify both known and unknown malware efficiently enough, without using signatures. In this paper, we propose a new machine-learning technique for open-world malware classification, using hyperspheres for the succinct representation of different malware families. For each malware sample that needs to be classified, we calculate the probability for it to belong to each hypersphere, then assign the sample to the family having the hypersphere with the highest probability of containing the sample point. Results from experiments have demonstrated the effectiveness of our proposed method on malware datasets for personal computers.