Skip to main content
SpringerLink
Log in

Parallel multichannel music source separation system

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

This paper presents a parallel low-latency multichannel source separation system designed to recover the original signals of the instruments that compound a multichannel music recording. Our approach is suitable for many applications based on interactive live broadcast classical music, where a latency of a few seconds can be assumed by online users. The obtained results show that it is possible to reach real time in the tested scenarios assuming a low-latency and combining multi-core architectures with parallel and high-performance techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. www.celemony.com.

  2. www.myoperaplayer.com.

  3. www.medici.tv.

References

  1. Alonso P, Cortina R, Rodríguez-Serrano FJ, Vera-Candeas P, Alonso-González M, Ranilla J (2017) Parallel online time warping for real-time audio-to-score alignment in multi-core systems. The Journal of Supercomputing 73(1):126–138. https://doi.org/10.1007/s11227-016-1647-5

    Article  Google Scholar 

  2. Alonso P, Vera-Candeas P, Cortina R, Ranilla J (2017) An efficient musical accompaniment parallel system for mobile devices. Journal of Supercomputing 73(1):343–353. https://doi.org/10.1007/s11227-016-1865-x

    Article  Google Scholar 

  3. Anderson E, Bai Z, Bischof C, Blackford S, Demmel J, Dongarra J, Du Croz J, Greenbaum A, Hammarling S, McKenney A, Sorensen D (1999) LAPACK users’ guide, 3rd edn. Society for Industrial and Applied Mathematics, Philadelphia

    Book  Google Scholar 

  4. Cabañas-Molero P, Cortina-Parajón R, Combarro EF, Alonso P, Bris-Peñalver FJ (2019) HReMAS: Hybrid real-time musical alignment system. The Journal of Supercomputing 75(3):1001–1013. https://doi.org/10.1007/s11227-018-2265-1

    Article  Google Scholar 

  5. Campbell DR, Palomaki KJ, Brown GJ (2005) A MATLAB simulation of “shoebox” room acoustics for use in teaching and research. Computing and Information Systems Journal 9(3):48

    Google Scholar 

  6. Canadas-Quesada F, Vera-Candeas P, Martinez-Munoz D, Ruiz-Reyes N, Carabias-Orti J, Cabanas-Molero P (2016) Constrained non-negative matrix factorization for score-informed piano music restoration. Digital Signal Processing 50:240–257. https://doi.org/10.1016/j.dsp.2016.01.004

    Article  Google Scholar 

  7. Carabias-Orti JJ, Cobos M, Vera-Candeas P, Rodríguez-Serrano FJ (2013) Nonnegative signal factorization with learnt instrument models for sound source separation in close-microphone recordings. EURASIP Journal on Advances in Signal Processing 2013(1):184. https://doi.org/10.1186/1687-6180-2013-184

    Article  Google Scholar 

  8. Carabias-Orti, J. J., Rodriguez-Serrano, F., Vera-Candeas, P., Ruiz-Reyes, N., & Canadas-Quesada, F. J. (2015). An audio to score alignment framework using spectral factorization and dynamic time warping. In ISMIR: Proceedings of the International Conference of Music Information Retrieval (pp. 742–748)

  9. Carabias-Orti JJ, Virtanen T, Vera-Candeas P, Ruiz-Reyes N, Canadas-Quesada FJ (2011) Musical instrument sound multi-excitation model for non-negative spectrogram factorization. IEEE Journal on Selected Topics in Signal Processing 5(6):1144–1158. https://doi.org/10.1109/JSTSP.2011.2159700

    Article  Google Scholar 

  10. Chordia, P., & Rae, A. (2009). Using source separation to improve tempo detection. In: Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009 (pp. 183–188)

  11. Dagum L, Menon R (1998) Openmp: An industry standard api for shared-memory programming. IEEE Computational Science and Engineering 5(1):46–55

    Article  Google Scholar 

  12. Dessein, A., Cont, A., & Lemaitre, G. (2010). Real-time polyphonic music transcription with non-negative matrix factorization and beta-divergence. In Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010, (pp. 489–494)

  13. Dessein, A., Cont, A., & Lemaitre, G. (2013). Real-time detection of overlapping sound events with non-negative matrix factorization. In Matrix information geometry, (pp. 341–371). Berlin: Springer. https://doi.org/10.1007/978-3-642-30232-9_14

  14. Disch, S., Ertel, C., Faller, C., Herre, J., Hilpert, J., Hoelzer, A., Kroon, P., Linzmeier, K., & Spenger, C. (2004). Spatial audio coding: Next-generation efficient and compatible coding of multi-channel audio. In: Audio engineering society convention 117. Audio engineering society

  15. Duan Z, Pardo B (2011) Soundprism: An online system for score-informed source separation of music audio. IEEE Journal of Selected Topics in Signal Processing 5(6):1205–1215. https://doi.org/10.1109/JSTSP.2011.2159701

    Article  Google Scholar 

  16. Duong NQK, Vincent E, Gribonval R (2010) Under-determined reverberant audio source separation using a full-rank spatial covariance model. IEEE Transactions on Audio, Speech, and Language Processing 18(7):1830–1840. https://doi.org/10.1109/TASL.2010.2050716

    Article  Google Scholar 

  17. Ewert, S., & Muller, M. (2011). Estimating note intensities in music recordings. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (pp. 385–388). IEEE. https://doi.org/10.1109/ICASSP.2011.5946421

  18. Frigo M (1999) A fast fourier transform compiler. ACM SIGPLAN Notices 10(1145/989393):989457

    Google Scholar 

  19. Goto, M. (2004). Development of the RWC music database. In Proceedings of the 18th International Congress on Acoustics (ICA 2004) (pp. 553–556)

  20. Goto M, Hashiguchi H, Nishimura T, Oka R (2002) RWC music database: Popular, classical and Jazz music databases. Ismir 2:287–288

    Google Scholar 

  21. Hennequin, R., David, B., & Badeau, R. (2011). Score informed audio source separation using a parametric model of non-negative spectrogram. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (Vol. 1, pp. 45–48). IEEE. https://doi.org/10.1109/ICASSP.2011.5946324

  22. Huang, P. S., Chen, S. D., Smaragdis, P., & Hasegawa-Johnson, M. (2012). Singing-voice separation from monaural recordings using robust principal component analysis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 57–60).

  23. Itoyama K, Goto M, Komatani K, Ogata T, Okuno HG (2008) Instrument equalizer for query-by-example retrieval: Improving sound source separation based on integrated harmonic and inharmonic models. Ismir. https://doi.org/10.1136/bmj.324.7341.827

    Article  Google Scholar 

  24. Li B, Liu X, Dinesh K, Duan Z, Sharma G (2019) Creating a multitrack classical music performance dataset for multimodal music analysis: challenges, insights, and applications. IEEE Transactions on Multimedia 21(2):522–535. https://doi.org/10.1109/TMM.2018.2856090

    Article  Google Scholar 

  25. Marxer, R. (2013). Audio source separation for music in low-latency and high-latency scenarios.

  26. Miron, M., Carabias, J. J., & Janer, J. (2015). Improving score-informed source separation for classical music through note refinement. In: Proceedings of the International Society for Music Information Retrieval (ISMIR) Conference (pp. 448–454).

  27. Miron M, Carabias-Orti JJ, Bosch JJ, Gómez E, Janer J (2016) Score-informed source separation for multichannel orchestral recordings. Journal of Electrical and Computer Engineering 2016:1–19. https://doi.org/10.1155/2016/8363507

    Article  Google Scholar 

  28. Munoz-Montoro AJ, Carabias-Orti JJ, Vera-Candeas P, Canadas-Quesada FJ, Ruiz-Reyes N (2019) Online/offline score informed music signal decomposition: Application to minus one. EURASIP Journal on Audio, Speech, and Music Processing 2019(1):23. https://doi.org/10.1186/s13636-019-0168-6

    Article  Google Scholar 

  29. Muñoz-Montoro AJ, Ranilla J, Vera-Candeas P, Combarro EF, Alonso-Jordá P (2018) Real-time soundprism. The Journal of Supercomputing. https://doi.org/10.1007/s11227-018-2703-0

    Article  Google Scholar 

  30. Pätynen J, Pulkki V, Lokki T (2008) Anechoic recording system for symphony orchestra. Acta Acustica United with Acustica 94(6):856–865. https://doi.org/10.3813/AAA.918104

    Article  Google Scholar 

  31. Rodríguez-Serrano FJ, Carabias-Orti JJ, Vera-Candeas P, Canadas-Quesada FJ, Ruiz-Reyes N (2014) Monophonic constrained non-negative sparse coding using instrument models for audio separation and transcription of monophonic source-based polyphonic mixtures. Multimedia Tools and Applications 72(1):925–949. https://doi.org/10.1007/s11042-013-1398-8

    Article  Google Scholar 

  32. Rodriguez-Serrano FJ, Duan Z, Vera-Candeas P, Pardo B, Carabias-Orti JJ (2015) Online score-informed source separation with adaptive instrument models. Journal of New Music Research 44(2):83–96. https://doi.org/10.1080/09298215.2014.989174

    Article  Google Scholar 

  33. Turetsky, R., & Ellis, D. (2003). Ground-truth transcriptions of real music from force-aligned MIDI syntheses. In: Proceedings of the 4th International Symposium on Music Information Retrieval (pp. 135–141). https://doi.org/10.7916/D8S472CZ.

  34. Vincent E, Araki S, Theis F, Nolte G, Bofill P, Sawada H, Ozerov A, Gowreesunker V, Lutter D, Duong NQ (2012) The signal separation evaluation campaign (2007–2010): Achievements and remaining challenges. Signal Processing 92(8):1928–1936. https://doi.org/10.1016/j.sigpro.2011.10.007

    Article  Google Scholar 

  35. Vincent E, Gribonval R, Fevotte C (2006) Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech and Language Processing 14(4):1462–1469. https://doi.org/10.1109/TSA.2005.858005

    Article  Google Scholar 

  36. Vincent, E., & Plumbley, M. (2005). A prototype system for object coding of musical audio. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (pp. 239–242). IEEE. https://doi.org/10.1109/ASPAA.2005.1540214.

  37. Viste, H., & Evangelista, G. (2001). Sound source separation: Preprocessing for hearing aids and structured audio coding. In: COST G-6 Conference on Digital Audio Effects (DAFX-01), (pp. 67–70).

  38. Woodruff, J., Pardo, B., & Dannenberg, R. (2006). Remixing stereo music with score-informed source separation. In Proceedings of the 7th International Society for Music Information Retrieval Conference (ISMIR).

Download references

Acknowledgements

This work has been supported by the Regional Ministry of the Principado de Asturias under Grants FC-GRUPIN-IDI/2018/000226, the Government of the Junta de Andalucía under Grants “PAIDI 2020. Convocatoria 2017 de Ayudas a Actividades de Transferencia de Conocimiento” with reference AT-6044, and the University of Jaén under the program “Acción 1. Apoyo a las estructuras de investigación de la Universidad de Jaén para incrementar su competitividad atendiendo a sus singularidades”.

Author information

Authors and Affiliations

  1. Department of Telecommunication Engineering, Universidad de Jaén, Linares, Spain

    A. J. Muñoz-Montoro, J. J. Carabias-Orti & F. J. Canadas-Quesada

  2. Department of Computer Science, Universidad de Oviedo, Oviedo, Spain

    D. Suarez-Dou & J. Ranilla

Authors
  1. A. J. Muñoz-Montoro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. D. Suarez-Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. J. Carabias-Orti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. F. J. Canadas-Quesada
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. J. Ranilla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. J. Muñoz-Montoro.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Muñoz-Montoro, A.J., Suarez-Dou, D., Carabias-Orti, J.J. et al. Parallel multichannel music source separation system. J Supercomput 77, 619–637 (2021). https://doi.org/10.1007/s11227-020-03282-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03282-2

Keywords

Search

Navigation