File Spooler and Copy System for Fast Data Transfer
Authors: 
Ionuţ Mihalache, Costin Grigoras, Mihai Carabas, Maria-Elena Mihailescu, Sergiu Weisz, Nicolae TapusAuthors Info & Claims
Ionuţ Mihalache, Costin Grigoras, Mihai Carabas, Maria-Elena Mihailescu, Sergiu Weisz, Nicolae TapusAuthors Info & ClaimsPages 63 - 68
Abstract
The ALICE (A Large Ion Collider Experiment) experiment at the CERN (European Organization for Nuclear Research) LHC (Large Hadron Collider) is preparing for the LHC Run3, beginning in 2021, with a detector and computing upgrade. On the computing side, a large, purpose-build computing farm (O2) consisting of CPU and GPU will process the data coming from the experimental setup at an average input rate of some 2TB/sec and output rate of 100GB/sec. The farm will consist of few hundred off-the-shelve servers, called Event Processing Nodes (EPN), collectively connected to a remote disk-based storage system. The EPNs will process data in near-real time during the ALICE detector operation with expected output rate to storage of ~100GB/sec. To avoid interruptions of processing due to network glitches or overload, we foresee to equip the EPNs with fast high-capacity SSDs for temporary data storage. The data stored on the SSDs must be transferred asynchronously to the remote storage element. The transfer operation is time-critical, as the SSDs will be able to hold at most a few hours of data accumulation. This paper presents a method for fast copying the files generated by the EPNs while ensuring no data loss and caching all the encountered errors.
References
[1]
University of Southern California Information Sciences Institute. Transmission control protocol. Online: https://tools.ietf.org/html/rfc793. Last accessed: 14 June 2020.
[2]
JAVA ORACLE. Class socket. Online: https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html. Last accessed: 14 June 2020.
[3]
IBM (International Business Machines). Why big data overwhelms enterprise networks. Online: https://www.ibm.com/cloud/smartpapers/aspera/ transfer-large-files/. Last accessed: 14 June 2020.
[4]
CERN (The European Organization for Nuclear Research). Fast data transfer - FDT. Online: http://monalisa.cern.ch/FDT/. Last accessed: 14 June 2020
[5]
FABRIZIO FURANO ANDREW HANUSHEVSKY ALVISE DORIGO, PETER ELMER. Xrootd - a highly scalable architecture for data access.
[6]
SLAC (Stanford Linear Accelerator Center) CERN (The European Organization for Nuclear Research). Xrootd. Online:https://xrootd.slac.stanford.edu/docs.html. Last accessed: 8 June 2020
[7]
G Adde AJ Peters, EA Sindrilaru. Eos as the present and future solution for data storage at CERN. 2015.
[8]
CERN (The European Organization for Nuclear Research). Eos service. Online: http://information-technology.web.cern.ch/services/eos-service. Last accessed: 8 June 2020.
[9]
CERN (The European Organization for Nuclear Research). Storage. Online: https://home.cern/science/computing/storage. Last accessed: 8 June 2020.
[10]
CentOS. About. Online:https://www.centos.org/about/. Last accessed: 16 June 2020.
[11]
CentOS. Yum. Online:https://wiki.centos.org/PackageManagement/Yum. Last accessed: 16 June 2020
[12]
CERN (The European Organization for Nuclear Research). Setup yum repository. Online:https://eos-docs.web.cern.ch/eos-docs/quickstart/setup_ repo.html#eosbase-setup-repos. Last accessed: 16 June 2020.
[13]
CERN (The European Organization for Nuclear Research). Rpm installation. Online:https://eos-docs.web.cern.ch/eosdocs/quickstart/install.html. Last accessed: 16 June 2020.
[14]
LiquidWeb, How to enable an epel repository. Online: https://www. liquidweb.com/kb/enable-epel-repository/. Last accessed: 16 June 2020
[15]
Jeffrey I. Schiller Jennifer G. Steiner, Clifford Neuman. Kerberos: An authentication service for open network systems.
[16]
R. Badlishah Ahmad Naemah Abdul Wahab Mohd Alif Hasmani Abd Ghani, Ong BiLynn. Grid authentication mechanisms survey
[17]
JAVA ORACLE. Interface executorservice. Online:https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html. Last accessed: 8 June 2020.
[18]
JAVA ORACLE. Class delayqueue. Online:https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/DelayQueue.html. Last accessed: 8 June 2020.
[19]
Google. Implementing exponential backoff. Online: https://cloud.google.com/iot/ docs/how-tos/exponential-backoff. Last accessed: 8 June 2020.
Index Terms
- File Spooler and Copy System for Fast Data Transfer
Recommendations
File Spooler and Copy System for Fast Data Transfer
ECBS 2021: 7th Conference on the Engineering of Computer Based SystemsCERN (derived from Conseil européen pour la recherche nucléaire) is a European research organization in nuclear physics, which has particle accelerators and other infrastructures necessary for high-energy physics research.
ALICE (A Large Ion Collider ...
A multiple-file write scheme for improving write performance of small files in Fast File System
Fast File System (FFS) stores files to disk in separate disk writes, each of which incurs a disk positioning (seek + rotation) limiting the write performance for small files. We propose a new scheme called co-writing to accelerate small file writes in ...
Comparing Grid Data Transfer Technologies in the Expand Parallel File System
PDP '08: Proceedings of the 16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008)Data management is one of the most important problems in grid environments. One important challenge facing grid computing is the design of a grid file system. The Global Grid Forum defines a Grid File System as a human-readable resource namespace for ...
Comments
Information & Contributors
Information
Published In
January 2020
109 pages
ISBN:9781450377324
DOI:10.1145/3423390
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
In-Cooperation
- University of Thessaly: University of Thessaly, Volos, Greece
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 November 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- IFA-MG
Conference
ICACS'20
ICACS'20: 2020 4th International Conference on Algorithms, Computing and Systems
January 6 - 8, 2020
Rabat, Morocco
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 21Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in