The edge-based multi-agent visual SLAM is crucial for emerging mobile applications like search-and-rescue, inventory automation, and industrial inspection. It uses a central node to manage the global map and schedule tasks for agents. However, as the number of agents increases, the system faces scalability challenges due to operational overhead, such as data redundancy, bandwidth consumption, and localization errors. In this paper, we introduce SwarmMap, a framework designed to enhance the scalability of collaborative visual SLAM service in edge offloading settings. SwarmMap consists of three system modules: a change log-based server-client synchronization mechanism, a priority-aware task scheduler, and a lean global map representation. These modules work together to address the challenges of data explosion problems. SwarmMap is open-source and compatible with the robotic operating system (ROS). Existing visual SLAM applications could incorporate SwarmMap through SwarmAPI, a set of well-packaged APIs, to compose SwarmMap’s function modules to enhance their performance and capacity in multi-agent scenarios. Comprehensive evaluations and a three-month case study at one of the world’s largest oilfields demonstrate that SwarmMap can serve $2\times$ more agents (> 20 agents) than the state-of-the-arts with the same resource overhead, meanwhile maintaining an average trajectory error of $38cm$ , outperforming existing works by > 55%.