research-article

Public Access

UnrealCV: Virtual Worlds for Computer Vision

Authors:

Yizhou WangAuthors Info & Claims

MM '17: Proceedings of the 25th ACM international conference on Multimedia

Pages 1221 - 1224

https://doi.org/10.1145/3123266.3129396

Published: 19 October 2017 Publication History

PDF eReader

Abstract

UnrealCV is a project to help computer vision researchers build virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a plugin by providing (1) A set of UnrealCV commands to interact with the virtual world. (2) Communication between UE4 and an external program, such as Caffe. UnrealCV can be used in two ways. The first one is using a compiled game binary with UnrealCV embedded. This is as simple as running a game, no knowledge of Unreal Engine is required. The second is installing UnrealCV plugin to Unreal Engine 4 (UE4) and use the editor of UE4 to build a new virtual world. UnrealCV is an open-source software under the MIT license. Since the initial release in September 2016, it has gathered an active community of users, including students and researchers.

References

[1]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. (2016). arXiv:arXiv:1606.01540

Google Scholar

[2]

Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. 2012. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision. Springer, 611--625.

Digital Library

Google Scholar

[3]

Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. IJCAI International Joint Conference on Artificial Intelligence 2016-January (2016), 4246--4247.

Digital Library

Google Scholar

[4]

Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. 2016. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelligence and Games (CIG), 2016 IEEE Conference on. IEEE, 1--8.

Digital Library

Google Scholar

[5]

Adam Lerer, Sam Gross, and Rob Fergus. 2016. Learning Physical Intuition of Block Towers by Example. (2016). arXiv:1603.01312 http://arxiv.org/abs/1603. 01312

Google Scholar

[6]

Siyuan Qiao, Wei Shen, Weichao Qiu, Chenxi Liu, and Alan Yuille. 2017. ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond. arXiv preprint arXiv:1704.06752 (2017).

Google Scholar

[7]

Weichao Qiu and Alan Yuille. 2016. UnrealCV: Connecting Computer Vision to Unreal Engine. In Computer Vision--ECCV 2016 Workshops. Springer International Publishing, 909--916.

Crossref

Google Scholar

[8]

German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. 2016. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3234--3243.

Crossref

Google Scholar

[9]

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2017. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. (2017). arXiv:arXiv:1705.05065

Google Scholar

[10]

Yi Zhang, Weichao Qiu, Qi Chen, Xiaolin Hu, and Alan Yuille. 2016. UnrealStereo: A Synthetic Dataset for Analyzing Stereo Vision. arXiv preprint arXiv:1612.04647 (2016).

Google Scholar

[11]

Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. 2017. Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. In International Conference on Robotics and Automation (ICRA) .

Crossref

Google Scholar

Cited By

View all

Xu XZhang XLiu YWang HZhang XZhuang Y(2025)Semantic-Driven Informed Planning and 3D Reconstruction for the Quadrotor Unmanned Aerial VehicleIEEE Transactions on Vehicular Technology10.1109/TVT.2024.348922974:3(3843-3853)Online publication date: Mar-2025
https://doi.org/10.1109/TVT.2024.3489229
Yang FYamao SKusajima IMoteki AMasui SJiang S(2025)YOWO: You Only Walk Once to Jointly Map an Indoor Scene and Register Ceiling-Mounted CamerasIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.345327735:1(445-460)Online publication date: Jan-2025
https://doi.org/10.1109/TCSVT.2024.3453277
Ma LWang YZhong FZhu SWang YSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Fast peer adaptation with context-aware explorationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693452(33963-33982)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693452
Show More Cited By

Index Terms

UnrealCV: Virtual Worlds for Computer Vision
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
2. Software and its engineering
  1. Software notations and tools
    1. Software libraries and repositories

Recommendations

First-Person Shooter Game for Virtual Reality Headset with Advanced Multi-Agent Intelligent System
MM '16: Proceedings of the 24th ACM international conference on Multimedia

We present a multiplayer first-person shooter (FPS) game with advanced intelligent non-playable characters (NPC) under computer control. The game is specially adapted for playing in VR headset so the simulator sickness symptoms are significantly ...
What’s Inside Unreal Engine? - A Curious Gaze!
ISEC '21: Proceedings of the 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)

Game developers often use game engines to efficiently develop games and reduce effort by reusing predefined components. However, game engines are complex and difficult to comprehend to extend and integrate with other components. Unreal Engine is one of ...
Robo recall
SA '16: SIGGRAPH ASIA 2016 VR Showcase

Robo Recall is an action-packed virtual reality first-person shooter with gratifying gameplay and an in-depth scoring system. Explore immersive environments as you take on a variety of rogue robots, unlock an expanding arsenal of weapons and access all-...

Comments

Information & Contributors

Information

Published In

MM '17: Proceedings of the 25th ACM international conference on Multimedia

October 2017

2028 pages

ISBN:9781450349062

DOI:10.1145/3123266

General Chairs:
Qiong Liu
FXPAL, USA
,
Rainer Lienhart
Universität Augsburg, Germany
,
Haohong Wang
TCL America, USA
,
Program Chairs:
Sheng-Wei "Kuan-Ta" Chen
Academia Sinica, Taiwan
,
Susanne Boll
University of Oldenburg, Germany
,
Phoebe Chen
La Trobe University, Australia
,
Gerald Friedland
Lawrence Livermore National Lab, USA
,
Jia Li
Google, USA
,
Shuicheng Yan
Qihoo 360, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

MM '17

Sponsor:

SIGMM

MM '17: ACM Multimedia Conference

October 23 - 27, 2017

California, Mountain View, USA

Acceptance Rates

MM '17 Paper Acceptance Rate 189 of 684 submissions, 28%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

123
Total Citations
View Citations
2,800
Total Downloads

Downloads (Last 12 months)631
Downloads (Last 6 weeks)69

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Xu XZhang XLiu YWang HZhang XZhuang Y(2025)Semantic-Driven Informed Planning and 3D Reconstruction for the Quadrotor Unmanned Aerial VehicleIEEE Transactions on Vehicular Technology10.1109/TVT.2024.348922974:3(3843-3853)Online publication date: Mar-2025
https://doi.org/10.1109/TVT.2024.3489229
Yang FYamao SKusajima IMoteki AMasui SJiang S(2025)YOWO: You Only Walk Once to Jointly Map an Indoor Scene and Register Ceiling-Mounted CamerasIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.345327735:1(445-460)Online publication date: Jan-2025
https://doi.org/10.1109/TCSVT.2024.3453277
Ma LWang YZhong FZhu SWang YSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Fast peer adaptation with context-aware explorationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693452(33963-33982)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693452
Santos DMontalvão JAraujo CLebre UFerreira TFreire E(2024)Performance Assessment of Object Detection Models Trained with Synthetic Data: A Case Study on Electrical Equipment DetectionSensors10.3390/s2413421924:13(4219)Online publication date: 28-Jun-2024
https://doi.org/10.3390/s24134219
Zhou LWang RZhang L(2024)Accurate Robot Arm Attitude Estimation Based on Multi-View Images and Super-Resolution Keypoint Detection NetworksSensors10.3390/s2401030524:1(305)Online publication date: 4-Jan-2024
https://doi.org/10.3390/s24010305
Rüter JDurak UDauer J(2024)Investigating the Sim-to-Real Generalizability of Deep Learning Object Detection ModelsJournal of Imaging10.3390/jimaging1010025910:10(259)Online publication date: 18-Oct-2024
https://doi.org/10.3390/jimaging10100259
Dar ASengupta NArora C(2024)Assessing the Feasibility and Ethics of Economic Status Prediction using Deep Learning on Household ImagesACM Journal on Computing and Sustainable Societies10.1145/36751602:3(1-20)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3675160
Chen ZZhao JYang MZhou WLi H(2024)Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object TrackingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364836920:7(1-19)Online publication date: 21-Feb-2024
https://dl.acm.org/doi/10.1145/3648369
Dubovskoi ABaimuratov ISokolov M(2024)Informed Object Detection for Computer GamesPattern Recognition and Image Analysis10.1134/S105466182470070634:3(810-818)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1134/S1054661824700706
Wang HYang MZheng N(2024)G2-MonoDepth: A General Framework of Generalized Depth Inference From Monocular RGB+X DataIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.334646646:5(3753-3771)Online publication date: May-2024
https://doi.org/10.1109/TPAMI.2023.3346466
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

First-Person Shooter Game for Virtual Reality Headset with Advanced Multi-Agent Intelligent System

What’s Inside Unreal Engine? - A Curious Gaze!

Robo recall