demonstration

Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines

Authors:

Na Li,

Ping Yu,

Alex OlwalAuthors Info & Claims

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Article No.: 76, Pages 1 - 3

https://doi.org/10.1145/3586182.3615817

Published: 29 October 2023 Publication History

Get Access

Abstract

We demonstrate Visual Blocks for ML, a visual programming platform that facilitates rapid prototyping of ML-based multimedia applications. As the public version of Rapsai [3], we further integrated large language models and custom APIs into the platform. In this demonstration, we will showcase how to build interactive AI pipelines in a few drag-and-drops, how to perform interactive data augmentation, and how to integrate pipelines into Colabs. In addition, we demonstrate a wide range of community-contributed pipelines in Visual Blocks for ML, covering various aspects including interactive graphics, chains of large language models, computer vision, and multi-modal applications. Finally, we encourage students, designers, and ML practitioners to contribute ML pipelines through https://github.com/google/visualblocks/tree/main/pipelines to inspire creative use cases. Visual Blocks for ML is available at http://visualblocks.withgoogle.com.

Supplemental Material

ZIP File

Supplemental File

Download
164.55 MB

References

[1]

Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3334480.3382839

Digital Library

Google Scholar

[2]

John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories With Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1–19. https://doi.org/10.1145/3491102.3501819

Digital Library

Google Scholar

[3]

Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Liu, Ahmed Sabie, Sergio Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Olwal. 2023. Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI). ACM. https://doi.org/10.1145/3544548.3581338

Digital Library

Google Scholar

[4]

Michael Gleicher, Aditya Barve, Xinyi Yu, and Florian Heimerl. 2020. Boxer: Interactive Comparison of Classifier Results. Computer Graphics Forum (Jun. 2020). https://doi.org/10.1111/cgf.13972

Crossref

Google Scholar

[5]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. https://doi.org/10.1145/2939672.2939778

Digital Library

Google Scholar

[6]

Thilo Spinner, Udo Schlegel, Hanna Schafer, and Mennatallah El-Assady. 2019. ExplAIner: a Visual Analytics Framework for Interactive and Explainable Machine Learning. IEEE Transactions on Visualization and Computer Graphics (2019). https://doi.org/10.1109/TVCG.2019.2934629

Crossref

Google Scholar

[7]

Bingyuan Wu and Yongxiong Wang. 2022. Rich Global Feature Guided Network for Monocular Depth Estimation. SSRN Electronic Journal (2022). https://doi.org/10.2139/ssrn.4057946

Crossref

Google Scholar

[8]

Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie Cai. 2022. PromptChainer: Chaining Large Language Model Prompts Through Visual Programming. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. ACM. https://doi.org/10.1145/3491101.3519729

Digital Library

Google Scholar

[9]

Tongshuang Wu, Michael Terry, and Carrie Cai. 2022. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3491102.3517582

Digital Library

Google Scholar

Cited By

View all

Das Antar AMolaei SChen YLee MBanovic N(2024)VIME: Visual Interactive Model Explorer for Identifying Capabilities and Limitations of Machine Learning Models for Sequential Decision-MakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676323(1-21)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676323
Zhou ZJin JPhadnis VYuan XJiang JQian XZhou JHuang YXu ZZhang YWright KMayes JSherwood MLee JOlwal AKim DIyengar RLi NDu R(2024)Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual ProgrammingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648656(1-5)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3648656
Truss MSchmitt M(2024)Human-Centered AI Product Prototyping with No-Code AutoML: Conceptual Framework, Potentials and LimitationsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2425454(1-16)Online publication date: 26-Nov-2024
https://doi.org/10.1080/10447318.2024.2425454

Index Terms

Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Simulation types and techniques
      1. Visual analytics
2. Software and its engineering
  1. Software notations and tools
    1. Context specific languages
      1. Visual languages

Recommendations

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications through Visual Programming
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

In recent years, there has been a proliferation of multimedia applications that leverage machine learning (ML) for interactive experiences. Prototyping ML-based applications is, however, still challenging, given complex workflows that are not ideal for ...
Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Foundational multi-modal models have democratized AI access, yet the construction of complex, customizable machine learning pipelines by novice users remains a grand challenge. This paper demonstrates a visual programming system that allows novices to ...
Experiencing Rapid Prototyping of Machine Learning Based Multimedia Applications in Rapsai
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

We demonstrate Rapsai, a visual programming platform that aims to streamline the rapid and iterative development of end-to-end machine learning (ML)-based multimedia applications. Rapsai features a node-graph editor that enables interactive ...

Comments

Information & Contributors

Information

Published In

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

October 2023

424 pages

ISBN:9798400700965

DOI:10.1145/3586182

Editors:
Sean Follmer
Stanford University, USA
,
Jeff Han,
Jürgen Steimle
Saarland University, Germany
,
Nathalie Henry Riche
Microsoft Research, USA

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

UIST '23

Sponsor:

UIST '23: The 36th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 1, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 355 of 1,733 submissions, 20%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
172
Total Downloads

Downloads (Last 12 months)115
Downloads (Last 6 weeks)8

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Das Antar AMolaei SChen YLee MBanovic N(2024)VIME: Visual Interactive Model Explorer for Identifying Capabilities and Limitations of Machine Learning Models for Sequential Decision-MakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676323(1-21)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676323
Zhou ZJin JPhadnis VYuan XJiang JQian XZhou JHuang YXu ZZhang YWright KMayes JSherwood MLee JOlwal AKim DIyengar RLi NDu R(2024)Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual ProgrammingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648656(1-5)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3648656
Truss MSchmitt M(2024)Human-Centered AI Product Prototyping with No-Code AutoML: Conceptual Framework, Potentials and LimitationsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2425454(1-16)Online publication date: 26-Nov-2024
https://doi.org/10.1080/10447318.2024.2425454

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplemental Material

References

Cited By

Index Terms

Recommendations

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications through Visual Programming

Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming

Experiencing Rapid Prototyping of Machine Learning Based Multimedia Applications in Rapsai

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations