skip to main content
10.1145/3623504.3623567acmconferencesArticle/Chapter ViewAbstractPublication PagessplashConference Proceedingsconference-collections
research-article

Branching Compositional Data Transformations in jq, Visually

Published: 19 October 2023 Publication History

Abstract

jq is a widely-used command-line tool for filtering and transforming JSON data, in the vein of sed and awk, including a bespoke programming language for writing the filters. The paradigm of that language is unusual: while its appearance is somewhere between a shell pipeline and JavaScript, the language is pure-functional and essentially concatenative, the pipelines branch and interleave data invisibly, implicit output flattening obscures all these effects, and most users are unaware of any of its semantics, leading to confusion when encountering any of these latent elements and difficulty in constructing appropriate non-trivial filters, all the while common debugging techniques are also obstructed by the paradigm. These confusions can be eliminated by visually demonstrating the recursively forking nature of evaluation on actual data, and allowing manipulations of the program or data to be reflected live, but no such tool exists for jq or any similar approaches. We present a visualisation of jq's execution model that makes the branching nature manifest, illustrating its effects on concrete values provided by the user, and editing affordances that allow manipulating the program with reference to real data.

References

[1]
G. P. Anders. 2023. ijq: Interactive jq git repository. https://sr.ht/ gpanders/ijq/
[2]
Jihad Boulos, Marcel Karam, Zeina Koteiche, and Hala Ollaic. 2006. XQueryViz: An XQuery Visualization Tool. In Proceedings of the 10th International Conference on Advances in Database Technology (EDBT’06). Springer-Verlag, Berlin, Heidelberg. 1155–1158. isbn:3540329609
[3]
Glen Chiacchieri. 2018. Flowsheets v2. https://github.com/Glench/Flowsheets-v2
[4]
Bryan W. C. Chung. 2013. Multimedia Programming with Pure Data. Packt Publishing. isbn:1782164642
[5]
fiatjaf. 2021. jiq: jid with jq. https://github.com/fiatjaf/jiq
[6]
Monica Figuera. 2017. ZenSheet Studio: A Spreadsheet-inspired Environment for Reactive Computing. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH Companion 2017). ACM, New York, NY, USA. 33–35. isbn:978-1-4503-5514-8 https://doi.org/10.1145/3135932.3135949
[7]
Muhammad Ali Gulzar, Matteo Interlandi, Seunghyun Yoo, Sai Deep Tetali, Tyson Condie, Todd Millstein, and Miryung Kim. 2016. BigDebug: Debugging Primitives for Interactive Big Data Processing in Spark. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16). Association for Computing Machinery, New York, NY, USA. 784–795. isbn:9781450339001 https://doi.org/10.1145/2884781.2884813
[8]
Daniel D Hils. 1992. Visual languages and computing survey: Data flow visual programming languages. Journal of Visual Languages and Computing, 3, 1 (1992), 69–101. issn:1045-926X https://doi.org/10.1016/1045-926X(92)90034-J
[9]
Michael Homer. 2022. Calling Cards: Concrete Visual End-User Programming. In Programming Experience Workshop. https://doi.org/10.1145/3532512.3535221
[10]
Michael Homer. 2022. Interleaved 2D Notation for Concatenative Programming. In ACM SIGPLAN International Workshop on Programming Abstractions and Interactive Notations, Tools, and Environments. https://doi.org/10.1145/3563836.3568722
[11]
Timothy Jones and Michael Homer. 2018. The Practice of a Compositional Functional Programming Language. In Asian Symposium on Programming Languages and Systems. https://doi.org/10.1007/978-3-030-02768-1_10
[12]
jq. 2023. jq is a lightweight and flexible command-line JSON processor. https://jqlang.github.io/jq/
[13]
Amy J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The State of the Art in End-user Software Engineering. ACM Comput. Surv., 43, 3 (2011), Article 21, April, 44 pages. issn:0360-0300 https://doi.org/10.1145/1922649.1922658
[14]
John McCarthy. 1959. A Basis for a Mathematical Theory of Computation. In Computer Programming and Formal Systems, P. Braffort and D. Hirschberg (Eds.) (Studies in Logic and the Foundations of Mathematics, Vol. 26). Elsevier, 33–70. issn:0049-237X https://doi.org/10.1016/S0049-237X(09)70099-0
[15]
Hisham H. Muhammad. 2017. Dataflow Semantics for End-User Programmable Applications. Ph. D. Dissertation. Pontifícia Universidade Católica do Rio de Janeiro. https://hisham.hm/thesis/thesis-hisham.pdf
[16]
Hisham H. Muhammad. 2019. Userland. http://www.userland.org/
[17]
Johnathan Robie, Michael Dyck, and Josh Spiegel. 2017. XQuery 3.1: An XML Query Language. W3C.
[18]
Advait Sarkar, Andy Gordon, Simon Peyton Jones, and Neil Toronto. 2018. Calculation View: multiple-representation editing in spreadsheets. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 85–93. https://doi.org/10.1109/VLHCC.2018.8506584
[19]
Paul Shen. 2021. natto website. https://natto.dev/
[20]
Steven L. Tanimoto. 1990. VIVA: A visual language for image processing. Journal of Visual Languages & Computing, 1, 2 (1990), 127–139. issn:1045-926X https://doi.org/10.1016/S1045-926X(05)80012-6

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
PAINT 2023: Proceedings of the 2nd ACM SIGPLAN International Workshop on Programming Abstractions and Interactive Notations, Tools, and Environments
October 2023
61 pages
ISBN:9798400703997
DOI:10.1145/3623504
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. dataflow programming
  2. jq
  3. object pipelines
  4. stream processing

Qualifiers

  • Research-article

Conference

PAINT '23
Sponsor:

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 63
    Total Downloads
  • Downloads (Last 12 months)27
  • Downloads (Last 6 weeks)2
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media