Abstract
In recent years, program synthesis research has made significant progress in creating user-friendly tools for Programming by example (PBE) and Programming by demonstration (PBD) environments. However, program synthesis from logical specifications, such as reactive synthesis, still faces large challenges in widespread adoption. In order to bring reactive synthesis to a wider audience, more research is necessary to explore different interface options. We present The SynthSynthesizer, a music-based tool for designing and testing specification interfaces. The tool enables researchers to prototype different interfaces for reactive synthesis and run user studies on them. The tool is accessible to both researchers and users by running on a browser on top of a docker-containerized synthesis toolchain. We show sample implementations with the tool by creating dropdown interfaces, and by running a user study with 21 users.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baumeister, T., Finkbeiner, B., Torfah, H.: Explainable reactive synthesis. In: Automated Technology for Verification and Analysis (2020)
Behrmann, G., et al.: Uppaal 4.0 (2006)
Bloem, R., Jobstmann, B., Piterman, N., Pnueli, A., Saar, Y.: Synthesis of reactive (1) designs. J. Comput. Syst. Sci. 78, 911–938 (2012)
Church, A.: Application of recursive arithmetic to the problem of circuit synthesis. J. Symbol. Logic 28, 289–290 (1963)
Coblenz, M., et al.: User-centered programming language design: a course-based case study (2020)
Cope, D.: An expert system for computer-assisted composition. Comput. Music J. 11(4), 30–46 (1987)
Crichton, W.: Human-centric program synthesis. CoRR abs/1909.12281 (2019)
Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., Polikarpova, N.: Small-step live programming by example. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 614–626 (2020)
Finkbeiner, B., Heim, P., Passing, N.: Temporal stream logic modulo theories. CoRR abs/2104.14988 (2021)
Finkbeiner, B., Klein, F., Piskac, R., Santolucito, M.: Synthesizing functional reactive programs. In: International Symposium on Haskell (2019)
Finkbeiner, B., Klein, F., Piskac, R., Santolucito, M.: Temporal stream logic: synthesis beyond the bools. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 609–629. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_35
Frid, E., Gomes, C., Jin, Z.: Music creation by example. In: CHI 2020. ACM (2020). https://doi.org/10.1145/3313831.3376514
Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming languages. ACM Sigplan Notices (2011)
Gulwani, S., Mayer, M., Niksic, F., Piskac, R.: Strisynth: synthesis for live programming. In: International Conference on Software Engineering (2015)
Gulwani, S., Polozov, O., Singh, R., et al.: Program synthesis. Foundations and Trends®.Prog. Lang. 4, 1–119 (2017)
Hempel, B., Lubin, J., Chugh, R.: Sketch-n-sketch: Output-directed programming for SVG. In: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technolog (2019)
Huang, C.A., Koops, H.V., Newton-Rex, E., Dinculescu, M., Cai, C.J.: AI song contest: Human-AI co-creation in songwriting. CoRR abs/2010.05388 (2020)
Jacobs, S.: Extended AIGER format for synthesis. arXiv:1405.5793 (2014)
Jacobs, S., et al.: The 4th reactive synthesis competition (SYNTCOMP 2017): Benchmarks, participants & results. In: SYNT@CAV (2017)
Jacobs, S., Klein, F., Schirmer, S.: A high-level ITI synthesis format: Tlsf v1. 1. Synthesis Workshop at CAV (2016)
Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: CHI (2011)
Lerner, S.: Projection boxes: On-the-fly reconfigurable visualization for live programming. In: CHI (2020)
Lubin, J., Collins, N., Omar, C., Chugh, R.: Program sketching with live bidirectional evaluation. In: ICFP (2020)
Maderbacher, B., Bloem, R.: Reactive synthesis modulo theories using abstraction refinement. arXiv preprint arXiv:2108.00090 (2021)
Mayer, M., Kuncak, V.: Game programming by demonstration. In: Proceedings of the 2013 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (2013)
Meyer, P.J., Sickert, S., Luttenberger, M.: Strix: explicit reactive synthesis strikes back! In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 578–586. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_31
Pnueli, A., Rosner, R.: On the synthesis of an asynchronous reactive module. In: International Colloquium on Automata, Languages, and Programming (1989)
Rogers, C.: Web audio API specification. World Wide Web Consortium (2021)
Ryzhyk, L., Walker, A.: Developing a practical reactive synthesis tool: experience and lessons learned. In: Workshop on Synthesis at CAV (2016)
Ryzhyk, L., et al.: User-guided device driver synthesis. In: OSDI (2014)
Santolucito, M.: Human-in-the-loop program synthesis for live coding. In: Proceedings of the 9th ACM SIGPLAN International Workshop on Functional Art, Music, Modelling, and Design (2021)
Santolucito, M., Goldman, D., Weseley, A., Piskac, R.: Programming by example: Efficient, but not “helpful”. In: PLATEAU@SPLASH (2018)
Schewe, S., Finkbeiner, B.: Bounded synthesis. In: International Symposium on Automated Technology for Verification and Analysis (2007)
Wang, C., Feng, Y., Bodik, R., Cheung, A., Dillig, I.: Visualization by example. In: Proceedings of the ACM on Programming Languages (POPL) (2019)
Wang, C., Feng, Y., Bodik, R., Dillig, I., Cheung, A., Ko, A.J.: Falx: synthesis-powered visualization authoring. In: CHI Conference on Human Factors in Computing Systems (2021)
Wilson, C., Kalliokoski, J.: Web midi API W3C, Working Draft (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Appendix
A Appendix
1.1 A.1 TSL\(_{\mu }\) and its Decidability
For our tool, we use the TSL fragment TS\(\text {L}_{\mu }\) that has no predicate term application on cell values. While our tool has many internal cell values – such as modulation frequencies or waveforms – predicate terms are only applied to fresh user inputs (i.e. which notes they pressed, the velocity of key press, etc.). This allows us to use the fragment TS\(\text {L}_{\mu }\), which is decidable, unlike the full syntax of TSL.
Here, we formalize the definition of TS\(\text {L}_{\mu }\) and prove the decidability of its synthesis problem.
Definition 1
(TSL\(_{\mu }\)). Let function terms \(\tau _{F} \) and update terms \(\tau _{U}\) be defined as in Sect. 3. Let predicate terms \(\tau _{P} \) be defined as follows:
where \(s_{i_j}\) refers to input signals, and \({{\texttt {p}}}\) any predicate. Then, a TS\(\text {L}_{\mu }\) formula is defined by the following syntax:
Intuitively, this is a fragment of TSL where predicate terms are evaluated only on input signals, and not cells. In particular, synthesizing this fragment of TSL is decidable.
We now show that synthesis of this fragment of TSL is decidable by showing that every TS\(\text {L}_{\mu }\) formula can be reduced to an LTL formula.
Theorem 1
(TSL\(_{\mu }\)-LTL Equivalence). Every TS\(\text {L}_{\mu }\) formula can be transformed to an equivalent LTL formula in polynomial time.
Proof
In TSL synthesis, the environment player chooses the predicate terms \(\tau _{P} \) and the system player chooses the update terms \(\tau _{U}\). In TS\(\text {L}_{\mu }\), the environment inputs \(\tau _{P} \)’s are always fresh at each timestep, and their values do not depend on previous outputs \(\tau _{U}\) of the system player.
Now, we can use the translation procedure from TSL to LTL presented in [11]:
Finkbeiner et al. show the soundness of this procedure, that the realizability of \(\varphi _{LTL}\) implies the realizability of \(\varphi _{TSL}\). In the full syntax of TSL, this procedure may still produce \(\varphi _{LTL}\) that returns unrealizable even though \(\varphi _{TSL}\) is realizable since the procedure removes the semantic meanings of update terms. However, in TS\(\text {L}_{\mu }\), the environment inputs do not depend on the previous system outputs, and no semantic interpretation of update terms is necessary; it follows that an unrealizable \(\varphi _{LTL}\) always implies an unrealizable \(\varphi _{TSL}\) formula.
Furthermore, this procedure is bounded in polynomial time with respect to the formula size. The first part of the equation partially reconstructs the semantic meaning of updates by ensuring that a signal is not update with multiple values at a time. This is bounded in the size of update terms, \({n \atopwithdelims ()2} \in \mathcal O(n^2)\). The second part of the equation simply transforms predicate terms to environment inputs and update terms to system outputs, and is in done in linear time, so the entire procedure is bounded in polynomial time.
Finally, we state the decidability as a corollary.
Corollary 1
(Decidability of TS\(\mathbf{L} _{\mu }\) Synthesis). The synthesis problem of TS\(\text {L}_{\mu }\) is decidable.
Proof
The syntheis problem of LTL is 2EXP-COMPLETE [27]. Therefore, it follows from Theorem 1 that the synthesis problem of TS\(\text {L}_{\mu }\) is also 2EXP-COMPLETE, and decidable.
1.2 A.2 Experimental Results
In order for users to interact with an interface, it is necessary that it synthesizes in a reasonable amount of time. Therefore, we decided to measure synthesis times of our TSL fragments by randomly generating 1,000 specifications using The SynthSynthesizer ’s random specification generator. The runtimes of random specifications is particularly relevant to our tool, as the interfaces for TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \) included a “generate random specification” button, allowing users to explore the specification design space without needing to have a goal in mind. The random specification generator chooses an option randomly from each dropdown menu in the UI, effectively doing a random search through the combinatorial space of all possible specifications in TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \). We did not run a experimental result on the TS\(\text {L}_{\mu }\) syntax as we did not include random generation of specifications for TS\(\text {L}_{\mu }\).
Synthesis was executed on a quad-core Intel Xeon processor (2.30 Ghz, 16 Gb RAM) running Ubuntu 64bit LTS 18.04. Timeout was defined as any synthesis request that took over 10 s. Average and median time exclude these timed out synthesis requests. The results are shown in Table 1.
Overall, we found that TS\(\text {L}_\alpha \) specifications synthesized much faster than TS\(\text {L}_\beta \) specifications, without any timeouts. This was an expected result, given the relative simplicity of TS\(\text {L}_\alpha \) ’s grammar compared to that of TS\(\text {L}_\beta \). However, we were surprised to find that only one TS\(\text {L}_\beta \) specification was unrealizable. After a careful investigation, we discovered that the additional complexity in the grammar more tightly constrained each specification. Since each specification made weaker requirements, the grammar had less probability to create mutually exclusive specifications.
We visualize the distribution of the synthesis times in Fig. 5. TS\(\text {L}_\alpha \) synthesis times follow a quasi-Gaussian distribution, but even the longest-taking query completes in under 2.4 s. On the other hand, the distribution of TS\(\text {L}_\beta \) specifications skew right; the number of specifications decreases with increasing synthesis time. The majority of specifications synthesize quickly, with 68.5% specifications taking less than 10 s to synthesize. From our experimental results, we see a clear tradeoff between expressivity and synthesis times. TS\(\text {L}_\alpha \) has a limited grammar, but on average synthesis takes less than two seconds to complete. On the other hand, TS\(\text {L}_\beta \) uses a larger fragment of TSL and provides more expressivity to the user, but at the cost of timeout; 7.1% of specifications timed out, and on average took almost 10 times as longer to synthesize than TS\(\text {L}_\alpha \).
1.3 A.3 User Study Questions
In this section, we present the full set of questions for the comprehensive user study in Tables 2, 3, and 4. Note that Q5 is repeated in the table because the question is phrased slightly different for TS\(\text {L}_{\mu }\). The question is meant to ask about the intuitiveness of the structure of the specification interface. For TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \), the specification interface is structured around dropdown menus. For TS\(\text {L}_{\mu }\), the specification interface is structured around a text box.
1.4 A.4 User Study Results Visualizations
In this section, we present visualizations of the user study results. Figure 6a shows the user responses for each question for each separate interface. Figures 6b and 6c demonstrate the tradeoff between flexibility and ease-of-use of TS\(\text {L}_\alpha \), TS\(\text {L}_\beta \), and TS\(\text {L}_{\mu }\).
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Choi, W., Vazirani, M., Santolucito, M. (2021). Program Synthesis for Musicians: A Usability Testbed for Temporal Logic Specifications. In: Oh, H. (eds) Programming Languages and Systems. APLAS 2021. Lecture Notes in Computer Science(), vol 13008. Springer, Cham. https://doi.org/10.1007/978-3-030-89051-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-89051-3_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89050-6
Online ISBN: 978-3-030-89051-3
eBook Packages: Computer ScienceComputer Science (R0)