Skip to main content

Program Synthesis for Musicians: A Usability Testbed for Temporal Logic Specifications

  • Conference paper
  • First Online:
Programming Languages and Systems (APLAS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 13008))

Included in the following conference series:

Abstract

In recent years, program synthesis research has made significant progress in creating user-friendly tools for Programming by example (PBE) and Programming by demonstration (PBD) environments. However, program synthesis from logical specifications, such as reactive synthesis, still faces large challenges in widespread adoption. In order to bring reactive synthesis to a wider audience, more research is necessary to explore different interface options. We present The SynthSynthesizer, a music-based tool for designing and testing specification interfaces. The tool enables researchers to prototype different interfaces for reactive synthesis and run user studies on them. The tool is accessible to both researchers and users by running on a browser on top of a docker-containerized synthesis toolchain. We show sample implementations with the tool by creating dropdown interfaces, and by running a user study with 21 users.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://tslsynthesissynthesizer.com/tutorial.html.

References

  1. Baumeister, T., Finkbeiner, B., Torfah, H.: Explainable reactive synthesis. In: Automated Technology for Verification and Analysis (2020)

    Google Scholar 

  2. Behrmann, G., et al.: Uppaal 4.0 (2006)

    Google Scholar 

  3. Bloem, R., Jobstmann, B., Piterman, N., Pnueli, A., Saar, Y.: Synthesis of reactive (1) designs. J. Comput. Syst. Sci. 78, 911–938 (2012)

    Google Scholar 

  4. Church, A.: Application of recursive arithmetic to the problem of circuit synthesis. J. Symbol. Logic 28, 289–290 (1963)

    Google Scholar 

  5. Coblenz, M., et al.: User-centered programming language design: a course-based case study (2020)

    Google Scholar 

  6. Cope, D.: An expert system for computer-assisted composition. Comput. Music J. 11(4), 30–46 (1987)

    Google Scholar 

  7. Crichton, W.: Human-centric program synthesis. CoRR abs/1909.12281 (2019)

    Google Scholar 

  8. Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., Polikarpova, N.: Small-step live programming by example. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 614–626 (2020)

    Google Scholar 

  9. Finkbeiner, B., Heim, P., Passing, N.: Temporal stream logic modulo theories. CoRR abs/2104.14988 (2021)

    Google Scholar 

  10. Finkbeiner, B., Klein, F., Piskac, R., Santolucito, M.: Synthesizing functional reactive programs. In: International Symposium on Haskell (2019)

    Google Scholar 

  11. Finkbeiner, B., Klein, F., Piskac, R., Santolucito, M.: Temporal stream logic: synthesis beyond the bools. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 609–629. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_35

    Chapter  Google Scholar 

  12. Frid, E., Gomes, C., Jin, Z.: Music creation by example. In: CHI 2020. ACM (2020). https://doi.org/10.1145/3313831.3376514

  13. Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming languages. ACM Sigplan Notices (2011)

    Google Scholar 

  14. Gulwani, S., Mayer, M., Niksic, F., Piskac, R.: Strisynth: synthesis for live programming. In: International Conference on Software Engineering (2015)

    Google Scholar 

  15. Gulwani, S., Polozov, O., Singh, R., et al.: Program synthesis. Foundations and Trends®.Prog. Lang. 4, 1–119 (2017)

    Google Scholar 

  16. Hempel, B., Lubin, J., Chugh, R.: Sketch-n-sketch: Output-directed programming for SVG. In: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technolog (2019)

    Google Scholar 

  17. Huang, C.A., Koops, H.V., Newton-Rex, E., Dinculescu, M., Cai, C.J.: AI song contest: Human-AI co-creation in songwriting. CoRR abs/2010.05388 (2020)

    Google Scholar 

  18. Jacobs, S.: Extended AIGER format for synthesis. arXiv:1405.5793 (2014)

  19. Jacobs, S., et al.: The 4th reactive synthesis competition (SYNTCOMP 2017): Benchmarks, participants & results. In: SYNT@CAV (2017)

    Google Scholar 

  20. Jacobs, S., Klein, F., Schirmer, S.: A high-level ITI synthesis format: Tlsf v1. 1. Synthesis Workshop at CAV (2016)

    Google Scholar 

  21. Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: CHI (2011)

    Google Scholar 

  22. Lerner, S.: Projection boxes: On-the-fly reconfigurable visualization for live programming. In: CHI (2020)

    Google Scholar 

  23. Lubin, J., Collins, N., Omar, C., Chugh, R.: Program sketching with live bidirectional evaluation. In: ICFP (2020)

    Google Scholar 

  24. Maderbacher, B., Bloem, R.: Reactive synthesis modulo theories using abstraction refinement. arXiv preprint arXiv:2108.00090 (2021)

  25. Mayer, M., Kuncak, V.: Game programming by demonstration. In: Proceedings of the 2013 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (2013)

    Google Scholar 

  26. Meyer, P.J., Sickert, S., Luttenberger, M.: Strix: explicit reactive synthesis strikes back! In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 578–586. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_31

  27. Pnueli, A., Rosner, R.: On the synthesis of an asynchronous reactive module. In: International Colloquium on Automata, Languages, and Programming (1989)

    Google Scholar 

  28. Rogers, C.: Web audio API specification. World Wide Web Consortium (2021)

    Google Scholar 

  29. Ryzhyk, L., Walker, A.: Developing a practical reactive synthesis tool: experience and lessons learned. In: Workshop on Synthesis at CAV (2016)

    Google Scholar 

  30. Ryzhyk, L., et al.: User-guided device driver synthesis. In: OSDI (2014)

    Google Scholar 

  31. Santolucito, M.: Human-in-the-loop program synthesis for live coding. In: Proceedings of the 9th ACM SIGPLAN International Workshop on Functional Art, Music, Modelling, and Design (2021)

    Google Scholar 

  32. Santolucito, M., Goldman, D., Weseley, A., Piskac, R.: Programming by example: Efficient, but not “helpful”. In: PLATEAU@SPLASH (2018)

    Google Scholar 

  33. Schewe, S., Finkbeiner, B.: Bounded synthesis. In: International Symposium on Automated Technology for Verification and Analysis (2007)

    Google Scholar 

  34. Wang, C., Feng, Y., Bodik, R., Cheung, A., Dillig, I.: Visualization by example. In: Proceedings of the ACM on Programming Languages (POPL) (2019)

    Google Scholar 

  35. Wang, C., Feng, Y., Bodik, R., Dillig, I., Cheung, A., Ko, A.J.: Falx: synthesis-powered visualization authoring. In: CHI Conference on Human Factors in Computing Systems (2021)

    Google Scholar 

  36. Wilson, C., Kalliokoski, J.: Web midi API W3C, Working Draft (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Wonhyuk Choi .

Editor information

Editors and Affiliations

A Appendix

A Appendix

1.1 A.1 TSL\(_{\mu }\) and its Decidability

For our tool, we use the TSL fragment TS\(\text {L}_{\mu }\) that has no predicate term application on cell values. While our tool has many internal cell values – such as modulation frequencies or waveforms – predicate terms are only applied to fresh user inputs (i.e. which notes they pressed, the velocity of key press, etc.). This allows us to use the fragment TS\(\text {L}_{\mu }\), which is decidable, unlike the full syntax of TSL.

Here, we formalize the definition of TS\(\text {L}_{\mu }\) and prove the decidability of its synthesis problem.

Definition 1

(TSL\(_{\mu }\)). Let function terms \(\tau _{F} \) and update terms \(\tau _{U}\) be defined as in Sect. 3. Let predicate terms \(\tau _{P} \) be defined as follows:

$$\begin{aligned} \tau _{P}:= {{\texttt {p}}}(s_{i_0}, s_{i_1}, \cdots s_{i_j}) \end{aligned}$$

where \(s_{i_j}\) refers to input signals, and \({{\texttt {p}}}\) any predicate. Then, a TS\(\text {L}_{\mu }\) formula is defined by the following syntax:

Intuitively, this is a fragment of TSL where predicate terms are evaluated only on input signals, and not cells. In particular, synthesizing this fragment of TSL is decidable.

We now show that synthesis of this fragment of TSL is decidable by showing that every TS\(\text {L}_{\mu }\) formula can be reduced to an LTL formula.

Theorem 1

(TSL\(_{\mu }\)-LTL Equivalence). Every TS\(\text {L}_{\mu }\) formula can be transformed to an equivalent LTL formula in polynomial time.

Proof

In TSL synthesis, the environment player chooses the predicate terms \(\tau _{P} \) and the system player chooses the update terms \(\tau _{U}\). In TS\(\text {L}_{\mu }\), the environment inputs \(\tau _{P} \)’s are always fresh at each timestep, and their values do not depend on previous outputs \(\tau _{U}\) of the system player.

Now, we can use the translation procedure from TSL to LTL presented in [11]:

Finkbeiner et al. show the soundness of this procedure, that the realizability of \(\varphi _{LTL}\) implies the realizability of \(\varphi _{TSL}\). In the full syntax of TSL, this procedure may still produce \(\varphi _{LTL}\) that returns unrealizable even though \(\varphi _{TSL}\) is realizable since the procedure removes the semantic meanings of update terms. However, in TS\(\text {L}_{\mu }\), the environment inputs do not depend on the previous system outputs, and no semantic interpretation of update terms is necessary; it follows that an unrealizable \(\varphi _{LTL}\) always implies an unrealizable \(\varphi _{TSL}\) formula.

Table 1. Synthesis times for different grammars

Furthermore, this procedure is bounded in polynomial time with respect to the formula size. The first part of the equation partially reconstructs the semantic meaning of updates by ensuring that a signal is not update with multiple values at a time. This is bounded in the size of update terms, \({n \atopwithdelims ()2} \in \mathcal O(n^2)\). The second part of the equation simply transforms predicate terms to environment inputs and update terms to system outputs, and is in done in linear time, so the entire procedure is bounded in polynomial time.

Finally, we state the decidability as a corollary.

Corollary 1

(Decidability of TS\(\mathbf{L} _{\mu }\) Synthesis). The synthesis problem of TS\(\text {L}_{\mu }\) is decidable.

Proof

The syntheis problem of LTL is 2EXP-COMPLETE [27]. Therefore, it follows from Theorem  1 that the synthesis problem of TS\(\text {L}_{\mu }\) is also 2EXP-COMPLETE, and decidable.

1.2 A.2 Experimental Results

In order for users to interact with an interface, it is necessary that it synthesizes in a reasonable amount of time. Therefore, we decided to measure synthesis times of our TSL fragments by randomly generating 1,000 specifications using The SynthSynthesizer ’s random specification generator. The runtimes of random specifications is particularly relevant to our tool, as the interfaces for TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \) included a “generate random specification” button, allowing users to explore the specification design space without needing to have a goal in mind. The random specification generator chooses an option randomly from each dropdown menu in the UI, effectively doing a random search through the combinatorial space of all possible specifications in TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \). We did not run a experimental result on the TS\(\text {L}_{\mu }\) syntax as we did not include random generation of specifications for TS\(\text {L}_{\mu }\).

Synthesis was executed on a quad-core Intel Xeon processor (2.30 Ghz, 16 Gb RAM) running Ubuntu 64bit LTS 18.04. Timeout was defined as any synthesis request that took over 10 s. Average and median time exclude these timed out synthesis requests. The results are shown in Table 1.

Overall, we found that TS\(\text {L}_\alpha \) specifications synthesized much faster than TS\(\text {L}_\beta \) specifications, without any timeouts. This was an expected result, given the relative simplicity of TS\(\text {L}_\alpha \) ’s grammar compared to that of TS\(\text {L}_\beta \). However, we were surprised to find that only one TS\(\text {L}_\beta \) specification was unrealizable. After a careful investigation, we discovered that the additional complexity in the grammar more tightly constrained each specification. Since each specification made weaker requirements, the grammar had less probability to create mutually exclusive specifications.

Fig. 5.
figure 5

Synthesis times of 1000 random specifications

We visualize the distribution of the synthesis times in Fig. 5. TS\(\text {L}_\alpha \) synthesis times follow a quasi-Gaussian distribution, but even the longest-taking query completes in under 2.4 s. On the other hand, the distribution of TS\(\text {L}_\beta \) specifications skew right; the number of specifications decreases with increasing synthesis time. The majority of specifications synthesize quickly, with 68.5% specifications taking less than 10 s to synthesize. From our experimental results, we see a clear tradeoff between expressivity and synthesis times. TS\(\text {L}_\alpha \) has a limited grammar, but on average synthesis takes less than two seconds to complete. On the other hand, TS\(\text {L}_\beta \) uses a larger fragment of TSL and provides more expressivity to the user, but at the cost of timeout; 7.1% of specifications timed out, and on average took almost 10 times as longer to synthesize than TS\(\text {L}_\alpha \).

1.3 A.3 User Study Questions

In this section, we present the full set of questions for the comprehensive user study in Tables 2, 3, and 4. Note that Q5 is repeated in the table because the question is phrased slightly different for TS\(\text {L}_{\mu }\). The question is meant to ask about the intuitiveness of the structure of the specification interface. For TS\(\text {L}_\alpha \) and TS\(\text {L}_\beta \), the specification interface is structured around dropdown menus. For TS\(\text {L}_{\mu }\), the specification interface is structured around a text box.

Table 2. Please rate the TSL_[x] interface for creating and synthesizing specifications from 1 to 7 (7 is highest) on the following
Table 3. On a scale from 1 to 7, how much do you agree with the following statements about TSL_[x]
Table 4. Paragraph responses

1.4 A.4 User Study Results Visualizations

In this section, we present visualizations of the user study results. Figure 6a shows the user responses for each question for each separate interface. Figures 6b and 6c demonstrate the tradeoff between flexibility and ease-of-use of TS\(\text {L}_\alpha \), TS\(\text {L}_\beta \), and TS\(\text {L}_{\mu }\).

Fig. 6.
figure 6

User study average ratings

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Choi, W., Vazirani, M., Santolucito, M. (2021). Program Synthesis for Musicians: A Usability Testbed for Temporal Logic Specifications. In: Oh, H. (eds) Programming Languages and Systems. APLAS 2021. Lecture Notes in Computer Science(), vol 13008. Springer, Cham. https://doi.org/10.1007/978-3-030-89051-3_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89051-3_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89050-6

  • Online ISBN: 978-3-030-89051-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics