Keywords

1 Introduction

Cooking is a fundamental activity in our daily lives. A good meal enriches our quality of life, and it promotes wellness as well as provides pleasure. It can act to ease family budgets. In some cases, a meal can have specific religious or cultural meanings. To address these multifaceted needs, various types of improvement to cooking and meal planning might be possible by designing systems with the support of information and communication technology.

Daily cooking activities include regular repetition of planning a menu, preparing food, and eating. For eating, the main target for improvement concerns health administration [1–6]. The challenge of these studies is to recognize a menu and estimate a user’s nutritional intake from a photo taken by the user’s mobile phone. To encourage good eating habits, Takeuchi et al. [7] designed an interactive tool on a social network service; users can share photos of their meals and receive remarks from their friends via this tool. This communication tool is designed to prompt users to make healthier menu choices, by secretly sorting positive remarks based on the healthiness of the meal.

Although there are a multitude of interesting studies concerning eating, it is difficult to introduce them all. In this study, we focus on recent attempts to design modern applications for assisting with menu planning and food preparation, as these are activities of interest for us.

There are many recipe portal sites, and they are accessible from all over the world, because menu planning is a task that is directly related to people’s purchasing behavior. In spite of the universal need of recipe sites, there are large cultural differences in recipes. As is clear from the fact that “Washoku” (Japanese cuisine) won world heritage status in 2013, cooking has deep cultural aspects. Religions, histories, climates, and industrial progress are all related to local foods. A typical cross-cultural problem is translation of recipes, which is one of the desired applications. Ingredients, culinary arts, cooking devices, and equipment are subject to translation as well as language. In this sense, cooking is a challenging target for cross-cultural computing.

This manuscript is organized as follows. Section 2 introduces the technical efforts of a recipe portal site “cookpad”Footnote 1 as a case study of a commercial assistive service for menu planning. Academic challenges for menu planning are overviewed in Sect. 3. Section 4 summarizes assistive systems for food preparation from the viewpoint of the relationship between recipe texts and human actions. Finally, Sect. 5 presents the conclusion.

2 Challenges in a Company

In this section, we describe challenges faced by cookpad, an Internet site comprising over 2.2 million recipes as of January 2016, making it one of the largest recipe sites in the world. Of the many challenges faced by the site, we specifically introduce those that are closely related to academic research.

Recipe Search. Like general recipe sites, cookpad provides users with a search box to help them find recipes efficiently. Because recipes in cookpad are written in Japanese, which is a language that does not delimit words by white-spaces, morphological analysis is necessary to recognize words in the recipes. The analyzer in cookpad uses a manually maintained dictionary consisting of a vast number of food-related words. In the recipe search, synonymous expressions are also recognized using a domain-specific synonym dictionary, which is also maintained manually.

Recipe Classification. To find recipes, it is not sufficient merely to provide a search box. In cookpad, recipes are automatically classified into various categories (e.g., meat dishes, seafood dishes, vegetable dishes) and users can limit their search results using this information. The classification is based on a machine learning method (using support vector machines) where tens of thousands of recipes are used as labeled data. To ensure service quality, the precision for each category is maintained at \(90\,\%\) or above so that users can find relevant recipes easily.

Content Selection. Search results consisting only of recipes may not always satisfy the user’s information need. According to users’ queries, major search engines show not only web pages but also other content (e.g., YouTube videos in Google search results). Likewise, the cookpad shows not only recipes but also various other content (e.g., tips, news, and videos related to the user’s queries). For each query, content is selected based on a multi-armed bandit algorithm to maximize its click-through rate.

Research Promotion. To promote food research, cookpad has made its recipes available [8] through the National Institute of Informatics (NII), a Japanese Research Institute with the goal of advancing informatics research. The 1.7 million recipes involved in this challenge are those that were uploaded to cookpad by the end of September 2014. This data collection was released in February 2015.Footnote 2 Any researcher in public institutions (e.g., university) can obtain access to the collection for research purposes and as of January 2016, 82 research groups at 56 universities had already done so.

3 Academic Challenges for a Smart Recipe Search

3.1 Cooking Recipe Recommendations Using Surrounding Information

One of the most stressful issues for homemakers is to decide the day’s menu [9]. Many homemakers browse the web to decide what they are going to cook each day. Although most existing recipe search systems request that the user submits a query, such as the name of the menu that they want or names of the ingredients that they want to use [10–12], people cannot always explain the property of the recipe that they are looking for, even if they have a preference for a recipe. To deal with these issues, we considered that what people want to eat must strongly depend on their daily circumstances and that a recipe cooked in a given situation must be preferred by someone who is in a similar situation. Therefore, we have proposed a method that recommends recipes not only according to a recipe’s properties but also according to the user’s situation.

On the web, there are many blog-type recipes that describe not only the recipe itself but also the reason why the recipe was selected. We analyzed 2,074 blog-type recipes which were randomly collected from “RECIPE BLOG” [13]. We found that 48.2 % blog-type recipes describe at least one reason for selecting recipes and that these reasons could be classified into 18 categories. We then devised an algorithm that extracts situations corresponding to the 18 categories from a user’s life log, such as his or her tweets, and recommends recipes that have similar reasons to the extracted ones. We evaluated the proposed method under three fictitious scenarios; in two of them, the user had vague requirements for recipes and they could only describe their situation as “it was so busy today.” We call it as “situational scenario.” The other had concrete requirements such as “I bought Pacific saury for a good price.” We call this a “procedural scenario.” We input the description of each scenario into the proposed system and obtained the top ten results. As a baseline method, we also made queries derived from each scenario and found ten recipes that contained the queries as a part of the recipes. Five examinees evaluated each recipe recommended by the two methods using a five-point scale. The proposed method obtained a higher score than the baseline method in the two situational scenarios, while the baseline method obtained a higher score in the procedural one. Therefore, the proposed method is useful when a user cannot make his or her requirements clear as mentioned above. For details, please refer to [14].

3.2 Recipe Comparison Using a Recipe Flow-Graph

As will be shown in the next section, we are constructing a method to translate the procedural text of a recipe into a flow-graph, which will represent the procedural workflow of a recipe, such as which cooking actions are performed to which ingredients, and which ingredients are mixed with each other. Although the flow-graph introduced in the next section can be described as a Directed Acyclic Graph (DAG), it can be simplified to an ordered labeled tree, as shown in Fig. 1, which we call a recipe tree, as we focus on the flow related to ingredients, tools and cooking actions. A recipe tree has significant advantages over a recipe flow-graph because it has multiple fast and effective graph-matching algorithms, including the editing distance calculation. Therefore, we have proposed a method to calculate similarities (distance) between two recipes by calculating the editing distance between their recipe trees.

Fig. 1.
figure 1

A recipe tree of one “NIKUJAGA” recipe.

Full size image

In general for unordered labeled trees, including recipe trees, the problem of computing the editing distance between two trees is difficult (more precisely, the problem is known to be NP-hard [15]). Therefore, we developed heuristic algorithms specialized for cooking recipe trees. The proposed method not only calculates the distance between two recipes but also derives the details of their differences. Figure 2 shows the mapping result between two “NIKUJAGA” recipes when one recipe is mapped onto the other, with minimal operation costs. According to the result, we can easily note differences such as “the timing of mixing meat differs between the two recipes” and the action “immerse the cut potato” in the left recipe is omitted in the right one. Moreover, we have proposed a method to find a common cooking flow of a specific dish by extracting the common structures of recipe trees that are translated from the top ten search results from the name of that specific dish. The details of these methods are provided in [16]. In a related study, Wang et al. [17] also proposed flow-graph representations of recipes. They defined the similarity between two recipes based on the number of common subgraphs shared by two recipes.

Fig. 2.
figure 2

Result of mapping one “Nikujaga” recipe to another.

Full size image

4 Assistive Applications for Food Preparation

Recipes are important information sources in food preparation. People prefer to eat a different menu everyday. Some external factors, such as childbirth or a sick family member, may force us to change our repertoire of dishes. Both a variety of recipes and comprehensive recipe presentation are important to enrich our food preparation experience.

Fig. 3.
figure 3

Translation of Recipe/Observation into an intermediate model and its applications.

Full size image

Figure 3 shows our image of the applications and agendas in this topic. This perspective is inspired by machine translation via an intermediate model; this is, once we map the recipe text and observed food preparation activities onto a semantic model, we can directly compare the text and activities. In this sense, the semantic model of the working process corresponds to an intermediate language model in the machine translation problem.

A typical application is recipe generation from observation, which is achieved by agendas C\(\rightarrow \)B in Fig. 3. There are many challenges to recognizing food preparation activities (agenda C) [18–27] and some to generating recipe texts (agenda B) [28].

Recipe translation is realized by A\(\rightarrow \)D\(\rightarrow \)B. Agenda D has not yet been studied in detail; a comparison in the structured recipe representation in Fig. 2 is the first step of D, i.e. this study is front-line research toward this goal. As mentioned in Sect. 3.2, the DAG (or tree) representation of a recipe is a natural structural organization of the cooking process, and some research teams have proposed methods to extract the structure from recipe texts (agenda A) [29–31]. Most of the challenges facing processes A and B are addressed in the Japanese language [16, 30, 31], and but little in other languages [29]. For recipe translation and localization, it is important to develop these techniques in multiple languages.

An example of a workflow structure provided by [30] is shown in Fig. 4 with vertex labels (Table 1) and edge labels (Table 2). This is the flow extracted from a recipe posted on cookpad. Because recipes on cookpad are user-generated, texts are informal and there are many difficulties: disparate notation, non-regularized manners of reference, and broken contexts of descriptions.

Fig. 4.
figure 4

An example of flowgraph.

Full size image

To extract the complete structure from informal recipes, it might be useful to complement the recipes with information derived from observation. One of our recent trials attempted to match recipe texts and cooking observations in the semantic model to obtain reliable and informative structural representations of the recipes. The number of matched pairs results in statistical information concerning human activities, i.e., the human tendency to cook a dish following a recipe. Such information should enhance performance in systems involving agendas A-E.

Table 1. Named entity tags.
Full size table
Table 2. Edge labels.
Full size table

While the above applications contribute to increasing the variety of recipes, there are a series of proposals to improve recipe presentation [18, 32–38]. Most of them [32–35, 37, 38] assume a simple recipe structure, which is a sequence of steps. A few other studies use the DAG or tree structure for a more intelligent recipe presentation [18, 36].

In [18], we have forecasted the next intended step of a user via human-object interaction on a cooking surface. A human interacts with objects to proceed with cooking tasks. Simultaneously, the human puts objects aside if they are in his or her way. The challenge was to identify the intended step even while such out-of-context interactions are observed with informative interactions. The proposed method estimates the progress of the cooking process from the history of the interactions together with the intended next step. The estimated progress narrows down the options for the next step. In the experiment, the proposed method achieved more than 70 % accuracy in its forecast. Because forecasting is generally a difficult task of pattern recognition, this is a remarkable score, nonetheless the accuracy should be enhanced by more sophisticated semantic models with statistical information.

5 Conclusions

In this study, we introduced challenges for assisting with everyday cooking activities, particularly menu planning and food preparation. There are many commercial websites that provide recipes. Accurate and easy-to-use recipe search and recommendation tools are important to increase the number of customers. We introduced challenges by cookpad as a practical example. An accurate and large corpus is critical to achieve a practical recipe search engine. Companies pay a lot of efforts to maintain their corpus and keep the system up-to-date.

We also introduced academic challenges for menu planning. There is a lot of room for academic researchers to enhance recipe recommendation. Food preferences are influenced by the user’s day; therefore, a highly customized recipe recommendation requires a daily context of the family members. From this perspective, both web mining and Life logging are within the scope of this application.

While there have been several assistive systems and related methods for food preparation, one of the most important challenges is to obtain a highly informative semantic model of the food preparation process. We believe that matching the recipe text and cooking observations will provide a breakthrough in this topic. Finally, more challenges in non-Japanese languages are waited for cross-cultural applications in everyday cooking.