Keywords

1 Introduction

Recently, there is a great interest in using ontologies for video processing. Ontologies have been developed in the domain of multi-media with different goals as information retrieval and annotation. Based on the ontology, one may might develop multimedia information query applications for classifying vietnam traditional dances for instance. The use of ontologies is important to understanding, managing and querying videos by using high level semantic concepts and their relations in the context of the video analysis.

There is a large number of works that uses ontologies in videos/images processing. For instance, ontologies have been using for analysis of soccer videos [4], image retrieval system and video events detection [25], video-surveillance [17]. In [20] the authors provide an algorithm that allows unsupervised segmentation of image sequences, video indexing and retrieval using ontologies.

Besides, different works have been developed the use of ontologies for modeling the human movements (specifically dance movements) from video sequences. For instance, the authors in [9] propose an outline of collective dance consciousness into a dance ontology. The authors in [11] presented a knowledge-based system for describing and storing dances that takes advantage of the expressivity of description logics. In [12] the authors discussed key elements of a semantic dance move representation model based on rule and extractions of logical descriptions from existing Labanotation system [6]. Furthermore, in [10] the authors presented an approach for semantic annotation of movement in the videos based on ontology model and semantic concept classifiers. They used the BMN (Benesh Movement Notation) system for building Video Movement Ontology (VMO). On other hand, an ontology-based approach for browsing the heritage collection of Indian dance videos using Multimedia Web Ontology Language (MOWL) is presented in [19]. In [18] the authors proposed an ontology for learning scheme, which is a combination of standard techniques for multimedia analysis with knowledge drawn from conceptual meta data to learn a domain specific multimedia ontology for exploration of multimedia content. Similarly, in [13] the authors created an ontology transferring the semantics of Labanotation into OWL entities. In addition, authors in [6] evaluated the impact of the choreographic language agent and the digital dance archives on dance ontology.

In this paper, we extend our previous work in [23], which proposed an OWL ontology called “VDM” (Vietnamese Dance Movements), to define a taxonomy for the traditional Vietnamese dances taking into account the semantics of its art and its cultural anthropologists. The objective of this extension is to develop a searchable knowledge base that enables us to search for non-elementary movements in dances.

Our starting inputs are sequences segmentation of videos (produce of movements). These movements are represented as a dataset (assertions) using the terminology ontology “VDM” for the purpose to develop a knowledge base that describes traditional Vietnamese dances and enables us to search for specific movements in dance. However, this “VDM” ontology only contains a description of elementary movements (or poses).

This paper goes one step further and proposes a tool that supports the “VDM” ontology with classification rules. These rules are developed with the OWL complementary language SWRL (Semantic Web Rule Language) to entail movement phrases, which are basic movements with complete meaning. The SWRL [16] is a language for the Semantic Web that can be used to express rules as well as logic, combining OWL DL or OWL Lite with a subset of the Rule Markup Language.

The dataset ontology containing pose descriptions can be queried using the query language SQWRL (Semantic Query Web-enhanced Rule Language), which is an extension of SWRL to retrieve implicit dance knowledge. The description of these dances will allow us to express complex relations for inferring on the domain of human movements to extract implicit knowledge from explicit one. These complex relations will be described as SWRL [16] rules. These SWRL rules represent additional description for the dance OWL ontology, to entail implicit knowledge as movement classification.

We specify queries on dance ontologies using the language SQWRL [21], which is based on the SWRL rule language and uses SWRL’s strong semantic, where set of operators can be used to perform closure operations as failure, counting, and aggregation. Then, a SQWRL query can be specified to retrieve particular dance information using inference on the dataset ontology, which imports the “VDM” ontology and its classification SWRL rules.

The rest of paper is organized as follows. Sections 2 and 3 cares about the classification of elementary and non-elementary dance movements with illustrative examples, and Sect. 4 concludes the paper.

2 Classification of Elementary Dance Movements

2.1 Motivation

Traditional Vietnamese intangible cultural heritage values are always present in life. Each cultural heritage has its own nuance in each region, clearly showing its own needs and identity. Traditional festivals, performances, folk games, customs, habits, beautiful lifestyle of people have been actively researched, collected, restored and protected. However, due to the impact of the market mechanism and the integration process, many cultural and social organizations are being doped and gradually eroded. Even many folk songs, traditional dances, traditional musical instruments, folk festivals, some traditional handicrafts and daily means of transport are lost, the rest are also being under great competition pressure. The documents and materials mentioned in each field of storage are not much, the rest are scattered, no conditions for collecting and gathering. Artists and artisans who are professional are older.

Therefore, it is important to preserve and promote the intangible cultural heritages and make that heritage “live” in the midst of life, as its nature, contributing to preserving ethnicity’s cultural identity, bringing culture to become the spiritual foundation of society. Vietnam has a huge folk dance source of massive ethnic groups exploited to build a professional dance industry. Like other forms of intangible culture, folk dance is also being eroded and needs to be preserved and promoted.

In the context of classifying videos dances, ontologies could be used to annotate dance videos automatically. The constructing elements of the ontology and their relationships to construct the dance model may be based on the semantics of the Labanotation system [5], a widely-applied language that uses symbols, which are identified by concepts and relationships created with the language OWL to denote and reason on dance choreographies.

2.2 Mõ Dance

Mõ is a very popular folk musical instrument, used usually by a very special messenger in old villages, when go through the village for announcing an important notice to villagers. Mõ is also hang on buffalos so the buffalos do not get lost. Mõ is used as an instrument in dance because of its joyful sound. Mõ dance is often a playful dance with many dancers. Mõ dance, a combination of Mõ dance basic units (Inviting, Toe touching, Foot dragging, Rotation jumping, and Exchanging), are actually existing only in the Red river delta, performed in old village’s festivals. “Mõ dance under the moon” is a Mõ dance danced in moon cake festival by children. The dance having 20 steps is performed by 4 groups of dancers [24].

2.3 Brief Refresher: VDM Ontology

This section provides a brief refresher on the ontology concepts defined in [23]. First, We chose the representation and analysis of aspects of the folk Vietnam dance movement type MõFootnote 1, which expresses cultural knowledge, spiritual and daily life.

The terminological box (TBox) in our dance ontology is the concepts and roles which are defined beforehand. While, the assertional box (ABox) is each pose of body parts described by its positions. We use this dataset ontology to entail implicit knowledge from the explicit knowledge to answer queries on dance movements. The reasoning procedure is based on classification rules, which are developed using training datasets.

In order to express complex relationships and rules of inference for the realm of human movements we used OWL to represent Mõ dance ontology (we called by VDM ontology). This ontology development process follows the steps proposed in [14] and [15]. We apply body parts based on Labanotation [8] division in the dance analysis. Then, we represented formally in description logic the dance where the reasoner supports answering different queries on Vietnamese folk-dance.

Description Logics (DLs) [2, 3, 22] are a family of knowledge representation languages that can be used for representing and reasoning with ontologies. They have two components to build a DL knowledge base: a terminological base TBox, which represents conceptual knowledge by providing a vocabulary for a domain of interest together with axioms that describe semantic relationships between the vocabulary items [7]. An assertional base ABox containing facts (assertions) storing data about the concept and role names introduced in the TBox [26]. We chose DL-Lite family [1] as logic for lightweight ontologies to represent knowledge of vietnamese traditional dances. The choice of this important tractable family of DLs is based on its query answering, which is done in an efficient way.

We defined each basic unit as the smallest movement with a complete meaning MovementPhrase:

$$\begin{aligned} Dance \equiv \forall hasPhrase.MovementPhrase \end{aligned}$$

This is means that the (Dance) concept is related to movement phrase by the abstract role hasPhrase.

There are two main types of movements, corresponding to basic actions:

  • BodyMovement: makes the position of the whole body of dancer changed in space where the dancers move and change their positions on a stage. They are composed of movement phrases MovementPhrase, which include displacement (a point), translation (a line) and rotation (an arc).

  • BodyPartMovement: makes the position of different parts of dancer’s body changed. the movements primitives represented by the concept MovementPrimitive are quick movements, which change the position of body parts (hand movements, arm movements, feet movements, leg movements, head movements, and combined arm-leg movements). it is a movement between main dance poses and it considers the part of body as a local coordinate.

We consider that each part of human body has its right side and its left side. Mõ dance includes concepts, concepts properties and relationships between concepts to provide a formal description of a domain. For instance:

  • Movement: considered as the root of the movement aspect (related to body and body part) of the dance.

  • Region, Message and Clothing: represent regional information, dance content, and transmitted messages.

  • TimeOf, Orientation and Pose: represent the position and orientation of body or body part in every moment.

On other hand, there are certain types of relationships between dance concepts such as:

  • HasSegment(a, b): means that Mouvement(a) belongs to Segment(b)

  • PosDancer(a, b) means that Dancer(a) is at the Position(b)

  • HasMessage(a, b) means that Dancer(a) transmit the Message(b)

Since a dance pose is a particular position of the dancer’s body. We defined the following setting of each movement, where \(Pose_{I}\) is one of predefined poses:

$$\begin{aligned} PoseDance&\equiv \left\{ \begin{aligned}&A: DancerBody ,P: MovementPhrase, \\&B: MovementBodyPart, \\&(A, P) : HasPose, (P, time) : timeOf, (P, B) : Pose_{I} \end{aligned} \right\} \end{aligned}$$

We suggest in this example the motion description of legs for the left part of dancer body, which are represented in the following Figs. 1a, b and c as follows:

$$\begin{aligned} 1a&\equiv \left\{ \begin{aligned}&A : DancerBody , P: MovementPhrase,(A, P_1) : HasPose, \\&(P_1, 1) : timeOf, (P_1, Legs) : Orientation1Pose, \\&(P_1, RightLeg, LeftLeg) : BeforePose \end{aligned} \right\} \end{aligned}$$
$$\begin{aligned} 1b&\equiv \left\{ \begin{aligned}&A : DancerBody ,P: MovementPhrase,(A, P_2) : HasPose, \\&(P_2, 2) : timeOf, (P_2, LeftLeg) : StraightPose, \\&(P_2, RightLeg) : HalfBrokenPose \end{aligned} \right\} \\ 1c&\equiv \left\{ \begin{aligned}&A : DancerBody ,P: MovementPhrase,(A, P_3) : HasPose, \\&(P_3, 3) : timeOf, (P_3, LeftLeg) : StraightPose, \\&(P_3, RightLeg) : BrokenPose \end{aligned} \right\} \end{aligned}$$
Fig. 1.
figure 1

Different positions of the action Mõ Legs-Brokens

Full size image

3 Classification of Complex Dance Movements

Till now, we described the elementary dance movements in the Vietnamese dance Mõ. The aim of this section is to describe complex (non-elementary) movements which can be viewed as parallel combinations of different poses of basic movements (elementary movements) from body parts. A complex movement can be also a result of a movement achieved by a set of dancers. The concepts Group, Groupshape, and memberOf encode the informations of groups of dancers in a dance.

3.1 Example of Non-elementary Movement

In Vietnamese dance Mõ, a group of dancers can have a shape created by two relations:

  • LineRelation represents the sequences of the positions of the dancers that form a line or an arc. The descriptions given in Figs. 2a, b and c for the movement directions of the dancers form a straight line. These movements poses are represented by the individuals \(LP_1\), \(LP_2\) and \(LP_3\) from the concept MovementBodyPart, wich correspond to the image of Fig. 2. To give complete formal description, we assume that each dancer (A : DancerBody) makes the same basic movements. Thus, their DL descriptions are as follows.

    $$\begin{aligned} 2a&\equiv \left\{ \begin{aligned}&(A, LP_1) : hasPose, (LP_1, 0) : timeOf, (LP_1, A) : LeftSideCentre, \\&(LP_1, arms) : UpPose, (LP_1, Legs) : StraightPose \end{aligned} \right\} \\ 2b&\equiv \left\{ \begin{aligned}&(A, LP_2) : hasPose, (LP_2, 2) : timeOf, (LP_2, A) : BackCentre, \\&(LP_2, arms) : DownPose, (LP_2, Legs) : StraightPose \end{aligned} \right\} \\ 2c&\equiv \left\{ \begin{aligned}&(A, LP_3) : hasPose, (LP_3, 3) : timeOf, (LP_3, A) : FacingCentre, \\&(LP_3, arms) : HexaPose, (LP_3, Legs) : StraightPose \end{aligned} \right\} \end{aligned}$$
Fig. 2.
figure 2

Positions formed the Line Relation

Full size image

  • CircleRelation can represent the sequences of the positions of the dancers that form a circle. All these relations are declared individuals as shown in Figs. 3a, b, c and d. Logically, this relation is formulated by:

    $$\begin{aligned} 3a&\equiv \left\{ \! \begin{aligned}&(A, CP_1) : hasPose, (CP_1, 0) : timeOf, (CP_1, A) : Orientation1Pose, \\&(CP_1, arms) : StraightPose, (CP_1, Legs) : StraightPose \end{aligned} \right\} \\ 3b&\equiv \left\{ \! \begin{aligned}&(A, CP_2) : hasPose, (CP_2, 2) : timeOf, (CP_2, A) : Orientation1Pose, \\&(CP_2, arms) : HexaUpPose, (CP_2, Legs) : StraightPose \end{aligned} \right\} \\ 3c&\equiv \left\{ \! \begin{aligned}&(A, CP_3) : hasPose, (CP_3, 3) : timeOf, (CP_3, A) : Orientation1Pose, \\&(CP_3, arms) : HexaPose, (CP_3, Legs) : StraightPose,\\&(CP_3, Hands): OrthUpPose \end{aligned} \right\} \\ 3d&\equiv \left\{ \begin{aligned}&(A, CP_4) : hasPose, (CP_4, 4) : timeOf, (CP_4, A) : Orientation1Pose, \\&(CP_4, arms) : HexaDowenPose, (CP_4, Legs) : StraightPose \end{aligned} \right\} \end{aligned}$$
Fig. 3.
figure 3

Positions formed of the Circle Relation

Full size image

Now, according to the example Sect. 2.2 we present the parallel combination between the actions of the feet with the actions of the hands or arms in a complete corporal movement. These non-elementary movements execute in a repetitive way that is what we apply in the annotation of the movements.

$$\begin{aligned} a&\equiv \left\{ \begin{aligned}&(A, P_1) : hasPose, (P_1, 1) : timeOf, (P_1, Legs) : Orientation1Pose, \\&(P_1, RightLeg, LeftLeg) : BreakPose,(P_1, Arms): UpPose,(P_1, Hands): OpenPose \end{aligned} \right\} \\ b&\equiv \left\{ \begin{aligned}&(A, P_2) : hasPose, (P_2, 2) : timeOf, (P_2, LeftLeg) : StraightPose, \\&(P_2, RightLeg) : HalfBrokenPose,(P_2, Arms): DowenPose,(P_2, Hands): OpenPose \end{aligned} \right\} \\ c&\equiv \left\{ \begin{aligned}&(A, P_3) : hasPose, (P_3, 3) : timeOf, (P_3, LeftLeg) : StraightPose, \\&(P_3, RightLeg) : BrokenPose,(P_3, Arms): DowenPose,\\&(P_3, Hands, Lengs): UnderPose,(P_3, Hands): ClosePose \end{aligned} \right\} \end{aligned}$$

3.2 Rule-Based Classification of Non-elementary Movements

We use the ontology concepts defined in [23] to classify the movement poses (MovementPose), the movement primitives (MovementPrimitive), the movement phrases (MovementPhrase) and dance movements. For the classification, we propose a set of rules written using declared abstract roles (poseClass, primitiveClass, and phraseClass). This allows to associate a class with each movement pose, primitive or phrase, respectively. Using abstract roles is important for a compact representation of rules.

Rules are of the form of an implication between an antecedent (body) and consequent (head). The intended meaning can be read as: whenever the conditions specified in the antecedent hold, then the conditions specified in the consequent must also hold. Both the antecedent (body) and consequent (head) consist of zero or more atoms. Atoms in these rules can be of the form C(x), P(x, y), sameAs(x, y) or differentFrom(x, y), where C is an OWL concept (class), P is an OWL property, and x,y are either variables, OWL individuals or OWL data values. There are many built-in atoms.

An example of a SWRL rule used to classify the poses is described below (the rules below are presented with the premise-conclusion form).

$$\begin{aligned} \frac{ \begin{aligned}&DancerBody(?d)\ \wedge \ hasPose(?d, ?p)\ \wedge orientation8Pose(?p, LeftFoot)\ \wedge \\&orientation2Pose(?p, RightFoot)\ \wedge straightPose(?p, Legs) \end{aligned} }{ poseClass(?p, LegsPose1) } \end{aligned}$$

If a dancer body specified by the variable d has a movement pose p and this movement pose satisfies the description specified by the atoms in the rest of rule body then it is classified by the class specified by the second argument of the atom poseClass. The same thing applies for classification of arms poses.

A sequence of movement poses can create a predefined movement primitive. The consecutive legs poses will create for example, a primitive of legs movements, which is of class LegsPrimitive. The SWRL rule to create this primitive is below.

$$\begin{aligned} \frac{ \begin{aligned}&DancerBody(?d)\ \wedge \ hasPose(?d, ?p1)\ \wedge \ poseClass(?p1, LegsPose1)\ \wedge \\&hasPose(?d, ?p2)\ \wedge \ poseClass(?p2, LegsPose2)\ \wedge \ hasPose(?d, ?p3)\ \wedge \\&poseClass(?p3, LegsPose3)\ \wedge \ hasPose(?d, ?p4)\ \wedge \\&poseClass(?p4, LegsPose4)\ \wedge \ hasPose(?d, ?p5)\ \wedge \\&poseClass(?p5, LegsPose5)\ \wedge \ timeOf(?p1, ?t1)\ \wedge \ timeOf(?p2, ?t2)\ \wedge \\&timeOf(?p3, ?t3)\ \wedge \ timeOf(?p4, ?t4)\ \wedge \ timeOf(?p5, ?t5)\ \wedge \\&add(?t2, ?t1, 1)\ \wedge \ add(?t3, ?t2, 1)\ \wedge \ add(?t4, ?t3, 1)\ \wedge \\&add(?t5, ?t4, 1)\ \wedge \ makeOWLThing(?pr, ?d) \end{aligned} }{ \begin{aligned}&MovementPrimitive(?pr)\ \wedge \ primitiveClass(?pr, LegsPrimitive)\ \wedge \\&hasPrimitive(?d, ?pr)\ \wedge \ hasPose(?pr, ?p1)\ \wedge \ hasPose(?pr, ?p2)\ \wedge \\&hasPose(?pr, ?p3)\ \wedge \ hasPose(?pr, ?p4)\ \wedge \ hasPose(?pr, ?p5)\ \wedge \\&beginTime(?pr, ?t1)\ \wedge \ endTime(?pr, ?t5) \end{aligned} } \end{aligned}$$

Queries on dance dataset ontologies are specified with the language SQWRL [21], which is based on the SWRL rule language and uses SWRL’s strong semantic. SQWRL takes a standard SWRL rule antecedent and treats it as a pattern specification for a query. It replaces the rule consequent with a retrieval specification. For example, the query:

$$\begin{aligned} Query&(pose, time, image)\ \leftarrow \ phraseClass(phrase, MoFootDragging)\ \wedge \\&hasPrimitive(phrase, primitive)\ \wedge \ hasPose(primitive, pose)\ \wedge \\&timeOf(pose, time)\ \wedge \ video(pose, image) \end{aligned}$$

retrieves the poses, their times and their video sequences (images) of the basic movement Mõ foot-dragging of the Vietnamese dance Mõ and it can be specified using the query language SQWRL as

$$\begin{aligned} Query :\,&phraseClass(?phrase, MoFootDragging)\ \wedge \\&hasPrimitive(?phrase, ?primitive)\ \wedge \ hasPose(?primitive, ?pose)\ \wedge \\&timeOf(?pose, ?time) \wedge video(?pose, ?image)\ \rightarrow \\&sqwrl\!:\!select(?pose, ?time, ?image)\ \wedge \ sqwrl\!:\!orderBy(?time) \end{aligned}$$

where video is OWL datatype relating a movement pose to a string representing a file. The left hand side of a SQWRL query operates like a standard SWRL rule antecedent with its associated semantics. The atoms in the SQWRL will not match only all direct OWL individuals in the ontology, but will match also individuals that are entailed by the ontology to be OWL individuals. The core SQWRL operator \(sqwrl\!:\!select(?pose, ?time, ?image)\) builds a table using its arguments as columns of the table. This query returns tuples of poses, times and video sequences with one row for each tuple. The results are ordered by time (\(sqwrl\!:\!orderBy(?time)\)).

4 Conclusions

This paper has presented a work for the classification of non-elementary movements in Vietnamese Mõ Dances. The elementary dance movements (called dance poses, which are extracted from video sequences) are described by an initial dataset ontology based on a defined terminology in [23] using the OWL description language. These pose descriptions are inputs for a set of classification rules to construct non-elementary movements classes.

Then, the extended dataset ontology by non-elementary movements classes can be used to answer specific queries for replaying complex dance movements. The perspective of this work is its integration within an application for reproducing dance movements as for dance training.