Interpreting spatial language in image captions

Abstract

The map as a tool for accessing data has become very popular in recent years, but a lot of data do not have the necessary spatial meta-data to allow for that. Some data such as photographs however have spatial information in their captions and if this could be extracted, then they could be made available via map-based interfaces. Towards this goal, we introduce a model and spatio-linguistic reasoner for interpreting the spatial information in image captions that is based upon quantitative data about spatial language use acquired directly from people. Spatial language is inherently vague, and both the model and reasoner have been designed to incorporate this vagueness at the quantitative level and not only qualitatively.

Appendix: Technical details of the vague field

Definition

Conceptually, the vague field is a two-dimensional, unbounded, continuous scalar field defined on a external coordinate system. Computationally, it is impossible to store an unbounded, continuous field; therefore, in its internal representation, the vague field is a bounded, discretised, floating-point field which can easily be stored in a two-dimensional, floating-point matrix. This matrix which forms the foundation for the vague field is augmented with further attributes that are required for the instantiation and processing of the vague field.

To enable the translation between the external coordinate system and the internal matrix representation, the field stores an external and an internal anchor location. The external anchor represents the location that the vague field is defined as being relative to in the external coordinate system. For the spatial preposition fields, this is the location of the ground toponym, such as the location of “Cardiff” in the case of the vague field for “near Cardiff”. The internal anchor represents the point where the field is attached to the external anchor location. The values in the internal matrix are always to be interpreted as specifying the vague phenomenon’s applicability relative to this internal anchor location. The internal and external anchors are used in the read function to translate between the external and internal coordinate systems (“Appendix”—“Accessing the field values”).

Instantiation

The experiment described in the previous section resulted in a sparse set of measurement points and an applicability value for each measurement point. An interpolation using ordinary kriging is used to transform these point measurements into the continuous field representation. Ordinary kriging was developed in the geostatistics field to estimate the distribution of natural resources based on a set of point measurements (Krige 1951; Matheron (1962; Hengl 2007). To calculate the vague field, a grid is placed over the area defined by the measurement points and the extent of each grid cell defined by the field’s desired scale. The interpolated value for each cell is then calculated using a weighted average as shown in Eq. 9. The advantage of ordinary kriging over other distance-based interpolations is that the weighting values λ are automatically derived from the values and spatial distribution of the measurement points p (Fig. 23). The interpolated results are in the range [1, 9] and in a final step are normalised to the [0, 1] range (Eq. 10) where kriging[x, y] is the result matrix produced by the kriging algorithm.

$$ kriging[x,y] = \sum_{i=0}^{n} \lambda \cdot \hbox{value}\left(p_i\right) $$

(9)

$$ values[x, y] = \frac{kriging[x, y] - 1}{\max(kriging - 1)} $$

(10)

Fig. 23

The source point measurement data and the field calculated using ordinary kriging. The darker the field, the higher the applicability value at that point

The quality of the interpolation depends on the number of measurement points, and even for kriging, the number of measurement points as derived from the human-subject experiment is low. The effect of that is that the fitted variogram model is less stable. To increase the number of measurement points and thus the quality of the resulting field, additional measurement points were created based on the properties that the analysis described in “Background” revealed. For the cardinal directions where direction plays a the primary role, the measurement locations were mirrored across the cardinal direction’s primary axis¹⁷ (Fig. 24). With “near” the analysis showed that angle played no significant role, it was thus possible to mirror the measurement locations across both axis, effectively quadrupling the number of measurement locations (Fig. 25) and making the resulting field more stable.

Fig. 24

The original measurement points for “north” on the left and the mirrored, duplicated measurement points on the right

Fig. 25

The original measurement points for “north” on the left and the quadrupled set of measurement points that is used in the final implementation on the right

Accessing the field values

The read operation provides access to the vague field’s applicability values. It translates between the external, unbounded, continuous representation and the internal, discrete, bounded field-value matrix. The translation from the external to the internal representation is performed using the internal and external anchor locations. The offsets of the external x and y coordinates relative to the external anchor location are calculated and then using the field’s scale value transformed into internal offset coordinates. These internal offset coordinates are then added to the internal anchor’s coordinates to determine the internal coordinates. The internal coordinates are then used to read a value from the field matrix, which is returned.

Combining fields

The field combination calculation (Eq. 1) is performed every time the combined field is accessed. While this is a computationally expensive approach, it has the advantage that fields of any scale can be combined without having to align their internal matrix representations, as the fields are accessed through the read function and can thus be treated as continuous and scale-free.

The normalisation factor nf is defined as the maximum combined field value and is calculated by placing a virtual grid over all fields, calculating the value at each grid point and taking the maximum of these values. One problem with this approach is that if the source fields’ cells overlap as shown in Fig. 26, then none of the maximum measurement points actually measure the combined maximum. This means that if the combined field is read at a location that would produce the actual maximum, then the calculated value would be larger than the normalisation factor and the resulting value would be larger than 1, which is not allowed. To avoid this, if the combined value is larger than the normalisation factor, then a value of 1 is returned, regardless of what the actual measurement value is. While this may seem to skew the data, if all the source fields are continuous, as is usually the case, then the difference between the calculated and the actual maximum is very small and can be disregarded.

Fig. 26

Three fields that overlap at their boundaries and where none of the measurement points used to calculate the field maximum (small white circles) measure the actual maximum where the three fields overlap

Crisping the vague field

The crisp operation is used to transform the continuous vague field into a crisp polygon for integration with existing GI systems and algorithms and is based on active contours.

Active contours

The concept of active contours was introduced by Kass et al. (1988) as a method of finding boundaries in image data, but have also been used in GIS for various purposes (Burghardt 2005; Steiniger and Meier 2004; Horvath et al. 2009). They are defined as controlled continuity splines (Terzopoulos 1986) upon which image and external forces act to move them into the desired shape. In the original method, the energies acting upon the active contour are defined as in Eq. 11, consisting of an internal energy, the image energy and external constraint energy, which the active contour then tries to minimise.

$$ \begin{aligned} E_{snake}^* &= \int\limits_0^1 E_{snake}(v(s)) ds \\ &=\int\limits_0^1 E_{int}(v(s)) + E_{image}(v(s)) + E_{con}(v(s)) ds \end{aligned} $$

(11)

The internal energy acts to maintain the active contour’s shape, image energy can be defined via the image intensity (Kass et al. 1988), image gradient (Lam and Yan 1994), or via more complex methods (Xie and Mirmehdi 2006), and the external energy defines constraints that the active contour needs to observe that are not directly defined by the active contour itself or the image data. An iterative method on a grid is used to move the active contour’s control points to their final solution. For each control point, the minimum energy neighbour is calculated, and the control point moved there immediately. The energy calculation for the next control point will thus take into account the updated position of the previous control point. This is repeated until the active contour’s final shape is found and means that the active contour will achieve a locally minimal solution, but not necessarily a globally minimal solution. Due to this iterative way of moving, active contours are often also referred to as snakes, as they seem to slither across their processing space (Fig. 30).

Crisping fields with active contours

To enable the use of active contours in creating a crisp representation of the vague field, a slightly modified energy function is used (Eq. 12). The first two energies, internal and field, are similar to the internal and image energies as defined earlier. The contract energy is an external energy that pulls the active contour towards the centre of the field.

$$ E_{snake} = \alpha \cdot E_{int} + \beta \cdot E_{field} + \gamma \cdot E_{contract} $$

(12)

Each energy is defined as a vector field, with the direction of each cell’s vector defining the direction in which an active contour control point at that location would be pushed (Fig. 27). The length of the cell’s vector defines how strongly the control point is being pushed in the specified direction; thus, the energy function (Eq. 12) can be implemented as a simple vector addition, with the final vector defining the direction the control point will move.

Fig. 27

The three vectors that define the direction the control point will move. The dashed line represents the field’s vector, the dotted line is the contraction field vector and the small final arrow is the internal energy vector. In the left-hand case, the control point will be moved one grid cell in the direction indicated, while in the right-hand case, the control point will not move, as a local minimum has been found for that control point

In this framework, the internal energy (Eq. 13) is defined as the vector from the control point (p _i ) to the centre point between the preceeding (p _i-1) and following (p _i+1) control points (Fig. 28). This definition ensures that the control points always move so as to create a snake where the control points are evenly spread, since the further a control point moves towards its predecessor and further away from is successor, the stronger it will be pulled towards the successor.

$$ E_{int} = \left(v_{prev} + \frac{v_{next} - v_{prev}}{2}\right) - v_{current} $$

(13)

Fig. 28

The internal energy is defined as the vector from the current point (p _i ) to the half-way point between the previous (p _i-1) and the next control point (p _i+1)

The scalar vague field is transformed into the required vector representation by applying the gradient operator. The gradient operator defines each cell’s vector so that it points in the direction of the neighbouring cell with the lowest scalar value. The length of the vector is determined by the value of the current cell, unless the minimum is equal to the cell’s value in which case the vector’s length is set to 0 as the cell is a local minimum (Fig. 29).

Fig. 29

The vague field for ”near“ and a simplified representation of its gradient field

Fig. 30

The active contour moving from the initial location to its final solution on a field for ”near“. To illustrate the principle, the snake was initialised at α = 0.4

The contraction energy is used to define how far the active contour will contract. It is defined as a constant vector field of the same extent as the vague field that is being crisped, with each cell’s vector pointing towards the centre of the vague field and of length 1. The centre of the vague field is defined as the centroid of all cells with the maximum value. This guarantees that the active contour will contract towards the strongest part of the field.

In the active contour energy function (Eq. 12), each component energy has a weight attached to it, to define their relative influences on the total energy. The weights have been tuned experimentally and in the results shown in this section are $\alpha=0.2745,\,\beta=1,\,\gamma=0.4314$. The weights were chosen so as to create crisp polygons that had extents that roughly matched the angles and distances observed in the initial Geograph experiment (Hall and Jones 2008). The snake is initialised and terminated so as to minimise the number of iterations it has to run through, while guaranteeing a valid result. Details of the methods used to enable this can be found in Hall and Jones (2009).