An Agent Framework Based on Signal Concepts for Highlighting the Image Semantic Content

Abstract

This paper addresses the image semantic gap (i.e. the difficulty to automatically characterize the image semantic content through extracted low-level signal features) by investigating the formation of semantic concepts (such as mountains, sky, grass...) in a population of image agents:abstract structures representing the image visual entities. Through the development of processes mapping extracted low-level features to concept-based visual information, our contribution is twofold. First, we propose a learning framework mapping signal (color, texture) and semantic concepts to highlight the image agents. Contrary to traditional architectures considering high-dimensional spaces of low-level extracted signal features, this framework addresses the curse of dimensionality. Then, at the image agent population level, the agents communicate about the perceived semantic concepts with no access to global information or to the representations of other agents, they only exchange conceptual information. While doing so they adapt their internal representations to be more successful at conveying the perceived semantic information in future interactions. The image content is therefore soundly inferred through these concept-based linguistic interactions.

The SIR Agent prototype implements our theoretical framework and its architecture revolves around functional modules enabling the characterization of concept-based linguistic structures, highlighting the image agents and enforcing interactions and coordination between them.