Our life is becoming heavily documented and expressed on the digital substrate, which leads to an increasing demand of multimedia analysis tools to organize those memories and retrieve related information when required. Previous systems for information retrieval from first-person-view (FPV) rely on purely visual features, leaving aside the semantic information and failing to extract the personal nature of FPV videos. We propose a system which is capable of organizing and retrieving our digital memories based on their semantic similarity to a given video query. Moreover, we present a novel segmentation method for egocentric vision using multimedia features such as location, time, interactions or activity. Results show its high potential for real FPV videos, making it useful for a wide range of applications for life logging: from personal memory retrieval, to egocentric video summarization.