Safety in railways is mostly achieved by automated operation using a specialized infrastructure. However, many tasks still rely on the decisions and actions of a human crew. Aiming at improving safety in such situations, we present an approach for recognizing railway signals and signs in video sequences taken by an in-vehicle camera. Our approach is based on a model automatically learned from examples, built from clusters of features extracted by a modified version of SIFT. It does not require the examples and inputs to be obtained under controlled conditions or with specific camera parameters/positioning, being robust to arbitrary weather and lighting, deterioration, motion blur and perspective distortion. We demonstrate the feasibility of our approach by showing that it performs better than a shape-based matching method when recognizing a railway signal with particularly challenging characteristics under realistic conditions.