The successes in artificial intelligence (AI) research in recent years have led to the enormous usage of AI applications in cyber-physical systems. In these systems, the execution of the AI algorithms must be safe, reliable and dependable. Dedicated hardware accelerators are needed to handle the high computational load of deep neural networks and real-time requirements. However, commonly used accelerator hardware is too complex to analyse its timing behavior statically. In this paper, we present a measurement-based approach to analyse the execution time variability of AI accelerators. This approach allows to get insights into the time variability of accelerators at all. We use this approach to characterize the temporal behaviour of three open-source AI accelerators: NVDLA, Gemmini and VTA. These accelerators are implemented on a FPGA and measurements are carried out with multiple individual layers for each accelerator. In addition, the variations in the execution time for representative applications are studied. This is the first such investigation specifically for open-source accelerators. The results indicate that especially the layers supported by the accelerator, the connection between accelerator and host as well as the operating system have significant influence on the variations of the execution time. The found insights and identified challenges can be used in the future to design accelerators that make the execution of AI in safety-critical systems more reliable.