Telemedicine systems that involve the use of video conferencing technologies have been available for more than three decades. Yet, they have primarily been used for specialist appointments or within health care facilities. We are now seeing a shift with the proliferation of commercial technologies, such as smartphone apps that allow people to have appointments with a general practitioner from nearly any location for various reasons. Telemedicine has also seen an uptake due to the COVID-19 pandemic. However, little is known about how doctors and patients perceive smartphone-based telemedicine systems, what types of medical ailments are best suited for these systems, what sociotechnical challenges might emerge through their usage, and how systems should be designed to best meet the needs of both doctors and patients. Thus, we applied a scenario-based design method by presenting a set of medical situations to both general practitioners and patients, and conducted contextual interviews with them to investigate their thoughts on video-based appointments for a range of medical situations. Results show that video consultations using smartphone apps could raise challenges in delivering appropriate care and utilization, conducting camera work to assist different types of examinations, supporting doctor–patient relationship creation and maintenance, allowing doctors to maintain control over the appointment, as well as protecting patients’ and doctors’ privacy. This suggests the need to create designs that can support particular workflows, relationship building, safety and privacy protection, and camera work for varying contexts.