The emergence of ChatGPT brought a lot of controversy in society and academia due to the ease with which students can effortlessly perform work. This paper investigates whether a freshman student in an introductory programming unit using Python benefits from using ChatGPT, even as an aid. The methodology used was the application of three written tests of one semester - using Python programming language -, answered by ChatGPT (free version), analysed, and compared with a “good” solution of each one. Strangely, or maybe not, errors and problems were found in the solutions provided by ChatGPT that are the same ones that self-taught student (those who learn on YouTube and websites around the Internet world) make.