In this article, we examine whether ChatGPT is trustworthy and use our “conversation” with ChatGPT as a pivot for the larger conversation concerning trustworthy AI. Through the example of our “conversation” with ChatGPT, we argue that the development of trustworthy AI requires both keeping the best interests of users at heart as well as addressing larger ethical concerns. In the process, we emphasize the distinction between trusting ChatGPT and trusting the information provided by it. Lastly, we highlight the role of critical inquiry and acknowledgment of functional limitations in fostering trust in AI systems.