Depending on when you tested it, by default on Heroku after an hour or so if they see no traffic Heroku will kill the dyno (basically the container) that is running the application. Then when new traffic is detected, it pulls the git source code and cold boots the application.
This means during that cold boot it has to spin up the Strapi server, and thus that first request is slow and once it’s booted the rest will be faster.
Beyond that, the issue could also be your distance and thus network latency from the server.