The only downside currently is if you have policies set on the handlers, the cache will respond before the policies/controllers can be fired (they won’t be at all).
Also the cache middleware doesn’t work on any endpoint that needs security as an authenticated request as it bypasses the users-permissions plugin entirely.
There is no option to cache GraphQL at the moment.