I do agree… I talked a lot with the strapi team, even with Pierre, and they all told me the goal was to become a CMS more than an API. And this is absolutely mandatory for a CMS. There has been numerous votes for this (both relationships and list view).
For now, the only solution is a repeatable component which holds the relationship. We already refactored a bunch of entities this way to be ready for V4, and this is, no surprise, way worse in terms of performance… 
We need to get a look on the new plugin API though, because maybe this is something that will be enabled by a plugin, in which case this is totally fine to ditch it from strapi’s scope, but we don need a solution for drag and drop, manual ordering 