As far as I know the reasons behind this behaviour were performance considerations. As you can see that an image response contains a lot attributes, most likely more than you have with all of other fields together. The same goes to the other kind of realtions.
With this behaviour you can opt in the performance heavy response types only if you really need it.