I don’t believe that’s possible for components within a given collection content-type, but you can split those components into multiple entity collection types and link them via Ids to control permissions for each entity. Unfortunately this makes working with your data a lot less intuitive and you’ll spend a lot of time populating the relational fields.
Alternatively, and a lot more work, you can build your own custom permissions for /api routes and build your own UI to manage the data.