I’m not sure I understand the entire issue here, if the relation to users only has the done_by Strapi isn’t aware of the not_done_by. In this case does this mean every user in the database without a relation? Or is there only a certain group of users?
In your case it sounds like you need another relation for not_done_by so you can request a group of users to do something, then once they have remove them from this relation and move them to the done_by.
As a side point, I generally recommend you avoid the m2m relation if possible and use the manyWay relation instead as m2m can be quite complex and have a significant performance hit. The manyWay relation is on the far right of the relation matrix within in the content-type builder.