Right, I think I undesrtand the question now.
I built a wrapper object for each upload. I have a middleware in place ehich runs everytime an upload is created. So this is assigned to an owner.
When retrieving uploads I have a policy in place to look up the asset and the owner.
It is really painful but it is the only way I found to solve this issue.