Order by not working on nested populate

Since you are using the query engine it user the KNEX syntax, so you would need to use orderBy, you can learn more from the docs here

strapi.db.query('api::article.article').findMany({
  orderBy: {
    author: {
      name: 'asc',
    },
  },
});

For your above example to work, instead of using db.query you would use entityService instead.

strapi.entityService.findMany('api::article.article', {
  sort: {
    author: {
      name: 'asc',
    },
  },
});

You can see the docs here

Hope this helps.