Alright I will answer my own question with what seems to have worked well.
I went through the migration guides:
Migration guide from 4.0.6+ to 4.1.8
Migration guide from 4.1.8+ to 4.1.10
Migration guide from 4.2.x to 4.3.x
Migration guide from 4.3.6 to 4.3.8
Migration guide from 4.4.3 to 4.4.5
Migration guide from 4.4.5 to 4.5.1
Then I only looked for the breaking changes section or database migration, and applied all those changes at once.
The I upgraded the package.json with 4.5.4
Then npm Install and npm run build.
Restart the server, everything was upgraded and working.