Hi,
Just caught my eye that you are still using Swarm 
Same we do. Either way, I don’t see a big deal with 6 min, but in this case, one thing that tends to make builds long is copying content inside the container. I can see you use COPY . . Something that might be able to help is creating a tar and unzipping inside, or reducing the amount of data that you copy.
But as said, it can be wrong, there are too many things that vary the time taken. Just as an idea usually runners and your local dev machine cannot be compared as CPU, and other specs are totally different. Either way, I think you might want to seek help outside here, as your problem is more related to Docker than Strapi.