Thanks for your feedback.
Indeed caching is one "tricky" part here. Even though it depends a bit on your process.
Unfortunately I don't have experience using Circle CI so I can't provide information with this provider.
But a great idea to speed up your builds would be to create a Docker image, because the caching layers there don't care about this and you start from scratch there anyway.
For a reference please check out this comment from to one of my other articles, I think this can be exactly what you might look for:
Beside that I plan to soon write another article about the approach mentioned there. But I've already verified it that it speeds up the build process significantly.