Clean VMs for container runners?

About running a pipeline in a container, instead of directly on the VM.

I was thinking would there be any significant improvement in the load and run times of the GitHub Actions runs, if a pipeline was runs-on: a clean VM with only Docker installed compared to the current configs with wide variety of tooling ?