Package managers generally don't do diff updates, so every go version is just another huge download.
Thankfully, go is really easy to install and manage. And once you're up + running, git source code diffs are much smaller, and you're free to checkout whatever commit you want and recompile.
First, you need a go1.4 bootstrap and a C compiler; go1.4 is the last one with a C toolchain1. Once compiled, go1.4's used to bootstrap newer versions of go.
curl -L -o- https://jrl.ninja/sw/go/go1.4-bootstrap-20171003.tar.xz | tar xJf - mv go go1.4-bootstrap cd go1.4-bootstrap/src CGO_ENABLED=0 ./make.bash
Then clone go and build it.
git clone https://go.googlesource.com/go cd go/src GOROOT_BOOTSTRAP=$PWD/../../go1.4-bootstrap ./make.bash
1I don't know like, the minimum GCC version or what C standard go1.4 source complies to.