Spring Cloud Config with public/private key git access - solving the "Auth fail" error

So I was setting up Spring Cloud Config for our project. Access your config through a public Git repo seems quite straight forward. And then it wasn't. This post is about solving the "Auth fail" error when using a public/private keypair. A basic (server) setup