gh ssh-key add failing

Following the guide to add ssh keys: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account

I have authenticated with

gh auth login

Generated keys and added to ssh-agent using the following guide:
https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

When I try to add using

gh ssh-key add key-file.pub

I get an error:

open /home/username/.ssh/key-file.pub: permission denied

the file has permission 0644

I’m not sure what is going wrong