In a private repository, repository owners can only grant write access to collaborators. Collaborators can’t have read-only access to repositories owned by a user account. More information about permission levels for a repository owned by a user account can be found here:
Having said that, I know that we’ve heard users request these kinds of granular permissions before, and I’ll add your +1 to that existing feature request.
I don’t want to give fork/clone/download rights.
It’s worth noting here that cloning a repository is a central operation in the collaboration model that GitHub provides. For example, the read-only access that we currently offer to GitHub Organizations still allows users to both clone and fork the repositories that they have permission to read:
With all of that in mind, I believe that it’s unlikely that we’ll change our permission model to disable the ability to clone a repository that someone has read access to; even if we do allow for read only access to private repositories owned by user accounts in the future.