Having a hard time untracking a file that's been pushed

OK here’s my problem: I accidentally tracked/pushed a file that I want to stop tracking.

I used rm --cached and then the file showed up as being changed in Git Desktop. So I committed and pushed the changes.

The file is represented by a red minus now. So when I load that branch, it’s now deleting the file from my hard drive. I was hoping this command would remove the file entirely from the branch, not just tell it to be deleted whenever the branch is activated.

I also tried adding it to .gitignore. When doing that, it shows up as being changed, but it says there’s no changes to commit, despite it showing up in the window as having changes. This also adds .gitignore to the files being tracks, so now I’m tracking that, which I don’t want to.

So I just need a way to get the branch to stop tracking these files. Not delete them when the branch is loaded.