Sharing a Unity project with my team

We’re mainly just wondering about sharing the code with each other on github. We created the repository on github but when we try to clone it locally, we get ~1600 “changed files” that mostly start with Library. I think I understand that problem so I removed the Library folder from the remote repository and added a .gitignore file and this fixed those ~1600 changed files but now the game doesn’t have assets anymore because it said it had to rebuild the Library and it also makes me change back from PC to Android when I open it. What’s the step I’m missing here?