External collaborators in a team

I’m trying to create a team with organization members and external collaborators, where the latter only have access to the repositories linked to the team. However, whenever the external collaborators are added to the team, they can see everything in our organization. Can you not have a team with members and externals, without them accessing all our repo’s?