Concurrency cancel-in-progress but not when ref is master

My workflow runs “on push” and has the following concurrency field:

  group: ${{github.ref}}
  cancel-in-progress: true

This works as expected when pushing multiple times from the same branch.
The problem is that when I merge 2 different PR’s then since github.ref is master in that case it will cancel one of them

is there a way to add a conditional? or a ‘to ignore’? to the concurrency block?