Snoring detection is not 100% reliable, it may get sometimes confused by other noises. The accuracy depends on many conditions - the particular type of the phone, acoustics in the room, positioning of the phone, etc. For some people it works very reliably, for some (like you, and I am sorry for your inconvenience) not so well. We keep working on improvements of the algorithm, and hopefully it will be 100% perfect for everybody some day.
The amount of snoring indeed does affect your sleep score. It can not be adjusted manually. If you remove the “snore” tag from a particular recording, it does not restrospectively adjust the score. Another thing to consider is how many of the false snore events there are, compared to true detected snoring. If you think that most of the detected snoring is false, and you do not want it to ruin your sleep score, you can disable the snoring detection altogether. If you just do not want to be disturbed by the anti-snoring sound, you can disable only the anti-snoring feature. It’s all in the menu “Setting/Sleep noise recording”.
You may also try to listen to your recordings (on a high volume, with headphones). Some phones generate audible artifacts (various kinds of whirring or buzzing) even if there is no real sound, and then the sound recognition gets very unreliable. If this is not your case, it might be a good idea to use fitbit for sleep tracking, as you suggested, and put your phone on a bedside table, where the sounds from your rustling sheets will be much weaker, compared to the snoring.
I hope some of my tips help,