I can pretty much confirm through comparing who's on the same level as me with the current standings that it isn't tied to level.
How can you confirm that the extra votes aren't tied to level?
If it isn't then you must have an ability to causes extra votes / extra pressure. From your post it sounds like two people have it. If you could clarify a bit more how you mean that; I don't know how it can be explicitly confirmed otherwise? Here is how I see things...
As an overall summary: I think that the only way you can confirm that it isn't tied to level is that you know someone else's ability; or that it is actually tied to level and you're bluffing.
You voted Scarlet, but nobody voted you or scarlet so there doesn't appear to be a paper trail to follow in that regard. That means it could be a negative ability for you, or someonoe can vote secretly. If you know that it isn't tied to the level, then it must be because you have some ability that lets you vote secretly. If you can confirm that, it seems like you also know the other players ability; this makes it seem like one of two things: you and that person are on the same team; or you were lying when you said you can confirm that its not tied to level; or both. That said, if it's an ability where people can vote secretly, it would make sense for them to try and vote each other off so that they don't have to worry about being secretly voted against.
Knowing that...
NeoSilk voted me and I voted him, it appears that there could be a potential revenge extra vote according to the paper trail. But knowing also what you said, it could also be that you AND someone else has the ability to vote secretly which would be the reason you could confirm it in the first place.