I get that too, occasionally I’ll get random downvotes from nowhere only for them to disappear a few days later. Presumably some troll is going around downvoting random people until the admins catch them and remove their platform manipulation.