There's definitely a legitimate problem with the way Antagonist mode is handled right now. There will always be a matchmaking discrepancy as long as it's 4 raiders and 1 antagonist - and to make matters worse, antagonists aren't always issued to a group of raiders, so the queues are even longer because of it. I would suggest making antagonist mode mandatory, but there seems to be quite a few people who don't like having an invader in their game. I think the only solution there would be to allow people to queue without an antagonist for a quicker, easier match and a lower reward. High risk; high reward. Having an antagonist always makes matches far more interesting, but I can see players getting frustrated when they don't know how to counter them. It's a shame, because that's where the real meat of the game is.
Sorry about the ping - it's strange that it's so bad for you. I usually end up as host, and I'm situated in the Middle East at the moment. You need to get into my games, fam. If you're playing on PS4, I could add you.