If you are significantly lower level than the Raiders this will affect a lot. They will have higher forged weapons, higher stats, better cards, etc...
If you aren't using certain characters that are good in PvP, or if they are using those characters instead/as well (Alicia with R&R, Doldren, Konstantin with Tolchock, etc...) this will affect the match as well.
Next the MMR. If you're MMR is significantly higher than their group MMR the allied AI will be weaker. If it's significantly lower, the AI will be much stronger.
Finally the mission. Certain missions are harder for antagonist and certain missions are a cake walk. Try queuing for Mind over Matter, White Noise, Fistful of Sand (and focus the old man if you are really mean), etc...