I see... I may be wrong, but my guess might be, misguided as it is, that he is taking some of the frustration of losing his original intent out on you. As if it is somehow your fault, though of course it isn't in the least.

[He can't help but feel as if his own Master is the one who should be distrustful of his Servant, not Lancer's Master. Their positions should be switched.]