As a guy I can understand his frustration, but I've been there on both sides of the situation and you have to understand the following: relationships are about trust, and its a matter of your boyfriend trusting you regarding this, but at the same time, put yourself in his shoes. If you would get equally upset, then you can't be a hypocrite and expect him to be cool with it. Relationships are hard work and its a 2 way street, that's why the call it commitment. Good luck with this.Relationship?
being faithful is one of the most important aspects of any relationship
your not going to want to hear this but you CAN NOT go to your exs house after a fight with bf. It would be ok for you to do things with your ex occasionally like meet for lunch if you truely are friends but if you have a boyfriend with trust issues who is insecure and when you guys fight you leave him to go hang out with your ex. That makes him feel like you are leaving him for your ex and that your ex is better for you than he is. If you know that a guy is flirting with you and your boyf is there and he doesnt like it you should make it clear to the guy that you have a boyfriend.
If you are not willing to do these things then you will end up losing him eventually. remember that it is usually the jealous guy in a relationship that is searching for attention and is not getting it. he will look for this attention if you do not provide it.