I can't join my brothers session on Xbox 360.

Hello, I'm so sick of trying to get in the game with my brother, every time i join him i get this message ( ERROR, Cant join session, check your internet connection) But when i try to join others it works completely fine. I didn't have the problem with Call of Duty 5, then i could join him..... But on Red dead redemption it dont work!!! This is so freaking irritating. Please help me !!!!!!!!!!