Connecting two computers into a router

Hello,
I'm a fairly big newbie when it comes to networking, and I was trying to set up a network among my cousins' two computers via ethernet cables and a router. I foolishly thought that it was as simple as plugging in the dsl modem ethernet cable into the router where indicated, and plugging in the two individual ethernet cables from both computers into one of the 4/5 slots. I believe I was using the BEFSR41 Linksys Router (http://www.linksysbycisco.com/US/en/products/BEFSR41 and it was bought brand-new. After having done so, and WITHOUT installing any software, I tested both computers by trying to open an internet browser and surfing the internet. Both worked for 20-30 minutes simultaneously, so I left assuming everything was fine. However, a half hour after I had left, I was told that only one computer could be online at one time, and the other would not connect at all. I probably shouldn't post before at least installing the router onto both computers, but I was curious if that was not all that it takes. Again, I'm a completely ignorant newbie when it comes to networking and I was hoping someone could give me some advice, or at least point me in the right direction. Thanks.