The three-way handshake is the way in which any "reliable" internet connection (connection using the TCP protocol) is made.
When a client establishes a connection to a server, the client sends an SYN request; the server responds with an SYN/ACK packet and the client validates the connection with an ACK (acknowledgement) packet.
A TCP connection cannot be established until these 3 steps have been completed. The SYN attack involves sending a large number of SYN requests via a host with a nonexistent or invalid IP address. As a result, the target machine cannot receive an ACK packet.
Machines vulnerable to SYN attacks queue up the open connections in a data memory structure and wait to receive an ACK packet. There is an expiration mechanism that makes it possible to reject packets after a certain amount of time has passed. However, with an extremely high number of SYN packets, if the resources used by the target machine to store queued requests are all used up, the machine risks entering a unstable state that can cause it to crash or restart.