Understanding the IPSec logs

Given the nature of the IPSec configuration, there are a high number of factors involved when setting up a VPN connection. Therefore we will discuss what areas of the configuration that should be checked, when certain types of messages are displayed in the IPSec log, because trying to map every single error message to a specific configuration error is all but impossible. Instead, this primer will offer a good start when troubleshooting VPN errors. Following the guidelines below, the troubleshooting process will have a good chance of starting in the area of the configuration where the problem lies.

Using the logs

You can view IPsec VPN logs to help diagnose any issues that you might have. As with other logs, you can filter and then export the logs.

On the REPORTS menu, under the Logs submenu, click IPsec.

  • To filter the log by a specific tunnel, from the Tunnel name list, select the tunnel and click Update.
  • To filter the logs for a specific time period, specify the date by using the Month and Day lists and click Update.
  • To export the IPsec logs, choose an Export format from the list, select the Export all dates option and click Export. Follow the browser instructions to save results.csv file to a local directory.

Solution

When examining the logs it is important to be able to compare the logs from both sides of the connection. Often the same error will display different messages, depending on which side the error originates from. A cryptic error message from one end of the tunnel can become quite clear, when looking at the log messages from the Peer end. It should be considered standard practice to have both logs available and only make changes after examining both of them.

Another good trick when examining the IPSec logs, especially on systems with more than one connection running, is to select the specific connection name in the Smoothwall log viewer and update the logs. That will only show log entries from the connection selected and thus improve the overview considerably.

Error messages

  • No answer messages, No acceptable response or similar
    Any messages stating that a connection is being tried (main mode initiated) and reporting that it's getting No answer is likely caused by the Peer side not responding at all. Common causes for this will be that the VPN engine is not running or the Peer IP address has been set wrong in the configuration of the Tunnel. Comparing the two logs should reveal this soon enough.
    If there are no entries in the Peer log, one of the above explanations should be examined.The No Acceptable Response messages can also be indicative of NAT problems.
  • Incomplete ISAKMP, Part of previous message, Phase 1 reply received but already at phase 2 or similar
    If any part of the above messages are displayed in the logs, this will most likely be due to NAT or network port access problems.
    If dealing with a Road Warrior type configuration the peer end will possibly have No Acceptable Response messages or No reply messages in the log.
    If you are dealing with a client behind a firewall doing NAT, the obvious answer would be that the firewall in front of the client has IPSec-Passthrough enabled.
  • No Proposal Chosen
    If this error shows up at either end of the connection, the most likely cause of problems are incompatible encryption settings.
    On a Smoothwall-to-Smoothwall tunnel, the error is most likely that compression has been enabled on either of the ends, and not enabled on the other.
    If you are trying a Road Warrior connection or a connection to a VPN Gateway from another vendor, double check all the encryption algorithm settings and make sure they are the same on both endpoints.
  • Invalid ID, No connection known errors or similar
    Check for any mismatch between IP Subnet settings and ID value settings in the tunnel configuration. The subnet definitions and, in case of a Road Warrior, the IP allocated to the Road Warrior, must match on both ends. Also check that the ID Value used by certificate authentication are entered correctly. Bring up both configuration screens at the same time to better be able to compare them.
    In case of an L2TP connection, make sure you have selected the certificate issued to the L2TP client in the Authenticate by field instead of Certificate Provided by Peer
  • No key known, Cannot load my private key, No public key known for, or similar
    Errors like those, popping up in the log, almost certainly has to do with invalid certificates. Check the time and date on the Smoothwalls and that the certificates are valid with the current date and time set. Also check that the Certificate Authority (CA) has been installed on each Smoothwall and Road Warrior that needs access to the CA certificate to be able to verify the validity of the created certificates.

The above pointers should get the troubleshooting started at the right track. Remember of course, that multiple configuration errors can exist, and that when one error is corrected, the next might show up in the log, but with the above pointers, you should at least be able to move on to the next type of error, instead of trying to correct the wrong parameters.