The following sections guide you through the most common problems encountered with VPNs.
Site-to-site Problems
All the PCs that are to participate in the VPN need to be fully operational and visible on the network before attempting to install and configure VPN software.
Check that it is possible to ping the IP address of the RED (Internet) NIC on both Smoothwalls. Failure to get a ping echo would indicate that:
- The remote Smoothwall is not running
- You have the wrong IP address for the remote Smoothwall
- There is a network connection problem – check routers, hubs and cables, and so on.
- There is a problem at your Internet Service Provider
- The Smoothwall has ping turned off via the administration interface
- Verify IP addresses by checking the Network > Configuration > Interfaces page for the appropriate Ethernet card.
- Check the routing information displayed in the Smoothwall's status page, there must be a default route (gateway).
- Verify with the ISP that VPN traffic is not being blocked by any firewall or router used by the ISP. Specifically, ESP mode uses IP protocol 50. AH mode uses IP protocol 50. In particular, if the tunnel goes into OPEN mode but no packets will flow between the two networks, it is possible that one of the ISPs involved is blocking the ESP or AH packets.
- To simplify the problem, attempt to get a connection with shared secrets before moving on to certificates.
- Verify the symmetry in the tunnel specification, that is, that the IDs, IP addresses and Remote network addresses are mirrored. This is where most people make mistakes.
- Each node on the VPN network must have its own unique certificate. At least one field in the subject must be different. The subject is a composite of the information fields supplied when the certificate is created. Likewise the Alt (Alternative) Name field must be unique for each certificate. Obviously fields like company name can be common to all certificates.
- A different local network address must be configured at both ends of the tunnel; they cannot both use the default of 192.168.0.0. Likewise, ensure there is no conflict with another network address. Be consistent with IDs. For example:
- Hosts on static IPs should use the hostname for the gateway as the ID.
- Hosts on dynamic IPs should use the administrator's email address.
- Clients should usually not use an ID, unless they are using an unusual client that requires one.
L2TP Road Warrior Problems
The most likely problem with L2TP road warriors is establishing the initial IPSec transport connection.
The most likely reason for a failure at this stage is an incorrect or invalid certificate. The same problems that can occur with any other type of IPSec connection can also occur with an L2TP road warrior. However, because the vast majority of parameter values are predefined it is generally not likely for an IPSec protocol error other then a certificate problem to occur.
First of all, verify the correct certificate is installed using the Microsoft MMC tool. There must be a CA certificate, as well as a host certificate, present in the system. Also verify the certificate is within its valid time window. If the certificate is newly created, and the time is set incorrectly by only an hour or so, the connection is refused because the certificate is not valid. MMC has facilities for verifying that a host certificate is recognized as being valid.
Note that the error messages produced by the L2TP client can be somewhat strange. Modem not responding can mean that there was an IPSec certificate error, for instance. Check the IPSec logs first when looking for causes of problems. As a last resort, you can also enable debug logging on the Windows client.
Enabling L2TP Debugging
In a default configuration, Microsoft's L2TP client does not produce any log files. This can make diagnosing problems difficult if the logs on the Smoothwall gateway are not sufficient for finding the cause or causes of connection issues.
To enable IPSec-level logging if you are using Windows 2000 or XP, you must create a registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley
Add a REG_DWORD value named 'EnableLogging'. Set the value to 1 to enable logging, or 0 to turn it off. After changing this value, the VPN service must be restarted. From the command line:
net stop policyagent followed by: net start policyagent
The log file is in Windows system directory:
\debug\oakley.log
Note: Smoothwall does not endorse manually editing the registry. Incorrectly altering registry values may result in registry corruption and render the computer unusable.
Windows Networking Issues
In order to facilitate network browsing under Microsoft Windows across the VPN, it is necessary to make sure both ends of the tunnel are properly configured.
In small, single subnet Windows networks, network browsing is facilitated via network broadcasts. In these small networks, network neighborhood will just work without any configuration required. If a road warrior were to connect in, though, it would be unable to browse the network unless the administrator has configured the network to enable it. This is because network broadcasts do not normally cross network boundaries, such as routers and VPNs.
This problem is exactly what Windows network administrators experience when connecting two or more subnets via a router. If you are familiar with setting up multiple subnets of Windows machines, then the problem to be solved is the same.
In the case of road warrior connections, the details depend on the client in use. The built in L2TP client for Windows can be configured to accept WINS and DNS server settings from the server. These parameters are configured in the Global Settings page.
For inexperienced Windows administrators, the following notes are provided to assist with configuring your network to enable network browsing across the VPN.
For NT networks, you will require a WINS server, normally running on your PDC. This WINS server is analogous to a DNS server for the Windows machines. Each of your desktop machines and servers should be configured to use the central WINS server in its network properties box. Any road warriors connecting in should also be set to use this WINS server. If this is done then when they are connected to the office network via the VPN, they should be able to browse the office network, attach to printers and shares, and so on.
In more complex arrangements, such as two subnets of Windows machines with a VPN between the two, it is necessary to set-up either one WINS server and share it between the subnets, or have one on each and configure a replicating system between the two. Again, the problem to be resolved is identical to that which the administrator would face with two normally routed networks.