cheat-sheets

Cheat sheets for various stuff

View the Project on GitHub bertvv/cheat-sheets

A checklist for troubleshooting a network service running on RHEL/CentOS 7. Basic knowledge of TCP/IP is assumed (addressing, port numbers, main protocols, etc.)

TL;DR Checklist

  1. Link layer
    • Check the cable/port LEDs
  2. Network layer
    • ip a
    • ip r (+ ping default gw)
    • cat /etc/resolv.conf (+ dig www.google.com @a.b.c.d +short)
  3. Transport layer
    • sudo systemctl status SERVICE.service
      • sudo systemctl start SERVICE.service
      • sudo systemctl enable SERVICE.service
    • sudo ss -tlnp
    • sudo firewall-cmd --list-all
      • sudo firewall-cmd --add-service=SERVICE --permanent
      • sudo firewall-cmd --add-port=PORT/tcp --permanent
      • sudo systemctl restart firewalld
  4. Application layer
    • sudo journalctl -f -u SERVICE.service
    • sudo systemctl restart SERVICE.service (after each config file change)

General guidelines

A few best practices when setting up and troubleshooting network services

A bottom-up approach

In this troubleshooting guide, we propose a bottom-up approach following the layers of the TCP/IP protocol stack:

  1. Link layer: cables, network ports, etc.
  2. Internet layer: ip address configuration, routing and DNS
  3. Transport layer: service, network ports, firewall settings
  4. Application layer: configuration, etc.

It is important to follow the same procedure systematically. When you skip steps, or start to troubleshoot too high up in the stack, you may miss the cause of the fault.

Link layer

In this phase, we check the network hardware, specifically network cables and ports.

Network layer

The network layer is responsible for being able to communicate with other hosts on the network. In order to be able to communicate, each host should have three settings configured correctly:

  1. The network interface should have an IP address assigned
  2. A default gateway should be set
  3. A DNS server should be set

Before “reaching out” to other hosts, first check local settings.

IP address

The IP address may have been set automatically (DHCP), or manually. Check this in /etc/sysconfig/network-scripts/ifcfg-IFACE, with IFACE the name of the network interface.

Use the command ip address (or shortcut ip a) to list the IP addresses for each interface.

You should know the expected value, if not the exact IP, at least the network range or network IP and network mask.

Possible problems and causes (automatic IP assignment with DHCP):

Possible problems and causes (manual IP setting):

Default gateway

Usually, a host is connected to a LAN through a switch. Network traffic to the outside world goes through a router, connected to the same LAN. Every host on the LAN should know this router, the “default gateway”.

Use the command ip route (or shortcut ip r). There should be a line starting with default via x.y.z.w.

Possible problems and causes (automatic IP assignment with DHCP):

DNS server

In order to be able to resolve host names to IP addresses, every host should be able to contact a DNS server. View the file /etc/resolv.conf. It usually has a header that mentions it was generated automatically, and should have one or more lines starting with nameserver.

# Generated by NetworkManager

Possible problems and causes are equivalent to those with the default gateway setting.

Check LAN connection

If the previous settings are correct, you can check whether other hosts on the LAN can be reached.

Be aware that ping (and other network troubleshooting tools like the traceroute family) may not always work. Some system administrators will block ICMP traffic on routers, rendering the results useless. A command like ping www.google.com (for some the first command they try in case of network connection problems) is not very suitable, in that it depends on too many things to work at once:

Transport layer

In the transport layer, we’ll check whether the network service is actually running, what port it uses, and whether the firewall allows traffic on that port. An example for httpd is given, but this can be applied to other services.

Service and port

Firewall setting

Does the firewall allow traffic on the service? sudo firewall-cmd --list-all.

$ sudo firewall-cmd --list-all
[sudo] password for USER:
public (default, active)
  interfaces: enp0s3 enp0s8
  sources:
  services: dhcpv6-client mdns samba-client ssh http https
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Check the output for the following items:

Application layer

On this phase, we check whether the application is configured correctly, and whether it is available to clients and responds correctly to requests.

Log files

Check the log, either using journalctl, or by looking at the log file in /var/log. The former is standard, the latter may be needed for services that keep a log file not managed/recognized by systemd.

Open a separate terminal with the relevant logs opened and watching for changes (-f option): - sudo journalctl -f -u httpd.service - sudo tail -f /var/log/httpd/error_log

Configuration files

Check the configuration file, somewhere in /etc/, e.g. /etc/httpd/httpd.conf. First, create a backup of the current configuration file(s), and if possible the default one (as created when installing the service)

Availability

You can start checking availability on the loopback interface, but it is important to repeat this from another host. The loopback interface is not firewalled, while physical network interfaces are.

Troubleshooting VirtualBox networking

Network interfaces

In a VirtualBox VM with EL7, the network interfaces are usually named as follows:

NAT interface

The VirtualBox NAT adapter is the default way of allowing your VM to access the Internet. However, it is normally not possible to access your VM from the host system over the network.

The network settings of your VM are predictable. Any VM with a NAT interface has:

You can ping the gateway and name server from the VM, but pinging 10.0.2.15 from your host system will never work. Generally, if your host system has Internet, your VM also should have Internet. One possible exception is when you switch your host system from a wired to wireless connection, or vice versa, while your VM is running.

Host-only interface

The VirtualBox host-only adapter allows you to access a VM over the network. A virtual Ethernet interface is created that is also visible on your host system. You can create multiple host-only adapters and configure network settings from the main window by opening File > Preferences > Network > Host-only Networks.

A host-only interface is named:

The default settings for a host-only interface are:

If a VM is configured to get an IP address automatically (i.e. via DHCP), it will probably get 192.168.56.101 (if it’s the first VM on this host-only network). If you want to assign static IP addresses to VMs on this network, take one from the range 2-99.

I suggest to leave the default host-only interface as is, and create new interfaces as needed. It may be needed to shut down all VMs and restart VirtualBox for configuration changes to come into effect, especially after creating a new interface.

Checklist:

  1. Check host-only network configuration: VirtualBox main window > File > Preferences > Network > Host-only Networks
    • Does the host system have an IP address in the form 192.168.NNN.1?
    • Is the DHCP server activated and configured consistently with the IP address of the host system?
  2. Check whether the host system actually has the expected IP address
  3. During the Link Layer phase, check the VM network adapters (VM Settings > Network)
    • Is the adapter enabled?
    • Is the cable connected (hidden setting under “advanced”)
    • Is it attached to the correct host-only network?
  4. During the Network Layer phase, in the VM, check the IP address of the adapter connected to the host-only network
    • Is it e.g. 192.168.56.101 (default host-only network, first IP address assigned by DHCP)
    • Is it e.g. 192.168.56.NN (with 2 <= NN <= 99, if you assigned a fixed IP address)
  5. Also during the Network Layer phase, ping the VM from your host system and vice versa. Remark that pinging the host system from the VM may not always work, e.g. if the firewall on the host system is configured to block ICMP traffic.

Other troubleshooting resources