Cheat sheets for various stuff
A checklist for troubleshooting a network service running on RHEL/CentOS 7. Basic knowledge of TCP/IP is assumed (addressing, port numbers, main protocols, etc.)
ip aip r (+ ping default gw)cat /etc/resolv.conf (+ dig www.google.com @a.b.c.d +short)sudo systemctl status SERVICE.service
sudo systemctl start SERVICE.servicesudo systemctl enable SERVICE.servicesudo ss -tlnpsudo firewall-cmd --list-all
sudo firewall-cmd --add-service=SERVICE --permanentsudo firewall-cmd --add-port=PORT/tcp --permanentsudo systemctl restart firewalldsudo journalctl -f -u SERVICE.servicesudo systemctl restart SERVICE.service (after each config file change)A few best practices when setting up and troubleshooting network services
journalctl -f)In this troubleshooting guide, we propose a bottom-up approach following the layers of the TCP/IP protocol stack:
It is important to follow the same procedure systematically. When you skip steps, or start to troubleshoot too high up in the stack, you may miss the cause of the fault.
In this phase, we check the network hardware, specifically network cables and ports.
ip link
UP: ok, the interface is connectedNO-CARRIER: no signal on this interfaceThe network layer is responsible for being able to communicate with other hosts on the network. In order to be able to communicate, each host should have three settings configured correctly:
Before “reaching out” to other hosts, first check local settings.
The IP address may have been set automatically (DHCP), or manually. Check this in /etc/sysconfig/network-scripts/ifcfg-IFACE, with IFACE the name of the network interface.
Use the command ip address (or shortcut ip a) to list the IP addresses for each interface.
You should know the expected value, if not the exact IP, at least the network range or network IP and network mask.
Possible problems and causes (automatic IP assignment with DHCP):
Possible problems and causes (manual IP setting):
Usually, a host is connected to a LAN through a switch. Network traffic to the outside world goes through a router, connected to the same LAN. Every host on the LAN should know this router, the “default gateway”.
Use the command ip route (or shortcut ip r). There should be a line starting with default via x.y.z.w.
Possible problems and causes (automatic IP assignment with DHCP):
In order to be able to resolve host names to IP addresses, every host should be able to contact a DNS server. View the file /etc/resolv.conf. It usually has a header that mentions it was generated automatically, and should have one or more lines starting with nameserver.
# Generated by NetworkManager
Possible problems and causes are equivalent to those with the default gateway setting.
If the previous settings are correct, you can check whether other hosts on the LAN can be reached.
ping a.b.c.ddig www.google.com @e.f.g.h +shortBe aware that ping (and other network troubleshooting tools like the traceroute family) may not always work. Some system administrators will block ICMP traffic on routers, rendering the results useless. A command like ping www.google.com (for some the first command they try in case of network connection problems) is not very suitable, in that it depends on too many things to work at once:
In the transport layer, we’ll check whether the network service is actually running, what port it uses, and whether the firewall allows traffic on that port. An example for httpd is given, but this can be applied to other services.
sudo systemctl status httpd.service
active (running)If the output contains: inactive (dead), start the service, and if necessary, make sure it starts automatically on boot
sudo systemctl start httpd.service
sudo systemctl enable httpd.service
sudo ss -tlnp (list TCP (-t) server (-l) port numbers (-n) with the process behind them (-p). The -p option requires root, hence the sudo)
/etc/services for standard port numbers for well-known network services.Does the firewall allow traffic on the service? sudo firewall-cmd --list-all.
$ sudo firewall-cmd --list-all
[sudo] password for USER:
public (default, active)
interfaces: enp0s3 enp0s8
sources:
services: dhcpv6-client mdns samba-client ssh http https
ports:
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
Check the output for the following items:
firewall-cmd --get-services.firewalld is not necessarily equal to the service name for systemd. E.g. BIND is called named.service by systemd, while it is referred to as dns by firewalld.firewalld)On this phase, we check whether the application is configured correctly, and whether it is available to clients and responds correctly to requests.
Check the log, either using journalctl, or by looking at the log file in /var/log. The former is standard, the latter may be needed for services that keep a log file not managed/recognized by systemd.
Open a separate terminal with the relevant logs opened and watching for changes (-f option):
- sudo journalctl -f -u httpd.service
- sudo tail -f /var/log/httpd/error_log
Check the configuration file, somewhere in /etc/, e.g. /etc/httpd/httpd.conf. First, create a backup of the current configuration file(s), and if possible the default one (as created when installing the service)
httpd: apachectl configtestsudo systemctl restart httpd.serviceYou can start checking availability on the loopback interface, but it is important to repeat this from another host. The loopback interface is not firewalled, while physical network interfaces are.
sudo nmap -sS -p 80,443 HOST (perform a TCP SYN scan on port 80 and 443)wget http://HOST/, wget https://HOST/curl http://HOST/, curl https://HOST/In a VirtualBox VM with EL7, the network interfaces are usually named as follows:
enp0s3enp0s8enp0s9enp0s10The VirtualBox NAT adapter is the default way of allowing your VM to access the Internet. However, it is normally not possible to access your VM from the host system over the network.
The network settings of your VM are predictable. Any VM with a NAT interface has:
You can ping the gateway and name server from the VM, but pinging 10.0.2.15 from your host system will never work. Generally, if your host system has Internet, your VM also should have Internet. One possible exception is when you switch your host system from a wired to wireless connection, or vice versa, while your VM is running.
The VirtualBox host-only adapter allows you to access a VM over the network. A virtual Ethernet interface is created that is also visible on your host system. You can create multiple host-only adapters and configure network settings from the main window by opening File > Preferences > Network > Host-only Networks.
A host-only interface is named:
vboxnet0, vboxnet1, …The default settings for a host-only interface are:
If a VM is configured to get an IP address automatically (i.e. via DHCP), it will probably get 192.168.56.101 (if it’s the first VM on this host-only network). If you want to assign static IP addresses to VMs on this network, take one from the range 2-99.
I suggest to leave the default host-only interface as is, and create new interfaces as needed. It may be needed to shut down all VMs and restart VirtualBox for configuration changes to come into effect, especially after creating a new interface.