Troubleshooting tools

Before, we look at the most significant commands for connectivity checking and troubleshooting, here is little reminder on how to check host computer's network interface parameters on .

The Microsoft windows have a whole set of helpful command line tools that helps testing and configuring LAN/WAN interfaces. We will look only at commonly used Windows networking tools and commands.

All of the tools are being ran from windows terminal. Go to Start/Run and enter "cmd" to open a Command window.

Some of commands on windows are:

ipconfig – used to display the TCP/IP network configuration values. To open it, enter "ipconfig" in the command prompt.

C:\>ipconfig Windows IP Configuration Ethernet adapter Local Area Connection:   Connection-specific DNS Suffix  . : mshome.net   Link-local IPv6 Address . . . . . : fe80::58ad:cd3f:f3df:bf18%8   IPv4 Address. . . . . . . . . . . : 173.16.16.243   Subnet Mask . . . . . . . . . . . : 255.255.255.0   Default Gateway . . . . . . . . . : 173.16.16.1

There are also a variety of additional functions for ipconfig. To obtain a list of additional options, enter "ipconfig /?" or “ipconfig -?”.


netstat – displays the active TCP connections and ports on which the computer is listening, Ethernet statistics, the IP routing table, statistics for the IP, ICMP, TCP, and UDP protocols. It comes with a number of options for displaying a variety of properties of the network and TCP connections “netstat –?”.

nslookup – is a command-line administrative tool for testing and troubleshooting DNS servers. For example, if you want to know what IP address is "www.google.com", enter "nslookup www.google.com" and you will find that there are more addresses 74.125.77.99, 74.125.77.104, 74.125.77.147.

netsh – is a tool an administrator can use to configure and monitor Windows-based computers at a command prompt. It allows configure interfaces, routing protocols, routes, routing filters and display currently running configuration.

Very similar commands are available also on unix-like machines. Today in most of Linux distributions network settings can be managed via GUI, but it is always good to be familiar with the command-line tools. Here is the list of basic networking commands and tools on Linux:

ifconfig – it is similar like ipconfig commands on windows. It lets enable/disable network adapters, assigned IP address and netmask details as well as show currently network interface configuration.

iwconfig - iwconfig tool is like ifconfig and ethtool for wireless cards. That also view and set the basic Wi-Fi network details.

nslookup – give a host name and the command will return IP address.

netstat – print network connections, including port connections, routing tables, interface statistics, masquerade connections, and more. (netstat – r, netstat - a)

ip – show/manipulate routing, devices, policy routing and tunnels on linux-machine.

For example, check IP address on interface using ip command:

$ip addr show

You can add static route using ip following command:

ip route add {NETWORK address} via {next hop address} dev {DEVICE}, for example:

$ip route add 192.168.55.0/24 via 192.168.1.254 dev eth1

mentioned tools are only small part of networking tools that is available on Linux. Remember if you want full details on the tools and commands options use man command. For example, if you want to know all options on ifconfig write command man ifconfig in terminal.


Check network connectivity

Using the ping command

Ping is one of the most commonly used and known commands. Administration utility used to test whether a particular host is reachable across an Internet Protocol (IP) network and to measure the round-trip time for packets sent from the local host to a destination host, including the local host's own interfaces.

Ping uses Internet Control Message Protocol (ICMP) protocol for echo response and echo request. Ping sends ICMP echo request packets to the target host and waits for an ICMP response. Ping output displays the minimum, average and maximum times used for a ping packet to find a specified system and return.

From PC:

Windows:

C:\>ping 10.255.255.4 Pinging 10.255.255.4 with 32 bytes of data: Reply from 10.255.255.4: bytes=32 time=1ms TTL=61 Reply from 10.255.255.4: bytes=32 time<1ms TTL=61 Reply from 10.255.255.4: bytes=32 time<1ms TTL=61 Reply from 10.255.255.4: bytes=32 time<1ms TTL=61 Ping statistics for 10.255.255.4:    Packets: Sent = 4, Received = 4, Lost = 0 (0% Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 1ms, Average = 0ms


Unix-like:

andris@andris-desktop:/$ ping 10.255.255.6 PING 10.255.255.6 (10.255.255.6) 56(84) bytes of data. 64 bytes from 10.255.255.6: icmp_seq=1 ttl=61 time=1.23 ms 64 bytes from 10.255.255.6: icmp_seq=2 ttl=61 time=0.904 ms 64 bytes from 10.255.255.6: icmp_seq=3 ttl=61 time=0.780 ms 64 bytes from 10.255.255.6: icmp_seq=4 ttl=61 time=0.879 ms ^C --- 10.255.255.6 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2999ms rtt min/avg/max/mdev = 0.780/0.948/1.232/0.174 ms

Press Ctrl-C to stop ping process.


From MikroTik:

[admin@MikroTik] > ping 10.255.255.4 10.255.255.4 64 byte ping: ttl=62 time=2 ms 10.255.255.4 64 byte ping: ttl=62 time=8 ms 10.255.255.4 64 byte ping: ttl=62 time=1 ms 10.255.255.4 64 byte ping: ttl=62 time=10 ms 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 1/5.2/10 ms

Press Ctrl-C to stop ping process.


Using the traceroute command

Traceroute displays the list of the routers that packet travels through to get to a remote host. The traceroute or tracepath tool is available on practically all Unix-like operating systems and tracert on Microsoft Windows operating systems.

Traceroute operation is based on TTL value and ICMP “Time Exceeded” message. Remember that TTL value in IP header is used to avoid routing loops. Each hop decrements TTL value by 1. If the TTL reaches zero, the packet is discarded and ICMP Time Exceeded message is sent back to the sender when this occurs.

Initially by traceroute, the TTL value is set to 1 when next router finds a packet with TTL = 1 it sets TTL value to zero, and responds with an ICMP "time exceeded" message to the source. This message lets the source know that the packet traverses that particular router as a hop. Next time TTL value is incremented by 1 and so on. Typically, each router in the path towards the destination decrements the TTL field by one unit TTL reaches zero.

Using this command you can see how packets travel through the network and where it may fail or slow down. Using this information you can determine the computer, router, switch or other network device that possibly causing network issues or failures.

From Personal computer:

Windows:

C:\>tracert 10.255.255.2 Tracing route to 10.255.255.2 over a maximum of 30 hops  1    <1 ms    <1 ms    <1 ms  10.13.13.1  2     1 ms     1 ms     1 ms  10.255.255.2 Trace complete.


Unix-like:

Traceroute and tracepath is similar, only tracepath does not not require superuser privileges.

andris@andris-desktop:~$ tracepath 10.255.255.6 1:  andris-desktop.local (192.168.10.4)                  0.123ms pmtu 1500 1:  192.168.10.1 (192.168.10.1)                          0.542ms  1:  192.168.10.1 (192.168.10.1)                          0.557ms  2:  192.168.1.2 (192.168.1.2)                            1.213ms  3:  no reply 4:  10.255.255.6 (10.255.255.6)                          2.301ms reached     Resume: pmtu 1500 hops 4 back 61


From MikroTik:

[admin@MikroTik] > tool traceroute 10.255.255.1     ADDRESS                                    STATUS   1       10.0.1.17 2ms 1ms 1ms    2    10.255.255.1 5ms 1ms 1ms [admin@MikroTik] >


Log Files

System event monitoring facility allows to debug different problems using Logs. Log file is a text file created in the server/router/host capturing different kind of activity on the device. This file is the primary data analysis source. RouterOS is capable of logging various system events and status information. Logs can be saved in routers memory (RAM), disk, file, sent by email or even sent to remote syslog server.

All messages stored in routers local memory can be printed from /log menu. Each entry contains time and date when event occurred, topics that this message belongs to and message itself.

[admin@MikroTik] /log> print 15:22:52 system,info device changed by admin 16:16:29 system,info,account user admin logged out from 10.13.13.14 via winbox 16:16:29 system,info,account user admin logged out from 10.13.13.14 via telnet 16:17:16 system,info filter rule added by admin 16:17:34 system,info mangle rule added by admin 16:17:52 system,info simple queue removed by admin 16:18:15 system,info OSPFv2 network added by admin

Read more about logging on RouterOS here>>


Torch (/tool torch)

Torch is real-time traffic monitoring tool that can be used to monitor the traffic flow through an interface.

You can monitor traffic classified by protocol name, source address, destination address, port. Torch shows the protocols you have chosen and tx/rx data rate for each of them.

Icon-note.png

Note: Unicast traffic between Wireless clients with client-to-client forwarding enabled will not be visible to torch tool. Packets that are processed with hardware offloading enabled bridge will also not be visible (unknown unicast, broadcast and some multicast traffic will be visible to torch tool).


Example:

The following example monitor the traffic generated by the telnet protocol, which passes through the interface ether1.

 [admin@MikroTik] tool> torch ether1 port=telnet SRC-PORT                     DST-PORT                     TX         RX 1439                         23 (telnet)                  1.7kbps    368bps  [admin@MikroTik] tool>


To see what IP protocols are sent via ether1:

 [admin@MikroTik] tool> torch ether1 protocol=any-ip PRO.. TX         RX tcp   1.06kbps   608bps udp   896bps     3.7kbps icmp  480bps     480bps ospf  0bps       192bps  [admin@MikroTik] tool>


In order to see what protocols are linked to a host connected to interface 10.0.0.144/32 ether1:

 [admin@MikroTik] tool> torch ether1 src-address=10.0.0.144/32 protocol=any   PRO.. SRC-ADDRESS     TX         RX  tcp   10.0.0.144      1.01kbps   608bps  icmp  10.0.0.144      480bps     480bps [admin@MikroTik] tool>

Icon-warn.png

Warning: Traffic that appears in torch is before it has been filtered by Firewall. This means you will be able to see packets that might get dropped by your Firewall rules.


IPv6

Starting from v5RC6 torch is capable of showing IPv6 traffic. Two new parameters are introduced src-address6 and dst-address6. Example:

admin@RB1100test] > /tool torch interface=bypass-bridge src-address6=::/0 ip-protocol=any sr c-address=0.0.0.0/0 MAC-PROTOCOL    IP-PROT... SRC-ADDRESS                                 TX         RX        ipv6            tcp        2001:111:2222:2::1                          60.1kbps   1005.4kbps ip              tcp        10.5.101.38                                 18.0kbps   3.5kbps   ip              vrrp       10.5.101.34                                 0bps       288bps    ip              udp        10.5.101.1                                  0bps       304bps    ip              tcp        10.0.0.176                                  0bps       416bps    ip              ospf       224.0.0.5                                   544bps     0bps                                                                             78.7kbps   1010.0kbps 

To make /ping tool to work with domain name that resolves IPv6 address use the following:

/ping [:resolve ipv6.google.com]

By default ping tool will take IPv4 address.

Winbox

More attractive Torch interface is available from Winbox (Tool>Torch). In Winbox you can also trigger a Filter bar by hitting the F key on the keyboard.

Image11001.gif

Packet Sniffer (/tool sniffer)

Packet sniffer is a tool that can capture and analyze packets sent and received by specific interface. packet sniffer uses libpcap format.

Packet Sniffer Configuration

In the following example streaming-server will be added, streaming will be enabled, file-name will be set to test and packet sniffer will be started and stopped after some time:

 [admin@MikroTik] tool sniffer> set streaming-server=192.168.0.240 \ \... streaming-enabled=yes file-name=test [admin@MikroTik] tool sniffer> print             interface: all          only-headers: no          memory-limit: 10             file-name: "test"            file-limit: 10     streaming-enabled: yes      streaming-server: 192.168.0.240         filter-stream: yes       filter-protocol: ip-only       filter-address1: 0.0.0.0/0:0-65535       filter-address2: 0.0.0.0/0:0-65535               running: no [admin@MikroTik] tool sniffer> start [admin@MikroTik] tool sniffer> stop

Here you can specify different packet sniffer parameters, like maximum amount of used memory, file size limit in KBs.

Running Packet Sniffer Tool

There are three commands that are used to control runtime operation of the packet sniffer:

/tool sniffer start/tool sniffer stop/tool sniffer save.

The start command is used to start/reset sniffing, stop - stops sniffing. To save currently sniffed packets in a specific file save command is used.

In the following example the packet sniffer will be started and after some time - stopped:
 [admin@MikroTik] tool sniffer> start [admin@MikroTik] tool sniffer> stop


Below the sniffed packets will be saved in the file named test:

 [admin@MikroTik] tool sniffer> save file-name=test


View sniffed packets

There are also available different submenus for viewing sniffed packets.

  • /tool sniffer packet – show the list of sniffed packets
  • /tool sniffer protocol – show all kind of protocols that have been sniffed
  • /tool sniffer host – shows the list of hosts that were participating in data exchange you've sniffed


For example:

 [admin@MikroTik] tool sniffer packet> print   # TIME    INTERFACE   SRC-ADDRESS  0 1.697    ether1    0.0.0.0:68 (bootpc)          1 1.82     ether1    10.0.1.17  2 2.007    ether1    10.0.1.18  3 2.616    ether1    0.0.0.0:68 (bootpc)  4 2.616    ether1    10.0.1.18:45630  5 5.99     ether1    10.0.1.18  6 6.057    ether1    159.148.42.138  7 7.067    ether1    10.0.1.5:1701 (l2tp)  8 8.087    ether1    10.0.1.18:1701 (l2tp)  9 9.977    ether1    10.0.1.18:1701 (l2tp) -- more


Figure below shows sniffer GUI in Winbox, which is more user-friendly.

Image11002.gif

Detailed commands description can be found in the manual >>

Bandwidth test

The Bandwidth Tester can be used to measure the throughput (Mbps) to another MikroTik router (either wired or wireless network) and thereby help to discover network "bottlenecks"- network point with lowest throughput.

BW test uses two protocols to test bandwidth:

  • TCP – uses the standard TCP protocol operation principles with all main components like connection initialization, packets acknowledgments, congestion window mechanism and all other features of TCP algorithm. Please review the TCP protocol for details on its internal speed settings and how to analyze its behavior. Statistics for throughput are calculated using the entire size of the TCP data stream. As acknowledgments are an internal working of TCP, their size and usage of the link are not included in the throughput statistics. Therefore statistics are not as reliable as the UDP statistics when estimating throughput.
  • UDP traffic – sends 110% or more packets than currently reported as received on the other side of the link. To see the maximum throughput of a link, the packet size should be set for the maximum MTU allowed by the links which is usually 1500 bytes. There is no acknowledgment required by UDP; this implementation means that the closest approximation of the throughput can be seen.

Remember that Bandwidth Test uses all available bandwidth (by default) and may impact network usability.

If you want to test real throughput of a router, you should run bandwidth test through the router not from or to it. To do this you need at least 3 routers connected in chain:

Bandwidth Server – router under test – Bandwidth Client.


Icon-note.png

Note: If you use UDP protocol then Bandwidth Test counts IP header+UDP header+UDP data. In case if you use TCP then Bandwidth Test counts only TCP data (TCP header and IP header are not included).


Configuration example:

Server

To enable bandwidth-test server with client authentication:

[admin@MikroTik] /tool bandwidth-server> set enabled=yes authenticate=yes [admin@MikroTik] /tool bandwidth-server> print                   enabled: yes             authenticate: yes  allocate-udp-ports-from: 2000             max-sessions: 100 
[admin@MikroTik] /tool bandwidth-server>


Client

Run UDP bandwidth test in both directions, user name and password depends on remote Bandwidth Server. In this case user name is ‘admin’ without any password.

[admin@MikroTik] > tool bandwidth-test protocol=udp user=admin password="" direction=both \ address=10.0.1.5                status: running              duration: 22s            tx-current: 97.0Mbps  tx-10-second-average: 97.1Mbps      tx-total-average: 75.2Mbps            rx-current: 91.7Mbps  rx-10-second-average: 91.8Mbps      rx-total-average: 72.4Mbps          lost-packets: 294           random-data: no             direction: both               tx-size: 1500               rx-size: 1500 
-- [Q quit|D dump|C-z pause]      

More information and all commands description can be found in the manual>>

Profiler

Profiler is a tool that shows CPU usage for each process running on RouterOS. It helps to identify which process is using most of the CPU resources.

Profiler.png