vitess troubleshooting, tcpdump instructions : Vitess

Occasionally, when a problem is application or application MySQL driver specific, PlanetScale Support may ask you to collect a tcpdump network trace of the data flowing from the application to the vtgate MySQL listener.

In a production environment, this may be complicated by the fact that you may have a network loadbalancer in front of multiple vtgate instances. In this case, you may have to run network captures across multiple hosts hosting the vtgate instances simultaneously to get all the information we need to debug the problem. However, the method for collecting the network trace on each host would remain the same.

To collect a network trace, let's review what you need:

You will typically need sudo or root access on the host in question to capture network traffic.
You need to determine which TCP port your vtgate instance is listening on. If you look at your vtgate start script or at a process listing via:

ps -ef | grep vtgate

You should see the vtgate port as the value of the -mysql_server_port parameter. Make a note of this port number.
Next, you need to determine which physical network interface the application traffic is coming into the vtgate server. Typically it could be something like eth0 or eno0, but you would verify by checking the output of:

ip addr

and matching up the ip address the application is using to access the vtgate instance.

To actually capture the traffic:

We assume you are using sudo. Run:

sudo tcpdump -i<interface> -s0 -n -nn -B 32768 -w /path/to/tempfile.dump port <tcpport>

Where:

<interface> is the physical network interface you determined earlier, e.g. eth0
/path/to/tempfile.dump is the filesystem path to a location where you have sufficient space for the dump file. Note that in a production environment, a tcpdump of live traffic can generate a dumpfile of many gigabytes pretty quickly, so be careful.
<tcpport> is the port number you determined earlier that vtgate is listening on for MySQL traffic.

When you are done, you can compress the dumpfile (e.g. using gzip, zstd or xz) and send the dump file to us. If the dumpfile is large, even after compression, it may be best to share it via a file service like Google Drive. Note that if this network trace is from a production system, it may contain sensitive information, so you may want to encrypt it before sending as well, and share the encryption key with us separately. You can use whichever method you have comfortable with, but we recommend something like PGP, using the PlanetScale Support (support@planetscale.com) public key:

Capturing a tcpdump network trace for vtgate Print

Related Articles