On relay search we show an amber dot next to the relay nickname when it is overloaded. This means that one or many of the following load metrics have been triggered:

• Any Tor OOM invocation due to memory pressure
• Any ntor onionskins are dropped
• TCP port exhaustion

Note that if a relay reaches an overloaded state we show it for 72 hours after the relay has recovered.

If you notice that your relay is overloaded please:

1. Check https://status.torproject.org/ for any known issues in the "Tor network" category.

2. Consider tuning sysctl for your system for network, memory and CPU load.

3. Consider enabling MetricsPort to understand what is happening.

Tuning sysctl for network, memory and CPU load

TCP port exhaustion

If you are experiencing TCP port exhaustion consider expanding your local port range. You can do that with

# sysctl -w net.ipv4.ip_local_port_range="15000 64000"

o

# echo 15000 64000 > /proc/sys/net/ipv4/ip_local_port_range

Keep in mind that tuning sysctl as described is not permanent and will be lost upon restart. You need to add the configuration to /etc/sysctl.conf or to a file in /etc/sysctl.d/ to make it permanent.

MetricsPort

To understand the well-being of Tor relays and the Tor network it is vital to provide and have access to relay metrics. Relay overload information has been added to relay descriptors since 0.4.6+ but it was not until Tor >= 0.4.7.1-alpha that an interface to the underlying relay metrics was available: the metrics port.

Enabling MetricsPort

Tor provides access to the metrics port via a torrc configuration option called MetricsPort.

It's important to understand that exposing the tor MetricsPort publicly is dangerous for the Tor network users, which is why that port is not enabled by default and its access has to be governed by an access policy. Please take extra precaution and care when opening this port, and close it when you are done debugging.

Let's assume you are the only user on a server that runs a Tor relay. You can enable the metrics port adding this to your torrc file:

MetricsPort 127.0.0.1:9035
MetricsPortPolicy accept 127.0.0.1

And then you will be able to easily retrieve the metrics with:

# curl https://127.0.0.1:9035/metrics

which are by default in a Prometheus format.

Note: every user on that server will be able to access those relay metrics in the example above. In general, set a very strict access policy with MetricsPortPolicy and consider using your operating systems firewall features for defense in depth.

For a more detailed explanation about MetricsPort and MetricsPortPolicy see tor's man page.

MetricsPort output

Here is an example of what output enabling MetricsPort will produce (we omitted any congestion control related metrics as we still need to stabilize that interface):

# HELP tor_relay_connections Total number of opened connections
# TYPE tor_relay_connections gauge
tor_relay_connections{type="OR listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Exit",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Exit",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Exit",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Exit",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent natd listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent natd listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent natd listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent natd listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="DNS listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="DNS listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="DNS listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="DNS listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics",direction="received",state="opened",family="ipv6"} 0
# HELP tor_relay_connections_total Total number of created/rejected connections
# TYPE tor_relay_connections_total counter
tor_relay_connections_total{type="OR listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="rejected",family="ipv6"} 0
# HELP tor_relay_flag Relay flags from consensus
# TYPE tor_relay_flag gauge
tor_relay_flag{type="Fast"} 0
tor_relay_flag{type="Exit"} 0
tor_relay_flag{type="Authority"} 0
tor_relay_flag{type="Stable"} 0
tor_relay_flag{type="HSDir"} 0
tor_relay_flag{type="Running"} 0
tor_relay_flag{type="V2Dir"} 0
tor_relay_flag{type="Sybil"} 0
tor_relay_flag{type="Guard"} 0
# HELP tor_relay_circuits_total Total number of circuits
# TYPE tor_relay_circuits_total gauge
tor_relay_circuits_total{state="opened"} 0
# HELP tor_relay_streams_total Total number of streams
# TYPE tor_relay_streams_total counter
tor_relay_streams_total{type="BEGIN"} 0
tor_relay_streams_total{type="BEGIN_DIR"} 0
tor_relay_streams_total{type="RESOLVE"} 0
# HELP tor_relay_traffic_bytes Traffic related counters
# TYPE tor_relay_traffic_bytes counter
tor_relay_traffic_bytes{direction="read"} 0
tor_relay_traffic_bytes{direction="written"} 0
# HELP tor_relay_dos_total Denial of Service defenses related counters
# TYPE tor_relay_dos_total counter
tor_relay_dos_total{type="circuit_rejected"} 0
tor_relay_dos_total{type="circuit_killed_max_cell"} 0
tor_relay_dos_total{type="circuit_killed_max_cell_outq"} 0
tor_relay_dos_total{type="marked_address"} 0
tor_relay_dos_total{type="marked_address_maxq"} 0
tor_relay_dos_total{type="conn_rejected"} 0
tor_relay_dos_total{type="concurrent_conn_rejected"} 0
tor_relay_dos_total{type="single_hop_refused"} 0
tor_relay_dos_total{type="introduce2_rejected"} 0
# HELP tor_relay_load_onionskins_total Total number of onionskins handled
# TYPE tor_relay_load_onionskins_total counter
tor_relay_load_onionskins_total{type="tap",action="processed"} 0
tor_relay_load_onionskins_total{type="tap",action="dropped"} 0
tor_relay_load_onionskins_total{type="fast",action="processed"} 0
tor_relay_load_onionskins_total{type="fast",action="dropped"} 0
tor_relay_load_onionskins_total{type="ntor",action="processed"} 0
tor_relay_load_onionskins_total{type="ntor",action="dropped"} 0
tor_relay_load_onionskins_total{type="ntor_v3",action="processed"} 0
tor_relay_load_onionskins_total{type="ntor_v3",action="dropped"} 0
# HELP tor_relay_exit_dns_query_total Total number of DNS queries done by this relay
# TYPE tor_relay_exit_dns_query_total counter
tor_relay_exit_dns_query_total 0
# HELP tor_relay_exit_dns_error_total Total number of DNS errors encountered by this relay
# TYPE tor_relay_exit_dns_error_total counter
tor_relay_exit_dns_error_total{reason="success"} 0
tor_relay_exit_dns_error_total{reason="format"} 0
tor_relay_exit_dns_error_total{reason="serverfailed"} 0
tor_relay_exit_dns_error_total{reason="notexist"} 0
tor_relay_exit_dns_error_total{reason="notimpl"} 0
tor_relay_exit_dns_error_total{reason="refused"} 0
tor_relay_exit_dns_error_total{reason="truncated"} 0
tor_relay_exit_dns_error_total{reason="unknown"} 0
tor_relay_exit_dns_error_total{reason="tor_timeout"} 0
tor_relay_exit_dns_error_total{reason="shutdown"} 0
tor_relay_exit_dns_error_total{reason="cancel"} 0
tor_relay_exit_dns_error_total{reason="nodata"} 0
# HELP tor_relay_load_oom_bytes_total Total number of bytes the OOM has freed by subsystem
# TYPE tor_relay_load_oom_bytes_total counter
tor_relay_load_oom_bytes_total{subsys="cell"} 0
tor_relay_load_oom_bytes_total{subsys="dns"} 0
tor_relay_load_oom_bytes_total{subsys="geoip"} 0
tor_relay_load_oom_bytes_total{subsys="hsdir"} 0
# HELP tor_relay_load_socket_total Total number of sockets
# TYPE tor_relay_load_socket_total gauge
tor_relay_load_socket_total{state="opened"} 0
tor_relay_load_socket_total 0
# HELP tor_relay_load_tcp_exhaustion_total Total number of times we ran out of TCP ports
# TYPE tor_relay_load_tcp_exhaustion_total counter
tor_relay_load_tcp_exhaustion_total 0
# HELP tor_relay_load_global_rate_limit_reached_total Total number of global connection bucket limit reached
# TYPE tor_relay_load_global_rate_limit_reached_total counter
tor_relay_load_global_rate_limit_reached_total{side="read"} 0
tor_relay_load_global_rate_limit_reached_total{side="write"} 0

Let's find out what some of these lines actually mean:

tor_relay_load_onionskins_total{type="ntor",action="dropped"} 0

When a relay starts seeing "dropped", it is a CPU/RAM problem usually.

Tor is sadly single threaded except for when the "onion skins" are processed. The "onion skins" are the cryptographic work that needs to be done on the famous "onion layers" in every circuits.

When tor processes the layers we use a thread pool and outsource all of that work to that pool. It can happen that this pool starts dropping work due to memory or CPU pressure and this will trigger an overload state.

If your server is running at capacity this will likely be triggered.

tor_relay_exit_dns_error_total{...}

Any counter in the "*_dns_error_total" realm (apart from the one for successful queries) indicates a potential DNS related problem. However, we realized during the 0.4.7 release cycle that DNS errors are way too noisy and contain too many false positives to be useful for overload reporting purposes. We therefore don't use them anymore for that purpose starting with 0.4.6.9 and 0.4.7.4-alpha. However, we still keep DNS metrics around to give the relay operator insight into what is going on with their relay.

DNS timeout issues and errors only apply to Exit nodes.

tor_relay_load_oom_bytes_total{...}

An Out-Of-Memory invocation indicates a RAM problem. The relay might need more RAM or it is leaking memory. If you noticed that the tor process is leaking memory, please report the issue either via Tor gitLab or sending an email to the tor-relays mailing list.

Tor has its own OOM handler and it is invoked when 75%, of the total memory tor thinks is available, is reached. Thus, let's say tor thinks it can use 2GB in total then at 1.5GB of memory usage, it will start freeing memory. That is considered an overload state.

To estimate the amount of memory it has available, when tor starts, it will use MaxMemInQueues or, if not set, will look at the total RAM available on the system and apply this algorithm:

    if RAM >= 8GB {
      memory = RAM * 40%
    } else {
      memory = RAM * 75%
    }
    /* Capped. */
    memory = min(memory, 8GB) -> [8GB on 64bit and 2GB on 32bit)
    /* Minimum value. */
    memory = max(250MB, memory)

To avoid an overloaded state we recommend to run a relay above 2GB of RAM on 64bit. 4GB is advised, although of course it doesn't hurt to add more RAM if you can. Note: If you are running a powerful server with lots of RAM then you might end up in an overloaded state due to the default queue size limit of 8GB even though you still have plenty of RAM unused. Add an appropriate MaxMemInQueues entry to your torrc configuration in that case.

One might notice that tor could be called by the OS OOM handler itself. Because tor takes the total memory on the system when it starts, if the overall system has many other applications running using RAM, it ends up eating too much memory. In this case the OS could OOM tor, without tor even noticing memory pressure.

tor_relay_load_socket_total

If the number of opened sockets is close to or the same as total sockets available then this indicates the relay is running out of sockets. The solution is to increase ulimit -n for the tor process.

tor_relay_load_tcp_exhaustion_total

These lines indicate the relay is running out of TCP ports.

Try to tune sysctl as described above.

tor_relay_load_global_rate_limit_reached_total

If this counter is incremented by some noticeable value over a short period of time, the relay is congested. It is likely being used as a Guard by a big onion service or for an ongoing DDoS on the network.

If your relay is still overloaded and you don't know why, please get in touch with [email protected]. You can encrypt your email using network-report OpenPGP key.