2 relay per IP limit / how to use more bandwidth

Support Relay Operator
1

Is there any way to get around the 2 relay per IP limit? For context, I have a symmetric 3 gig business fiber connection which I have been running a middle relay for ~4 years. Unfortunately, it has never used more than ~10Mbit/s, leaving my connection with 2900+ Mbit idle most of the time (I can benchmark the full 3 gig simultaneously so I’m not running into a connection limit). I started up extra instances but discovered the 2 node per IP limit, and I can’t even offer to pay my ISP for extra IPs.

Any suggestions on how to use more bandwidth? I was hoping to spin up 10-15 nodes to hopefully achieve 80-100Mbit but the IP limit obviously is stopping that. I was previously hoping that my single node would grow to use closer to 100Mbit on its own but after 3 years that has not happened.

1 Like
2

What do you run the relay on? Fingerprint?
It depends on your CPU but modern machines can use much more than 10Mbit/s with a single IP and double that with a second relay.

There is no workaround as the limit is designed to protect the network against some attacks. But you should be able to use much more bandwidth even with the current limit.

3

(Main long running relay fingerprint B5053ABFF845C96B1DD8F45DCF32E6BE1E63F127 - recently it’s declined as I’ve had some outages and hardware failure but even in the 2ish uninterrupted years of operation it still stagnated)

4

I’m running in a 2 core proxmox vm, the host of which has an E3-1280 V2 @ 3.60GHz. CPU usage is normally around 25% and ram usage is about 2 gigs out of 4 gigs for the vm out of the host’s 32. It doesn’t seem like the VM is resource bound at all. My firewall is a Dell R210 II running opnsense on bare metal.

5

Your are currently not using 10Mbit/s but closer to 100 Mbit/s with the two relays (bits vs bytes confusion).
https://metrics.torproject.org/rs.html#details/A6D56D2919598C9B81CFDB7FA0434216E6EA308B
https://metrics.torproject.org/rs.html#details/B5053ABFF845C96B1DD8F45DCF32E6BE1E63F127

Check from inside the VM that you can use 1Gbit/s towards the internet and are not limited to 100Mbit for example by the virtual network interface or a bridge. Changing the virtual network driver in Proxmox for the VM is another thing to try. Two cores in the VM and that CPU should be good for more traffic.

1 Like
6

does the notices.log indicate some kind of limiting factor?

an E3-1220 V2 @ 3.10Ghz (without proxmox involved) can push around 45 MByte/s in both directions

did you set a maximum RelayBandwidthRate in torrc?

7

The network type on the VM is virtIO and I can speed test at 900/800 (haven’t added 10gig to this host yet). I can tell you for sure that the actual BW used as shown by opnsense, which is “bps” is less than 10Mbit/s normally. Maybe the metrics show a peak of higher, the second relay is new so I haven’t been taking it into account since I haven’t seen what real throughput it will add.

@atari I have no RelayBandwidthRate in my torrc, I think I previously tried to set it high (>1Gig) but I just removed it eventually.

8

As far as I understand, no.

Yes, but this is a long answer so bear with me.

I have asked a very similar question in the last tor relay operator meetup a short while ago. Spinning up high-bandwidth nodes outside Europe (especially in Asia and on the US West Coast) appears to be a challenge. No matter how much computing power and/or bandwidth I threw at the problem, the results were underwhelming.

The response I got is that it has to do with network latency. A large portion of the network is located in Europe with the 10 largest AS by Consensus Weight making up just over 50% alone all being in Europe (source: OrNetStats | OrNetStats ). Even if you have tons of bandwidth, it will be difficult to to get good utilization if you are “far away” from a majority of the network.

Why is that the case? Because of the way the network measures it’s relays. Long story short: It’s old and assuming of the way the internet was 10+ years ago, but it is working. I understand that there are some improvements on the way, but it will take time (years).

How did I deal with it? I first tried the same as you - more relays per host. I got a /28 IPv4 address block for each server, ran 2 relays per IP and got “decent” overall utilization across all of them. Not great, not terrible.

What has worked much better for me is to run the relays on a fast host in Europe for a couple of weeks, getting a high consensus and advertised bandwidth rate while doing so, and then transferring them back to my server in the US. If the US server is able to handle the increased load, it will pretty much stay there.

Example (server has a 10 Gbit/s link):

 eno1  /  daily

          day        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2021-12-12     3.07 TiB |    3.10 TiB |    6.17 TiB |  627.84 Mbit/s
     2021-12-13     3.23 TiB |    3.28 TiB |    6.52 TiB |  663.39 Mbit/s
     2021-12-14     4.03 TiB |    4.08 TiB |    8.11 TiB |  825.78 Mbit/s
     2021-12-15     4.23 TiB |    4.29 TiB |    8.52 TiB |  867.33 Mbit/s
     2021-12-16     3.04 TiB |    3.10 TiB |    6.13 TiB |  624.52 Mbit/s
     2021-12-17     3.07 TiB |    3.11 TiB |    6.18 TiB |  629.40 Mbit/s
     2021-12-18     3.02 TiB |    3.05 TiB |    6.07 TiB |  618.05 Mbit/s
     2021-12-19     3.32 TiB |    3.34 TiB |    6.65 TiB |  677.30 Mbit/s
     2021-12-20     3.56 TiB |    3.58 TiB |    7.14 TiB |  727.36 Mbit/s
     2021-12-21     3.41 TiB |    3.43 TiB |    6.85 TiB |  697.23 Mbit/s
     2021-12-22     4.16 TiB |    4.18 TiB |    8.34 TiB |  848.84 Mbit/s
     2021-12-23     6.67 TiB |    6.71 TiB |   13.38 TiB |    1.36 Gbit/s
     2021-12-24     6.51 TiB |    6.55 TiB |   13.06 TiB |    1.33 Gbit/s
     2021-12-25     6.74 TiB |    6.78 TiB |   13.52 TiB |    1.38 Gbit/s
     2021-12-26     8.35 TiB |    8.41 TiB |   16.76 TiB |    1.71 Gbit/s
     2021-12-27    10.01 TiB |   10.11 TiB |   20.12 TiB |    2.05 Gbit/s
     2021-12-28     9.77 TiB |    9.87 TiB |   19.64 TiB |    2.00 Gbit/s
     2021-12-29    10.59 TiB |   10.67 TiB |   21.26 TiB |    2.16 Gbit/s
     2021-12-30    11.36 TiB |   11.39 TiB |   22.75 TiB |    2.32 Gbit/s
     2021-12-31    10.97 TiB |   11.05 TiB |   22.02 TiB |    2.24 Gbit/s
     2022-01-01    12.45 TiB |   12.56 TiB |   25.01 TiB |    2.55 Gbit/s
     2022-01-02    13.08 TiB |   13.21 TiB |   26.29 TiB |    2.68 Gbit/s
     2022-01-03    13.06 TiB |   13.19 TiB |   26.25 TiB |    2.67 Gbit/s
     2022-01-04    12.72 TiB |   12.86 TiB |   25.58 TiB |    2.60 Gbit/s
     2022-01-05    13.25 TiB |   13.40 TiB |   26.65 TiB |    2.71 Gbit/s
     2022-01-06    12.34 TiB |   12.47 TiB |   24.80 TiB |    2.53 Gbit/s

=> The steady-state I could achieve by running a ton of relays on one US-based machine was somewhere around 6 to 8 TB per day.
=> On Dec 22nd and on Dec 26th I transferred some high-bandwidth fingerprints from my European hosts to this server. Each time, I could see a big jump in utilization. First it doubled to around 13 TB /day and then it jumped the 2nd time to over 20 TB/day. Since the server can handle the load, it stayed.

If you have only one public IP and transfer two high-consensus & bandwidth relays, I’d expect you’d be able to push up to 1.0 Gbit/s - assuming everything else checks out. At that point, you will probably be limited by the single-core performance of your Xeon CPU. Tor is mostly single-threaded.

4 Likes
9

Thanks for that great info - that sounds like it could be related here so I may try transferring my newer instance to europe and back, and if that helps then I’ll do my main instance. What hoster did you use for this?

10

I have various and anything with a good network in Western Europe and fast-enough cores will do. The most cost-effective solution appears to be Hetzner Cloud though. It works well, is rather cheap, billed per hour, has 10 Gbit/s networking and you can choose AMD Epyc cores. Just select one of their Germany locations and you are good to go.

11

Old archived list for low cost dedicated servers in Europe:
https://web.archive.org/web/20160329121849/https://wiki.nyr.es/servidores_dedicados_low_cost

1 Like