[tor-relays] Overload (dropped ntor) due to DDoS??

Hi All

I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet)
I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB

Usually it takes less than 1 CPU core, and like 1 GB of RAM.
But recently my relay is foten shown as obverloaded.
I have these LOG entries:
Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%

Is this due to DDoS attacks or a misconfigration on my side?
Is there something that I can do to aleviate this issue?

CU, Ricsi

ยทยทยท

_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Richard Menedetter wrote:

Hi All

I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet)
I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB

Thanks for running a relay!

didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?

Usually it takes less than 1 CPU core, and like 1 GB of RAM.
But recently my relay is foten shown as obverloaded.
I have these LOG entries:
Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%

You are not the only one, it's an ongoing DoS attack on the network, targeting onion services.

Is this due to DDoS attacks or a misconfigration on my side?

Besides the question above about RelayBandwidthRate I don't see anything wrong.

Is there something that I can do to aleviate this issue?

Nope, there is nothing you can do, unfortunately. Tor has some defenses against DoS and will blacklist / mark the abusing addresses, etc. as much as it can. But as you know DoS is a never ending battle, usually won by having "larger pipe", and it's something hard to tickle in an environment where anonymity is the grounding law.

What you can do is maintain your relay up and running in good shape with the latest version of Tor until this "attack" gets through. As I said, I guess most of relays are getting this at present times. The DoS "attack" is not targeted at your relay, what you are seeing is just a side effect of someone creating large amounts of circuits (heavy usage of Tor) which is reflected network-wide anyways.

ยทยทยท

CU, Ricsi

1 Like

Richard Menedetter wrote:

> I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB
> RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2,

Why? Do you have other services on the server? Otherwise, omit num CPU. Let
the tor daemon use all CPU's for crypto stuff.

> relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB,
> maxmeminqueues 3GB

Thanks for running a relay!

didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?

>
> Usually it takes less than 1 CPU core, and like 1 GB of RAM.
> But recently my relay is foten shown as obverloaded.
> I have these LOG entries:
> Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is
> above threshold of 0.5000%

You are not the only one, it's an ongoing DoS attack on the network,
targeting onion services.

>
> Is this due to DDoS attacks or a misconfigration on my side?

Besides the question above about RelayBandwidthRate I don't see anything
wrong.

> Is there something that I can do to aleviate this issue?

Nope, there is nothing you can do, unfortunately. Tor has some defenses
against DoS and will blacklist / mark the abusing addresses, etc. as
much as it can. But as you know DoS is a never ending battle, usually
won by having "larger pipe", and it's something hard to tickle in an
environment where anonymity is the grounding law.

What you can do is maintain your relay up and running in good shape with
the latest version of Tor until this "attack" gets through. As I said, I
guess most of relays are getting this at present times. The DoS "attack"
is not targeted at your relay, what you are seeing is just a side effect
of someone creating large amounts of circuits (heavy usage of Tor) which
is reflected network-wide anyways.

Sometimes 100.000-1.000.000 connections from one IP!
I block the worst with 2 nftables egress rules.

toralf has developed some smarter ddos scripts:

ยทยทยท

On Friday, August 5, 2022 1:11:27 AM CEST s7r wrote:

--
โ•ฐ_โ•ฏ Ciao Marco!

Debian GNU/Linux

It's free software and it gives you freedom!

1 Like

Hi

Thanx for the explanation.
I have 0.4.7.8 and try to run the latest version.

So it seems the overload is entirely due to the DDoS and not my config.
I have removed the maxadvertised bandwidth limit, it will now again send the measured value instead of being limited to 10 MB.

I have these limits:
RelayBandwidthRate 15 MB
RelayBandwidthBurst 30 MB
BandwidthRate 50 MB
NumCPUs 2
MaxMemInQueues 3072 MB

CU, Ricsi

ยทยทยท

Gesendet: Freitag, 05. August 2022 um 01:11 Uhr
Von: "s7r" <s7r@sky-ip.org>
An: tor-relays@lists.torproject.org
Betreff: Re: [tor-relays] Overload (dropped ntor) due to DDoS??

Richard Menedetter wrote:
> Hi All
>
> I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet)
> I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB

Thanks for running a relay!

didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?

>
> Usually it takes less than 1 CPU core, and like 1 GB of RAM.
> But recently my relay is foten shown as obverloaded.
> I have these LOG entries:
> Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%

You are not the only one, it's an ongoing DoS attack on the network,
targeting onion services.

>
> Is this due to DDoS attacks or a misconfigration on my side?

Besides the question above about RelayBandwidthRate I don't see anything
wrong.

> Is there something that I can do to aleviate this issue?

Nope, there is nothing you can do, unfortunately. Tor has some defenses
against DoS and will blacklist / mark the abusing addresses, etc. as
much as it can. But as you know DoS is a never ending battle, usually
won by having "larger pipe", and it's something hard to tickle in an
environment where anonymity is the grounding law.

What you can do is maintain your relay up and running in good shape with
the latest version of Tor until this "attack" gets through. As I said, I
guess most of relays are getting this at present times. The DoS "attack"
is not targeted at your relay, what you are seeing is just a side effect
of someone creating large amounts of circuits (heavy usage of Tor) which
is reflected network-wide anyways.

>
> CU, Ricsi
_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
tor-relays Info Page

_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Hi

> > I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB
> > RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2,
Why? Do you have other services on the server? Otherwise, omit num CPU. Let
the tor daemon use all CPU's for crypto stuff.

Yes, there are other services running on that host, and that are the ressources I am willing to contribute to tor.
2 EPYC cores can deal easily with the crypto, especially as they are using HW crypto acceleration.

CU, Ricsi

ยทยทยท

_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

lists@for-privacy.net wrote:

> Richard Menedetter wrote:

> > I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB
> > RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2,
Why? Do you have other services on the server? Otherwise, omit num CPU. Let
the tor daemon use all CPU's for crypto stuff.

     The OP has a EPYC chip with four cores and presumably eight CPU threads,
so he should specify at least "NumCPUs 8" as a bare minimum. "NumCPUs 16",
"NumCPUs 24", or "NumCPUs 32" would be better. These onion skin decryption
threads use hardly any actual CPU time throughout the course of a busy day, but
having many extras available for work can mean drastically lower times for a
decryption task to remain in the work queue and therefore a much greater chance
of getting through a high connection count period without dropping any due to
the high load. I've never seen an overload message like the one the OP showed,
and my system is ridiculously slower than the OP's EPYC.
     When I last started tor 41 days ago, it was started with "NumCPUs 12", up
from 10 in the previous iteration. I've since changed torrc to "NumCPUs 16"
because of the connection and circuit counts during the recent troubles, so the
next time I start it it will be with four onion skin decryptors per core. This
is on an ancient QX9650 (Core 2 Extreme quad-core, so only one CPU thread per
core) bumped up to 3.3 GHz. I hope that will be sufficient to get it through
such attacks by making my Internet connection speed, rather than my tor config,
be the weak spot. During the recent events I've seen tor's main thread take up
15% to 16% of CPU time on one core, while each of the worker threads consumed
far less than 1% apiece. FWIW, mprime runs nice'd to 20 on all four cores, as
well. Also, I normally renice tor to -6 about 15 seconds after starting it, to
keep it mostly prioritized higher than anything I might be doing interactively
and also other stuff running all the time (e.g., top(1), privoxy(1)).

> > relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB,
> > maxmeminqueues 3GB
>
> Thanks for running a relay!
>
> didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?
>
>
> >
> > Usually it takes less than 1 CPU core, and like 1 GB of RAM.

     During the couple of hours of the attack the other day my tor instance
never seemed to use more than ~2700 MB.

> > But recently my relay is foten shown as obverloaded.
> > I have these LOG entries:
> > Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is
> > above threshold of 0.5000%
>
> You are not the only one, it's an ongoing DoS attack on the network,
> targeting onion services.
>
>
> >
> > Is this due to DDoS attacks or a misconfigration on my side?
>
>
> Besides the question above about RelayBandwidthRate I don't see anything
> wrong.
>
>
> > Is there something that I can do to aleviate this issue?
>

     Again, I suggest increasing NumCPUs by several times to try to avoid the
overload message.

>
> Nope, there is nothing you can do, unfortunately. Tor has some defenses
> against DoS and will blacklist / mark the abusing addresses, etc. as
> much as it can. But as you know DoS is a never ending battle, usually
> won by having "larger pipe", and it's something hard to tickle in an
> environment where anonymity is the grounding law.
>
> What you can do is maintain your relay up and running in good shape with
> the latest version of Tor until this "attack" gets through. As I said, I
> guess most of relays are getting this at present times. The DoS "attack"
> is not targeted at your relay, what you are seeing is just a side effect
> of someone creating large amounts of circuits (heavy usage of Tor) which
> is reflected network-wide anyways.
Sometimes 100.000-1.000.000 connections from one IP!
I block the worst with 2 nftables egress rules.

toralf has developed some smarter ddos scripts:
GitHub - toralf/torutils: Few tools for a Tor relay.

                                  Scott Bennett, Comm. ASMELG, CFIAG

ยทยทยท

On Friday, August 5, 2022 1:11:27 AM CEST s7r wrote:

**********************************************************************
* Internet: bennett at sdf.org *xor* bennett at freeshell.org *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good *
* objection to the introduction of that bane of all free governments *
* -- a standing army." *
* -- Gov. John Hancock, New York Journal, 28 January 1790 *
**********************************************************************
_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays