Recently I decided to install Snowflake proxy to better understand how it works.
After looking at logs, I began to suspect that its work kind of single threaded and it is not using all available resources:
What I see is lots of “sdp offer successfully received” messages, followed by “Timed out waiting for client to open data channel” message.
The problem is that proxy make no new connections until client successfully connects or times out.
Looks like timeout duration is 25s, here is histogram from my logs:
Which means that it is not possible to have more than 144 connections per hour. Is this limit made intentionally? If yes, then why? Why not processing several sdp offers simultaneously?
From logs I see that proxy is almost always reaching this cap:
(line of 12 connections per 5 minutes is clearly seen)
(yellow is sdp offers, blue is connection successes)
I saw explanation about timeouts in different topic:
But it looks like @meskio is wrong: SDP offers are coming from clients, it is not just timeout without clients, it’s a failures.
It looks more like different frequency of (single-threaded?) polling of the same broker, than using several brokers.
Right – there is one Snowflake broker, which clients reach via domain fronting and which Snowflakes reach directly.
Different frequency of polling is half of the story. The other half is that the Snowflakes running in browser extensions have a limit of one client they can serve at once, whereas the headless Snowflake have a configurable limit of how many clients to serve in parallel. See the “-capacity” argument to the ./proxy command:
which looks like it defaults to 0:
which I believe means “no limit”.
Load balancing is handled at the broker end: snowflakes say how many clients they’re handling right now when they check in with the broker, so the broker has the opportunity to assign clients to less-loaded snowflakes. I think the load balancing approach is very simple currently.
And as a last note, you can read a big pile of wishlist items on the gitlab tickets:
Thanks for clarifications.
1 client at a time is a very strong restriction.
I think it is better to use standalone version when possible.
Yes, I know about it.
What I see with my proxy: 1. Requests are coming almost non-stop. 2. Most of them are failing with timeout.
It means that network want to put load on my proxy, but proxy don’t want to accept such load, waiting in timeouts most of the time.
Failing connections may be the problem by itself, but what prevents proxy from handling timeout in it’s own thread and keep asking for new connections at the same time? It looks like too many resources are wasted while being very needed at the same time.