Snowflake bridge does not work in China since days ago

solids · May 13, 2023, 9:32pm

wrong rusults

The block looks similar to prior block to github.com and store.steampowered.com: multiple times of TLS client hello containing the specific servername (in this case it’s cdn.sstatic.net) will trigger a temporary packet dropping to the IP’s 443 port for 3 minutes.

The difference is that block to cdn.sstatic.net only happens on Fastly CDN IPs, instead of all foreign IPs of China.

Fastly CDN IP range: Public IP List | Fastly Developer Hub

OK, the problem is not on cdn.sstatic.net domain. Mysterious.

See also: Confirmed block of default Snowflake in China · Issue #249 · net4people/bbs · GitHub

solids · May 14, 2023, 9:41am

I built the Snowflake client manually and run it, got Tor logs below, the connection hangs on 10% forever.

May 14 13:50:54.728 [notice] Tor 0.4.7.13 running on Darwin with Libevent 2.1.12-stable, OpenSSL 1.1.1t, Zlib 1.2.11, Liblzma N/A, Libzstd N/A and Unknown N/A as libc.
May 14 13:50:54.728 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
May 14 13:50:54.728 [notice] Read configuration file "/Users/v/git/snowflake/client/torrc".
May 14 13:50:54.730 [warn] Path for DataDirectory (datadir) is relative and will resolve to /Users/v/git/snowflake/client/datadir. Is this what you wanted?
May 14 13:50:54.730 [notice] Opening Socks listener on 127.0.0.1:0
May 14 13:50:54.731 [notice] Socks listener listening on port 55209.
May 14 13:50:54.731 [notice] Opened Socks listener connection (ready) on 127.0.0.1:55209
May 14 13:50:54.000 [warn] Your log may contain sensitive information - you disabled SafeLogging. Don't log unless it serves an important reason. Overwrite the log afterwards.
May 14 13:50:54.000 [warn] Cannot find maximum file descriptor, assuming: 256
May 14 13:50:54.000 [notice] Parsing GEOIP IPv4 file /usr/local/Cellar/tor/0.4.7.13/share/tor/geoip.
May 14 13:50:54.000 [notice] Parsing GEOIP IPv6 file /usr/local/Cellar/tor/0.4.7.13/share/tor/geoip6.
May 14 13:50:54.000 [notice] Bootstrapped 0% (starting): Starting
May 14 13:50:55.000 [notice] Starting with guard context "bridges"
May 14 13:50:55.000 [notice] Delaying directory fetches: No running bridges
May 14 13:50:55.000 [notice] new bridge descriptor 'flakey10' (cached): $2B280B23E1107BB62ABFC40DDCC8824814F80A72~flakey10 [1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko] at 192.0.2.3
May 14 13:50:55.000 [notice] new bridge descriptor 'crusty6' (cached): $8838024498816A039FCBBAB14E6F40A0843051FA~crusty6 [tO9nYvNCAdAh9lPoEEv2pZ9BJq+YzmPAMY6pxoFrLuk] at 192.0.2.4
May 14 13:50:56.000 [notice] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
May 14 13:50:56.000 [notice] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
May 14 13:50:56.000 [notice] Bootstrapped 10% (conn_done): Connected to a relay
May 14 13:50:57.000 [notice] Managed proxy "./client": offer created
May 14 13:50:57.000 [notice] Managed proxy "./client": offer created
cMay 14 13:52:12.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:52:12.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:52:13.000 [notice] Managed proxy "./client": offer created
May 14 13:52:13.000 [notice] Managed proxy "./client": offer created
May 14 13:52:57.000 [notice] Delaying directory fetches: No running bridges
May 14 13:53:28.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:53:28.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:53:28.000 [notice] Managed proxy "./client": offer created
May 14 13:53:29.000 [notice] Managed proxy "./client": offer created
May 14 13:54:43.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:54:44.000 [notice] Managed proxy "./client": offer created
May 14 13:54:44.000 [notice] Managed proxy "./client": broker failure dial tcp [scrubbed]: connect: operation timed out
May 14 13:54:44.000 [notice] Managed proxy "./client": offer created

torrc file content is as below.

UseBridges 1
DataDirectory datadir

ClientTransportPlugin snowflake exec ./client -log snowflake.log

Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=fastly.jsdelivr.net ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn

Bridge snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=fastly.jsdelivr.net ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.net:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn

SocksPort auto

SafeLogging 0

(I changed the front URL to fastly.jsdelivr.net, other than that there are no other changes. )

The connection to snowflake-broker.torproject.net.global.prod.fastly.net is always going through during the connection process, and Tor never tries to connect to this domain name, so it is certain that the problem occurs before connecting to the CDN server .

Below is logs of Snowflake client. It shows something wrong with STUN server and it can’t determine a correct NAT type(my NAT is in fact restricted), don’t know if it’s relevant.

2023/05/14 07:50:54 snowflake-client 2.5.1
2023/05/14 07:50:54 Started SOCKS listener at [scrubbed].
2023/05/14 07:50:56 SOCKS accepted: {[scrubbed] fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72;url=https://snowflake-broker.torproject.net.global.prod.fastly.net/;front=fastly.jsdelivr.net;ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:347 8,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478;utls-imitate=hellorandomizedalpn map[fingerprint:[2B280B23E1107BB62ABFC40DDCC8824814F80A72] front:[fastly.jsdelivr.net] ice:[stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478] url:[https://snowflake-broker.torproject.net.global.prod.fastly.net/] utls-imitate:[hellorandomizedalpn]]}
2023/05/14 07:50:56 


 --- Starting Snowflake Client ---
2023/05/14 07:50:56 SOCKS accepted: {[scrubbed] fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA;url=https://snowflake-broker.torproject.net.global.prod.fastly.net/;front=fastly.jsdelivr.net;ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:347 8,stun:stun.epygi.com:3478,stun:stun.sonetel.net:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478;utls-imitate=hellorandomizedalpn map[fingerprint:[8838024498816A039FCBBAB14E6F40A0843051FA] front:[fastly.jsdelivr.net] ice:[stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.net:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478] url:[https://snowflake-broker.torproject.net.global.prod.fastly.net/] utls-imitate:[hellorandomizedalpn]]}
2023/05/14 07:50:56 


 --- Starting Snowflake Client ---
2023/05/14 07:50:56 Using ICE servers:
2023/05/14 07:50:56 url: stun:stun.voipgate.com:3478
2023/05/14 07:50:56 url: stun:stun.antisip.com:3478
2023/05/14 07:50:56 url: stun:stun.sonetel.com:3478
2023/05/14 07:50:56 url: stun:stun.uls.co.za:3478
2023/05/14 07:50:56 url: stun:stun.dus.net:3478
2023/05/14 07:50:56 Using ICE servers:
2023/05/14 07:50:56 url: stun:stun.l.google.com:19302
2023/05/14 07:50:56 Rendezvous using Broker at: https://snowflake-broker.torproject.net.global.prod.fastly.net/
2023/05/14 07:50:56 Domain fronting using: fastly.jsdelivr.net
2023/05/14 07:50:56 url: stun:stun.uls.co.za:3478
2023/05/14 07:50:56 url: stun:stun.sonetel.net:3478
2023/05/14 07:50:56 url: stun:stun.voipgate.com:3478
2023/05/14 07:50:56 url: stun:stun.epygi.com:3478
2023/05/14 07:50:56 ---- SnowflakeConn: begin collecting snowflakes ---
2023/05/14 07:50:56 ---- SnowflakeConn: starting a new session ---
2023/05/14 07:50:56 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:50:56 snowflake-1e28a4abb953337e  connecting...
2023/05/14 07:50:56 Rendezvous using Broker at: https://snowflake-broker.torproject.net.global.prod.fastly.net/
2023/05/14 07:50:56 Domain fronting using: fastly.jsdelivr.net
2023/05/14 07:50:56 redialing on same connection
2023/05/14 07:50:56 ---- SnowflakeConn: begin stream 3 ---
2023/05/14 07:50:56 ---- SnowflakeConn: begin collecting snowflakes ---
2023/05/14 07:50:56 ---- SnowflakeConn: starting a new session ---
2023/05/14 07:50:56 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:50:56 redialing on same connection
2023/05/14 07:50:56 ---- SnowflakeConn: begin stream 3 ---
2023/05/14 07:50:56 WebRTC: DataChannel created.
2023/05/14 07:50:56 snowflake-e07c068518ebf35e  connecting...
2023/05/14 07:50:56 WebRTC: DataChannel created.
2023/05/14 07:50:56 WebRTC: Created offer
2023/05/14 07:50:56 WebRTC: Set local description
2023/05/14 07:50:56 WebRTC: Created offer
2023/05/14 07:50:56 WebRTC: Set local description
2023/05/14 07:50:56 Warning: NAT checking failed for server at stun.l.google.com:19302: NAT discovery feature not supported: attribute not found
2023/05/14 07:50:56 NAT Type: unrestricted
2023/05/14 07:50:57 NAT Type: unrestricted
2023/05/14 07:50:57 WebRTC: PeerConnection created.
2023/05/14 07:50:57 WebRTC: PeerConnection created.
2023/05/14 07:50:57 Negotiating via HTTP rendezvous...
2023/05/14 07:50:57 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:50:57 Front URL:   fastly.jsdelivr.net
2023/05/14 07:50:57 Negotiating via HTTP rendezvous...
2023/05/14 07:50:57 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:50:57 Front URL:   fastly.jsdelivr.net
2023/05/14 07:52:12 WebRTC: closing DataChannel
2023/05/14 07:52:12 WebRTC: closing PeerConnection
2023/05/14 07:52:12 WebRTC: Closing
2023/05/14 07:52:12 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:52:12 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:52:12 snowflake-2f2c64911c525dc3  connecting...
2023/05/14 07:52:12 WebRTC: closing DataChannel
2023/05/14 07:52:12 WebRTC: closing PeerConnection
2023/05/14 07:52:12 WebRTC: DataChannel created.
2023/05/14 07:52:12 WebRTC: Closing
2023/05/14 07:52:12 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:52:12 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:52:12 snowflake-26a33bf07094535f  connecting...
2023/05/14 07:52:12 WebRTC: DataChannel created.
2023/05/14 07:52:12 WebRTC: Created offer
2023/05/14 07:52:12 WebRTC: Set local description
2023/05/14 07:52:12 WebRTC: Created offer
2023/05/14 07:52:12 WebRTC: Set local description
2023/05/14 07:52:13 WebRTC: PeerConnection created.
2023/05/14 07:52:13 Negotiating via HTTP rendezvous...
2023/05/14 07:52:13 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:52:13 Front URL:   fastly.jsdelivr.net
2023/05/14 07:52:13 WebRTC: PeerConnection created.
2023/05/14 07:52:13 Negotiating via HTTP rendezvous...
2023/05/14 07:52:13 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:52:13 Front URL:   fastly.jsdelivr.net
2023/05/14 07:53:28 WebRTC: closing DataChannel
2023/05/14 07:53:28 WebRTC: closing PeerConnection
2023/05/14 07:53:28 WebRTC: Closing
2023/05/14 07:53:28 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:53:28 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:53:28 snowflake-3b1122f32eb4bbfd  connecting...
2023/05/14 07:53:28 WebRTC: DataChannel created.
2023/05/14 07:53:28 WebRTC: Created offer
2023/05/14 07:53:28 WebRTC: Set local description
2023/05/14 07:53:28 WebRTC: closing DataChannel
2023/05/14 07:53:28 WebRTC: closing PeerConnection
2023/05/14 07:53:28 WebRTC: Closing
2023/05/14 07:53:28 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:53:28 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:53:28 snowflake-0c50c7674065d54b  connecting...
2023/05/14 07:53:28 WebRTC: DataChannel created.
2023/05/14 07:53:28 WebRTC: Created offer
2023/05/14 07:53:28 WebRTC: Set local description
2023/05/14 07:53:28 WebRTC: PeerConnection created.
2023/05/14 07:53:28 Negotiating via HTTP rendezvous...
2023/05/14 07:53:28 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:53:28 Front URL:   fastly.jsdelivr.net
2023/05/14 07:53:29 WebRTC: PeerConnection created.
2023/05/14 07:53:29 Negotiating via HTTP rendezvous...
2023/05/14 07:53:29 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:53:29 Front URL:   fastly.jsdelivr.net
2023/05/14 07:54:43 WebRTC: closing DataChannel
2023/05/14 07:54:43 WebRTC: closing PeerConnection
2023/05/14 07:54:43 WebRTC: Closing
2023/05/14 07:54:43 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:54:43 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:54:43 snowflake-910d8e086ff7f92e  connecting...
2023/05/14 07:54:43 WebRTC: DataChannel created.
2023/05/14 07:54:43 WebRTC: Created offer
2023/05/14 07:54:43 WebRTC: Set local description
2023/05/14 07:54:44 WebRTC: PeerConnection created.
2023/05/14 07:54:44 Negotiating via HTTP rendezvous...
2023/05/14 07:54:44 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:54:44 Front URL:   fastly.jsdelivr.net
2023/05/14 07:54:44 WebRTC: closing DataChannel
2023/05/14 07:54:44 WebRTC: closing PeerConnection
2023/05/14 07:54:44 WebRTC: Closing
2023/05/14 07:54:44 WebRTC: dial tcp [scrubbed]: connect: operation timed out  Retrying...
2023/05/14 07:54:44 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/05/14 07:54:44 snowflake-dc4a3c077d7948be  connecting...
2023/05/14 07:54:44 WebRTC: DataChannel created.
2023/05/14 07:54:44 WebRTC: Created offer
2023/05/14 07:54:44 WebRTC: Set local description
2023/05/14 07:54:44 WebRTC: PeerConnection created.
2023/05/14 07:54:44 Negotiating via HTTP rendezvous...
2023/05/14 07:54:44 Target URL:  snowflake-broker.torproject.net.global.prod.fastly.net
2023/05/14 07:54:44 Front URL:   fastly.jsdelivr.net

Up to now, I still can’t figure out where the problem is.

dcf · May 14, 2023, 10:51pm

If it is not rendezvous TLS, and it is not STUN, then it may be DTLS fingerprinting, as it was in Russia in 2021 and 2022. You may be able to test various UDP payloads to discover a matching rule. Some past examples:

“All right, finally managed to circumvent the censorship.
Russian DPI check supported_groups extension in ServerHello payload (byte 0x5a in udp packet).
It looks for DTLS packet header “magic” “16 FE FD” and then looks for “1D 00 17 00 18” at 0x5a offset.”
“The filter is applied for both DTLS v1.0 in record (16 fe ff), which is used by both Chrome and Firefox in ClientHello, and DTLS v1.2 in record (16 fe fd), which is used at least by Jitsi Meet, all at offset 0. Then, 1D 00 17 00 18 at 0x6f offset is checked. This is indeed different than the previous offset 0x5a (which is also blocked).”
" They removed the blocking of Client Hello, now they block Hello Verify Request"

You can see some past brainstorming about potential distinguishers at Make Snowflake's DTLS fingerprint more similar to popular WebRTC implementations (#40014) · Issues · The Tor Project / Anti-censorship / Pluggable Transports / Snowflake · GitLab.

solids · May 15, 2023, 9:45am

Probably both STUN and DTLS are targeted.

Snowflake logs showed wrong NAT type and there’re NAT checking failed messages.
Another tool I used to check NAT type before: SagerNet failed to check, too. As below screenshot.

IMG_20230515_050239838×1094 25.5 KB

But since STUN does not use TCP, I have no idea how to test them, or how many of them are blocked, how to find usable STUN server etc.

solids · May 15, 2023, 9:45am

Problem solved.

It’s just the IP of front domain cdn.sstatic.net (Fastly CDN) is blocked in my area. I changed it to mirrors.fastly.net and it works fine.

solids · May 15, 2023, 9:45am

See my this comment for more information.

WofWca · May 15, 2023, 10:04am

Idk, to me it clearly looks like an issue with connecting to the broker. Here’s the part where Negotiating via HTTP rendezvous... is printed. This very function is supposed to also print HTTP rendezvous response: ..., but it doesn’t. I believe there’s nothing stopping the Snowflake client from trying to ask for a client even if NAT check fails.

Now, about the NAT check. Looks like it’s trying each ICE server that your provided in the config until it succeeds (see this). So, for you it failed for stun.l.google.com:19302 and succeeded for the next one (apparently stun:stun.antisip.com:3478), which determined that your NAT is unrestricted.

In summary, to me it looks that STUN works and broker connection does not, and we don’t know about DTLS because your client never got to that point because signaling never finished.

Just as I finished typing…

dcf · May 15, 2023, 6:26pm

But your BBS comment says:

Two repeated HTTPS request with any same SNI (even including www.gov.cn) to that IP will result in a temporary ban of that IP, even the ICMP won’t go through, requests without SNI won’t trigger that.

The idea that the firewall uses two consecutive equal SNIs to the same Fastly IP address is plausible. Why would an SNI of mirrors.fastly.net work, if www.gov.cn does not work? Did you use only 1 bridge in this test, not 2?

Thanks, I think that’s correct.

dcf · May 15, 2023, 7:34pm

I’ve opened an issue in the censorship-analysis project.

system · February 9, 2024, 3:54pm

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.