Updates from the Network Team, October 2021

General Discussion
1

Welcome to this first monthly update from Tor’s Network Team here on our shiny new Tor Project forum.

These updates will appear monthly and summarize what the team has been working on in the previous month. Today’s update is only from the last two weeks of October, though, as we decided on this new reporting format as an experiment at that time.

The Network Team is responsible for writing the Tor network’s software and supporting tools to test and experiment with the network. In addition to the classic Tor implementation, the team has also recently started working on a Rust implementation of Tor named Arti.

General Updates on the Team

This month, the team got a new member – a big welcome to eta, who will work on Tor’s Arti project with Nick.

Tor

The team worked on preparation for the 0.3.5.17, 0.4.5.11, 0.4.6.8, and 0.4.7.2-alpha releases of the Tor software. The team has recently rotated roles such that David and Alex will generally be trying to take care of the maintenance of tor.git such that Nick can focus more of his time on the Arti project. As part of the release preparation, backporting of patches took place, and some new patches were introduced to the respective releases to get them ready for release. This work means we were down to around 14 open Merge Requests towards Tor at the end of the release preparation.

After the releases were out, we spent time automating the processes around the preparation of the releases to reduce time spent on this in the future. The work included:

As part of the daily work on maintaining Tor, the following newsworthy updates happened:

  • David worked together with the Network Health Team on an issue where relays would incorrectly report themselves as being overloaded due to a bug in the DNS subsystem in Tor. The problem was tracked in ticket tor#40491.

  • As part of the above overload issue, the team handled another issue in how Tor reports metrics on DNS requests in tor#40490.

  • Nick wrote a couple of Guard-related proposals found in Proposal #336 and Proposal #337 based on conversations with Mike et al.

  • Roger and Cecylia from the Anti-censorship Team debugged and fixed a long-time UX issue that affects Tor’s internal Bridgestrap system and users using Tor together with bridges. The team handled this work in tor#40396 and its respective sub-tickets. Backporting is planned for these issues once the changes have been living in our main branch for a while. Next step, once that ticket is finished, is tor#40499.

Arti

The Arti project is The Tor Project’s upcoming implementation of a Tor client in the Rust programming language. This month the big goal was to release Arti version 0.0.1 and get eta up-to-speed on hacking on the Arti codebase. The team released Arti 0.0.1 ahead of schedule, and you can read the release announcement here.

Some updates for the Arti project from the team:

  • General release preparation for Arti 0.0.1.

  • The team dealt with the fallout from RUSTSEC-2020-0071 and RUSTSEC-2020-0159.

  • Added a new target-address API. See: arti#81.

  • Allow multiple Arti instances to share a state directory. See: arti#194.

  • Rename tor-client to arti-client to avoid name collision. See: arti#130.

  • Added some example code using Hyper to fetch a webpage.

  • Clean the client initialization API.

  • Improve the circuit isolation API. See: arti#73.

  • “Path-bias lite” implementation. See: arti#185.

  • wait_for overhaul and follow-ups. See: arti#149.

  • future-proofing on state serialization. See: arti#175.

  • Future-proof CLI and move to clap. See: arti#206.

  • Rework top-level documentation.

  • Lots of test improvements. Arti’s currently at 80.03% coverage.

Shadow

The Shadow team is responsible for writing and maintaining the Shadow network simulator used by the Tor research community and the different internal groups to simulate significant changes before rolling them out in the production network.

This month the team have the following updates:

  • Shadow 2.0.0 is now technically released. Rob Jansen is working on some final design documents before announcing the new release more widely. See: the Shadow Github release page.

  • Added an end-to-end CI for testing tornettools (the wrapper script to setup, run, and analyze Shadow Tor simulations) in pull request #44.

Simple Bandwidth Scanner

The Simple Bandwidth Scanner project (sbws for short) is used by the Tor network’s Directory Authorities to scan the Tor network for available bandwidth. juga is the maintainer of sbws, and they work closely with both the Network Health Team and the Network Team.

This month’s updates from the sbws project:

  • Graph transition from the former bandwidth scanner Torflow to Simple Bandwidth Scanner.

  • Follow-up on Simple Bandwidth Scanner deployment on the Directory Authority named gabelmoo.

  • Follow-up on Simple Bandwidth Scanner deployment on the Directory Authority called faravahar on whether it reports more relays around 00:00.

  • Log attempts to create 2-hop circuits exiting to the exit-local webserver.

Congestion Control Project

The Congestion Control project is a big cross-team project going on within The Tor Project right now. The Network Team’s Mike Perry leads the efforts, but with help from several other members of The Tor Project, including everybody in the Network Team, Georg from the Network Health Team, Jim from both the Network Team and the Shadow project, juga from the Simple Bandwidth Scanner project, and Hiro from the Metrics Team.

Updates on the Congest Control project this month:

  • Jim iterated on the Shadow simulation CI script. We tried using cache volume directly and later Gitlab cache API for storing “full” results, but still debugging.

  • Jim experimented with bringing up ephemeral Gitlab CI runners to run more simulations at once using Chameleon Cloud; it seems to work reasonably well.

  • Jim created a separate Gitlab project to store the final post-processed data of each simulation and the parameters; simulation runs now use a token to push to this repository automatically.

  • We have begun testing and tuning the congestion control algorithms in Shadow. Preliminary test results indicate that TOR_VEGAS with the original RTT-adjusted congestion window estimation is the most stable algorithm. Additionally, it appears that the optimal queue length parameters for TOR_VEGAS are a function of the TLS connection outbuf size and the path length of the circuit, in order to fully utilize all hops in a circuit. Further details on our simulation experiment plans and progress are documented in our Shadow simulation plan document.

  • Hiro worked on patching tornettools to generate better throughput graphs based on the last 1 MB of a 5 MB download, instead of the last 500 KB of a 1 MB download. Additionally, we split the tornettools graphs by perfclient source location (HK/DE). These changes to tornettools are helpful to get a clearer picture on the changes in steady-state throughput, and throughput as a function of latency/location.

  • We discovered that CircEWMA appears to correctly prioritize circuits in Shadow, so long as we set TOR_VEGAS queue parameters large enough to fill all of the output buffers of a path. This is a huge relief, as CircEWMA is necessary to protect TOR_VEGAS from cheaters. However, it also means that our TOR_VEGAS queue parameters will depend upon path length to ensure that CircEWMA is active and that we fully utilize the path. This will require some patches for the negotiation and Vegas parameterization.

  • Jim iterated on workflow for running simulations in Gitlab CI:

    • Complete data (too big for the one GB artifact limit) is stored in the cache (locally on the Gitlab runner’s machine). Working with Tor’s System Administration Team on the way to access that cache out of band. See: tpa/team#40476.

    • All “pipeline variables” are now exposed in UI for manually running a job, with defaults pre-filled in. There are some hard-coded “presets” for overriding several at once, e.g., a “test” preset for running a short sim to test the pipeline itself. See: sponsor-61-sims!2.

    • We now run a static web server as a non-anonymous onion service that serves the simulation directory while it’s running. This is helpful for early evaluation and verifying that things are running correctly. See sponsor-61-sims!5 and sponsor-61-sims!6.

Community

This month some members of the team joined the Community Team training on forum moderation.

Epilogue

Would you mind letting us know if you found these updates helpful? I’ll try to answer any questions people may have here on the forum in the next few days. If I can’t answer your questions, I’ll try to get somebody from the team to look.

Thanks for reading this far!

7 Likes