[tor-project] What's happening in Shadow 2022-09

Mirrored from What's happening in Shadow 2022-09 · Discussion #2428 · shadow/shadow · GitHub :

This is part of a series of periodic updates of development in Shadow. This work is sponsored by the [NSF](NSF Sponsorship - The Shadow Simulator). Previous update: [2022-06](What's happening in Shadow 2022-06 · Discussion #2243 · shadow/shadow · GitHub).

We've merged [103 non-dependabot pull requests](https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2022-06-27..2022-09-28+-author%3Aapp%2Fdependabot+) and closed [12 issues](Issues · shadow/shadow · GitHub) since our previous update.

Release status

···

==============

We've released [Shadow 2.2.0](Release v2.2.0 · shadow/shadow · GitHub)! Notable user-facing changes, from the release notes:

* We have removed ptrace-mode, and the associated experimental options `use-o-n-waitpid-workaround` and `--interpose-method`. ptrace-mode was an alternative to Shadow's current interposition mechanism that uses `LD_PRELOAD` and `seccomp`. This change should be transparent to most users, since it hasn't been the default for several releases, and was only accessible via experimental options. See Remove --interpose-method=ptrace · Issue #1945 · shadow/shadow · GitHub

* `dup()` and related syscalls are now supported for all file descriptors

* Fixed behavior when multiple threads are blocked in `epoll_wait` on the same epoll file description. epoll_wait never delivers events to multiple waiters · Issue #2260 · shadow/shadow · GitHub

* Fixed bugs causing `timerfd_settime` to not reset the internal timer's expiration count (Ensure timerfd_settime resets expiration count by sporksmith · Pull Request #2279 · shadow/shadow · GitHub), and not cancel previously scheduled timer-fire events (timerfd: ensure previously scheduled events are cancelled when re-arming by sporksmith · Pull Request #2282 · shadow/shadow · GitHub).

* Fixed a panic when patching the VDSO in newer kernels, such as those in Ubuntu 22.04. Size of `__vdso_gettimeofday` is too small for trampoline · Issue #2273 · shadow/shadow · GitHub

* Fixed the errno returned from calling `connect()` on a unix socket. This fixes a `getaddrinfo()` test failure on some systems. Cannot pass getaddrinfo-shadow test · Issue #2286 · shadow/shadow · GitHub

* Fixed minor memory leaks. Fixed misc memory leaks by stevenengler · Pull Request #2249 · shadow/shadow · GitHub

Other user-facing improvements since last update

We also continue to make compatibility improvements:

* Added support for escaping additional "busy loops" - ones that only use `rdtsc` without making any actual syscalls. This affected some versions of [libopenblas](Failing to run Traffic Generation simulation due to unsupported system calls · Discussion #2299 · shadow/shadow · GitHub). Handle busy-loops that only use rdtsc by sporksmith · Pull Request #2314 · shadow/shadow · GitHub

* Replaced custom cmake code for locating glib with pkg-config. This makes shadow easier to compile in environments with non-standard layouts such as guix or nix. cmake: improve dependency processing by sporksmith · Pull Request #2331 · shadow/shadow · GitHub

* Fixed a bug handling files opened with `O_NOFOLLOW`. Remove O_NOFOLLOW flag when mmap file in plugin by Congyu-Liu · Pull Request #2353 · shadow/shadow · GitHub

* Fixed our documentation and CI to support mapping pages with `PROT_EXEC` when run under Docker, fixing [2400](Shadow panics when mmaping executable pages · Issue #2400 · shadow/shadow · GitHub). When using `docker run`, you should also pass the `--tmpfs /dev/shm:rw,nosuid,nodev,exec,size=1024g`. Add documentation and tests to support mmap with `PROT_EXEC` by stevenengler · Pull Request #2402 · shadow/shadow · GitHub

* Made the shadow installation directory relocatable. e.g. installing to `$HOME/opt/shadow` and later moving to `$HOME/opt/shadow-old` now behaves as expected. This also makes it easier to share and reuse pre-built shadow binaries, e.g. for CI, though we still do not yet publish pre-compiled binaries. Make shadow installation directory relocatable by sporksmith · Pull Request #2391 · shadow/shadow · GitHub

* Added support for emulating `PR_SET_DUMPABLE`, which fixes compatibility with managed programs that "harden" themselves against debuggers, including [arti](The Tor Project / Core / Arti · GitLab). Emulate PR_SET_DUMPABLE and misc improvements by sporksmith · Pull Request #2370 · shadow/shadow · GitHub

* Fixed a bug affecting getaddrinfo on some systems. Fix `getaddrinfo()` error on some systems by stevenengler · Pull Request #2292 · shadow/shadow · GitHub

* Partly implemented fcntl. Re-enable fcntl locking for regular files by sporksmith · Pull Request #2259 · shadow/shadow · GitHub

* Made failures due to running out of tmpfs space easier to understand. Pre-allocate space in shared memory files by sporksmith · Pull Request #2267 · shadow/shadow · GitHub

Other notable changes since last update

Since the 2.2.0 release, our current focus is primarily on migrating Shadow's C code to Rust. These changes should be mostly invisible to Shadow's end users, but may be of interest to folks interested in hacking on Shadow itself. Along those lines we've:

* Migrated Shadow's scheduler to Rust. This is a fairly complex and performance-sensitive component, and we're taking care to prevent performance regressions during the migration:
   * Move the scheduler's priority queue to rust by stevenengler · Pull Request #2340 · shadow/shadow · GitHub
   * Add tasks to the host instead of the worker by stevenengler · Pull Request #2343 · shadow/shadow · GitHub
   * Rename `HostSinglePolicyData` to `SchedulerPolicy` and remove vtable by stevenengler · Pull Request #2344 · shadow/shadow · GitHub
   * Simplify misc scheduler-related code by stevenengler · Pull Request #2352 · shadow/shadow · GitHub
   * Add rust synchronization primitives needed for the scheduler by stevenengler · Pull Request #2412 · shadow/shadow · GitHub
   * Rust thread-per-host scheduler by stevenengler · Pull Request #2414 · shadow/shadow · GitHub
   * Rearrange scheduler-related code, and add scheduler wrapper by stevenengler · Pull Request #2416 · shadow/shadow · GitHub

* Migrated other modules to Rust.
   * Event: Move the `Event` to rust by stevenengler · Pull Request #2327 · shadow/shadow · GitHub
   * TokenBucket: Refactor token bucket from netiface to a rust module by robgjansen · Pull Request #2306 · shadow/shadow · GitHub
   * Manager: Replace the C `Manager` with a rust version by stevenengler · Pull Request #2277 · shadow/shadow · GitHub

* Implemented a Rust mutex suitable for use in shared memory, and compatible with [rkyv](https://rkyv.org/). https://github.com/shadow/shadow/pull/2386

* Moved most of the compilation and linking of Shadow's remaining C code from cmake into Cargo build scripts. This fixes some dependency issues and will make it easier to start migrating some of our additional support libraries from C to Rust. Cargo once by sporksmith · Pull Request #2404 · shadow/shadow · GitHub

* Improved organization of global state in Rust.
   * Move some of `Controller`'s global state to the worker module by stevenengler · Pull Request #2361 · shadow/shadow · GitHub
   * Fix UB in the controller, and move the plugin error count to the worker module by stevenengler · Pull Request #2363 · shadow/shadow · GitHub
   * Move the runahead data from `Controller` to `WorkerShared` by stevenengler · Pull Request #2367 · shadow/shadow · GitHub
   * Move more global state to the worker module by stevenengler · Pull Request #2372 · shadow/shadow · GitHub
   * Improve design of rust global shared state by stevenengler · Pull Request #2373 · shadow/shadow · GitHub

* Refactored other components to simplify migration to Rust.
   * Controller, Manager, Scheduler: Changes to controller, manager, and scheduler by stevenengler · Pull Request #2255 · shadow/shadow · GitHub
   * Thread: Simplify thread by sporksmith · Pull Request #2328 · shadow/shadow · GitHub

Shadow in Tor

We've been working on using Shadow to help test and evaluate [arti](The Tor Project / Core / Arti · GitLab) - The Tor Projects's new Rust-based implementation of tor. This has driven many of the compatibility fixes above, and we are happy to report that arti now runs under Shadow! We plan to merge a shadow-based continuous integration test into arti itself within the next few days: Draft: Shadow ci (!634) · Merge requests · The Tor Project / Core / Arti · GitLab

Happy simulating!
The Shadow team

_______________________________________________
tor-project mailing list
tor-project@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project

1 Like