New to server operation

Support Relay Operator
1

As title suggests, im new to server administration. Not new to tor, or compiling source code, working with linux, and OS X unix, etc…

I am interested in getting involved with the Tor project as a community member, and relay operator.

Im running into issues with tor not being able to autodetect IPv4 addresses, I figured a work around by using a dynamic DNS updater and registering a hostname. But im still running into issues with my router dropping packets, occasionally sending RST packets and closing sockets. Things being wonky as shit. So I’ve been wanting to put it at the front of the network, to act as both a gateway to my local network, and putting my wifi router behind it, with my wireless devices behind another firewall beyond my relay. That said, what can I do to increase security and stability? Im open to switching OS’s to something different if need be, but would like to stick with OS X Snow leopard if at all possible as its stable, and my relay node is running on an old, heavily modified Mac Pro.

Im also interested in running a Dir Mirror too, I’ve got an unmetered connection, and unlimited bandwidth, and im willing to donate half or a bit less of my bandwidth for this endeavor, and for now, don’t intend on running another relay until I’ve got this one sorted, pen tested, and stable.
Im most interested in running either a guard, or middle relay, not an exit.

Server nickname, MidWorldRelay9
Fingerprint: 51A3394C59BF5E414D57722335CD1E838C6EE986

Just as a heads up, in case you (whoever) searches for this relay nick, you may find up to 3. One I did a test run on a Linode hosted in Canada, and the other was a failure in compiling or config, not sure which, so I wiped the install, erased the keys and started from scratch. Which is why MidWorldRelay9 may show 2 results. I don’t have the fingerprint for the Linode, as I didn’t want to risk potentially de-anonymizing users that had circuits through it. So I completely wiped it before taking the Linode down, and redeploying on my Mac Pro.

2

Update.
Router kept dropping packets, sometimes would close circuits it didn’t like for evidently no reason (at least one I couldn’t find and verify anyway) so I went ahead and did what I was thinking about, and put the machine at the front my network, and put my wifi behind it. This way 2 firewalls are between the relay and all my personal shit. So that’s working, and the machine is also acting as the router coming out of the modem. Seems much more stable now than it was, which is good. Progress. Now im onto the pen testing stage.

Already found and patched a few vulnerabilities accessible via WAN.

Any tips, or advice relay management, and hardening services would be highly appreciated.

3

So cool Mephisto :slight_smile: I wish I could help you with the relay management but I can barely get snowflake sorted properly at this point lol. I do however have similar aspirations to yours so I hope you don’t mind me asking what modifications you’ve made to your old Mac Pro.

At the moment I’m running the snowflake proxy remotely on an old MBP 9,2 running 10.15.7, but I do have my old Snow Leopard install disc and I’d love to dig up a desktop machine to run it on as I work my way up to a skill level sufficient enough to actually run a relay. Reliable uptime is such a huge issue with laptops running more recent OS versions it seems, so a Mac Pro running SL might solve that problem (one hopes).

Thanks. And best of luck!

4

Nice to see another Mac enthusiast, seems to be rare. Well, among the mods I’ve made to my machine, one of them is getting my hands on a copy of OS X SL server addition (wouldn’t suggest, updates are… unstable. and Active directory master/standalone break permissions. Though I did figure out a work around, more of a hack really. When booted after new update, it freezes on spinning wheel of death. Booted into verbose mode to see what the issue is, mDNS responder was getting hung on a request due to “Permission denied”
So I booted into single user mode with a root SH prompt, ran a “chmod -r 755 /” and hard reboot, that fixed it and made it want to boot. So that’s cool, though id highly suggest staying away from this particular issue as I nearly went bald trying to figure out why OSX was so unhappy.

But that aside, I built a raid 10 config with 3 500GB hard drives, and another 4tb backup drive (defcon zoz has taught us many things, in that keeping daily redundant paranoid backups will save your ass)
Flashed firmware from 1,1 to 2,1, changed instruction from 32bit EFI to 64bit x86 EFI, so much more stable with that.
Upgraded the dual socket xeon dual cores to quad cores, for a total of 8 logical cores (no multithreading, old as shit yo)

I will warn, nearly all software used on this machine is compiled from open source. Most software that’s made for this OS is old and deprecated, badly and for good reason I might add. The default SSH and remote management client is of a version that’s old enough to cause problems. So I took those offline and made firewall rules saying anyone connecting from the external (wan) interface can’t connect on those ports, and went about compiling VNC and openSSH from source to patch those old vulnerabilities.

Just to be on the safe side, I removed pretty much all baked in software aside from the terminal, sys utilities, and Xcode (gcc compiler needed for open source software) just to be sure that no pre-installed software would attempt to connect or run, or be visible as a running service. This is an effort to reduce attack surface as much as possible.

Something to keep in mind compared to a server edition of OSX as apposed to a standard edition, is the server edition has an extra (what apple refers to as) smart firewall, that dynamically assigns open sockets for devices behind the router so keep state, and streaming can happen without interruption. The regular version of OSX does not have a smart firewall, or dynamic rules the same way. So in order to accomplish this, you’d need to build a firewall from scratch using the client built into MacOS (which for SL is the openBSD firewall prior to PF) and that’s accessible via terminal command, iptables

For more info, either google the man pages for iptables, or open a terminal in macOS and type ‘man iptables’

Also put in a lot more ram, installed a dynamic DNS client (im on a home connection, so dynamic DHCP assignment from my ISP, my public facing IP changes every couple days or so) and pointed the torrc config to that dynamic hostname. For whatever reason, the auto ipv4 was non functional. Though I couldn’t begin to speculate why, I tried to debug but eh, no need when ddns for a single domain is free anyway.

So I’d say my copy of OSX is closer to a linux distro than Mac. Most of the software available as source code for linux can be compiled on macOS.

Running a tor relay node on a MacBook running SL would probably be just fine most of the time, but the reason Im using a Mac Pro instead is because of the server/workstation grade hardware being better equipped the handle what tor needs. Registered ECC memory and xeon processors are just better at those sorts of workflows compared to standard intel core and non ECC memory. But I digress, for most machines not handling a large amount of traffic a setup like this is overkill. My Mac pro is not only just a tor relay, its also the router and DHCP server for my home network, and then my wifi router plugged in behind it, running its own DHCP server and NAT. So instead of the whole network on one subnet, its on 3. The public facing comcast subnet, the internal subnet, and then finally a 3rd layer subnet just for the wifi (network segregation is important for access security)

This way my router can ask the Mac for sockets it needs, and those won’t be shared with tor or the second layer, just tunneled directly and handled all by itself. The idea (I hope anyway) is that if an attacker were to gain access to the Mac, or the node, they couldn’t dig any deeper and infect devices on the wifi side of things, at least without some serious effort.

So this is how I solved the router dropping and sending RST packets to circuits it didn’t like (im fairly sure it was a limit in the max number of concurrent connections bogging it down)
So if you intend to run a relay on the internal side of your network, running it on a MacBook would probably be fine. But if you plan to run a relay that’s going to take in any serious amount of traffic, it needs to be at the front of the network, or given a dedicated cable to prevent collisions, and general… wonkiness. But for a reasonably small operation, say less than 3k concurrent connections and maybe… 2MB/s of transfer consistently per second should run fairly stable on what your after. (A third party or a mod may want to correct or add to this if inaccurate)

That said, I don’t want to give you false info, because many of the points I’ve made have been situationally dependent to me personally, and my clusterfuck of machines.
The only way to get decent at it, is it go do it and see what happens, and fix things as you go along, and remember to pen test your shit. Nmap/zenmap is a wonderfully versatile tool, use it. Use it often and use it on everything, this nmap -sS -sV --script vuln (insert target hostname or IP here) will tell you a lot about what’s going on.

If your intention is to use macOS and only macOS for most of your stuff, you should seriously look into both macports and homebrew.
MacPorts is much more reliable and stable for older machines, like my clusteruck, and brew is great for newer machines. MacPorts includes a -s or -b switch to specify whether you want to install from precompiled binaries, or compile from source using MacPorts (which I highly suggest on older machines, as binaries will sometimes install, then break for no apparent reason), building from source, and it failing, will teach you a lot more, and it tends to work a whole lot more stably than binaries do in a lot of cases. Again, this is a point made from the perspective of a long since deprecated OS. Building from source on newer machines isn’t necessary in a lot of cases, as newer machines, OS, and hardware specs are much more compatible with binaries, old shit like my Mac (which the processors DO NOT support multithreading/hyperthreading) will have a shitfit if a program tries to spawn another thread with a modern instruction that won’t work on that old machine. At least when its compiled from source, its compiled to work on your specific host and architecture.

I hope this helps answer some of your questions, and I hope I didn’t give you a turbo aneurism with this info dump. Please let me know if you have questions, or need clarification on some points.

5

Wow. I hardly know where to start except with a big thanks! That was quite the tutorial and I appreciate you taking the time. No turbo aneurism was experienced :wink:

I managed to pick up a tricked out Mac Pro 5,1 yesterday for a decent price and your tips will be very helpful on that score (especially with repect to hardening against vulns and using iptables). Total overkill for the time being but lots of headroom for future endeavors :slight_smile:

Thanks also for the tip about MacPorts having an option to compile from source. I’ve become quite familiar with learning from failures lol! This’ll be another chance.

I do have nmap in my toolkit so thanks for the script. I expect that will come in handy too.

In case your interested in a handy GUI network analyser for Mac I’ve come to rely on WhatRoute by Bryan Christianson. There’s also Patrick Wardle’s Netiquette (Objective See). Perhaps redundant for you but useful to recommend to the less terminal savvy.

Anyhow, Thanks again!

6

Congrats, and no problem man.

Something to consider with the 5,1. Mac OS X Mojave is probably the best version of macOS for that machine, not that with a bit of effort it couldn’t run newer versions, but that Mojave was the last official version of OS X with the unix underbelly before apple rewrote a lot of the base code for macOS version 11 and onward. Some useful things like the Network Utility (which was deprecated in later versions)
You can still use network utility on every version of macOS 10.14 and prior.

Open by
COMMAND + SPACE for spotlight, and type in Network Utility
You can also find it somewhat buried in the system applications folder, and utilities subfolder. This is a prebaked utility, and its got a nice GUI and runs a lot of the really basic network commands for administration, and troubleshooting problems. Very very handy.

Be very careful about what GPU’s you try to put into your cheesegrader. Apple is still really salty with NVIDIA, and most of their cards will make your Mac very unhappy. So, as much as I hate to say it, stick with Radeon cards for it unless your going to wipe the drive and run a version of linux or something. The Radeon Vega VII is probably one of the more common cards for the 5,1 as it works and plays well with Mac OS and is a workstation grade GPU anyway.

You can also upgrade the processors in it, if you have a single socket board, 14 cores is the max I believe, and for a dual socket board, 28 cores and 128GB of ram (or 96 if you take advantage of triple channel memory)
Oh, speaking of which. The 5,1 can handle 128GB of ram max, BUT. BUT. 128 will NOT run in triple channel mode. If your going for speed and stability, you’ll want triple channel memory and that is capped at 96GB. I can find the guide for that if you need.

Also, I can’t stress this enough. BACKUP YOUR SHIT.
Keep AT LEAST one drive in either another machine on the network, or in a dedicated drive bay for JUST backups and nothing else. I promise, when compiling, and fucking with shit, and SIP and so on, eventually you’ll brick your OS and have to load from clean backups. Something I’ve discovered, is even after a processor upgrade, you don’t have to recompile source code for things to work. As long as the source code and programs your trying to run are loaded into your home directory, that is
/Users/you/Documents or something similar.

My advice is make a folder in your home directory called bin.
Then add that folder to your path, and when you compile source code by hand, use the
./configure --prefix=/Users/you/somefolderyouwant/ --bindir=/Users/you/bin/ --sbindir=/Users/you/bin
To explain, in case you (or another reader that isn’t you) is confused and yelling at me.
–bindir= moved the programs compiled USER executables to the directory indicated, and runs from there
–sbindir= moves the programs compiled ROOT executables to the directory indicated, and runs from there when the program is started with sudo nmap or whatever the name of the program is.
–prefix= moves the entire container of compiled and build code from where its default install directory would be, for macports its usually /opt/local/bin or /opt/bin and for root executables its /opt/local/sbin or /opt/sbin/

Protected directories like /usr/bin or the root directory like /bin stay untouched, and are very inconsistent to work with depending on what your doing. but generally its advisable to stay out of those directories and paths unless you know exactly what your doing.

Now this isn’t generally required by most people, and macports picks its own place to install stuff and the directory is already in your system $PATH when macports finishes installing, but to add /Users/you/bin to your path, you have to do It by hand, or use a prompt or script and navigate there before the programs run. This helps segregate things and keep them macOS from interfering to much. This is less of a problem in older version of macOS than newer versions, if I recall Mojave doesn’t have SIP, but 11.0 onward does.

Now, as a bit of a disclaimer. I didn’t change the default path for macports install directories and such, most of the time those programs executing from there default locations is 100% fine.

But for programs like tor, or nmap, or something like that and your looking at sensitive data, in the case of tor, if mishandled could result in good folks getting put up against a bulkhead and shot.

The reason I mention all this is because I compiled tor as a standalone static program running and executed from my home directory, which is encrypted by FileVault with a paranoid password, on top of the entire drive being encrypted, for the base OS. Take advantage of apples built in encryption. It works.

My primary concern at the time, and currently being that, its never impossible to break into someones server or desktop. Make whatever an attacker could potentially get they hands on totally and completely unreadable.

7

It’s a 12 core 2x2.66 processor with a Radeon 5870 GPU and 32 GB RAM. I haven’t had a chance to spin it up yet but it does have 256 gig SSD for a working drive and a 1 TB HDD for storage etc.

Yes, Backup backup backup. TM + SuperDuper + Backblaze. Lots of slots for extra drives but I’m not sure if I’ll run a RAID on this machine as it won’t be my day to day workhorse and it’s at my studio and I access remotely. I’ll just run TM on the HDD and add a drive for bootable clones with SD.

To start with I’m going to use Snow Leopard because I really miss it. It was the last stable OS in many respects and it has a couple of advantages over Mojave. One is the consol which is so much more user friendly than the unified log. The other is the lack of SIP (which Mojave does have by the way). A bit dangerous I know, but that’ll depend on what you want to do with kernel extensions etc. Of couse…File Vault and a paranoid password :slight_smile:

Thanks again for the wealth of excellent suggestions. Much appreciated!

8

You are absolutely right about the console app, that thing is amazingly flexible all things considered.
Its nice to know that Mojave does in fact have SIP, I thought it didn’t. Thank you for letting me know that.
I would like to mention though that disabling SIP is actually not that hard, and overall not that dangerous (so long as you head the advice to never fuck with the root directories without a very good, and specific reason, and only after backing up your shit twice)

Ive recently found that I pretty much had to disable it on my newer machines. Some things like torify, proxychains-ng and a few others just flatly refused to work because it had to get into the protected directories to access libraries, and other times the command would trigger, then SIP would just flat bypass protected directories entirely and the program won’t even print an output. Very frustrating.

I finally arrived at the ultimate screw it point, and just completely disabled SIP, enabled all the extra fun red buttons in developer mode, and enabled the root user account, and added my primary user to the wheel group.

Now it does what I fucking tell it to, which is glorious. Dont regret it. 10/10, would break my OS again.

If you can stick with SL and not have to mess with SIP at all, that would be most advisable.
Remember too, built in SSH and VNC clients for SL are… well, old and insecure. But new versions still work, im running the newest openssh and openvnc (forget the exact name of the VNC client at the moment) versions compiled from source, and assigned them to talk on non-default port numbers.

Like SSH likes 22, but its considered a privileged port, which I don’t know if SIP would have an effect on non-apple services running on privileged ports, so instead of finding out the hard way somethings fucky, I just assigned them high numbers and that seems to work very well and stably.

Anyway. I hope all goes well with your new Mac, let me know how it works out! And your welcome man, anytime.

9

Oh right, ssh. I use a third party VNC called Screens (Edovia) which works on SL and is quite flexible about port mapping. It’ll let assign a port number for ssh. I’ll see how that goes and keep you posted.
Cheers!

10

Thank you both @Mephistopheles @Quartermarsh :purple_heart:
for this insightful and informative communication Kind
regards Gabriel

11

Ah yes, there’s a fly in the ointment it seems. The Mac Pro has been updated all the way to High Sierra and the firmware (boot rom) update to 144.000.000 has made it excessively tricky to downgrade to SL. Tmalss I’m going to have to run SL in a VM. This conclusion reached after 7 or 8 hours of hacking around for a fix. I just don’t feel like replacing hardware to get back to SL. Now I just have to find an inexpensive/reliable VM to use. I’ve ruled out VMware fusion and Parallels desktop so far. An older version of VirtualBox might do it but…Oracle :scream: This might be a rabbit hole I should back out of.

12

:thinking:
[Time Machine backup tutorial ]
(Comment obtenir d’anciennes versions de macOS - Assistance Apple (FR))
or Save time by making
:face_with_raised_eyebrow:
an online support request.
it costs nothing :wink:

13

Oracle » Vm Virtualbox : Security Vulnerabilities 2022-07-19 2022-07-25

Vulnerabilities :wink: If the can help in your virtualizations + OS choices Have a nice day

14

Yeah. That sounds about right.

MacOS really doesn’t like trying to be jammed into a VM, you’ll probably run into a lot more problems than you want to deal with tryin to make that work. Though I can’t say that with certainty, I haven’t tried, but that’s mostly because im not that masochistic.

My honest advice is choose a version of MacOS that your machine won’t shit itself over and works without having to hack it on. 10.10 isn’t bad, and Lion 10.7 may work for you, its not SL though.

15

Though, It just occurred to me. Sometimes the Mac installer gets upity and bitchy about downgrading, or OS installations when there is an OS present on the bare metal.

You might have some luck backing up your shit, and then loading a disk utility to erase the entire drive. Reboot twice, clear VRam, and then see if it’ll take SL. No promise itll work though.

16

Yes, uppity indeed. In this case I wonder if it’s the boot rom version on the bare metal which is preventing the install of SL. Meanwhile, I reformatted the ssd on the Mac Pro during my efforts to install SL and at present there is no OS on it, and an attempt to install High Sierra from a bootable USB ran into a weird issue where it seems to want a magic mouse and keyboard to continue the install. (Strangest warning screen I’ve ever seen during an installation—and I have a magic trackpad and keyboard and bluetooth is working because holding the option key does work to pick a startup disk. I haven’t tried resetting nvram yet which might help with the startup disk. Thanks for the reminder :slight_smile:
Next up, try Sierra. :roll_eyes:

17

Merci Gabriel. Salút!

1 Like
18

Update:

Annoying update. The last couple of times my relay has gotten the HSDir flag, someone somewhere must be doing a metric pull. Both times I got the flag, the relay went unresponsive in less than 4 hours after getting DDoS’ed.

@Quartermarsh got any suggestions on anti DoS measures? I know tor has its own DoS mitigation, and I’ve done a lot to keep it online, but still having issues. Not enough resource left around to run snort in the background all the time. Sad face.

19

Oy! Who’s ddos-ing you? I guess you could play whack-a-mole with host files, although that would be a lot of work. You might try hitting up the tor dev team for suggestions at their git-lab?
What’s the resource heavy part of snort? I know it can do intrusion detection but can it do mitigation as well? Would running the relay from a VM give you any better network management options, or is that even possible? Would a VPS be an option? I’m afraid I’m reduced to ‘blue sky’ conceptual thinking here as I’m sure your network management skills are considerablly ahead of mine. Hopefully, someone here on the forum will have some practical ideas. Good luck!

20

Lol probably not as ahead as you might think. But I appreciate that :stuck_out_tongue:
Snort does both IDS and IPS, its CPU hungry as hell for it. Mac OS X is nice and all, but the number of problems I continue to bump into, and the stability issues I’ve encountered since the DDoS began, im actually going to go forth and move over to BSD, and run the relay there. Its lighter than Mac, stable as fuck, and well known for routing jobs.
Part of the reason I picked macOS over unix or linux originally is because so many of the relays operated by the community are Linux. My relay is one of only 7 Darwin relays, and there are a few hundred FreeBSD/OpenBSD scattered about but the vast majority of them are linux of some sort.

Variation is important, even mentioned in the tech considerations section of the Relay Operator community page that more BSD hosts are preferable because of how many linux nodes there are compared to unix. That’s actually most of the reason I was inclined to give OS X a shot as far as relay and light server duties. Nope. The DDoS attack they (whoever, didnt take names) corrupted my OS image, and was causing kernel panics and crashes. Bad joo joo.

So I used my Oh shit plan, I disconnected from the internet entirely, plugged in my MacBook to my cable modem, and forcefully released my IP address, and disabled the dynamic DNS client running on my network to keep the domain record from updating. Can’t DDoS what aint listening.

Now I can take my time to fix the issue, since the attackers no longer have access to IP used to attack my machine.

Already running into a headache trying to get BSD, or any linux for that matter to install. God damn apple engineers did a hacky work around for 32bit EFI boot. So the machine and most of its hardware is x86_64 compliant, but the bootrom will only take and boot EFI 32bit kernels, or i386 MBR (no efi). So the machine will run and take advantage of 64 bit pathways and such after boot, and even run 64 bit programs and compile 64 bit source no problem, but it flatly refuses to boot on a 64bit kernel.
That’s been a challenge.
Hopefully, with FreeBSD being much lighter (compared to Mac OS anyway) running an IDS and packet sniffer shouldnt be a problem.