To DoH or not to DoH?

DNS over HTTPS (DoH) like so many other emerging protocols in a “post privacy” world is an attempt to wrap more layers of abstraction (crypto) around basic internet applications and functions.

The premise behind DoH is this: Instead of using the standard DNS protocol, your DoH client (mostly commonly a web browser) reaches out to a DoH provider/server and both agree to send DNS-like queries and responses over standard HTTPS.

The problem is that DNS was never designed with security in mind. As with E-Mail some additions were later added to help smooth over some security edges.

Just because you can do something, it does not always mean that you should. DNS is a very foundational protocol; are you okay with your browser using a different DNS mechanism instead of the one built into your OS? Are you still able to access your internal applications? If you suspect DNS issues how are you going to troubleshoot?

How does this thing work?

Cloudflare and Google provide really good documentation on DoH and you can always read the RFC if you need help sleeping at night.

Essentially, you have a DoH client built into your web browser (or even some routers) that is responsible for brokering all of the requests and replies. This traffic is little snippets of JSON, with the MIME content type of “application/dns-json”.

To generate a sample request, you can use curl to query a DoH resolver like the one Cloudflare operates.

Request/Response example based on Cloudflare’s documentation
It’s not DNS, its regular ole HTTPS. See “Server Name Indicator” field highlighted. This will be useful to look at later.

As demonstrated, DoH has the potential to bypass any DNS filtering as the actual query is encrypted. The only thing that you can see is someone using Cloudflare DoH (Server Name Indicator, highlighted in screenshot above). Because DOH rides alongside HTTPS you can’t just block it. I mean you could block port 443/tcp, if you want to have a bad time.

I will circle back to the technical aspects, now we should talk about why some people don’t like DoH.

My problem with DoH

DoH does work and I have not seen an appreciable overhead or degradation in performance. However, its important to be honest about what DoH does and doesn’t do. Specifically, some of the false hope it can give users about security and privacy.

Being concerned with privacy/digital rights, some aspects of DoH are appealing to me. The use of a Separate HTTPs broker to send and receive DNS queries in little bits of encryption JSON is very clever. It even seems to be upsetting large ISPs such as Comcast and AT&T. The enemy of my enemy is my friend right? Not always…

I prefer simple, plain ole DNS. The reason is Trust.

  1. Trusting your DoH provider – In instances of malware, you may have no idea who that provider is. Yes, you would have to trust your upstream resolvers in normal DNS. However, your local resolver/gateway can override the answers to queries , cache the records, and keep logs. 
  2. Going blind –  The lack of visibility not only creates trust issues with your DoH provider (should they be compromised), but it also circumvents internet filtering policies for legitimate use cases such as home or work monitoring.
  3. (Not specific to DoH) – For large organizations (proxy/decryption), It can be trivial to view sensitive HTTP/HTTPs information.

    Even without decryption you will still be able to see the Sever Name Indication “SNI” field mentioned above. You can loosely think of SNI as hostname, but for the purposes of certificate verification/TLS handshake. Take a look at the screenshot below. There is no DNS query, but why is a TLS connection being established with Hysler.net? Could it be user activity?
Boom Roasted

In short, DoH can provide some privacy benefits as DNS information is no longer plain text en massé. However, these can be mitigated by HTTP headers and partially by SNI field for HTTPs traffic. Metadata analysis on the destination IP addresses, source/destination ports can also reveal a great deal about the traffic in question.

You have to trust providers such as Cloudflare to make sure they are sending you the right responses. Something you can have slightly more control over with a “closer to home” resolver or technologies like DNSSEC or DNSCrypt.

Blocking DoH: Home Network and Small/Medium Business Edition

Blocking DoH in a large enterprise is straightforward with if you have a managed external DNS service such as Cisco Umbrella (viva la OpenDNS!), a web proxy, and some GPOs.

However, in a home or Small/Medium Business (SMB) setting you are going to find unmanaged devices with various levels of security controls. The only reliable control you are likely to have is the network. Here are some proactive steps you can take to defeat DoH.

  1. Upstream DNS provider
  2. Blocking DoH Providers (IP/DNS)
  3. Preventing circumvention of #1 and 2
  4. (Optional/Opportunistic) – Endpoint controls

Using an Upstream DNS provider that is filtering out DoH domains

Services like OpenDNS prosumer have an option to block “DoH providers” if you block the “Proxy/Anonymization” category or if you use their DNS servers that filter out adult content. I know Cloudflare has a similar offering, but I have not been able to see if it blocks DoH or not.

There is a caveat to this approach, if you make the appropriate DoH request via direct IP you can still get around the DNS level protections.

Blocking DoH Providers

You can also use public block lists to prevent communication with DoH servers via hostname or IP address. The public lists may not be current, but they contain a few dozen servers including many I had never heard of.

Even if your router does not allow for block lists you can always use firewall rules to block some of the most common resolvers (as a last resort).

I use the pfBlocker-ng package for pfSense to automatically grab the latest version of the following lists from this Github project.

https://github.com/bambenek/block-doh

  1. DoH Hosts


2. DoH IPv4 and IPv6 lists

Preventing circumvention of existing DNS

This one is pretty straight forward. Make sure your gateway/router is giving your clients the right internal DNS server and that your gateway/router is using the right external DNS servers to resolve/respond to queries (Example: Google being 8.8.8.8, OpenDNS being 208.67.222.222, etc.)

Now you can use the built in firewall for your gateway/router to block any outbound DNS traffic that is not going to Google, OpenDNS, etc. You may break something here, I have seen some IOT devices try and use Google DNS for example.

Optional/Opportunistic: Blocking at the endpoint

Note: Information current as of July 2020

Does your router not let you do any of the above? Is there some other constraint but you still want to block DoH? There is good news, you can still block DoH manually. This is a good approach for a small number of devices and/or children with devices that have parental controls.

These methods require setting some hidden flags and options within browsers:

The obvious problems with this approach are that the locations for these settings may change or get inadvertently flipped by software updates and/or bugs. Secondly, it appears that Firefox is more likely to honor the end user going into Settings GUI and explicitly enabling DoH instead of about:config or network telemetry (such as canary domain).

Google, seems to have a more nuanced approach detailed in its blog post. Google is not going to change your DNS provider by default. Mozilla will change your DNS provider to one of its partners in its “Trusted Recursive Resolvers” program.

I hope that this post has been helpful in understanding DNS over HTTPS. For those of you who like to tinker and really like to control your own home network, I hope that you found some practical advice.

For me this blog post was largely inspired by being an enterprise defender and how to think through mitigating command and control traffic/DNS queries using DOH. It is also a useful exercise to think through blocking DoH where it makes sense as a parent, sharpen your InfoSec skills, and/or maintaining visibility on your network.

What do you think? Is there something I missed or got wrong? Want to talk about security? Shoot me an e-mail or lets talk on Twitter.

Thanks for reading,

-Justin
justin@hysler.net
@difr_justin

Homelab isolation with pfSense

If you are continuing on from my earlier homelab post, I mentioned that doing proper network isolation is very important when working with malware samples or doing any kind of reverse engineering work. There are other practical reasons you may want a separate network for your virtual machines, a good example is building your own Active Directory Domain and Domain Controller(s).

Like the first post, this contains some tips that I wish I had known starting out. I hope that it helps you out. This guide is very similar to pfSense’s virtualization guide, but with some additional commentary around InfoSec specific labs/virtualization.

Note: We are only going to discuss virtual IPv4 networking, IPv6 is disabled on my pfSense interfaces, but link-local addresses are still enabled for compatibility.

Moving from VMware Workstation/VirtualBox to Type 1 Hypervisor

When you move from VMWare Workstation / VirtualBox / Parallels (Type 2) to something like ESXi, you may notice that some key networking options have disappeared. The thing I neglected to take into account was NAT’d networking options

This is what you are probably used to

Notice when you create a new VM in ESXi, the default network connection for a VM is going to be bridged and your VM will grab a DHCP lease from your local network if available.

VMkernel / Virtual NICs.

So how do you get around this constraint and into something more familiar? The answer is Virtual Switches and Port Groups!

By default ESXi, has a vSwitch0 that is tied to your management network interface. You can create a second vSwitch that does not have an uplink to another network connection. Below is an example

Look at the vSwitch topology, no physical adapters or other uplinks!

You can see some sample options and that my virtual machines are not connected to a physical NIC or other interface that gives access to my local network. This vSwitch1 will become my isolated “Internal Lab Only” network. You can do this by associating this virtual switch with a “Port Group”.

This is all fine and well, but how am I going to get routing/DHCP/NAT (optional) and all of the other cool stuff that my Type 2 Hypervisor did for me?

Enter pfSense

You don’t have to use pfSense, but you need something that allows for proper routing and a firewall (the most hardcore will use iptables). pfSense is a really good option as it has a large community of users, is well documented, and is open source.

Since the intention of this post is isolating a homelab, I am not going to cover the installation process in detail. You can find that information via the Google or just use the official guide. I do suggest you double check your MAC addresses as you build your WAN and LAN interfaces to make sure they are correct.

Here is what I wanted to accomplish: (Internet) <-> (Home Network: 192.168.1.0/24) <-> (pfSense) – (Isolated Lab Only Network: 10.0.0.1/24).

For my WAN interface I made sure that my home network router provided my pfSense a reserved or static IP address. I defined the LAN as a different private IP space (Its recommended to make these distinct to aid in any troubleshooting).

Here we go!

Once you get pfSense up and running, you can then start creating Virtual Machines there are assigned to the “Internal Lab Only” port group. With my particular configuration I did want to allow my VMs to have internet access. At this point we are still not isolated!

Note: For the scenarios where I dont want internet access, I can turn off my WAN interface and/or use something like “inetsim” along with static routing in my VMs to make sure traffic goes only where I want to it to (I am sure there is a better way to do this).

Let’s take some steps to keep our 192.168.1.0/24 network from talking to our 10.0.0.1/24 network and vice versa. What do we need here?

  1. DNS – We want to make pfSense the default gateway and DNS server. We won’t be talking to 192.168.1.0/24 anymore
  2. DHCP – Same story as above, but you may want to have a small pool of IPs dedicated for management on other VMs. (Ex: 10.0.0.2 – 10.0.0.150 can be assigned, but 151-254 are reserved.)
  3. Firewall Rules! – Lets do some simple enforcement in both directions to make sure these networks can’t talk to each other.

Basic Services

Here are some sample settings for reference as of pfSense 2.4.5, as with anything you find on the internet. Do not just copy and paste. Use my blog as a reference but verify and test!

System -> General Setup -> DNS

You can use any public DNS provider here, I also choose not to let my pfSense box get its assigned DNS from the wan interface (DNS Server Override). I also want to use pfSense as a DNS resolver.

Also take a look under Services -> DNS Resolver and make sure you have the options you want selected. You may want to handle how DHCP hostnames are registered, or disable DNSSEC for example.

System -> Services -> DHCP Server

Here is where you can define where your range starts and begins. At the bottom of this page you can set your DHCP reservations/static mappings.

Firewall

Here is where we can enforce our network boundaries. If your WAN and LAN net are in private (RFC1918) IP space you want to make sure you change some of the pfSense defaults. Go to the interfaces page and check the following setting for your interfaces.

Yes, these networks reside in private IP space.

Now onto the firewalls page…

I have created the rules below and modified/removed some of the default rules to make this work for me. I am sure there are more elegant ways to do this, but it makes sense to me and my small lab. Note that my WAN and LAN nets are already predefined per their interfaces.

I recommend making sure you have a labeled config backup (Diagnostics -> Backup & Restore) incase you need to go back to a previous version of the firewall rules.

WAN Rules – I deleted all of the rules here. Traffic should be outbound only!

LAN Rules (and a word about NAT) – This is where you get most of your isolation. You can prevent LAN -> WAN communication in two ways. One is adjusting the default NAT rules so that any outbound traffic is not translated to 192.168.1.0/24. The second is to create a firewall rule, I prefer using both as an extra safety measure.

LAN Isolation #1: Manual NAT

For your LAN Firewall rules, make sure you have a rule to block any LAN -> WAN Traffic (Line #3).

LAN Isolation #2: A sample. Note the second rule is not necessary. Its for pfBlocker-ng

If you are new to reading firewall rules the bottom two lines can seem counter intuitive. pfSense like many other firewalls are “first match” meaning that anything on the LAN destined for the WAN will hit the block rule (3rd line) and not be allowed (4th line). If you need to troubleshoot, pfSense has great documentation on rule processing.

After hitting save and reloading your filter, you should not be able to access your firewall from the WAN side. Now it is time to test out our isolation!

Isolation: Verify!

Whatever your configuration or firewall rules are like, you need to verify them for yourself and use an automated means of doing so! This is great practice in general, don’t trust some random guy on the internet and copy settings blindly.

You can try using ping or SSH to hoists on either side of your network, but its best to use something like an nmap scan. Thankfully, there is a nmap package for pfSense! Go into pfSense’s package manager and search for nmap and then install it.

You can use the pfSense GUI (Diagnostics -> nmap) , but it is somewhat slow and I prefer the CLI version.

We want to try and scan the WAN IP space using the LAN interface and vice versa.

Here is how we go about it, go ahead and drop to a shell. See how vmx0 is our WAN interface. We can tell nmap to scan our lab network (LAN) from the WAN interface. If you did this right, you should get zero 0 hosts up.

You should see something like this!

You should also try this in the other direction and see similar output. “nmap -e vmx1 -A -v 192.168.1.0/24”.

Conclusion

Now you should have an isolated Lab network on your ESXi machine that you attach to your guest VMs and play around with all sorts of software that you wouldn’t necessarily want floating around your “trusted” home network.

As always, remember your OPSEC if playing around with malware.

If you wanted to get more into virtualization and InfoSec research/homelab, I hope this guide has helped you.

Until next time,

-Justin

If you have any comments or corrections, you can reach me at @dfir_justin or any of my contact methods.

InfoSec homelab for the frugal

Until recently, I had always run my Virtual Machine labs (SANS courses, CTFs, Hackthebox, Vulnhub, etc..) on a physical host with a general purpose operating system (Windows, macOS, Linux). However, the limitations of using VirtualBox, VMWare Fusion, Workstation Pro, etc.. become very clear when setting up a more complex lab with its own special networking requirements (especially a span or mirror port). I hope this post is useful for others like me who wanted to take the homelab jump, but did not know how to get started.

There is also the matter of Reverse Engineering Malware (REM) work that benefits immensely from running traffic across a real IDS (Disconnected from the internet of course!) Its a more mature environment, but one that was intimidating given the number of support posts related to getting ESXi working on “unsupported” hardware. Thankfully, after more hours of research and asking the kind folks on the SANS mailing list, I had a better idea of what to look for.

My criteria: Workstation form factor PC that is capable of running 4-5 Virtual Machines for light InfoSec work (IE: pfSense, Windows Server (as a DC), A Windows client or two).

My budget: Approximately $250, I figured this would at least get me a 4 Core/8 Thread processor, a decent amount of RAM, and if ESXi/vSphere Hypervisor did not work I could repurpose the machine for something else.

In looking at workstations, I found that Xeon based workstations with Intel NICs usually had the best compatibility. For these reasons, I narrowed my search down to HP Z Series Workstations and Dell PowerEdge towers. I also found this extremely useful presentation/slide deck by Jeff McJunkin called “Building a kick ass home lab”.

Now that I had my requirements and budget, I needed to find a place that sold used workstations. $250 was not going to get me the latest hardware and you need to expect a machine with several years of use and likely sold to a third party reseller after a company did a hardware refresh/end of life for their corporate workstations.

Note: Don’t be too concerned about old hardware, it should be plenty for this use case. Secondly improvements in CPU performance year-over-year have slowed down tremendously over the past decade. Intel has been stuck on 14nm chips for almost five years!

I found the best deals on eBay and the following subreddits (r/homelabsales and r/hardwareswap). I recommend doing an advanced search on Reddit and sorting by “New” to get an idea of prices on the gear you want. My experience was that Reddit had better deals, but you can get lucky on eBay with enough persistence.

What I ended up buying: An HP Z420 with a 4c/8t Xeon Processor (Sandy Bridge), 20GB of DDR3 ECC RAM, and 500GB of spinning rust. – $150

See the source image
HP Z420 Workstation

I took out the 500GB HDD and replaced it with a spare SSD I had lying around and purchased a 480GB Kingston SSD from Amazon. So approximately $200 for a solid performing workstation.

After seeing what was possible with ESXi, I quickly became greedy and realized that 20GB of RAM and a 4c/8t CPU was not going to cut it. Back to Reddit and eBay!

This is fun and addictive!

My next two acquisitions were an Xeon E5 2660 v1 /w 8c/16c for $30 shipped via Reddit and another pair of 8GB ECC DIMMs for $70. Now I am up to $300, but feeling comfortable about the amount of space I have to work with!

Here is what I have running so far and after some trial and error with learning ESXi, I feel comfortable with this setup. The screenshot below is with all VMs powered on and at idle.

I hope to expand on this post via a “lessons learned” and how I configured pfSense to isolate my VMs from the rest of my network and vice versa. Now get out there and learn and don’t be afraid to break things in your lab!

Interview at InfoSec Nashville 2019

See the SoundCloud link below for some thoughts on Medical Device Security, the importance of security research, and especially the importance of networking for students or other newcomers to the Information Security industry.

Also go listen to several of the speakers/attendees at InfoSec Nashville this year.

Excerpt below from Middle Tennessee’s ISSA 2019 Podcast Compilation page.

Security Threat Engineer Justin Hysler, unpacked why cybersecurity is a patient safety issue for healthcare companies, explaining that vulnerabilities in essential healthcare systems must be exposed and addressed before they’re exploited. He discussed the challenges of implementing cybersecurity measures in the healthcare world, but shared his optimism that cybersecurity will be increasingly prioritized.

Something old and something new

Hello visitor! I started this website as half portfolio/resume and half miscellany posts on (mostly Cyber Security) topics that I find interesting.

Thank you for visiting. Please check out some of my favorite resources on the right sidebar. You can also check out my security focused review of a pretty awesome prosumer router below.

Synology RT2600ac Security Features Review

TLDR; For $200 you get 802.11ac, Suricata, VPN capabilities in a single box.

Disclosure: This review is not supported/endorsed by Synology. This is my perspective on some security specific features not covered by other reviews.

Home networking for the InfoSec curious or veteran is a topic that seems to come up regularly. Some people take great pride in their pfSense, Ubiquity switched/multi AP environment, with multiple VLANs (for IOT device network segmentation) setups. These are great for the more experienced Network Engineer and/or InfoSec guy or gal, but what about the more casual (SOHO) power user? Or even someone on a tight budget?

Enter the Synology RT2600ac (yes, the NAS company) which has plenty of networking and security prowess hiding in a $200 dual core ARM based appliance. This has been my go to recommended gateway for co-workers and family who are tech savy but not ready to make the deep dive and expensive into prosumer home networking. This review focuses exclusively on the security features. If you want some reviews that go over the NAT performance, wireless range/speed, go check out the folks over at SmallNetBuilder.

The RT2600ac will be two years old at the time of this writing (February 2019). An important question to ask is how often are security vulnerabilities being addressed and how often does the router receive updates?

Synology seems to be on top of this as an December 2018 update fixed an RCE vulnerability (no authentication required!) and the list of release notes is impressive. WPA3 support was also added a few months ago.

Basic Configuration: automatic Updates, default configuration, and authentication.

Before moving onto the more advanced features, lets get the essentials of out of the way. For your non-technical or non security user, having an automatic update mechanism is essential. Thankfully you can choose between security updates and/or operating system (SRM) feature updates. This is similar to the Windows feature update settings in Windows 10. I have found the SRM releases to be quite stable. However, if you are going to be using this router in a mission critical setting . I wouldn’t fault anyone for going with security updates only.

Gracefully, external access to the administrator page is only by “opt-in” (seriously though never do it!) and the router supports 2FA via the use of Time-based One Time Passwords (TOTP). This type of second factor is compatible with popular TOTP clients such as Google Authenticator, Authy, and others. This protection also extends to other apps in the SRM specifically Synology’s Web VPN.

Firewall and Geo-ip blocking

After you go through all of your configuration options and make sure everything works, you can start hardening your perimeter (also wise to turn off uPNP). Beginning with blocking several countries that you have no interest in exchanging traffic with and/or avoiding malicious infrastructure (botnets).

Each firewall rule listed above can have up to 15 countries in it. I just stick with 15, however you could go crazy and add 30, 45, 60 to cover even more territory. I wish that Synology would allow you to add as many countries as you want within one rule. The SRM UI is limited in the data it can display (more on that later).

Synology suricata

How in the heck does a $200 ARM based router run Suricata inline with little performance overhead? Well it took Synology almost two years to figure it out and after some tweaking it really does work! Prior to the November 2018 update, I was only pulling 30 Mbps from LAN <-> WAN on a family member’s 300 Mbps Comcast connection. Post update am seeing consistent 300 Mbps speeds. Really excellent for a 1.7 Ghz dual core ARM processor with only 512MB of RAM!

The downside is that managing Suricata from the GUI in its current state is less than ideal. The web server is fairly slow, there are several drill downs to find specific rules, and I see no place to write your own rules and/or download packet captures. I hacked away at it in the command line (Busybox shell) and could not get my custom rules to take effect in Synology’s version of Suricata (perhaps further testing will lead to another post).

I should also mention the “Safe Browsing” application that launched in November 2018 as well. Its a competent parental control style app that uses some data from Google’s safe browsing project and an unknown source of “Threat Intelligence”. I have had a few false positives, but the lists are updated daily and I have not noticed a performance hit.

You may be better off using an DNS provider that does filtering such as OpenDNS (family shield/security) or Quad9 (security).

VPN

Synology’s SRM supports many of the most popular tunneling protocols such as SSTP, OpenVPN, IPSec, and PPTP VPN in addition to Synology’s proprietary apps.

SRM does create some other private subnets that allow you to segment VPN users from your primary systems and/or the guest network. I briefly used OpenVPN, which was easy to set up and came with great instructions for getting the necessary certificates/configs exported.

If you want to fly a little lower under the radar, you could run Synology’s SSL VPN over a more common port (443) and your traffic would look like a normal HTTPs connection to the rest of the world. The Synology app also supports 2FA which is another bonus.

PROS and CONs

Pros

  • Extremely stable
  • Incredible value at $200
  • Regular security updates
  • Late 2018 update allowed for Suricata to run with no appreciable performance hit.
  • 2 Factor Authentication for admin access AND VPN
  • Easy to use GUI
  • Syslog support

Cons

  • Slow to reboot (10 minutes) for a consumer router
  • Odd UI bugs such LED on/off preferences not working or DNS settings greyed out for IPv6. (To their credit this was eventually fixed)
  • Hardware limited, I would love to see what an extra $100 would allow for in terms of CPU and Memory (more cores? higher clocks? x86 base 1-2GB of RAM?)
  • Easy to use GUI (Very few command line options)