Letsencrypt SSL Certificates by DNS Challenge with Lighttpd

LAN HTTPS Everywhere

The aim of this investigation was to find a way to deploy HTTPS certificates on my LAN.  This might seem over the top, but a) Troy Hunt thinks it is a good idea and b) I was getting fed up of the ‘insecure’ messages.

After much fiddling and experimenting I have settled on this process.

Things to note:

  • You do not need any web server on the internet. You just need to be able to edit your domain DNS record.
  • Use a subdomain for each certificate/server – these do not need to actually exist on the DNS.
  • I’m not going to go into detail about things that are easily found with Google search.

Prerequisites

  • Domian – you need to have one available – I bought mine for $10 from Google Domains.
  • Host the DNS for the domain where you can edit the records – I was going to use Google Domains (hence the purchase) but ended up with the domain on CloudFlare DNS servers.

For me I am using Lighttpd so some of this is that server specific and my OS is DietPi.

Set Up DNS Access

Assuming you have got your CloudFlare account all setup, go to your profile page, scroll down and click on ‘View’ next to Global API Key.  You will need it in the next step.

Install & Configure certbot

You may need  sudo for these commands if not on DietPi as root. Also remember that any scripts need to be made executable  chmod +x .

First thing is to install certbot with the right plugin.

If like me you use DietPi, then dietpi-software install 92 else apt-get install certbot .  Then:

apt-get install python3-certbot-dns-cloudflare

Create a folder called  .secrets and create /edit a file called  cloudflare.ini . Include the following lines from setting up access to CloudFlare DNS above. Treat this like a password so  chmod 600

Now create a small script to generate the certificate (you will need to change the domain and the path to match your ini file) – you will only need this once (hopefully):

You can run this script, answer the questions and the certificate should be generated.

Note the  cat  command in the script.  certbot does not generate the right combined certificate for lighttpd so this needs to be done manually.  That is a bit of a pain and requires a little more setting up to get renewal to work automatically.

Setting Up Renewal

To get a new combined certificate on renewal a small script is needed that will execute once the certificate has been renewed and deployed.

and in the file place the following code (no modification necessary):

The only way to test this is to force the renewal

A modern install of certbot will install systemd timer for you so it should just renew.  To check the timer

Lighttpd Configuration

This is included for me more than anything, as it can easily be found, but this is the addition to the standard  /etc/lighttpd/lighttpd.conf  file that I find works.

Restart lighttpd (and clear cache).

Setting up an internal FQDN

Outside the scope of this but, if you have PiHole installed simply add the domain and IP to the  /etc/hosts file on the PiHole, restart PiHole and you should be able to just type in the domain name and be directed to a secure https address.

HTH

Facebooktwitterby feather
Posted in DietPi, Technical - General | Tagged , , , , , | Leave a comment

BT Broadband – Faster Speeds – Yes Really

Well BT have done what they said.  I’ve even noticed a slight up tick today so it is obviously still training to the fastest possible stable rate.

Facebooktwitterby feather
Posted in Technical - General | Leave a comment

Ubiquiti USG Home Setup

I treated myself to an Ubiquiti UniFi Security Gateway (USG) this week, bought off Netxl but Via Amazon (as it was cheaper).  The setup was not quite as straightforward as the AP I installed a couple of months ago.  In doing that I had played with installing the Controller software on a DietPi VM (what a surprise) and hit a couple of issues around Java.

Although that setup worked, I was never comfortable with it and I have subsequently discovered this amazing script to install the UniFi Controller – thanks Glenn.  Being on a VM, I just rolled it back to the base snapshot and started afresh having first exported the configuration and then just loaded the backup in to the new instance – easy.  He also has an update script which will also update the client firmware if you like – awesome. They both just work!  The Controller software was asking to be updated so I did that first.

I didn’t write up anything about setting up the AP; it just worked. I created WLANs with an SSID/Password combo to match the existing networks so there was no need to go round reconnecting everything.  The AP is so much better – I used to have major issues with my work VPN as it kept dropping out when on WiFi (fine on a wired connection). It is now rock solid.

I’d hoped that the USG, like the AP, would just plug into the network and off it would go – but it didn’t.  This was partly a lack of understanding on my part and a little more info required in the Quick Start. To cut a long story short, this is what you need to understand:

  1. The factory firmware is probably way behind and the UI is much improved and expanded so you need to log in by SSH locally to do that.
  2. The USG wants the WAN network and LAN network to be on different subnets (assuming you are behind a home router). You therefore end up with 2 DHCP servers one on the Home Router for the WAN port on the USG, and the other on the USG for the rest of your network.
  3. The USG takes a while to boot – be patient.
  4. This also assumes you have disabled all the WiFi connections to the Home Router.

I have got quite a lot of hard-coded IPs on my network and they often expect a specific default gateway so I needed it to slot in where the Home Router left off.

To get it up and running:

  1. Make sure only one thing is plugged in directly to the Home Router so you can insert the USG between your network and the Home Router (your WAN).
  2. Connect the LAN1 port on the USG to a standalone computer (need a cable and turn off the wireless so forcing it onto the wired connection), and the WAN port of the USG to your normal network (or the Home Router).
  3. The USG will boot with the WAN connected to a DHCP IP from your network, and the LAN1 IP (and the computer) on a different subnet.
  4. Get the IP of the USG from the computer (it will be the gateway) and SSH into it using the default credentials ubnt:ubnt.
  5. Upgrade the firmware as per these instructions (just type ‘upgrade and the URL of the firmware).
  6. Once it has rebooted, go to https://setup.ubunt.com and use the UI to do some initial configuration.
  7. For me this consisted of
    1. Setting the USG LAN IP to the current IP address of the Home Router and configure the DHCP range again to match the Home Router setup (don’t save yet).
    2. Changing the IP address of my Home Router to a different subnet (e.g. from 192.168.1.1 to 192.168.10.1) and the DHCP range appropriately (don’t save yet).
    3. Changing the USG WAN settings to work with the new subnet of the Home Router (you could just leave it to get a DHCP address).
  8. First then, save the configuration on the USG – it will reboot. You will need to unplug the network cable from the computer to force the computer to pick up a new IP – be patient at this point. You will not have any internet on the computer (as the WAN cannot connect – yet).
  9. Save the Home Router configuration – you might need to force a reboot.  Unplug your network from the Home Router and plug it into LAN1 port of the USG. Now the Home Router is connected to the USG WAN port, and the rest of your network to the LAN1 port.
  10. You should now be able to adopt the USG in your UniFi Controller and everything still works. The one fly for me was that the Network had not been configured correctly and it moved from my specified IP to a X.X.X.1 IP. Manually changing that in the controller was easy enough.

I just now need to work out how to harness the power! My next task is use it for DDNS and a VPN connection.

References:

  1. Follow the steps on this KB to update the USG offline or SSH if have the internet access on USG https://help.ubnt.com/hc/en-us/articles/204910064-UniFi-Changing-the-Firmware-of-a-UniFi-Device#USG2
  2.  Then follow this steps to change the LAN IP of the USG https://help.ubnt.com/hc/en-us/articles/236281367-UniFi-How-to-Adopt-a-USG-into-an-Existing-Network
Facebooktwitterby feather
Posted in Hardware, Network, Technical - General | Tagged , | Leave a comment

BT Broadband – Faster Speeds

BT have said they are going to increase my speed

This I find really interesting as I hardly use what I have.  I monitor my speed on an hourly basis, As you can see, it is pretty reliable.  The dips are relatively small, usually plenty in reserve.  I have had one outage although that may have been when I was fiddling with my LAN setup.  This limiting factor – external bandwidth / site speed.

As ever, the press release is bull, aimed at the uninitiated. It will not improve the number of devices that can connect reliably.

Facebooktwitterby feather
Posted in Technical - General | Leave a comment

Cloudflared DNS over HTTPS (DoH) and Pi-Hole

(Using the new WordPress Editor and I am not sure I like it – in fact I hate it.  Gone back via the ‘Classic Editor’ plugin)

Background

I have been following Troy Hunt and Scott Hulme, 2 well known security experts and decided to try and employ DNS over HTTP (DoH).

I already had Pi-Hole installed on a DietPi VM (my favoured building block) so the logical thing was to put the cloudflared daemon on the same machine.

I used a combination of the following 3 references

Installation & Configuration

First step is to install the cloudflared package

To check it is installed correctly;

Next step is to configure the package correctly.  There are 2 means of providing a configuration to cloudflared; a config file or a yaml file.  I did not find the yaml method before I did the config.  I may change it at some point.

To setup the config file, create the following file

With this content

Setup a Service

Once configured, the next step is to run cloudflared as a service.  Create the required file;

and insert the following text;

Note I have commented out the reference to the user as this is DietPi and I simply run everything as root.

Once the file has been created, enable the service and start it;

To check that the service is running;

should return something like this

To test if this is working use this command;

The final piece of the puzzle is to now get Pi-Hole to use this for the DNS queries.  This is quite easy to do, simply add in  127.0.0.1#5053 to the custom DNS entry in the settings.

Change the Listener IP Address

There is one final thing.  Because of the way I wanted to set my network up, some devices bypass Pi-Hole but I still wanted the DNS queries to go via DoH.  In order to do that I wanted to point those machines DNS directly to the IP address hosting the Cloudflared daemon.  However it didn’t work.  After some discussion on the forum and looking at the output of  netstat -lnp I established that cloudflared was only listening on the loopback interface (127.0.0.1).

The solution was to add an  --address 0.0.0.0 parameter to the config line.

While the output of the netstat command still does not reflect the setup of a service like lighttpd, it does work.

Facebooktwitterby feather
Posted in DietPi, Linux, Pi-Hole, Technical - General | Tagged , | Leave a comment

UniFi Controller on a Pi or DietPi

Having just invested in a UniFi AP UAP-AC-Lite (it is absolutely fantastic), and refusing to spend £70 on the USB UniFi controller or wanting to use the cloud system, I needed to get the UniFi controller installed.  I didn’t expect any issues.  Well there were.

I first ran up a new DietPi VM on my micro HP server running an old 5.5 version of Esxi – Ain’t broke don’t fix. I now have a policy of not combining systems but running individual VMs.  I was burned recently when an installation of HomeAssistant went Pete Tong and I had to rebuild from scratch.  That also took out Node-Red and Mosquitto so they all (broardly) have their own VMs.

I tried a number of different ways of installing the controller via apt.  The problem I kept hitting was Java.

[Edit 9th March 19] I have discovered a much better way via these scripts.

To cut a long and tedious story short, I stumbled upon a community post that linked to a docker version of the Controller (project links https://github.com/ryansch/docker-unifi-rpi and https://hub.docker.com/r/ryansch/unifi-rpi/).

To install this on DietPi (after my usual setup) I simply installed docker and then from apt docker-compose, before building the container.

And that was it.

The controller appears on port 8443, you need to use an https address and there is the usual stupidity around browsers complaining about no SSL cert for non-routable addresses but it all works fine.

Epilogue: I subsequently realised that there is a second method to install the docker that might actually be better as it has instructions on how to upgrade. I do not understand the difference in the methods and I will leave it until I need to upgrade.

Facebooktwitterby feather
Posted in DietPi, Hardware, Network, Technical - General | Tagged , , | Leave a comment

Pi-Hole and Disabling dnsmaq

Pi-Hole no longer uses the dnsmasq DNS proxy directly. However, existing installations will still have it installed and the service will still attempt to run and fail (as it should).

I was keen to get rid of it as I’m not a fan of unused services especially ones that should always fail. It was not as straightforward as I’d hoped as it is an init.d service. After a discussion on the forum, and some trial and error I think this is what I did. Unfortunately, I could not go back to where I was before I started so I cannot be 100% confident it is right (or the order).

So the issue was that, as this is a service that is started via update-rc.d a normal ‘disable’ does not work as it leaves the various run level files hanging around as seen by

To remove those you need to execute

Secondly, disabling a service simply stops it from running at boot, but does leave open the possibility of it being initiated by another service. To stop the possibility of that happening, a

is required.

This then removes it from the output of a systemctl status and avoids confusion.

There is a suggestion that dnsmasq could be uninstalled. I decided not to as it felt a little risky (I may take a VM snapshot and try).

To ‘uninstall’ a package, you need to do one of 4 things with apt.

remove– just removes the package from the os (i.e. uninstalls it) and not from the disk (the cached package).

purge – removes the installed package and associated files like personal settings, configuration files etc

auto-remove – removes any packages that are no longer required as a result of removing packages that had previously caused other dependencies to be installed. If used with purge will remove associated files as well.

clean – actually removes all of the above from disk, think emptying a recycle bin in windows.

Facebooktwitterby feather
Posted in DietPi, Linux, Pi-Hole, Technical - General | Leave a comment

Systemd Timer and monit

Why is it that, so often, a key bit of information is assumed by documentation writers?

I’ve had an issue for a while where monit will fail to start properly on boot.  The issue is that with DietPi, it seems that the system boots too quickly and monit trys to create a network interface before the network is ready.  Calling a systemctl restart monit solves the problem but that is a pain to remember to do after every boot.

I started looking at systemd solutions.  There is an obvious one; use a timer file.

In theory you create a file with the extension .timer and that then triggers the related service to start.  Good in theory but I could not get it to work; the monit service just started straight off.

The clue is a small line here – “The .service does not require an [Install] section as it is the timer units that are enabled. ”

Off goes the light.

So the monit.service file looks like this (note no install section)

and the monit.timer file looks like this

both reside in /lib/systemd/system/

To enable them you simply enable the timer systemctl enable monit.timer

One other thing I did and I’m not sure if it makes a difference, I removed the file /etc/init.d/monit

As a recap, a vanilla install of monit is simply apt-get install monit.

To access the status, you need to setup the internal http server. Add a file /etc/monit/conf-enabled/monitconfig with the contents;

There are other examples in the main config file  /etc/monit/monitrc.  This setup only allows access from my local subnet and as I used a fixed IP, I specify that IP.

I then access this from Node-Red (fill in your IP address) and output onto an MQTT broker as I explained in the past https://tech.borpin.co.uk/2017/03/21/node-red-and-xml-to-json/

Facebooktwitterby feather
Posted in Linux, Technical - General | Leave a comment

Contributing to Open Source Projects on GitHub

If it has crossed your mind to contribute to an Open Source project, but GitHub just seems too daunting, then I’d suggest reading this tutorial.

The bit that got me was how to actually contribute and this section explains it perfectly (I think). Before, there were too many origins the odd upstream and far too much pushing and pulling!

The key steps (which I have got wrong on numerous occasions before reading this tutorial) are:

  • Create a Fork of the repository you wish to contribute to (the upstream repository), to your GitHub account (the origin repository)
  • Clone the origin repository to your working machine (the local repository)
  • Add the upstream as a remote
  • Create a local branch off the master branch
  • add in your features and test them
  • commit to your local repository
  • push those changes to the origin (your GitHub repository)
  • Create a Pull Request to include it in the main project (i.e. ask the maintainer to pull in your changes).

This tutorial does refer to BitBucket but the principles are exactly the same and is a reference I keep on coming back to.

Facebooktwitterby feather
Posted in Coding and Programming, Technical - General | Leave a comment

Using nmcli to set a static IPV4 address in Hass.io

This method of setting a static IP for Hassio has been tested with a Hass.io vmdk image on an Esxi server. Log into the HASSOS base system via a console – note this is not the same as an SSH login via the add-on.

Welcome to HassOS
Hassio login:

Login as root (no password needed)

At the hassio > prompt, type login (as instructed).

From here you will use the nmcli configuration tool.

# nmcli connection show will list the “HassOS default” connection in use.

# nmcli con edit “HassOS default” will put you in a position to edit the connection.

nmcli> print ipv4 will show you the ipv4 properties of this connection.

To add your static IP address (select ‘yes’ for manual method);

nmcli> set ipv4.addresses 192.168.100.10/24
Do you also want to set 'ipv4.method' to 'manual'? [yes]:
nmcli> save
nmcli> exit

In addition I have found it is wise to set the dns server and the local gateway. For most home routers these will be the same address. If you are using Pi-Hole you can set the dns to that.

nmcli> set ipv4.dns 192.168.100.1
nmcli> set ipv4.gateway 192.168.100.1
nmcli> save
nmcli> exit

If you now view the default connection cat /etc/NetworkManager/system_connections/default you should see the method is manual and the address is set.

Doing a nmcli con reload does not always work so restart the VM.

Facebooktwitterby feather
Posted in Hass.io, HASSOS | Leave a comment