nftfw logo

nftfw - Installation Guide

Prerequisites

This document assumes that you are installing on Debian Buster.

nftables

$ sudo apt install nftables

The standard version of nftables at the time of writing is: 0.9.0-2. Buster backports has a more recent version - 0.9.3-2~bpo10+1, and it's a good idea to upgrade to that. Look in /etc/apt/sources.list, and if necessary append:

# backports
deb <YOUR SOURCE> buster-backports main contrib non-free

then

$ sudo apt update
...
$ sudo apt upgrade
...
$ sudo apt -t buster-backports install nftables

which will upgrade your system to the most recent nftables release. The install action will tell you that a library is not needed before it installs things. There is a systemctl entry for nftables, that will probably be disabled now. Check with

$ sudo systemctl status nftables

I've got a live iptables installation

Well, what you need to do depends on what's running in your system. Buster comes ready installed with two flavours of iptables: the legacy version and a bridging version using the iptables API, which maps and loads the iptables formatted rules into nftables. You can tell which one you are using by:

$ sudo iptables -V
iptables v1.8.2 (nf_tables)

if your version output looks like that, then you are OK and can just skip over what follows to incron.

If the words in brackets says legacy then you need to swap to the nf_tables version. Here's what you do:

FIrst, save the current iptables settings, you'll need to put the settings into the kernel's nftables system when you switch.

$ sudo iptables-save > ipsaved
$ sudo ip6tables-save > ip6saved

Now change iptables and ip6tables to use the nftables versions:

$ sudo update-alternatives --config iptables
# select selection 0, /usr/sbin/iptables-nft, auto mode
$ sudo update-alternatives --config ip6tables
# select selection 0, /usr/sbin/iptables-nft, auto mode

Each gives you a menu of options: select the nftables compatible version. I used option 0 - auto. Check your iptables command is now at the correct version using -V. Finally, reload the iptables data you saved earlier.

$ sudo iptables-restore < ipsaved
$ sudo ip6tables-restore < ip6saved

You now have two sets of rules loaded into the kernel, the new ones using nftables and the previous iptables set. The kernel will use the new rules but will complain. To remove the old rules:

$ sudo iptables-legacy -F
$ sudo ip6tables-legacy -F

All done, and it's painless. You have a system that works for both iptables commands and nftables nft command.

incron

nftfw uses incron. It monitors files on the file system and triggers events when the files change. If you don't have it:

$ sudo apt install incron

As I write, the available version of incron is 0.5.12-1. It has a nasty bug where it will leave processes hanging around on the system. On a busy system, dormant processes can cause problems. There is a fixed version 0.5.12-2, stuck in the Debian testing system. It apparently has a bug which doesn't affect system use, but as a consequence is not released for mortals at the time of writing. If it is now available, I suggest you install it.

I am using a work-around. I have an entry for cron which restarts the program every 24 hours. This removes all the dormant processes. See the sample cron entry in the cronfiles directory.

nftfw doesn't need_incron_, see the note How do I: do without incron? in the How do I document.

sympl no longer installs incron, but I think I like the idea of instant actions when the nftfw control directories change.

Python

nftfw is coded on Python 3 and the standard Python version on Buster is 3.7. The nftfw package was developed in part using Python 3.6, and so the nftfw package may run using that version.

We will use pip3 to install the nftfw package.

$ sudo apt install python3-pip

The nftfwls command uses Python's prettytable, which may not be installed:

 $ sudo apt install python3-prettytable

nftfw Installation

You now need the nftfw distribution. I put mine into /usr/local/src, it can be your home directory.

$ sudo apt install git
...
$ cd /usr/local/src
$ sudo git clone https://github.com/pcollinson/nftfw

which will create an nftfw directory.

Now change into the nftfw directory and use pip3 to install the package. pip3 will allow you to uninstall the package at a later date, if you wish.

$ sudo pip3 install .
...
Successfully installed nftfw-<version>

To uninstall, sudo pip3 uninstall nftfw.

pip3 installs four commands: nftfw, nftfwls, nftfwedit and nftfwadm in /usr/local/bin. Since these are system commands, they ought to be in /usr/local/sbin, but the Python installation system doesn't allow that.

Take a moment to see if the installation worked by asking nftfw for help:

$ nftfw -h
...

The next step is to install the basic control files in /usr/local/etc/nftfw, the working directories in /usr/local/var/lib/nftfw, and the manual pages in /usr/local/share/man.

The Install.sh script will copy files from the distribution into their correct places. It asks several questions and permits you to control the installation phases. It's safe to run the script again, it will not replace the contents of any directory ending with .d, or the two control files in /usr/local/etc/nftfw. The script uses the standard system install program to do its work.

It's a good idea to make files in /usr/local/etc/nftfw owned by a non-root user, so they are easier to change without using sudo. For Symbiosis, the user should be admin, for Sympl it will sympl. The script asks for a user name and will create these files owned by that user. Later, it's important to edit /usr/local/etc/nftfw/config.ini to tell nftfw the user that you selected.

Take care, and slowly...

$ sudo sh Install.sh
...

Answers for default installation:

  • Install under /usr/local? yes
  • See the files installed? your choice
  • Install? yes
  • User to replace root? 'admin' for Symbiosis, 'sympl' for Symbl, 'return' for root
  • Install Manual pages? yes

In /usr/local/etc/nftfw, you will find two files: config.ini and nftfw_init.nft. config.ini provides configuration information overriding coded-in settings in the scripts. All entries in the distributed files are commented out using a semi-colon at the start of the line. nftfw_init.nft is the framework template file for the firewall. It's copied into the build system whenever a nftfw creates a firewall. Also, you'll find the original directory holding all the original settings for the files. The intention is to provide a place for later updates to supply new and fixed default files.

install.sh also creates the necessary directories into /usr/local/var/lib/nftfw.

The final stage of the installation is to copy manual pages into /usr/local/share/man. There are six pages:

  • nftfw(1) - manage the Nftfw firewall generator. Describes the main command that creates and manages firewall tables.
  • nftfwls(1) - list the sqlite3 database used for storing IP addresses that have shown themselves to be candidates for blocking.
  • nftfwedit(1) - provides a command line interface to inspect IP addresses (both in and not in the blacklist database), and tools to add and delete IP addresses in the database, optionally adding them to the active blacklist.
  • nftfwadm(1) - provides some tools that may be useful when installing the system.
  • nftfw-config(5) - describes the contents of the ini-style config file tailoring settings in nftfw.
  • nftfw-files(5) - the format, names and contents of the files used to control the system.

The man command may need '5' in the command line to display the section 5 manual pages. Incidentally, the distribution also has these manual pages in HTML format (see docs/man).

As distributed, Debian Buster comes out of the box using nfttables as the basic firewall with a compatibility mode for iptables installed. Your system may vary. nftfw introduces specific nftables constructs, perhaps adding sets into the mix, the iptables interface will break down. You may have listed the kernel firewall with:

$ sudo iptables -L -v -n

and now you need to re-educate yourself to run:

$ sudo nft list ruleset

You are now ready to create firewall to suit your needs.

Paying attention to config.ini

Find the Owner section in the file and change settings for owner and group to fit the user you selected when installing the etc/nftfw files.

If you are running nftables and have a live /etc/nftables.conf file, you may need to alter the nftables_conf setting in config.ini and please read on. If not, skip to Logging.

Debian expects systems using nftables to keep a configuration file in /etc/nfttables.conf. The file sets up nftables when the system reboots, or when systemctl restarts the nftables service. nftfw will write this file after creating its rule set but depends on configuration in its config.ini file to set its location. As distributed, the value of nftables_conf in config.ini is relative to the installation root. This means you need to take different actions depending on where your nftfw is installed:

  • For nftfw installed in /usr/local: The default setting of nftables_conf will be /usr/local/etc/nftables.conf, which is the 'wrong' location, but is safe for now. You will eventually need to change the nftables_conf setting in config.ini to /etc/nftables.conf. Once you are happy with nftfw, you will then change the setting to its correct location in /etc.

  • For nftfw installed in /: The default setting of nftables_conf will be /etc/nftables.conf, which is the 'right' location, but maybe dangerous now. You probably should change the default setting in nftfw/config.ini to prevent it writing or installing /etc/nftables.conf until you are happy. Change the setting to place the file in perhaps /etc/nftfw/nftables.conf.new for now, and change it back later.

Logging

All nftfw programs will write logging message to syslog, and also to the terminal. Error messages are output using logging level ERROR, and information messages using INFO. The scripts turn off direct printing output unless they are talking to a terminal. The scripts all have a -q (quiet) flag suppressing terminal output.

The logging level displayed by the scripts is set by a value in the configuration file config.ini, and this defaults to ERROR, so only error messages are displayed. The scripts have a -v (verbose) flag that raises the output level to INFO, showing the information messages. Alternatively, the loglevel setting can be set in config.ini to always show these messages.

;loglevel = ERROR

to

loglevel = INFO

Using Symbiosis/Sympl control files

nftfw uses the same format for the control files found in the Symbiosis/Sympl firewall directory /etc/symbiosis/firewall.d or /etc/sympl/firewall.d. There is a setting in config.ini which tells nftfw where to look for these files, and this may point at the current directory. The five directories, incoming.d, outgoing.d, blacklist.d, whitelist.d and patterns.d are usable by nftfw.

nftfw makes three changes to the file format for patterns that make nftfw pattern files incompatible with those used by Symbiosis.

  • It's possible to set the ports= value in a pattern file to the word 'update'. The idea is to use the pattern scanning system to look in /var/log/syslog for messages logged by the firewall and use that information to update counts in the blacklist database. The 'update' action doesn't create a new record, it simply updates counts. A blacklisted site will often continue to hammer at the closed door, and the 'update' scan keeps the blacklisting from timing out until the site stops sending packets.

  • It's possible to set the ports= value in a pattern file to the word 'test'. See Testing regular expressions in the Users Guide.

  • It's possible to use shell-style 'glob' expressions in the file= statement, enabling the scanning of several related log files by the same set of regular expressions. This is useful for writing one set of rules for several websites whose log files are in separate files.

nftfw has its own set of rules used by actions to create control lines in the firewall. Symbiosis keeps its rules in /usr/share/lib/symbiosis/firewall/rule.d and nftfw has moved its files to its own directory in /usr/local/etc/nftfw/rule.d. Several of nftfw's rules are there to provide compatibility with Symbiosis and some of these return nothing because the functionality has migrated into the firewall framework provided by nftfw_init.nft.

Migrating to nftfw

The nftfw command builds the firewall, installs it in the kernel and saves a copy of what it has created in /etc/nftables.conf. Debian expects to reload nftables from the file in /etc on a reboot.

If your system has a running firewall that's not nftfw then you probably don't want to be too hasty about installing the system. I have no expectation that things will go wrong, but the ability to go back is important. The steps below provides ways to revert, and explain how to do the installation safely.

First, make sure you open another window to the system and login to the machine with it, this will keep the connection open and you can recover if something goes wrong. See the Users Guide for An example of incoming.d to see how to do that.

Let's look at where you might be now:

  • The system runs a Symbiosis or Sympl system using iptables. In this case, the running firewall can be re-installed using the {symbiosis|sympl}-firewall command. You should move /etc/cron.d/symbiosis-firewall and /etc/incron.d/symbiosis-firewall (or equivalent sympl files if present) to a safe place to stop their firewalls from running for now.
  • Or: The system runs a different system using iptables, you can usually use iptables_save to save the settings somewhere, and iptables_restore to put the old rules back.
  • Or: Your system already runs an nftables based firewall, and you want to try nftfw out. In this case, do make sure that nftfw won't overwrite your /etc/nftables.conf file.
  • Or: There may be other options.

If you have a live iptables system, check the information about iptables versions and how to set things up at the top of this document.

Having set up the directories in /usr/local/etc/nftfw and the working directories in /usr/local/var/lib/nftfw, you can run:

$ sudo nftfw -x -v load

The -x flag makes nftfw compile the files used in setting up the firewall, tests their syntax using the nft command, but doesn't install them. The -v flag prints information messages, and is a good idea if you've not altered logging levels in config.ini.

Assuming the tests show that the installation is OK, then you will have a working firewall to install. At this point, if you want to know what will happen, you can change to /usr/local/var/lib/nftfw/test.d and look at the files. The starting point for the firewall is nftfw_init.nft, which then includes further files. We cannot get a fully compiled set without loading the actual kernel tables, and you could do that if you are feeling confident, but not until you've taken the next step.

Installing

If you are on a Symbiosis or Sympl system, I recommend at this point that you move /etc/cron.d/{symbiosis|sympl}-firewall to a safe place. You don't want this firing when you are doing the next sequence of commands, so remove it from cron. On a Symbiosis system, you also need to move /etc/incron.d/symbiosis-firewall to a safe place, you don't want this starting Symbiosis if files in /etc/symbiosis/firewall.d change.

If you have a running nftables or iptables installation, now is the time to run:

$ sudo nftfwadm save

this saves your nftables settings into nftfw's backup system, so if the install does fail, it will revert to what you had there before you started meddling. If you are using an iptables based system, fear not, the nft command doing the work will save the iptables settings in nftables format, assuming you have the nf_tables version of iptables installed and have working tables.

If all is well, you can try loading the rules made by nftfw.

$ sudo nftfw -f -v load

The -f forces a full load and you'll need it if you've run the test. The point of saving the original settings will be apparent if something bad happens on this load, nftfw will reload the kernel state from your saved settings.

If there was no error, you can now see all the tables using:

$ sudo nft list ruleset

If you need to revert, then now's the time.

$ sudo nftfwadm restore

replaces the newly installed rules with the tables you saved. The restore command will delete the backup file you stored, so you will need to run save again if you plan to change things and try again.

If you are happy with nftfw, then you are nearly all set. If you've used the save command to nftfwadm, then you need to remove the backup version of your old settings from the system.

$ sudo nftfwadm clean

simple deletes the backup file. Don't leave it installed, nftfw makes a backup file on every run so it can backtrack. However, it doesn't create a new file if it exists, and you need to remove your original settings from the system. If you don't remove the file and there's a problem some time in the future, you may find yourself wondering why the firewall in the system is an ancient version. Just where did that come from?

Final steps

To tidy up, check that the setting of nftables_conf in nftfw's config file /usr/local/etc/nftfw/config.ini reads:

#  Location of system nftables.conf
#  Usually /etc/nftables.conf
nftables_conf = /etc/nftables.conf

and run

$ sudo nftfw -f load

again to make sure that the new rules are written into /etc/nftables.conf.

Tell systemctl to enable and start its nftables service.

$ sudo systemctl enable nftables
$ sudo systemctl start nftables
$ sudo systemctl status nftables

On a boot of a Symbiosis or Sympl system, the firewall starts at network up time and closes at network down time. Change into /etc/network and delete if-up.d/{symbiosis|sympl}-firewall and if-down.d/{symbiosis|sympl}-firewall}. The file is a symbolic link to the firewall script. This turns out to be an important step, rebooting without having this done results in a bad combination of two firewalls, because the nftables settings are loaded before the Symbiosis/Sympl ones.

Setting cron and incron

Like Symbiosis/Sympl, nftfw uses cron to drive regular polls by the firewall loader, and blacklist and whitelist scanners. nftfw will rebuild its tables when triggered from incron monitoring the four control directories in /usr/local/etc/nftfw. You'll find sample cron and incron control files in the cronfiles directory in the distribution. Hopefully, by now, you've moved the Symbiosis or Sympl versions to somewhere where they are not activ4.

Look in cronfiles in the nftfw distribution. The files there have /usr/local/ in them, if your system is installed from root, you'll need to edit both files to point to the correct locations. Install cron.d-nftfw in /etc/cron.d/nftfw, and if you are using incron, install incron-nftfw in /etc/incron.d/nftfw (if not, remember to edit config.ini to tell nftfw).

Geolocation

The listing program nftfwls will print out the country that originated packets in the firewall using the geoip2 country database available from MaxMind. MaxMind don't charge but want you to create an account with them to access their files.

See Installing Geolocation.

Sympl users: Update your mail system after installation

A repository that steps through the changes I make to the standard exim4/dovecot systems on Sympl to improve feedback and detection of bad IPs - see Sympl mail system update.

You Are There

Now look at:

Acknowledgement

All of this is made possible by shamelessly borrowing ideas from Patrick Cherry who created the Symbiosis hosting package for Bytemark of which the firewall system is part.