Faster IPSET loading

By | May 19, 2013

IPSETs are a very efficient way to manage a large list of IP addresses for your iptables firewall. Rather than have an individual rule for each and every address or network that needs to be dropped or rejected, with ipsets you can have a single iptables rule that tests an entire list of addresses. The rule would look something like this:

-I INPUT -m set --match-set blacklist src -j DROP

where the -m set tells iptables to look for an ipset with a name given by the --match-set option. In this case, the ipset’s name is blacklist. (Note: this will give you an error if you haven’t already created an ipset by that name). You can create an ipset by scripting something like this:

ipset --create blacklist iphash

After the ipset has been created, the next task is to load the IP addresses from the blacklist into it. Bonekracker has a nice post on the Gentoo forums with information on how to do this from a text file with blacklisted IPs losartan potassium 100mg.

When I tried to use his approach for a 3,000+ entry blacklist, derived from my recent botnet attack, I found that it took a very long time (more than a minute) to load the blacklist into the ipset. A primitive approach to profiling showed that the slowdown was in the following code:

while [ $((--i)) -ge 0 ]; do
ipset --add temp_ipset ${networks[i]}
done

where ipset --add ipset_name ip is the code that adds IP addresses, one at a time, to the ipset. Unfortunately, this is very slow.

A faster approach is to use the ipset restore command. In order to use it, the text file needs to be formatted in a special way by adding add ipset_name in front of each IP address in the list, like so:

add temp_ipset 223.205.23.116 -exist
add temp_ipset 223.206.41.73 -exist
add temp_ipset 223.207.124.81 -exist
add temp_ipset 223.207.179.208 -exist

Then you can replace the while loop above with the following code, and voila, a 10x to 100x speedup.

$ipset restore < $ip_list

With this change, loading the ipset is nearly instantaneous and I have no qualms about loading my 3,696 entry bad boy list into an ipset on each bootup, effectively locking the barn door now that the horse is gone.

Hope this helps you make use of ipsets for your firewall.

9 thoughts on “Faster IPSET loading

  1. Josh

    Hi Jeff,

    I am just starting to learn and play with ipset and I was wondering if is there a way to test if all the IP’s from the blacklist have been loaded. I have a script which is supposed to load IPs from China, but I still see in my logs that IPs from China are trying to login on my server. I searched the logged IP’s in my blacklist and they are there, so it should have been blocked.

    Reply
    1. Jeff Davis Post author

      HiJosh,

      You can readily sheck individual addresses:

      sudo ipset test blacklist

      Hope this helps
      Jeff

      Reply
  2. Linux_lover

    Awesome! Thanks a million, I was dealing with adding a great range of ports to an special set because it took around 11 seconds now I add a 10X bigger range in just less than a second

    Reply
    1. Jeff Davis Post author

      Glad this helped. Good luck with your project!

      Reply
      1. Linux_lover

        Hi Jeff, sorry for late reply 😀 . Just to improve your solution, it is a great way of doing but if it sees an existing entry in set it would pop an error and not continue reading rest of file. For solving this add an “-exist” to each line. Then it would work perfectly awesome! 🙂

        Reply
        1. Jeff Davis Post author

          Great suggestion, thanks! I’ve revised the posting to reflect it. In my Ruby application, I flush the ipset before doing the restore, and have code to check for dupes, so I never saw errors from existing entries. I probably could have avoided a bunch of coding if I had paid more attention to the options! Thanks for sharing.

          Reply
        2. NOYB

          Or you could just add the – exist flag on the ipset command line. Like one of these…
          ipset restore -exist < block.set
          ipset restore -exist -file block.set
          cat block.set | ipset restore -exist

          Reply
  3. NOYB

    Can avoid the file redirections altogether by piping directly to ipset restore command.
    I find this to be about 30x faster than discrete “ipset add” commands for a set of about 115000. (30 seconds vs. 15 minutes.

    #! /bin/bash

    # Create the permanent set if it does not exist.
    # Do this at beginning of process to ensure a failure doesn’t prevent the set from getting created. Which could prevent iptables from loading.
    ipset create -exist ${firewall_ipset} ${ipset_params}

    # Create a temp set and ensure it is empty.
    temp_ipset=”${firewall_ipset}_temp”
    ipset create -exist ${temp_ipset} ${ipset_params}
    ipset flush ${temp_ipset}

    # Load the list into the temp set. Ignoring any duplicate entries.
    cat ${data_file} \
    | while read network; do
    # ipset add -exist ${temp_ipset} ${network} # This is very slow (pipe to ipset restore command instead)
    echo ‘add ‘${temp_ipset}’ ‘${network} # This is about 30x faster (faster even than file redirect)
    done \
    | ipset restore -exist

    # Activate the new set (swap the temp set with the permanent set).
    ipset swap ${temp_ipset} ${firewall_ipset}
    ipset destroy ${temp_ipset}

    # Save the ipset so it will be reloaded automatically at startup.
    ipset save -file ${ipset_dir}/${firewall_ipset}.set ${firewall_ipset}

    # Or the stdout save method if SELinux blocks ipset save -file operation
    ipset save > ${ipset_dir}/${firewall_ipset}.set ${firewall_ipset}

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.