IPSETs are a very efficient way to manage a large list of IP addresses for your iptables firewall. Rather than have an individual rule for each and every address or network that needs to be dropped or rejected, with ipsets you can have a single iptables rule that tests an entire list of addresses. The rule would look something like this:
-I INPUT -m set --match-set blacklist src -j DROPwhere the -m set tells iptables to look for an ipset with a name given by the --match-set option. In this case, the ipset’s name is blacklist. (Note: this will give you an error if you haven’t already created an ipset by that name). You can create an ipset by scripting something like this:
ipset --create blacklist iphashAfter the ipset has been created, the next task is to load the IP addresses from the blacklist into it. Bonekracker has a nice post on the Gentoo forums with information on how to do this from a text file with blacklisted IPs losartan potassium 100mg.
When I tried to use his approach for a 3,000+ entry blacklist, derived from my recent botnet attack, I found that it took a very long time (more than a minute) to load the blacklist into the ipset. A primitive approach to profiling showed that the slowdown was in the following code:
while [ $((--i)) -ge 0 ]; do
ipset --add temp_ipset ${networks[i]}
donewhere ipset --add ipset_name ip is the code that adds IP addresses, one at a time, to the ipset. Unfortunately, this is very slow.
A faster approach is to use the ipset restore command. In order to use it, the text file needs to be formatted in a special way by adding add ipset_name in front of each IP address in the list, like so:
add temp_ipset 223.205.23.116 -exist
add temp_ipset 223.206.41.73 -exist
add temp_ipset 223.207.124.81 -exist
add temp_ipset 223.207.179.208 -existThen you can replace the while loop above with the following code, and voila, a 10x to 100x speedup.
$ipset restore < $ip_listWith this change, loading the ipset is nearly instantaneous and I have no qualms about loading my 3,696 entry bad boy list into an ipset on each bootup, effectively locking the barn door now that the horse is gone.
Hope this helps you make use of ipsets for your firewall.
Hi Jeff,
I am just starting to learn and play with ipset and I was wondering if is there a way to test if all the IP’s from the blacklist have been loaded. I have a script which is supposed to load IPs from China, but I still see in my logs that IPs from China are trying to login on my server. I searched the logged IP’s in my blacklist and they are there, so it should have been blocked.
HiJosh,
You can readily sheck individual addresses:
sudo ipset test blacklist
Hope this helps
Jeff
Awesome! Thanks a million, I was dealing with adding a great range of ports to an special set because it took around 11 seconds now I add a 10X bigger range in just less than a second
Glad this helped. Good luck with your project!
Hi Jeff, sorry for late reply 😀 . Just to improve your solution, it is a great way of doing but if it sees an existing entry in set it would pop an error and not continue reading rest of file. For solving this add an “-exist” to each line. Then it would work perfectly awesome! 🙂
Great suggestion, thanks! I’ve revised the posting to reflect it. In my Ruby application, I flush the ipset before doing the restore, and have code to check for dupes, so I never saw errors from existing entries. I probably could have avoided a bunch of coding if I had paid more attention to the options! Thanks for sharing.
I’m glad I could be useful 🙂
Or you could just add the – exist flag on the ipset command line. Like one of these…
ipset restore -exist < block.set
ipset restore -exist -file block.set
cat block.set | ipset restore -exist
Can avoid the file redirections altogether by piping directly to ipset restore command.
I find this to be about 30x faster than discrete “ipset add” commands for a set of about 115000. (30 seconds vs. 15 minutes.
#! /bin/bash
# Create the permanent set if it does not exist.
# Do this at beginning of process to ensure a failure doesn’t prevent the set from getting created. Which could prevent iptables from loading.
ipset create -exist ${firewall_ipset} ${ipset_params}
# Create a temp set and ensure it is empty.
temp_ipset=”${firewall_ipset}_temp”
ipset create -exist ${temp_ipset} ${ipset_params}
ipset flush ${temp_ipset}
# Load the list into the temp set. Ignoring any duplicate entries.
cat ${data_file} \
| while read network; do
# ipset add -exist ${temp_ipset} ${network} # This is very slow (pipe to ipset restore command instead)
echo ‘add ‘${temp_ipset}’ ‘${network} # This is about 30x faster (faster even than file redirect)
done \
| ipset restore -exist
# Activate the new set (swap the temp set with the permanent set).
ipset swap ${temp_ipset} ${firewall_ipset}
ipset destroy ${temp_ipset}
# Save the ipset so it will be reloaded automatically at startup.
ipset save -file ${ipset_dir}/${firewall_ipset}.set ${firewall_ipset}
# Or the stdout save method if SELinux blocks ipset save -file operation
ipset save > ${ipset_dir}/${firewall_ipset}.set ${firewall_ipset}