How to effectively block nearly 300K subnets in Apache

Question

Let me start about Why...

Why

Since most people agree that the most effective way of blocking Ip's is by using a solution like iptables. So with that in mind why would i wanna block such a huge list directly in Apache. As a maintainer of servers and after investigating "unwanted traffic" trying to do all kinds of things that the online service isnt made for i started looking for a solution to stop that once and for all....

So in the past 2 years i identified all/most possible networks that produce "unwanted traffic". These networks have no use on the online service at all and are bassicly bad for the environment when u look at all the energy and resources that they consume!

So in my attempt to remove all the power from these services i made a list of all worldwide hosting addresses and bussines connections that are outside of the targetted area from the online service (which means all business connections unless they are inside 2 targetted countries and all hosting addresses).

This resulted in almost 300k subnets which after combining the subnets resulted in almost 290k subnets left that i wanna block.

When using this list in a solution like iptables it would also block updating services so to prevent this it's better to use apache since it will then only affect apache and nothing else on this server, protecting an entire server is not inside the scope of this question.

Possible solutions

Adding the subnets to .htaccess (Don't rlly wanna go there since this will make apache reload the list on every request)

Using

RewriteEngine on
RewriteMap    hosts-deny  "txt:/path/to/hosts.deny"
RewriteCond   "${hosts-deny:%{REMOTE_ADDR}|NOT-FOUND}" "!=NOT-FOUND" [OR]
RewriteCond   "${hosts-deny:%{REMOTE_HOST}|NOT-FOUND}" "!=NOT-FOUND"
RewriteRule   "^"                                      "-"           [F]

Works very well when using single addresses but converting 300k subnets to individual addresses doens't look very appealing to me, so unless this can be transformed to work with subnets this isn't going to work either.

Using expressions

RewriteCond expr "-R '12.345.678.90/28'"

This works untill i try to add more then 1 subnet, i think it goes wrong with RequireAll, RequireAny, RequireNone or the location it is placed.

Hopefully someone can shed a light on this and guide me towards the right way of implementing my solution.

Welcome to Stack Overflow. Wow! Thanks for showing your efforts on your first question :) — Example person, Jan 31 '22 at 12:59
Me myself, would create a proxy server program that rejects all the connections from those IPs using a [Hash Table](https://en.wikipedia.org/wiki/Hash_table), instead of making Apache serve all the requests with 403 forbidden status — Example person, Jan 31 '22 at 13:03
In this particular case i want the server to be able to reach those blocked addresses, as in example csf would block the entire in and outgoing traffic where apache doesn't. Can u clarify a bit more how a proxy would be better then apache? Also apache seems to have hashing options when looking at their docs, — JeroenL, Jan 31 '22 at 13:35
mainly to prevent server update issues, and i got quite some server resources in this case — JeroenL, Jan 31 '22 at 13:39
"*In this particular case i want the server to be able to reach those blocked addresses, as in example csf would block the entire in and outgoing traffic where apache doesn't.*", the *proxy* thing that I refer to, is a program that sits in between the client and Apache, and affects only Apache. — Example person, Jan 31 '22 at 13:51
"*Also apache seems to have hashing options when looking at their docs,*", let me know where they have mentioned it. I have never seen that — Example person, Jan 31 '22 at 13:52
While i'm very tempting to look into the proxy solution in the comming days, i also have cases where the proxy won't be an option if that's the case how could it be done in apache? — JeroenL, Jan 31 '22 at 13:56
"let me know where they have mentioned it. I have never seen that " https://stackoverflow.com/questions/15579620/how-to-block-100-000-individual-ip-addresses/15580768 Also apache have proxy options, was that what u ment? — JeroenL, Jan 31 '22 at 14:12
No. You said that Apache has hashing options. I cannot see it anywhere in the docs. Also, does that link answer your question? — Example person, Jan 31 '22 at 14:13
when u look at the answer it links to http://httpd.apache.org/docs/current/rewrite/rewritemap.html#dbm His answer works for ip's not cidr. I did manage to get it working like Require all granted Require not ip 4.0.0.0/9 Require not ip 223.255.227.0/24 — JeroenL, Jan 31 '22 at 15:45
The answer links to http://httpd.apache.org/docs/current/rewrite/rewritemap.html#dbm His answer works for ip's not cidr. I did manage to get it working like Require all granted Require not ip 4.0.0.0/9 Require not ip 223.255.227.0/24 This file can then be included in httpd or vhost that way it works with 50-75k lines on a small testing machine, only bottleneck i have seen so far is if it can start the apache service fast enough with all the rules, the test server failed with 300k rules by running out of time. — JeroenL, Jan 31 '22 at 15:50
I'm still working on it, as soon as i got all the results ill create an answer! So far, in an Nginx -> Apache config where nginx kinda functions as the proxie described above it works perfectly, on that testmachine no performance penalty at all and since it has been running for 1.5 days no abuse detected on that website at all, i believe this will replace all the google captcha integrations and remove another annoying feature for visitors. But ill update when i'm done testing in different environments etc. — JeroenL, Feb 03 '22 at 10:06

How to effectively block nearly 300K subnets in Apache

0 Answers0