Whitelist Design Pattern
When designing a whitelist mechanism for an HTTP Intrusion Detection System (IDS), several aspects need to be considered: rule definition, rule management, traffic matching, performance optimization, and security. The design approach is as follows:
1. Rule Definition
Whitelist rules define which requests are legitimate and normal. Rules can be based on the following aspects:
- IP Address: Allow requests from specific IP addresses or IP ranges.
- URL Path: Allow specific URL paths or URL patterns.
- HTTP Method: Allow specific HTTP methods (such as GET, POST, etc.).
- Request Headers: Allow specific request headers or header patterns.
- Parameters and Values: Allow specific parameter and value patterns.
Example Rules
{
"ip_whitelist": ["192.168.1.1", "192.168.1.0/24"],
"url_whitelist": ["/index.html", "/api/v1/*"],
"method_whitelist": ["GET", "POST"],
"header_whitelist": {
"User-Agent": ["Mozilla/*", "Chrome/*"]
},
"param_whitelist": {
"token": ["^[a-zA-Z0-9]{32}$"]
}
}
2. Rule Management
Whitelist rules need to be easily managed and updated. Rules can be stored and managed using configuration files or databases, and an interface can be provided to dynamically update the rules.
Configuration File Method
Use configuration files in formats like JSON or YAML to store rules.
{
"ip_whitelist": ["192.168.1.1", "192.168.1.0/24"],
"url_whitelist": ["/index.html", "/api/v1/*"],
"method_whitelist": ["GET", "POST"],
"header_whitelist": {
"User-Agent": ["Mozilla/*", "Chrome/*"]
},
"param_whitelist": {
"token": ["^[a-zA-Z0-9]{32}$"]
}
}
Database Method
Use relational databases or NoSQL databases to store rules and provide an API interface for management.
3. Traffic Matching
Traffic matching involves comparing HTTP requests with whitelist rules to determine if they are legitimate. It can be divided into the following steps:
- IP Address Matching:
- Check if the source IP of the request is in the whitelist.
- URL Path Matching:
- Check if the request URL path is in the whitelist.
- HTTP Method Matching:
- Check if the request HTTP method is in the whitelist.
- Request Header Matching:
- Check if the request headers meet the whitelist rules.
- Parameter Value Matching:
- Check if the request parameters and values meet the whitelist rules.
Example Code
import re
class Whitelist:
def __init__(self, rules):
self.ip_whitelist = rules["ip_whitelist"]
self.url_whitelist = rules["url_whitelist"]
self.method_whitelist = rules["method_whitelist"]
self.header_whitelist = rules["header_whitelist"]
self.param_whitelist = rules["param_whitelist"]
def is_whitelisted(self, request):
if not self.match_ip(request.ip):
return False
if not self.match_url(request.url):
return False
if not self.match_method(request.method):
return False
if not self.match_headers(request.headers):
return False
if not self.match_params(request.params):
return False
return True
def match_ip(self, ip):
# Check if the IP is in the whitelist
for ip_rule in self.ip_whitelist:
if self.ip_in_range(ip, ip_rule):
return True
return False
def ip_in_range(self, ip, ip_rule):
# Implement IP range matching logic
pass
def match_url(self, url):
# Check if the URL is in the whitelist
for url_rule in self.url_whitelist:
if self.url_match(url, url_rule):
return True
return False
def url_match(self, url, url_rule):
# Implement URL matching logic, support wildcards
pass
def match_method(self, method):
# Check if the HTTP method is in the whitelist
return method in self.method_whitelist
def match_headers(self, headers):
# Check if the headers match the whitelist rules
for header, patterns in self.header_whitelist.items():
if header not in headers:
return False
if not any(re.match(pattern, headers[header]) for pattern in patterns):
return False
return True
def match_params(self, params):
# Check if the parameters match the whitelist rules
for param, patterns in self.param_whitelist.items():
if param not in params:
continue
if not any(re.match(pattern, params[param]) for pattern in patterns):
return False
return True
# Example usage
rules = {
"ip_whitelist": ["192.168.1.1", "192.168.1.0/24"],
"url_whitelist": ["/index.html", "/api/v1/*"],
"method_whitelist": ["GET", "POST"],
"header_whitelist": {
"User-Agent": ["Mozilla/.*", "Chrome/.*"]
},
"param_whitelist": {
"token": ["^[a-zA-Z0-9]{32}$"]
}
}
whitelist = Whitelist(rules)
# Simulated request object for demonstration
class Request:
def __init__(self, ip, url, method, headers, params):
self.ip = ip
self.url = url
self.method = method
self.headers = headers
self.params = params
request = Request(
ip="192.168.1.100",
url="/api/v1/resource",
method="GET",
headers={"User-Agent": "Mozilla/5.0"},
params={"token": "1234567890abcdef1234567890abcdef"}
)
print(whitelist.is_whitelisted(request)) # Should print True if request matches all rules
Initial Whitelist
Creating a universally applicable whitelist involves covering common legitimate traffic patterns while avoiding being too strict to prevent excessive false positives, and not too lenient to allow malicious traffic to evade detection. Below is an initial design for a universally applicable whitelist suitable for most HTTP servers.
1. IP Address Whitelist
The initial whitelist can exclude IP address restrictions unless you have specific trusted IP address ranges. For example, you can allow internal network IP addresses:
{
"ip_whitelist": ["192.168.0.0/16", "10.0.0.0/8"]
}
2. URL Path Whitelist
Allow common static resources and API paths:
{
"url_whitelist": [
"/index.html",
"/about.html",
"/contact.html",
"/css/*", // Allow all CSS files
"/js/*", // Allow all JavaScript files
"/images/*", // Allow all image files
"/api/v1/*" // Allow all API v1 endpoints
]
}
3. HTTP Method Whitelist
Allow common HTTP methods:
{
"method_whitelist": ["GET", "POST", "PUT", "DELETE"]
}
4. Request Header Whitelist
Allow common request header fields and patterns:
{
"header_whitelist": {
"User-Agent": ["Mozilla/.*", "Chrome/.*", "Safari/.*"],
"Accept": ["*/*", "text/html", "application/json"],
"Content-Type": ["application/json", "application/x-www-form-urlencoded", "multipart/form-data"]
}
}
5. Parameter and Value Whitelist
Allow common parameter and value patterns:
{
"param_whitelist": {
"token": ["^[a-zA-Z0-9]{32}$"],
"id": ["^[0-9]+$"],
"name": ["^[a-zA-Z ]+$"]
}
}
Comprehensive Configuration Example & Implementation Code Example
Combining all the above parts into a comprehensive whitelist configuration, and using Python with the Flask framework to implement an HTTP server, you can apply the whitelist filtering during request processing:
import re
from flask import Flask, request, abort
import ipaddress
app = Flask(__name__)
whitelist_rules = {
"ip_whitelist": ["192.168.0.0/16", "10.0.0.0/8"],
"url_whitelist": [
"/index.html",
"/about.html",
"/contact.html",
"/css/*",
"/js/*",
"/images/*",
"/api/v1/*"
],
"method_whitelist": ["GET", "POST", "PUT", "DELETE"],
"header_whitelist": {
"User-Agent": ["Mozilla/.*", "Chrome/.*", "Safari/.*"],
"Accept": ["*/*", "text/html", "application/json"],
"Content-Type": ["application/json", "application/x-www-form-urlencoded", "multipart/form-data"]
},
"param_whitelist": {
"token": ["^[a-zA-Z0-9]{32}$"],
"id": ["^[0-9]+$"],
"name": ["^[a-zA-Z ]+$"]
}
}
def ip_in_whitelist(ip):
for ip_rule in whitelist_rules["ip_whitelist"]:
if ipaddress.ip_address(ip) in ipaddress.ip_network(ip_rule):
return True
return False
def url_in_whitelist(url):
for url_rule in whitelist_rules["url_whitelist"]:
if re.match(url_rule.replace("*", ".*"), url):
return True
return False
def method_in_whitelist(method):
return method in whitelist_rules["method_whitelist"]
def headers_in_whitelist(headers):
for header, patterns in whitelist_rules["header_whitelist"].items():
if header not in headers:
return False
if not any(re.match(pattern, headers[header]) for pattern in patterns):
return False
return True
def params_in_whitelist(params):
for param, patterns in whitelist_rules["param_whitelist"].items():
if param not in params:
continue
if not any(re.match(pattern, params[param]) for pattern in patterns):
return False
return True
@app.before_request
def check_whitelist():
if not ip_in_whitelist(request.remote_addr):
abort(403)
if not url_in_whitelist(request.path):
abort(403)
if not method_in_whitelist(request.method):
abort(403)
if not headers_in_whitelist(request.headers):
abort(403)
if not params_in_whitelist(request.args):
abort(403)
@app.route('/index.html')
def index():
return "Welcome to the index page."
@app.route('/api/v1/resource', methods=['GET', 'POST'])
def api_resource():
return "API v1 resource."
# Additional route definitions...
if __name__ == '__main__':
app.run(debug=True)
How to Implement: Using Flask Hook Functions
- Use the
@app.before_requesthook function to perform whitelist checks before processing each request. If a request does not comply with the whitelist rules, return a 403 Forbidden status code.
Deployment and Usage
- Starting the Server:
- Run the above code to start the Flask HTTP server.
- Testing Access:
- Access
http://localhost:8080/index.htmlto verify if it can be accessed normally. - Access other paths or use request headers or parameters not included in the whitelist to verify if they are blocked.
- Access
By following this approach, we can create an initial, universally applicable HTTP whitelist mechanism. Further configurations should be customized based on the specific HTTP server protection needs.
Another Way to Implement: Filter Module inside SentinelGuard Project
By compositing the whitelists into procedures of detecting intrusion network traffic, it is convenient to implement whitelist function (only done ip whitelist at the current circumstance).