Scaling IP Intelligence: Architecture for 10M+ Daily Lookups

Ingesting IP intelligence data—such as VPN detection, geolocation, and proxy status—is a standard requirement for modern security stacks. However, integrating these lookups into high-traffic authentication flows or firewall rules introduces significant latency challenges. When scaling from thousands to millions of requests per day, a naive implementation of synchronous HTTP calls will degrade application performance and exhaust thread pools.

This guide outlines the architectural patterns required to integrate IPASIS (or similar IP intelligence APIs) at scale.

1. Implement a Cache-Aside Strategy

IP metadata is relatively static. Geolocation data rarely changes for a specific subnet, and while IP reputation (VPN/Proxy status) is dynamic, it does not fluctuate by the millisecond. Querying the API for every request from the same IP is redundant and costly.

The Architecture:

1. L1 Cache (In-Memory): Store highly active IPs in local memory (e.g., an LRU cache) for microsecond access. Set a short TTL (e.g., 5 minutes).
2. L2 Cache (Distributed): Use Redis or Memcached to share lookups across service instances. Set a longer TTL (e.g., 1-6 hours).

Python Implementation (Redis Cache-Aside)

import redis
import requests
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def get_ip_data(ip_address):
    # Check L2 Cache
    cache_key = f"ip_data:{ip_address}"
    cached_data = r.get(cache_key)
    
    if cached_data:
        return json.loads(cached_data)
    
    # Cache Miss: Call IPASIS API
    response = requests.get(f"https://api.ipasis.com/v1/{ip_address}", timeout=2)
    
    if response.status_code == 200:
        data = response.json()
        # Store in Redis with 4-hour TTL
        r.setex(cache_key, 14400, json.dumps(data))
        return data
    
    return None

2. Connection Pooling and Keep-Alive

Establishing a new TCP/TLS handshake for every API request adds significant overhead (often 50ms+ per request). When handling high throughput, you must reuse connections.

Ensure your HTTP client is configured to use Persistent Connections (Keep-Alive). This maintains an open socket to the API provider, drastically reducing latency after the initial request.

Node.js Example (Axios with HTTP Agent)

const axios = require('axios');
const http = require('http');
const https = require('https');

// Configure Agents to keep sockets alive
const httpAgent = new http.Agent({ keepAlive: true });
const httpsAgent = new https.Agent({ keepAlive: true });

const ipasisClient = axios.create({
  baseURL: 'https://api.ipasis.com/v1/',
  httpAgent,
  httpsAgent,
  timeout: 2000 // Strict timeout prevents cascading failures
});

async function lookupIP(ip) {
  try {
    const response = await ipasisClient.get(`/${ip}`);
    return response.data;
  } catch (error) {
    console.error(`Lookup failed for ${ip}`);
    return null;
  }
}

3. Asynchronous Analysis vs. Blocking Gates

Not all IP checks need to block the user. Distinguish between Blocking Gates (login, payments) and Passive Analysis (analytics, fraud scoring update).

For passive analysis, decouple the API lookup from the request lifecycle using a message queue (e.g., Kafka, RabbitMQ). This ensures that network fluctuations in the IP lookup do not impact the user's response time.

Go Example (Concurrent Worker Pool)

For high-volume ingestion, use Go routines to process batches of IPs concurrently.

package main

import (
	"fmt"
	"net/http"
	"sync"
	"time"
)

func worker(ips <-chan string, wg *sync.WaitGroup) {
	defer wg.Done()
	client := &http.Client{
		Timeout: 2 * time.Second,
	}

	for ip := range ips {
		// In production, handle errors and JSON parsing
		resp, err := client.Get("https://api.ipasis.com/v1/" + ip)
		if err == nil {
			fmt.Printf("Processed %s: %d
", ip, resp.StatusCode)
			resp.Body.Close()
		}
	}
}

func main() {
	ips := []string{"1.1.1.1", "8.8.8.8", "1.2.3.4"} // ... millions of IPs
	ipChannel := make(chan string, 100)
	var wg sync.WaitGroup

	// Start 50 workers
	for i := 0; i < 50; i++ {
		wg.Add(1)
		go worker(ipChannel, &wg)
	}

	for _, ip := range ips {
		ipChannel <- ip
	}
	close(ipChannel)
	wg.Wait()
}

4. Circuit Breaking

When scaling systems, you must assume external dependencies can experience degradation. If the API provider experiences latency, your application should not hang.

Implement a Circuit Breaker:

1. If API failure rate exceeds X% or latency exceeds Y ms, "open" the circuit.
2. Fail open (allow the user) or fail closed (block the user) based on your risk appetite.
3. After a cooldown period, allow a single request through to test system health.

FAQ

Q: What is the ideal TTL for IP Intelligence data? For Geolocation, 24 hours is standard. For Threat Intelligence (VPN/Proxy detection), we recommend 1 to 4 hours. IP reputation is fluid; caching for too long may result in false negatives.

Q: How do I handle rate limits when scaling? Always respect the X-RateLimit headers. Implement exponential backoff for 429 errors. However, at enterprise scale (millions/day), you should negotiate a custom quota rather than relying on standard tier limits.

Q: Should I perform IP lookups on the client side? Never. Exposing your API keys in frontend code is a security risk. Always proxy requests through your backend architecture.

---

Enterprise-Grade IP Intelligence

Scaling your architecture is only half the battle. You need a data provider that can handle the load. IPASIS is engineered for high concurrency, offering sub-50ms latency and 99.9% uptime for enterprise workloads.

Get your API Key and start scaling your security stack today.