Semantic Log Search: Eliminating the "Splunk Tax" in Network Operations
During a core network disruption, telemetry data is your most critical asset. Syslog feeds, BGP state changes, and DHCP lease events contain the exact forensic breadcrumbs needed to diagnose root causes.
Yet, traditional DDI (DNS, DHCP, IPAM) and network management platforms treat unstructured logs like toxic waste. Lacking the database architecture to store and query heavy event streams, legacy systems dump uncompressed raw text directly onto external Security Information and Event Management (SIEM) tools like Splunk or Datadog.
This handoff has created a painful financial reality for enterprise IT teams: the Splunk Tax. Organizations pay thousands to tens of thousands of dollars per gigabyte in annual ingestion and indexing fees just to store basic network logs. Worse, during critical outages, engineers are still left manually writing complex regular expressions (regex) or proprietary query strings to hunt for clues.
The approach we are building toward is to embed local vector search in the core control plane, so engineers can search logs in natural language without third-party SIEM costs. This post describes that design. Semantic log search is in development and is not yet available in the product.
The Failures of Traditional Log Analytics
The traditional model of offloading network telemetry to external observability stacks creates two operational problems:
- Ingestion Taxes: SIEM providers charge based on daily data ingestion volume. When a core switch experiences a link flap storm, it generates millions of repetitive syslog lines, driving up the ingestion bill.
- Brittle Keyword Searching: Traditional search tools rely on exact string matching or complex regex syntax. If an engineer searches an enterprise log aggregator for "connection dropped", but the router log records "TCP session reset by peer", traditional string matching returns nothing.
Local Embeddings via bge-small
To deliver real-time intelligence without external cloud APIs or high processing overhead, the design runs a small embedding model directly at the data ingress layer.
With a small CPU-bound model such as bge-small, incoming unstructured syslog lines and network events are converted into 384-dimensional floating-point vectors. We have not published encoding times.
Local encoding has three intended benefits:
- Zero Cloud API Costs: Vector embeddings are calculated locally on your control plane CPU, eliminating external LLM API fees.
- Semantic Context Preservation: Technical log payloads are transformed into mathematical coordinate points that capture the underlying operational meaning of the event rather than just literal characters.
- Immediate Indexing: Vector arrays are written alongside raw JSON log payloads in a unified database step, so telemetry is searchable shortly after it arrives.
Conceptual Root Cause Matching in Postgres 17
Using pgvector inside PostgreSQL 17, the design gives network operations teams semantic query matching.
Instead of writing complex SQL queries with nested LIKE operators, engineers and automated agents can query the event ledger using natural language concepts or broad operational questions. The database executes a cosine vector similarity match (using distance operators like <=>) to find conceptually related events.
- Query: "Find all IP conflicts and DNS resolution drops in Austin"
- Matched Log 1: "DHCP IP 10.10.4.12 duplicate MAC detected on port 2"
- Matched Log 2: "Upstream resolver 10.10.0.1 unreachable timeout"
Because vector similarity evaluates mathematical proximity in coordinate space, the database can surface related log lines even when they use different terminology than the query. The matches above are an illustration of the intended behavior, not output from a shipped feature.
Bound Storage Footprints via Smart Vector Partitioning
A common concern with storing vector embeddings locally is storage growth. Storing multi-dimensional vector arrays for millions of events per day can rapidly expand disk usage if managed poorly.
To keep database footprints lightweight and cost-effective, the design uses telemetry filtering and automated partition pruning:
- Ingress Anomaly Filtering: Routine, non-essential "heartbeat" logs are filtered at the edge. Only telemetry representing state changes, protocol updates, configuration drifts, or operational errors is passed to the vector encoder.
- Automated Table Partitioning: Logs and vector indexes are stored in range-partitioned tables bound by time windows (such as daily partitions).
- Hot/Cold Lifecycle Management: As partitions age past active retention windows (such as 30 days), old vector partitions are dropped as whole partitions, which avoids long-running delete queries and table fragmentation.
Lower-Cost Observability
Enterprise networks generate a lot of telemetry, and raw data is of little use without fast retrieval.
By bringing local vector embeddings and semantic search directly into your primary control plane database, you can avoid exporting routine network logs to third-party aggregators. That keeps the ingestion bill down and gives engineers a natural language way to search during an incident, once the feature ships.