Trigpoint puts a lightweight agent in every POP and cross-measures latency,
loss, jitter, DNS and HTTP across your mesh. It learns what normal looks like
on every path, and when something drifts, it tells the NOC what broke, where,
who's affected, and why — before the customer calls.
Device telemetry is a solved problem. Delivery isn't.
Operators have excellent visibility into devices and links, and almost none into
what the network actually delivers between locations. Interface counters are
nominal, optics are in spec, every BGP session is established — and a customer
in Atlanta is quietly getting 36 ms to Chicago on a path that has run at 19 ms
for two years. Router health is green; the customer experience is a question mark.
Trigpoint closes that gap by measuring the thing you actually sell — path
performance — continuously, from inside your own POPs. The measurement mesh is
table stakes. The product is the intelligence layer that explains why
a measurement changed.
How it works
Measure. Learn. Detect. Explain.
01
Measure everything, from everywhere
A single static Rust binary — under a megabyte, no runtime dependencies —
registers with the control plane over mTLS and executes a signed test
schedule. ICMP, UDP with DSCP marking, TCP connect, DNS against the
resolvers your customers actually use, HTTP with full TLS phase timing,
STAMP with hardware timestamping where the reflector offers it, and Paris
traceroute — UDP, keyed on source port — that enumerates the ECMP paths a
LAG fabric really hashes on instead of smearing them. Router-destined ICMP
is kept honest as control-plane reachability and never folded into
dataplane latency, because a CoPP-rate-limited echo measures the control
plane, not transit. Agents ship 10-second aggregates and keep raw 1-second
samples in a ring buffer, pulled on demand when the correlator wants
forensic detail. When the control plane is unreachable, results spool
locally — the moments you most need data are the moments the network is broken.
Fixed points, cross-measured — the way a country gets surveyed.
02
Learn what normal means, per path, per hour
The magic is never "latency is 42 ms". It's "latency is normally 18–22 ms
on this path at this hour". The baseliner builds per-path, per-metric,
hour-of-week baselines using robust statistics — median and MAD, not mean
and standard deviation, because network latency distributions are
heavy-tailed and often multimodal. A path that ECMPs across two fiber
routes has two normal latencies; Trigpoint stores both modes and
measures deviation from the nearest one. Anomalous windows are excluded
from baseline updates, so an incident never teaches the system that broken
is normal.
03
Detect deviations without crying wolf
Deviation scoring with hysteresis and multi-window confirmation — no
flapping page at 03:14 because one probe crossed a threshold. The detector
recognizes distinct signatures: a level shift (the routing-change shape),
variance increase with p99 inflating before p50 (congestion), loss onset
with flat RTT (hardware or optics, not congestion), and correlated
multi-path anomalies — when many paths through one POP degrade together,
the POP is the problem, not the paths.
04
Explain it with your own routing state
Given an anomaly window and the affected path set, the correlator queries
the event record: did BGP paths change? Did the IGP reconverge? Did an
interface on the inferred path show drops, errors, or optical degradation?
Did the forwarding path itself change — a one-comparison check, because
every traceroute carries a stable path fingerprint? In an MPLS core, where
no ttl-propagate collapses the backbone into one invisible hop,
live IGP topology is the primary path source rather than traceroute: SPF
over the link graph proves which paths crossed a failed link
instead of merely coinciding with it in time. The output is a ranked
hypothesis with evidence and a confidence score. Correlation is rule-based
and explainable first; every root-cause claim ships with the checks behind
it, and "unexplained" is an honest, first-class category — never a guess
dressed up as an answer.
Where the agent sits
Four planes, one mesh
"Deploy an agent in every POP" hides a design question: attached to what,
measuring through what? Trigpoint models four distinct measurement planes,
each with its own targets and cadence — and the agent attaches to the
production forwarding plane, never the management network, because measuring
the OOB network measures the wrong network entirely.
Device
Agent → each local core, aggregation, and PE router — STAMP to line-card reflectors with hardware timestamps.
1 s · localizes a sick device in seconds
Infrastructure
Agent ↔ agent across the backbone, global table, one measured path per attachment pair.
1–10 s primary · 30–60 s cross-attachment
Service
Agent in a customer VRF → remote PE loopbacks, riding the same LSPs and QoS classes your customers do.
30–60 s · the measurement that backs an SLA
External
Agent → transit and peering next-hops, eyeball targets, resolvers, and CDNs, per exit.
30–60 s · per upstream, scorecard-ready
The payoff is localization without inference. Each agent is dual-homed on two
routed uplinks — one per aggregation router, each with its own source address —
not a LAG that deliberately hides which router forwarded the packet. So when
every path sourced via agr2 degrades while the agr1 twins to the same
destinations stay clean, the aggregation uplink is the fault and the alert
says exactly that: a group-by, not a guess. Path identity carries the
attachment, the VRF, and the target's role, so "which layer is sick" becomes a
query — and the cheap intra-POP device plane means the extra dimensions barely
move the probe budget.
Features
Built the way operators run networks
A full probe suite
ICMP, UDP with per-class DSCP marking, TCP connect, path MTU discovery, DNS,
HTTP(S) with DNS/connect/TLS/TTFB phase timing, source-port Paris traceroute,
and STAMP with hardware timestamping. STAMP and TWAMP-Light interoperate with
the reflectors already in your routers — Nokia, Juniper, and Arista line cards
become schedulable vantage points, so aggregation and PE routers get measured
without an agent sitting on them.
Baselines that respect reality
Median + MAD per path, metric, and hour of week. Multimodal baselines for
ECMP twin-route paths. Operator-declared physics floors (fiber-distance
latency) so gross violations alert on day one, while the system is honest
about reduced confidence during its learning period.
Alerts that carry their evidence
Every alert states what deviated and by how much, when it started to the
sample, the blast radius, each evidence source checked with its result, and a
ranked cause with confidence. If Trigpoint can't say most of that, it holds
the alert and keeps gathering — a wrong root cause destroys trust faster than
a slightly late alert.
Route changes as a first-class signal
Each traceroute's hop sequence hashes to a stable path fingerprint, so
detecting a forwarding change is a single comparison — the correlator's
cheapest, highest-value input. "Latency shifted and the path changed
40 seconds earlier" is a root cause, not a mystery.
Your control plane as context
BMP feeds from your route reflectors, IS-IS topology via BGP-LS, gNMI
interface and optics telemetry, RPKI validation state over RTR, inventory
from NetBox. Trigpoint delivers value with the mesh alone and gets
dramatically smarter with each feed you attach.
Impact, not just anomaly
Anomalous paths joined against topology and customer-port inventory answer
the question that makes a NOC take an alert seriously: which services and
which customers are riding the thing that just broke.
An agent you'd let near production
One static Rust binary, under 1 MB, no GC pauses polluting microsecond
timing, kernel timestamping where available. Agents are deliberately dumb:
they execute signed schedules and never decide what to test, so fleet
behavior stays predictable and auditable. Clock-sync quality ships with
every sample; one-way numbers are only trusted when the clocks deserve it.
A mesh that doesn't melt
Full mesh is quadratic: 50 POPs is 1,225 paths, 500 is 124,750. Trigpoint
tiers the mesh — core paths at 1–10 s, regional at 30–60 s, a background
sweep feeding baselines — then escalates frequency and depth automatically
where something smells wrong. With an IGP feed it prunes to the distinct
link-path cover, testing your links rather than your permutations.
Your data, your metal
Self-hosted first — operators don't ship their topology to a SaaS.
ClickHouse stores measurements and routing events side by side so correlation
is a SQL join, not a cross-system export. One modest node (8 vCPU, 32 GB,
NVMe) carries a 50-POP mesh with years of rollups. NOCs live in Grafana, so
Trigpoint meets them there.
Incidents, not alert storms
When forty paths through one link degrade together, that's one event.
Trigpoint rolls anomalies sharing a device, link, or attachment into a single
incident with a timeline — thirty path alarms become one — and exports it as
an evidence pack, the RFO artifact engineers otherwise assemble by hand at
4 a.m.
Maintenance-aware
Declare a window and anomalies inside it are suppressed and annotated,
never dropped — detection never stops, paging does. When the window closes, an
automatic post-change report answers the question every change ticket begs:
did everything return to baseline, per path and device touched?
On-demand bursts
Mid-incident, fire a traceroute storm, a STAMP burst, or an MTU sweep at a
suspect region straight from the alert, the API, or a trig CLI — the
same adaptive escalation the scheduler runs on its own, now under an
engineer's hand. The first tool you reach for at 03:14.
Per-class truth
The mesh runs per QoS class — every EF and AF path has a best-effort twin,
so "EF is slow while best-effort is clean" reads as a policer fault, not a
transport mystery. And STAMP reflectors echo the DSCP they received:
when a hop silently strips markings, latency stays perfect, the SLA quietly
dies, and Trigpoint is the only witness — "sent EF, arrived best-effort" is
an alert, with the hop range to check.
The contract with the NOC
Anatomy of an alert
An alert you can't act on is noise with a timestamp. Every Trigpoint alert
answers six questions, or it doesn't fire:
What deviated. Metric, path set, and magnitude against the learned baseline — "+18 ms vs the nearest mode, ≈60σ", not "threshold exceeded".
When it started. To the sample, including the gap between onset and confirmation.
Blast radius. POPs, services, and customers affected — from inventory, not guesswork.
Evidence checked. BGP: no change. IS-IS: no events. Interface et-0/0/1: output drops rising. Every source consulted, with its result — including the exculpatory ones.
Ranked likely cause, with confidence. Congestion, route change, hardware, service failure — or, honestly, unexplained.
The drill-down. Links straight to path forensics: raw samples around the window, traceroute history, the routing events that coincided.
And the loop closes: operators confirm or reject every hypothesis, that verdict
is recorded next to the evidence, and the correlator's hit rate is itself a
tracked metric. The system earns trust the same way an engineer does — by being
right, and by being checkable when it isn't.
One platform
Three questions every operator has to answer
Service Trust
"Is the network delivering the experience it should — POP to POP, POP to Internet?"
The continuous measurement mesh, self-learning baselines, anomaly
detection, and path forensics. Deploy a container in every POP; get a
self-baselining anomaly map of your network.
Routing Trust
"Is the control plane doing what it should?"
Observed BGP compared against RPKI validation state. Hijack and leak
detection on your prefixes and your customers'. External vantage answers
"does the Internet see us correctly?" while the internal mesh answers
"are we seeing ourselves correctly?"
Configuration Trust
"Are the devices configured the way we intend?"
Golden-config drift detection, compliance and security-posture checks,
and digital-twin queries — "if this link fails, what happens to these
paths?" — grounded in the same live topology the correlator already keeps.
Why another monitoring tool
Inside-out, not outside-in
Enterprise synthetics platforms watch the Internet from the outside: useful if
you're a company consuming networks, incomplete if you're the company running
one. They don't speak your IGP, they can't see your route reflectors, and they
have no idea which customers ride which links.
Trigpoint is built for the other side of the demarc. It understands
your AS, your IS-IS topology, your POPs, and
your customer inventory — so when a path degrades, the answer isn't
a red dot on a world map. It's "the ATL–CHI link changed transport path at
02:13, no IGP or BGP involvement, 14 customers affected, and here is the
evidence for each claim."
No product today — commercial or open source — combines a synthetic
measurement mesh with the operator's own routing state for root-cause
correlation. That's the gap Trigpoint exists to close.
Architecture
Small parts, sharp edges
POPs
trig-agenticmp · udp/dscp · tcp dns · http/tls · trace
trig-agentstatic binary, mTLS, signed schedules, spool
× every POPLinux box, container host, or k8s edge
resultsmTLS gRPC
Control plane
Schedulertiered mesh · adaptive escalation · signed tests
Ingest gatewayresult streams → storage
ClickHousemeasurements + events, one store, joinable
Correlatorrule-based · live IGP + BMP evidence · ranked cause
Impact mapperpaths × topology × customer ports
Incident aggregatoranomalies → one incident · timeline · evidence pack
alerts
NOC
Alertingwebhooks · Slack · email
API + UItopology · forensics · verdict feedback
Grafanawhere NOCs already live
Storage that matches the questions
Measurement tables cluster by path and time — the shape of "this path,
this window". Event tables cluster by time — the shape of "what happened
between 02:11 and 02:14". Raw data keeps 14 days, minute rollups 90,
hourly rollups two years, because "was it always like this?" is a real
question and deserves a real answer.
Deployment without ceremony
Docker Compose for small footprints, Helm for large ones; agents as
containers or systemd units. Everything is Rust — the same properties that
make the agent trustworthy make the control plane cheap to run. Ingest for
a 50-POP tiered mesh is about 38 rows a second; sizing is not the risk.
Where it stands
Being proven the honest way
Trigpoint is in active development and validated continuously against a lab
network with thirteen scripted, labeled fault classes: latency steps, bufferbloat
congestion, clean packet loss, IS-IS reroutes, dead DNS at 03:14 with the WAN
untouched, an aggregation-uplink degrade only one attachment sees, a customer
prefix withdrawn upstream, a killed LSP, a starved QoS class, a degrading
transit provider, a hop that silently strips DSCP markings, and a resolver
that quietly stops validating DNSSEC. The detector catches every one with the
correct signature and the correct discrimination — and has held zero false
positives on a quiet mesh throughout.
The concept alert fires end-to-end, evidence and blast radius included. An
IS-IS reroute is caught as an adjacency-down event within a second, and SPF
over the live topology proves which paths crossed the failed link —
including a cross-pair set no time-window could have attributed. A customer
outage traced to a BMP withdraw event says "routing pull, not fault," naming
the prefix and the peers that saw it vanish. Mesh loss joins against interface
counters to name the dropping port. A dead LSP surfaces as the alert only a
service-plane mesh can produce: the VPN is down and the underlay beneath
it is provably clean. A starved EF class is called a class fault because
its best-effort twins stay healthy; stripped DSCP markings are caught by the
reflector echo with latency perfect; a degrading upstream is named because its
sibling transit is clean. A paired probe — one validly-signed name, one
deliberately bogus — tells a resolver that stopped validating apart from a
zone whose signatures expired, faults that are invisible to every
reachability check because resolution keeps "working". Each alert ships as a
plain-language narrative —
what happened, why we believe it, what to check — with the affected customers
attached and the raw evidence one click below.
The loop closes in the tooling that exists today: a NOC web UI with a live
mesh matrix and one-click verdicts, a trig CLI for bursts and
triage, and a tracked per-hypothesis hit rate — the correlator's accuracy is
itself a metric, judged by the operators it serves. The operator plane is
production-shaped too: directory (LDAP) sign-in with PingID two-factor,
every verdict and rebaseline attributed to the operator who made it, and an
intelligence layer that survives reboots honestly — a restarted correlator
reconciles every claim it left open, so history never shows an alert that
nobody closed.
That bar — a zero-false-positive week followed by caught, explained,
customer-attributed degradations — is the standard the production release has
to clear. If you run POPs and want a self-baselining anomaly map of your own
network, we'd like to talk while the roadmap is still wet.