Relationship & Event Correlation Engine

CV-Scope’s relationship engine connects what the cameras observe into time-bounded, confidence-scored relationships, and correlates them into higher-level events:

Person track #92   IDENTIFIED_AS         Recognized person (Employee-017)
Vehicle track #51  IDENTIFIED_BY_PLATE   License plate
License plate      REGISTERED_AS         Registered vehicle (Delivery Van 04)
Vehicle track #51  PARKED_IN             Parking Zone        0:12 - 3:40
Person track #92   APPROACHED            Vehicle track #51   1:05 - 1:14
Person track #92   ASSOCIATED_WITH       Vehicle track #51   1:14 - 1:32  (likely, 88 %)
Person track #92   ENTERED_VEHICLE       Vehicle track #51   1:33
Vehicle track #51  CROSSED               Gate West           1:58
=> correlated event "Departed with vehicle" (supported by the four facts above)

It is not an alert system. It builds a structured record of who or what was where, with what, when, with the evidence for every link. Alerts, webhooks and recorded events are optional actions of the rules.

Design principle. Every relationship comes from a deterministic rule, a measurement, a recognition result, a sensor observation or a configured pattern. A language model never creates one. It may only reword a summary of facts the engine already stored, if an administrator allows it.

Across several cameras - the location model, cross-camera correlation, journeys, the site view and the interactive time-aware graph - see docs/locations.md.

Contents

1. Concepts
2. Pipeline
3. Entities and identity resolution
4. Relationship types
5. Rules
6. Measurements: calibrated and uncalibrated
7. Confidence
8. Correlation and time
9. Patterns and deviations
10. Versioning and re-analysis
11. Storage and the graph API
12. Privacy and access control
13. Sensors and external registries
14. HTTP API
15. Limitations

Concepts

Pipeline

camera / video / sensor
  -> detector + tracker                       (vision)
  -> recognition modules (licensed)           (face, plate)
  -> spatial engine: zones, lines, routes     (interactions)
  -> rule engine: ordinary events             (counts, routes, dwell)
  -> relationship engine                      (pathscope/relationships/engine.py)
       observations -> identity resolution -> relationship rules
       -> relationships -> correlation rules -> correlated events
  -> relationship service (API process)       storage, pattern deviations, actions
  -> graph API, explorer, timeline, search, rule clauses, exports

The engine runs inside the camera worker for live runs (in step with the frames, so rules can use it immediately), and in the API process when a stored run is analysed again. It depends only on the normalized inputs: tracked boxes, the spatial engine’s interactions, the rule engine’s events and the EntityResolver interface of the recognition modules. It never imports a detector, tracker or recognition model.

Code layout (backend/pathscope/relationships/):

Entities and identity resolution

Entity keys are <type>:<reference>. Tracks are person_track:r12.t182 (run 12, track 182). Identities never carry a name or plate in their key:

Recognition does not replace the track. The track stays an entity and gets an identity link:

Person track #182   IDENTIFIED_AS        recognized_person:<id>     (face module, confidence)
Vehicle track #91   IDENTIFIED_BY_PLATE  license_plate:<hash>       (plate module)
license_plate:<h>   REGISTERED_AS        registered_vehicle:<id>    (vehicle registry)

A face possible match is never an identity (as everywhere in CV-Scope). When the recognized identity of a track changes, the old link is closed and a new one opened.

An identity can also come from a person rather than a module: a stored recognition event with context.method = "annotation" (a reviewer’s identification, for example the ground truth of a study). Re-analysing the run turns it into the same identity link, with the source annotation and the reason “A reviewer identified … (manual annotation, not face recognition)”; cross-camera moves built on it say so too (docs/locations.md).

Graph queries on an identity are projected: the identity’s neighbourhood is the union of the neighbourhoods of its tracks (identity links of at least likely), and the other side is lifted to its identity too. Every projected relationship says which track it came through (via). Only viewers allowed to see identities get projections.

New entity, observation and relationship types can be registered in code; user-defined relationship types are managed in Relationship settings.

Relationship types

The engine describes observable behaviour only. Personal, social and ownership relations (FRIEND, FAMILY, PARTNER, COWORKER, OWNS, OWNER_OF, EMPLOYEE_OF…) are refused as rule outputs. They can exist only as external only types imported from an authorized registry by an administrator (see Sensors and external registries). ASSOCIATED_WITH means “observed together by a configured rule”, never ownership.

Built in (no rules needed, per experiment):

• Places: ENTERED / EXITED / CROSSED from the scene’s zones, checkpoints and lines; USED_ROUTE from completed routes; REMAINED_IN once a visit lasts remained_min_s; MOVED_FROM + MOVED_TO when the next place is reached within transition_s (also for adjacent zones, where the next entry is reported before the last exit); optionally OCCUPIED for every visit.
• Identities: the identity links above, when a recognition module is licensed and allowed for the graph.

Rules

Rules are built on Relationships › Rules with structured form controls (or as JSON under Advanced). Four kinds:

Subjects and objects are role filters: object classes plus the same recognition clause the rule builder uses (any, anonymous, recognized, registered with groups, specific identities, plates or vehicles). A rule whose recognition module is not active for the run stays inactive (the run says why).

Example (JSON, the Advanced tab):

{
  "kind": "pair",
  "name": "Person beside vehicle",
  "subject": {"classes": ["person"]},
  "object": {"classes": ["car", "truck", "bus", "motorcycle"]},
  "condition": "near",
  "distance": {"value": 2.0, "unit": "m", "fallback_fw": 0.06},
  "for_s": 5,
  "gap_s": 1.0,
  "conditions": [{"kind": "in_zone", "role": "both", "zone_ids": ["zone_parking"]}],
  "relation": "ASSOCIATED_WITH",
  "event_label": "Beside a vehicle",
  "act_min_state": "likely",
  "actions": [{"kind": "record_event"}]
}

A sequence rule (“departed with vehicle”):

{
  "kind": "sequence", "name": "Departed with vehicle", "window_s": 180,
  "subject": {"classes": ["person"]}, "object": {"classes": ["car", "truck"]},
  "steps": [
    {"role": "A", "what": "relation", "relation": "ASSOCIATED_WITH", "min_state": "likely"},
    {"role": "A", "what": "relation", "relation": "ENTERED_VEHICLE", "within_s": 120},
    {"role": "B", "what": "place", "place_event": "crosses", "places": ["gate_west"], "within_s": 60}
  ],
  "relation": "DEPARTED_WITH", "event_label": "Departed with vehicle"
}

Step kinds: relation (A and B have a relationship, either direction unless any_direction is false, at least min_state), place (enters, exits, crosses, uses route; places empty = any), appears, disappears, event (an ordinary event type such as anomaly). A step with role both needs A and B within together_s of each other.

Templates cover common cases: person beside a vehicle, person approaches a vehicle, people near each other, walking together, following, getting into and out of a vehicle, vehicles travelling together, parking, departed with, arrived with, shared route movement.

Actions (record an ordinary run event, send a webhook) run once, when a relationship or correlated event first reaches act_min_state (default likely). Published events have type relationship, correlated or relation_deviation and carry ids, never names.

In the ordinary rule builder a rule can require a relationship:

WHEN car ENTERS Restricted Parking
AND HAS RELATIONSHIP ASSOCIATED_WITH  with person (Recognized person is Employee-017)
    at least likely, holding or ended at most 300 s ago
CREATE EVENT Restricted Vehicle Association

The clause is evaluated when the event would be emitted. A relationship that forms a moment later (often from the same crossing that triggers the rule) is waited for during the recognition grace period (5 s). The rule stays inactive when relationships are off for the experiment.

Measurements: calibrated and uncalibrated

Distances use the scene calibration:

• Calibrated (four ground points, or one known distance): ground-plane distance in metres, marked physical.
• Not calibrated: image distance in frame widths (pixels / frame width), marked scene-relative. It is never shown as metres. A rule gives its distance in metres plus an optional fallback in frame widths; without the fallback it does not run on an uncalibrated camera, and the run says why.

People are measured by their ground point (bottom centre of the box). Vehicles are measured by the bottom edge of their box, so a person at a car’s side is near the car even though its centre is metres away. Speeds and headings come from the last second of ground positions.

Each spatial relationship records its calibration (mode, unit, scene version, how vehicles were measured) and its measurements (closest, mean and farthest distance, samples meeting the condition, duration, lag).

Confidence

Every inferred relationship has a confidence in 0..1 built from evidence components:

The score is a weighted geometric mean of the components that apply, capped at the weakest component + 0.3, times the support factor.

On an uncalibrated camera a relationship can reach likely but rarely confirmed: the measurement is less certain, and the state says so.

Correlation and time

Sequence rules match steps in order, each at most within_s after the previous one, all within window_s. A match stores the correlated event, its roles, and the explicit temporal relations between consecutive supporting facts (FOLLOWED_AFTER, OVERLAPPED_WITH, STARTED_AFTER with the gap). Recognition that arrives after the steps (a face matched two seconds later) is waited for; a role that turns out not to match drops the match.

follow_route rules watch places only: B reaches the same zones or lines as A, in the same order, each at most max_lag_s later, min_checkpoints times within window_s.

Patterns and deviations

Per experiment (Experiments › Relationships):

• Expectations: “vehicles of group Delivery are expected in Loading Zone only” or “people are not expected in the Centre floor”. A matching entity that ENTERS (CROSSES, PARKS IN, REMAINS IN, USES ROUTE) a place outside the expectation gives a deviation.
• Learned history: after pattern_min_sessions earlier sessions of an identity, a place it used in fewer than pattern_rare_share of them, or a partner (vehicle, person) it was never observed with, gives a deviation, for example “Recognized person entered Gate C; in 12 earlier sessions it did so in 0 (usually Gate A, 11 of 12)”.

Deviations describe what was observed, never intent. Their text never contains a name; names appear only for authorized viewers through the linked entities. With Record deviations as run events they appear in the run’s live event list, its events table and its webhooks.

Versioning and re-analysis

• Rules are stored per (key, version). Editing stores the next version with a note; nothing is overwritten.
• An experiment uses a rule’s latest version (picked up from the next run) or a pinned version.
• A run records the versions it used (run snapshot and analysis).
• Analyse again (run analysis page, Relationships tab) replays the stored trajectories through the run’s own scene version, with the same spatial and rule engines a live run uses, and the rules chosen now. The new analysis becomes current when it finishes; earlier ones stay available (Make current switches back). Re-analysis needs stored trajectories (Settings › Store sampled trajectories).

Storage and the graph API

Relational tables (migration 0006_relationships), indexed for the graph questions CV-Scope asks:

A graph database was not introduced: the questions (neighbours, two-step subgraphs, timelines, histories, time and place filters) are one or two indexed joins, the data lives next to the runs it came from (deleting a run deletes its relationships), and SQLite and PostgreSQL both serve it. graph.py defines the GraphBackend contract that the API uses; a graph database can be added behind it if traversal depth or scale require it.

Privacy and access control

Relationship data can reveal routines and associations, so it has its own policy (Relationships › Settings):

• Roles come from the recognition access tokens (viewer < operator < admin). Each action has a minimum role: read the graph (default: anyone on this computer, like the rest of CV-Scope), see identities (viewer), export (anyone), export with names and plates (admin), change rules and analyse (anyone), delete (anyone), change the settings (admin; while no token exists, the local user on this computer).
• Identities are shown by name only with the identity role. Everyone else sees “Recognized person”, “License plate” or “Registered vehicle” with no key: it cannot be opened, searched or exported by name.
• Modules: face-derived and plate-derived links can be switched off for the graph (hidden everywhere, not created by new runs), as can external sensor observations.
• Retention (hourly): relationship data after n days, identity links (and deviations about identities) after n days (default 30, as for recognition events), audit records after n days.
• Audit: rule creation and versions, archive/restore, analyses, which analysis is current, deletions, exports, settings changes, registry imports, sensor observations, model-written summaries, and every identity-level view (entity, neighbours, graph, history, timeline, search).
• Language models: optional, off by default, for rewording summaries only. The model receives the anonymous deterministic summary (no names, no plates, no pictures). Summaries of an identity are never sent.

Sensors and external registries

External sensors publish observations through the API (the sensors module must be on; the rules role is needed):

POST /api/relationships/observations
{
  "sensor_id": "depth-01", "sensor_type": "depth", "run_id": 12, "media_time_s": 75.2,
  "subject": {"track_id": 1}, "object": {"track_id": 2},
  "measurement": {"kind": "distance", "value": 0.9, "unit": "m"}, "confidence": 0.95
}

Kinds: distance (compared with a relationship’s metre threshold), presence (thermal: confirms or not), movement (radar speed in m/s: confirms moving or standing-still relationships). The observation is linked to each relationship of the pair that holds at that time, the sensor type is added to its sources (“RGB, DEPTH”) and its confidence is recomputed with the sensor component (agreeing 1.0, contradicting 0.3 or 0.4).

Personal or organisational relations can only be imported by an administrator, and only for types defined as external only:

POST /api/relationships/registry
{"source": "HR registry", "items": [{"relation": "COWORKER", "subject": "recognized_person:<id>", "object": "recognized_person:<id>"}]}

They are marked as supplied by the registry, not inferred, and can be withdrawn (DELETE /api/relationships/relationship/{id}).

HTTP API

All under /api/relationships. Send the recognition token (X-Recognition-Token) to be recognized as a viewer.

Common filters: types (comma-separated), time_from, time_to, last_hours, camera_id, experiment_id, run_id, zone_id, min_state, analysis_id, include_superseded.

Limitations

• Relationships are formed within a run (one camera). Across cameras and sessions, entities connect through identities (history, projected neighbourhoods), not through anonymous tracks.
• Re-analysis measures vehicles as points unless a live analysis of the same run stored their box width; the provenance says which.
• A run on a video file is analysed faster than real time. Durations and summaries use the video’s own clock (media time); the wall-clock times of such runs are processing times.
• Detection and tracking errors carry through: two people merged into one track cannot be NEAR each other. Check a sample of relationships against the video (every one links to it).
• The recognition modules decide identities; the graph only records them. A face possible match is never an identity.