Manual/connector reconciliation
Log Lens lets you get logs in two ways: import a file by hand, or let a connector sync it automatically. If both methods touch the same underlying log, a naive importer would index every event twice. Reconciliation prevents that by recognising when a manually imported file and a connector-synced stream are really the same log, and stitching them into a single source.
How matching works
Instead of comparing whole files, Log Lens compares a prefix hash - a SHA-256 over the shared opening bytes of two files, as far as the shorter one reaches. Two logs that start with identical content are treated as the same stream, because a log file only ever grows at the tail.
- Connector adopting a manual import. When a connector first sees a remote file with no existing stream, it looks for manually imported sources that have no stream yet, in the same module, no larger than the remote file, preferring a matching channel or file name. It hashes each candidate's bytes against the remote file's matching prefix. If exactly one candidate matches, that manual file becomes the connector's canonical mirror - no re-fetch, no re-index of what's already there.
- Manual import adopting a connector stream. When you import a file that a connector already manages (same module, log type, and channel), Log Lens hashes the shared prefix. On a unique match it either skips the file (if it is not longer than what's already stored) or merges only the new tail bytes and re-indexes those.
A match only counts when it is unambiguous: if zero or several candidates match, the file is treated as brand new rather than risk merging the wrong log.
Seeing it in the preview
The sync preview labels each file with the action it will take:
| Action | Meaning |
|---|---|
reconcile | Matches a manual import; the connector will adopt it and fetch only newer bytes |
already_ingested | Matches a manual import of the same size; nothing to do |
new | No match found - will be fetched and indexed fresh |
After a sync, the result includes a manual_sources_reconciled count so you can confirm files were merged rather than duplicated.
Why it matters
You can start by dragging a log file in, then later connect a local directory or an SSH host to keep it current - without losing history or double-counting occurrences. Reconciliation is what makes switching methods safe.
See also incremental ingestion and log rotation and generations.