Duplicate fields

​

Signals

• Repeated values across timestamp, severity, host, service, or environment fields.
• Field pairs with equivalent names from different parts of the telemetry pipeline.
• Redundant fields added by SDKs, agents, collectors, or exporters.
• A canonical field that can preserve the value after the duplicate field is removed.

​

Example

{
  "time": "2024-01-15T10:30:00Z",
  "@timestamp": "2024-01-15T10:30:00Z",
  "level": "ERROR",
  "severity_text": "ERROR",
  "host": "checkout-api-7d8f9",
  "hostname": "checkout-api-7d8f9",
  "message": "Connection timeout"
}

{
  "@timestamp": "2024-01-15T10:30:00Z",
  "severity_text": "ERROR",
  "hostname": "checkout-api-7d8f9",
  "message": "Connection timeout"
}

id: remove-duplicate-fields-checkout-api
name: Remove duplicate fields from checkout-api
description: Drop duplicate fields that contain the same data as their canonical equivalents.
log:
  match:
    - resource_attribute: service.name
      exact: checkout-api
  transform:
    remove:
      - log_attribute: time
      - log_attribute: level
      - log_attribute: host

​

Recommended enforcement

​

Detection notes

• Tero compares field values across logs.
• Tero flags exact string matches when the fields represent the same concept.
• It flags semantic equivalents when the values share a normalized meaning, such as a numeric severity and a text severity.
• Fields with similar values but different meanings are not duplicates. For example, request_id and trace_id are related identifiers, not duplicate fields.
• Fields with different representations are not duplicates unless they normalize to the same value. For example, UTC and local timestamps are different representations.