> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usetero.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Duplicate fields

> The same field stored in multiple locations

Duplicate fields occur when multiple fields in a log record carry the same information.

Common sources include:

| Duplicate pair              | Common source                                    |
| --------------------------- | ------------------------------------------------ |
| `time` and `@timestamp`     | Application timestamp plus collector timestamp   |
| `level` and `severity_text` | Application severity plus OpenTelemetry severity |
| `host` and `hostname`       | Service field plus agent field                   |

Duplicate fields require either repeated field names or clear semantic equivalents such as `level` and `severity`. Fields with matching values but different meanings are not duplicates.

## Signals

* Repeated values across timestamp, severity, host, service, or environment fields.
* Field pairs with equivalent names from different parts of the telemetry pipeline.
* Redundant fields added by SDKs, agents, collectors, or exporters.
* A canonical field that can preserve the value after the duplicate field is removed.

## Example

The following log has three duplicate pairs that carry the same values under different names: `time` and `@timestamp`, `level` and `severity_text`, `host` and `hostname`.

<Tabs>
  <Tab title="Before">
    ```json theme={null}
    {
      "time": "2024-01-15T10:30:00Z",
      "@timestamp": "2024-01-15T10:30:00Z",
      "level": "ERROR",
      "severity_text": "ERROR",
      "host": "checkout-api-7d8f9",
      "hostname": "checkout-api-7d8f9",
      "message": "Connection timeout"
    }
    ```
  </Tab>

  <Tab title="After">
    ```json theme={null}
    {
      "@timestamp": "2024-01-15T10:30:00Z",
      "severity_text": "ERROR",
      "hostname": "checkout-api-7d8f9",
      "message": "Connection timeout"
    }
    ```
  </Tab>
</Tabs>

Tero generates a scoped policy:

```yaml theme={null}
id: remove-duplicate-fields-checkout-api
name: Remove duplicate fields from checkout-api
description: Drop duplicate fields that contain the same data as their canonical equivalents.
log:
  match:
    - resource_attribute: service.name
      exact: checkout-api
  transform:
    remove:
      - log_attribute: time
      - log_attribute: level
      - log_attribute: host
```

## Recommended enforcement

<Card title="Enforce at edge" icon="server" href="/policies/enforcement/edge" horizontal>
  Drop duplicate fields before data leaves your network.
</Card>

Use edge enforcement when duplicate fields come from agents, collectors, SDKs, or exporters rather than application code.

## Detection notes

* Tero compares field values across logs.
* Tero flags exact string matches when the fields represent the same concept.
* It flags semantic equivalents when the values share a normalized meaning, such as a numeric severity and a text severity.
* Fields with similar values but different meanings are not duplicates. For example, `request_id` and `trace_id` are related identifiers, not duplicate fields.
* Fields with different representations are not duplicates unless they normalize to the same value. For example, UTC and local timestamps are different representations.

Tero keeps the more standard field, such as `@timestamp` or `severity_text`, and removes the duplicate.