Keep the trace and span an OTLP log was written in, and query logs by an OTLP or Zipkin span

docs/en/changes/changes.md

@@ -20,6 +20,9 @@
 * Fix a webhook alarm hook not being posted when the alarm also names a hook of another type: the webhook callback looked up the other hook's name in the webhook settings and threw a `NullPointerException`. Depending on the hook names, the webhook was never posted (for example `webhook.default` with `gRPC.default`), or an error was logged on every alarm.
 * Add `alarm.default.grpcTimeout` / `SW_ALARM_GRPC_TIMEOUT`, the deadline in seconds of each gRPC alarm hook stream (default `10`, which was the fixed deadline before).
 * Record gRPC alarm hook deliveries on the alarm delivery self-observability metrics (`alarm_http_delivery_total` and `alarm_http_delivery_latency_seconds`) with `hook_type=gRPC`, so the OAP dashboards show the gRPC hook as one more channel.
+* Keep the trace an OTLP log was written in: the OTLP log receiver now stores a record's `trace_id` as the log's trace ID, which it dropped before, and its `span_id` in a new `otlp_span_id` column. The log query's `TraceScopeCondition` takes `otlpSpanId` to read the logs one OTLP or Zipkin span wrote, which the numeric `spanId` cannot name, and a LAL rule sets it with `otlpSpanId`, an output field of the default output type. An existing BanyanDB log stream and Elasticsearch log index gain the column at boot.
+* A log query's `relatedTrace` is one of `traceId` alone, `traceId` with `segmentId` and `spanId` for a SkyWalking native span, or `traceId` with `otlpSpanId` for an OTLP or Zipkin span; any other mix is refused rather than read. This also refuses two forms accepted before: `segmentId` without `spanId`, which read the logs of a whole segment, and `spanId` without `segmentId`. The debugging query API refuses `segmentId`, `spanId` or `otlpSpanId` without `traceId`, which it ignored before.
+* **Breaking Change** — JDBC storage (MySQL, PostgreSQL): the log tables gain the `otlp_span_id` column, and OAP adds it at boot only to the current day's table. Every other existing log table keeps its old shape — including the next day's, when OAP had already created it before the upgrade — and inserting a log into one fails, for logs from every source. When upgrading, add the column to the existing log tables, `ALTER TABLE log_<yyyyMMdd> ADD COLUMN otlp_span_id VARCHAR(64);`, or drop them, which deletes their logs, so OAP creates them anew.
 
 #### UI
 

docs/en/concepts-and-designs/lal.md

@@ -394,7 +394,30 @@ not dropped) and is used to associate with traces / metrics.
 - `spanId`
 
 `spanId` extracts the span ID from the `parsed` result, and set it into the `LogData`, which will be persisted (if not
-dropped) and is used to associate with traces / metrics.
+dropped) and is used to associate with traces / metrics. It is a SkyWalking span ID: a number, counted within the segment
+`segmentId` names.
+
+- `otlpSpanId`
+
+`otlpSpanId` sets the span ID of an OTLP or Zipkin span, 16 lowercase hex characters, which `spanId` cannot hold. It is
+persisted with the log, so the log can be queried by the span through `relatedTrace { traceId, otlpSpanId }`. It is an
+[output field](#output-fields) of the default output type rather than a keyword, so it works in rules without a custom
+`outputType`:
+
+```
+filter {
+    text {
+        regexp $/trace_id=(?<traceId>[0-9a-f]{32}) span_id=(?<spanId>[0-9a-f]{16})/$
+    }
+    extractor {
+        traceId parsed.traceId as String
+        otlpSpanId parsed.spanId as String
+    }
+}
+```
+
+The [OTLP log receiver](../setup/backend/log-otlp.md) sets it from the record's own `span_id`; a rule reads it as
+`log.traceContext.otlpSpanId`.
 
 - `timestamp`
 

docs/en/debugging/query-tracing.md

@@ -474,8 +474,9 @@ debuggingTrace:
   | serviceInstance            | The service instance name                                               | No, require service           |               
   | endpoint                   | The endpoint name                                                       | No, require service           |               
   | traceId                    | The trace ID                                                            | No                            |               
-  | segmentId                  | The segment ID                                                          | No, require traceId           |
-  | spanId                     | The span ID                                                             | No, require traceId           | 
+  | segmentId                  | The segment ID                                                          | No, require traceId and spanId |
+  | spanId                     | The span ID                                                             | No, require traceId and segmentId |
+  | otlpSpanId                 | The OTLP or Zipkin span ID, 16 lowercase hex characters                 | No, require traceId; not with segmentId or spanId |
   | queryOrder                 | The order of the query result, `ASC`, `DES`                             | No, default `DES`             |               
   | tags                       | The tags of the trace, `key1=value1,key2=value2`                        | No                            |  
   | pageNum                    | The page number of the query result                                     | Yes                           |

docs/en/setup/backend/log-otlp.md

@@ -55,3 +55,12 @@ And several attributes are optional as add-on information for the logs before an
 Note, that these attributes should be set manually through OpenTelemetry SDK or through [attribute#insert in OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md).
 
 The OTLP log record body is delivered to [LAL](../../concepts-and-designs/lal.md) as a plain-text body, even when the string content is JSON-shaped. A LAL rule can still parse such content with the `json {}` parser — when the JSON body is empty and a text body is present, `json {}` tries the text as JSON. On a successful parse the log is normalized to a JSON body, so it is stored and rendered as JSON. See [`json`](../../concepts-and-designs/lal.md#json) for details.
+
+A log record written inside a span carries that span's `trace_id` and `span_id`. Both are stored with the log, as
+lowercase hex: the trace ID as the log's trace ID, and the span ID as its OTLP span ID — not the `spanId`, which is a
+SkyWalking span's number within its segment. So the log can be queried by the trace that wrote it, through
+`relatedTrace { traceId }`, and by the span, through `relatedTrace { traceId, otlpSpanId }`. A LAL rule reads the span ID as
+`log.traceContext.otlpSpanId`, and may set it with [`otlpSpanId`](../../concepts-and-designs/lal.md#extractor).
+
+The trace ID is written the way this OAP stores the OTLP trace itself: with `otlpTraceStorage: zipkin`, an ID whose upper
+64 bits are zero — a trace started by a tracer with 64-bit IDs — keeps only its lower 16 hex characters, as Zipkin does.

oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALCodegenHelper.java

@@ -53,6 +53,7 @@ final class LALCodegenHelper {
         METADATA_TRACE_GETTERS.put("traceId", "getTraceId");
         METADATA_TRACE_GETTERS.put("traceSegmentId", "getTraceSegmentId");
         METADATA_TRACE_GETTERS.put("spanId", "getSpanId");
+        METADATA_TRACE_GETTERS.put("otlpSpanId", "getOtlpSpanId");
 
         LOG_GETTERS.put("body", "getBody");
         LOG_GETTERS.put("tags", "getTags");

oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptExecutionTest.java

@@ -38,6 +38,7 @@
 import org.apache.skywalking.oap.server.core.config.ConfigService;
 import org.apache.skywalking.oap.server.core.config.NamingControl;
 import org.apache.skywalking.oap.server.core.source.LALOutputBuilder;
+import org.apache.skywalking.oap.server.core.source.LogMetadata;
 import org.apache.skywalking.oap.server.core.source.SourceReceiver;
 import org.apache.skywalking.oap.server.library.module.ModuleManager;
 import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder;
@@ -188,6 +189,18 @@ private void executeAndValidate(
                     srcAttrs != null
                         ? org.apache.skywalking.oap.server.core.source.LogMetadataUtils.fromLogData(testLogBuilder, srcAttrs)
                         : org.apache.skywalking.oap.server.core.source.LogMetadataUtils.fromLogData(testLog);
+                final Object otlpSpanId = inputData != null ? inputData.get("otlp-span-id") : null;
+                if (otlpSpanId != null) {
+                    // No native protocol carries an OTLP span id; the OTLP receiver puts it on the metadata.
+                    final LogMetadata.TraceContext tc = v2Metadata.getTraceContext();
+                    v2Metadata.setTraceContext(
+                        LogMetadata.TraceContext.builder()
+                            .traceId(tc.getTraceId())
+                            .traceSegmentId(tc.getTraceSegmentId())
+                            .spanId(tc.getSpanId())
+                            .otlpSpanId(String.valueOf(otlpSpanId))
+                            .build());
+                }
                 final Object v2Input = extraLog != null ? extraLog : testLogBuilder;
                 v2Ctx = new org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext();
                 v2Ctx.init(v2Metadata, v2Input);

oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.data.yaml

@@ -270,6 +270,30 @@ trace-context-read:
     save: true
     abort: false
 
+trace-otlp-span-read:
+  body-type: json
+  body: '{"service":"trace-svc"}'
+  trace-id: 0a1b2c3d4e5f60718293a4b5c6d7e8f9
+  otlp-span-id: 0123456789abcdef
+  expect:
+    service: trace-svc
+    tag.osid: 0123456789abcdef
+    traceId: 0a1b2c3d4e5f60718293a4b5c6d7e8f9
+    outputField.otlpSpanId: 0123456789abcdef
+    save: true
+    abort: false
+
+trace-otlp-span-write:
+  body-type: json
+  body: '{"service":"trace-svc","tid":"4bf92f3577b34da6a3ce929d0e0e4736","osp":"00f067aa0ba902b7"}'
+  otlp-span-id: 0123456789abcdef
+  expect:
+    service: trace-svc
+    traceId: 4bf92f3577b34da6a3ce929d0e0e4736
+    outputField.otlpSpanId: 00f067aa0ba902b7
+    save: true
+    abort: false
+
 trace-segment-override:
   body-type: json
   body: '{"service":"trace-svc","seg":"segment-from-body"}'

oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.yaml

@@ -417,6 +417,36 @@ rules:
         sink {}
       }
 
+  # An OTLP or Zipkin span id has no keyword: it is an output field of the default output
+  # type, LogBuilder (setOtlpSpanId). Read it from the metadata's TraceContext, where the OTLP
+  # receiver puts it, and let LogBuilder copy it into the output untouched.
+  - name: trace-otlp-span-read
+    layer: GENERAL
+    dsl: |
+      filter {
+        json {}
+        extractor {
+          service parsed.service as String
+          tag 'osid': log.traceContext.otlpSpanId
+        }
+        sink {}
+      }
+
+  # The write direction: a span id written into the log text — a Zipkin or OpenTelemetry
+  # logging context — wins over the one the metadata carries.
+  - name: trace-otlp-span-write
+    layer: GENERAL
+    dsl: |
+      filter {
+        json {}
+        extractor {
+          service parsed.service as String
+          traceId parsed.tid as String
+          otlpSpanId parsed.osp as String
+        }
+        sink {}
+      }
+
   # segmentId alone — the single-statement shape that used to NPE.
   - name: trace-segment-override
     layer: GENERAL

oap-server/server-admin/status/src/main/java/org/apache/skywalking/oap/server/admin/status/DebuggingHTTPHandler.java

@@ -551,6 +551,7 @@ public String queryLogs(@Param("service") Optional<String> service,
                             @Param("traceId") Optional<String> traceId,
                             @Param("segmentId") Optional<String> segmentId,
                             @Param("spanId") Optional<Integer> spanId,
+                            @Param("otlpSpanId") Optional<String> otlpSpanId,
                             @Param("tags") Optional<String> tags,
                             @Param("pageNum") int pageNum,
                             @Param("pageSize") int pageSize,
@@ -559,11 +560,15 @@ public String queryLogs(@Param("service") Optional<String> service,
                             @Param("queryOrder") Optional<String> queryOrder) {
         LogQueryCondition condition = new LogQueryCondition();
         condition.setPaging(new Pagination(pageNum, pageSize));
+        if (traceId.isEmpty() && (segmentId.isPresent() || spanId.isPresent() || otlpSpanId.isPresent())) {
+            return "traceId is required with segmentId, spanId or otlpSpanId";
+        }
         if (traceId.isPresent()) {
             TraceScopeCondition traceScopeCondition = new TraceScopeCondition();
             traceScopeCondition.setTraceId(traceId.get());
             segmentId.ifPresent(traceScopeCondition::setSegmentId);
             spanId.ifPresent(traceScopeCondition::setSpanId);
+            otlpSpanId.ifPresent(traceScopeCondition::setOtlpSpanId);
             condition.setRelatedTrace(traceScopeCondition);
         } else {
             if (startTime.isEmpty() || endTime.isEmpty() || step.isEmpty()) {

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java

@@ -44,6 +44,7 @@ public abstract class AbstractLogRecord extends Record {
     public static final String TRACE_ID = "trace_id";
     public static final String TRACE_SEGMENT_ID = "trace_segment_id";
     public static final String SPAN_ID = "span_id";
+    public static final String OTLP_SPAN_ID = "otlp_span_id";
     public static final String CONTENT_TYPE = "content_type";
     public static final String CONTENT = "content";
     public static final String TAGS_RAW_DATA = "tags_raw_data";
@@ -79,6 +80,16 @@ public abstract class AbstractLogRecord extends Record {
     @Column(name = SPAN_ID)
     @BanyanDB.NoIndexing
     private int spanId;
+    /**
+     * The span that wrote the log when it is an OTLP or Zipkin span: its 16 lowercase hex characters.
+     * A SkyWalking native span is {@link #traceSegmentId} + {@link #spanId} instead, as a native span id
+     * is a number counted within its segment.
+     */
+    @Setter
+    @Getter
+    @Column(name = OTLP_SPAN_ID, length = 64)
+    @BanyanDB.NoIndexing
+    private String otlpSpanId;
     @Setter
     @Getter
     @Column(name = CONTENT_TYPE, storageOnly = true)
@@ -120,6 +131,7 @@ protected void map2Data(T record, final Convert2Entity converter) {
             record.setTraceId((String) converter.get(TRACE_ID));
             record.setTraceSegmentId((String) converter.get(TRACE_SEGMENT_ID));
             record.setSpanId(((Number) converter.get(SPAN_ID)).intValue());
+            record.setOtlpSpanId((String) converter.get(OTLP_SPAN_ID));
             record.setContentType(((Number) converter.get(CONTENT_TYPE)).intValue());
             record.setContent(new LongText((String) converter.get(CONTENT)));
             record.setTimestamp(((Number) converter.get(TIMESTAMP)).longValue());
@@ -134,6 +146,7 @@ protected void data2Map(final T record, final Convert2Storage converter) {
             converter.accept(TRACE_ID, record.getTraceId());
             converter.accept(TRACE_SEGMENT_ID, record.getTraceSegmentId());
             converter.accept(SPAN_ID, record.getSpanId());
+            converter.accept(OTLP_SPAN_ID, record.getOtlpSpanId());
             converter.accept(TIME_BUCKET, record.getTimeBucket());
             converter.accept(CONTENT_TYPE, record.getContentType());
             converter.accept(CONTENT, record.getContent());

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/LogRecord.java

@@ -33,7 +33,7 @@
 import static org.apache.skywalking.oap.server.core.storage.StorageData.TIME_BUCKET;
 
 @SuperDataset
-@Stream(name = LogRecord.INDEX_NAME, scopeId = DefaultScopeDefine.LOG, builder = LogRecord.Builder.class, processor = RecordStreamProcessor.class)
+@Stream(name = LogRecord.INDEX_NAME, scopeId = DefaultScopeDefine.LOG, builder = LogRecord.Builder.class, processor = RecordStreamProcessor.class, allowBootReshape = true)
 @SQLDatabase.ExtraColumn4AdditionalEntity(additionalTable = AbstractLogRecord.ADDITIONAL_TAG_TABLE, parentColumn = TIME_BUCKET)
 @BanyanDB.TimestampColumn(AbstractLogRecord.TIMESTAMP)
 @BanyanDB.Group(streamGroup = BanyanDB.StreamGroup.RECORDS_LOG)

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/LogRecordDispatcher.java

@@ -37,6 +37,7 @@ public void dispatch(final Log source) {
         record.setTraceId(source.getTraceId());
         record.setTraceSegmentId(source.getTraceSegmentId());
         record.setSpanId(source.getSpanId());
+        record.setOtlpSpanId(source.getOtlpSpanId());
         record.setContentType(source.getContentType().value());
         record.setContent(new LongText(source.getContent()));
         record.setTagsRawData(source.getTagsRawData());

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/LogQueryService.java

@@ -69,6 +69,7 @@ public Logs queryLogs(String serviceId,
                           final List<Tag> tags,
                           List<String> keywordsOfContent,
                           List<String> excludingKeywordsOfContent) throws IOException {
+        validateRelatedTrace(relatedTrace);
         DebuggingTraceContext traceContext = TRACE_CONTEXT.get();
         DebuggingSpan span = null;
         try {
@@ -98,6 +99,32 @@ public Logs queryLogs(String serviceId,
         }
     }
 
+    /**
+     * A related trace is one of three forms: {@code traceId} alone for the whole trace, {@code traceId} +
+     * {@code segmentId} + {@code spanId} for a SkyWalking native span, or {@code traceId} + {@code otlpSpanId} for an
+     * OTLP or Zipkin span. Any other mix names a partial or contradictory span, which would answer "no logs", or a
+     * different set of logs, rather than report the mistake. A condition with nothing set, as the LogQL handler passes
+     * without a trace ID label, is no condition.
+     */
+    static void validateRelatedTrace(final TraceScopeCondition relatedTrace) {
+        if (relatedTrace == null) {
+            return;
+        }
+        final boolean segment = StringUtil.isNotEmpty(relatedTrace.getSegmentId());
+        final boolean span = nonNull(relatedTrace.getSpanId());
+        final boolean otlpSpan = StringUtil.isNotEmpty(relatedTrace.getOtlpSpanId());
+        if (!segment && !span && !otlpSpan) {
+            return;
+        }
+        final boolean nativeSpan = segment && span && !otlpSpan;
+        final boolean otlpSpanOnly = otlpSpan && !segment && !span;
+        if (StringUtil.isEmpty(relatedTrace.getTraceId()) || !(nativeSpan || otlpSpanOnly)) {
+            throw new IllegalArgumentException(
+                "relatedTrace takes traceId alone, traceId with segmentId and spanId for a SkyWalking native span, "
+                    + "or traceId with otlpSpanId for an OTLP or Zipkin span");
+        }
+    }
+
     private Logs invokeQueryLogs(String serviceId,
                           String serviceInstanceId,
                           String endpointId,

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/TraceScopeCondition.java

@@ -29,4 +29,5 @@ public class TraceScopeCondition {
     private String traceId;
     private String segmentId;
     private Integer spanId;
+    private String otlpSpanId;
 }

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/AbstractLog.java

@@ -38,6 +38,7 @@ public abstract class AbstractLog extends Source {
     private String traceId;
     private String traceSegmentId;
     private int spanId;
+    private String otlpSpanId;
     private ContentType contentType = ContentType.NONE;
     private String content;
     private byte[] tagsRawData;

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Log.java

@@ -43,6 +43,7 @@ public String toJson() {
         obj.addProperty("traceId", getTraceId());
         obj.addProperty("traceSegmentId", getTraceSegmentId());
         obj.addProperty("spanId", getSpanId());
+        obj.addProperty("otlpSpanId", getOtlpSpanId());
         obj.addProperty("contentType", contentType == null ? null : contentType.name());
         obj.addProperty("content", getContent());
         obj.addProperty("error", isError());