Skip to content

feat(tracer): send traces with OpenTelemetry over OTLP - #822

Open
mwbrooks wants to merge 10 commits into
mainfrom
mwbrooks-tracer-otlp
Open

mwbrooks wants to merge 10 commits into
mainfrom
mwbrooks-tracer-otlp

Conversation

@mwbrooks

@mwbrooks mwbrooks commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Changelog

N/A - internal tracing change with no user-facing changes.

Summary

This pull request sends the CLI's traces with OpenTelemetry instead of the archived jaeger-client-go, so maintainers keep tracing on a library that still gets fixes. Traces now go to slackb.com/traces/v1/otlp, the OTLP endpoint other Slack services already use, and every existing span call keeps working through the OpenTracing bridge.

  • SetupTracer builds an OpenTelemetry TracerProvider with a batching OTLP exporter and registers the bridge as the global OpenTracing tracer
  • Spans keep service.name and add a service_name resource attribute, because the Jaeger endpoint set that tag and Honeycomb queries filter on it
  • Spans no longer carry the raw hostname and ip tags that jaeger-client-go added on its own. hashed_hostname stays
  • SetTraceHeaders and TraceID now inject through a B3 propagator, and x-b3-sampled stays 0
  • Export errors are dropped, like the Jaeger reporter did, so a blocked network prints nothing
  • Ignores the developer's own OTEL_* variables. The SDK reads OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_INSECURE, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, and OTEL_BSP_*, which would send someone's collector credentials or attributes to slackb.com, or crash the CLI with OTEL_BSP_MAX_QUEUE_SIZE=-1. Every exporter option, the resource, the sampler, and the batch options are pinned instead, and OpenTelemetry's internal logger is discarded so a malformed variable prints nothing
  • Flush shuts the provider down with a 2s timeout

Builds on #819, which added the flush before exit that the batching exporter needs.

Testing

  1. Point SetupTracer at a local HTTP server that logs each POST (add otlptracehttp.WithInsecure()), then build
  2. Run ./bin/slack version and ./bin/slack notacommand, and confirm each sends one protobuf POST to /traces/v1/otlp with the main span
  3. Stop the server, rerun ./bin/slack version, and confirm nothing new prints to stderr
  4. Run go test ./internal/tracer/ -run Test_newProvider, which sets those OTEL_* variables and checks the export has no authorization header, no extra resource attributes, and still goes over TLS

Notes

  • Ran the steps above: both commands sent the main span with service.name and service_name, and the payload had no raw hostname
  • Sent spans from a build pointed at dev.slackb.com for version and notacommand, and the exporter reported no errors
  • The trace ID grows from 16 to 32 hex chars. webapp reads x-b3-traceid as an opaque string (src/tracing/TraceContext.hack)
  • x-b3-parentspanid is no longer sent, because OpenTelemetry span contexts don't carry a parent ID. webapp ignores that header
  • Reran the local check with OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_TRACES_SAMPLER=always_off set: the payload had only the two service attributes and still nested every span under main
  • About 210 lines without go.sum, well over the usual size for this series. The exporter swap and the OTEL_* guards can't land separately without leaking a developer's settings to slackb.com in between. It also drops jaeger-client-go from go.mod

Requirements

Error and interrupt paths exit through clients.Os.Exit, which skips the
deferred span.Finish and reporter Close in main.go. Adds tracer.Flush
and calls it before each of those exits so the root span is sent.
A Ctrl+C during the InitConfig error-path flush made the second Flush
return early and exit while the reporter was still closing.
Replaces the archived jaeger-client-go reporter with the OpenTelemetry
SDK and OTLP exporter, sending to slackb.com/traces/v1/otlp. The
OpenTracing bridge keeps every existing span call working unchanged.
@mwbrooks mwbrooks added this to the Next Release milestone Nov 20, 2026
@mwbrooks mwbrooks added code health M-T: Test improvements and anything that improves code health semver:patch Use on pull requests to describe the release version increment labels Nov 20, 2026
@mwbrooks mwbrooks self-assigned this Nov 20, 2026
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.40984% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.57%. Comparing base (700a112) to head (8f1998c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/tracer/tracer.go 75.40% 15 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #822      +/-   ##
==========================================
+ Coverage   80.46%   80.57%   +0.10%     
==========================================
  Files         246      246              
  Lines       18882    18913      +31     
==========================================
+ Hits        15194    15239      +45     
+ Misses       3688     3674      -14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The OTLP exporter and SDK read OTEL_EXPORTER_OTLP_HEADERS, INSECURE,
OTEL_RESOURCE_ATTRIBUTES, and OTEL_TRACES_SAMPLER from the developer's
shell, which could send their own collector credentials or attributes
to slackb.com. Pins every exporter option, the resource, and the
sampler instead.
OTEL_BSP_MAX_QUEUE_SIZE=-1 panicked in SetupTracer and 0 dropped every
span.
A malformed OTEL_EXPORTER_OTLP_* variable in the developer's shell
printed parse errors on every command.
@mwbrooks
mwbrooks marked this pull request as ready for review November 20, 2026 18:32
@mwbrooks
mwbrooks requested a review from a team as a code owner November 20, 2026 18:32
@mwbrooks
mwbrooks requested review from zimeg and removed request for a team November 20, 2026 18:32
Base automatically changed from mwbrooks-tracer-flush-on-exit to main November 20, 2026 18:50

@zimeg zimeg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔭 @mwbrooks Exciting to move tracing off the archived Jaeger client! The OTEL_* guards are really thoughtful, and dropping the raw hostname and ip tags from spans is a nice privacy win.

📚 suggestion: AGENTS.md still describes tracing as "OpenTracing (Jaeger)" so let's update that line with this change too.

Comment thread internal/tracer/tracer.go
// SetupTracer sets up the tracer to send data to honeycomb with OpenTelemetry
func SetupTracer(isDev bool) (io.Closer, opentracing.Tracer) {
var collectorEndpoint = "https://slackb.com/traces/v1/jaeger"
var collectorEndpoint = "https://slackb.com/traces/v1/otlp"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ question: The testing notes mention dev.slackb.com, but main.go always calls SetupTracer(false). Have spans from a build pointed here shown up in Honeycomb with the service_name queries still matching? Since export errors are dropped, I think a check on the receiving end is the only signal before release.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in Honeycomb yet, I can't reach it from here. A build of this branch pointed at dev.slackb.com sent trace e4c9fb598d7cb43fd699407882a85a41 (session 85e6fbd5-1820-4d0f-aa2d-ea8f78326aca) at 05:55 UTC on 2026-11-21, and the exporter returned no errors. I'll hold the merge until we see it there with service_name = slack-cli matching 👀

Comment thread internal/tracer/tracer.go
sdktrace.WithBatchTimeout(sdktrace.DefaultScheduleDelay*time.Millisecond),
sdktrace.WithExportTimeout(sdktrace.DefaultExportTimeout*time.Millisecond),
),
sdktrace.WithSampler(sdktrace.AlwaysSample()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔭 thought: The span limits still read OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, and friends from the environment. Perhaps we pin these to the defaults too, to match the other guards?

Suggested change
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithSampler(sdktrace.AlwaysSample()),
// The defaults, set here so OTEL_*_LIMIT environment variables can't change them
sdktrace.WithRawSpanLimits(sdktrace.SpanLimits{
AttributeValueLengthLimit: sdktrace.DefaultAttributeValueLengthLimit,
AttributeCountLimit: sdktrace.DefaultAttributeCountLimit,
EventCountLimit: sdktrace.DefaultEventCountLimit,
LinkCountLimit: sdktrace.DefaultLinkCountLimit,
AttributePerEventCountLimit: sdktrace.DefaultAttributePerEventCountLimit,
AttributePerLinkCountLimit: sdktrace.DefaultAttributePerLinkCountLimit,
}),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned in commit 8f1998c, great catch! 🙇

@mwbrooks

Copy link
Copy Markdown
Member Author

Updated the tracing line in AGENTS.md in commit 8f1998c, thanks for the reminder @zimeg! 🙇

@mwbrooks
mwbrooks requested a review from zimeg November 21, 2026 05:56

@zimeg zimeg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔭 @mwbrooks LGTM! The pinned span limits and the new AGENTS.md tracing line both read right. Holding the merge until that dev.slackb.com trace shows up in Honeycomb with service_name matching sounds perfect to me 🎁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

code health M-T: Test improvements and anything that improves code health semver:patch Use on pull requests to describe the release version increment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants