(Archived) Using Jaeger or Zipkin

You can collect traces with Jaeger or Zipkin and send the trace data to Operations for Applications, which:

• Provides managed, highly scalable storage for your trace data.
• Allows you to examine and alert on RED metrics that are derived from the spans.

Suppose you have already instrumented your application using Jaeger or Zipkin with OpenTracing or OpenTelemetry. You can continue using that system for application development, and then switch to using Operations for Applications by changing a few configuration settings.

Tracing-System Integrations and Exporters

You can use the Jaeger or Zipkin integrations, but you can also specify custom tags.

To get data flowing:

• Follow setup steps for the Jaeger integration
• Follow setup steps for the Zipkin integration

Each integration

1. Configures your distributed tracing system to send trace data to a Wavefront proxy. During integration setup, follow the prompts to create a new integration or use an existing integration.
2. The proxy processes the data and sends it to your Operations for Applications service.

Part of setting up the integration is to configure the Wavefront proxy to listen for the trace data on an integration-specific port.

Using an integration is the simplest way - but not the only way - to send trace data to Operations for Applications from a 3rd part tracing system.

Trace Data from an Integration

When you use a tracing-system integration, the Wavefront proxy receives the trace data that your application emits. If you configured your distributed tracing system to perform sampling, the proxy receives only the spans that the 3rd party sampler has allowed.

The Wavefront proxy:

• Checks for required span tags on each received span, and adds them if needed.
• Automatically derives RED metrics from the received spans.

Required Span Tags

Operations for Applications requires certain span tags on well-formed spans. The following spans tags enable you to filter and visualize trace data from the different services in your instrumented application:

The proxy preserves any tags that you assigned through your distributed tracing system. You can explicitly instrument your code to add an application tag with a preferred application name.

The mandatory span tags cannot have multiple values. Ensure that your application does not send spans with multiple application or service tags. For example, a span with two span tags service=notify and service=backend is invalid.

Derived RED Metrics

Operations for Applications automatically derives RED metrics from the spans that are sent from the instrumented application services. RED metrics are measures of the request Rate, Errors, and Duration that are obtained from the reported spans. These metrics are key indicators of the health of your services, and you can use them as context to help you discover problem traces.

Operations for Applications stores the RED metrics along with the spans they are based on. For more details, see RED Metrics.

Custom Tags for RED Metrics

Include custom tags for RED metrics using the Wavefront proxy. To do so, add:

traceDerivedCustomTagKeys=<comma-separated-custom-tag-keys>

to the proxy configuration file (/etc/wavefront/wavefront-proxy/wavefront.conf by default).

The Wavefront proxy generates custom tags for the specified keys.

For example, if you add the following properties, the Wavefront proxy generates 3 custom tags:

traceDerivedCustomTagKeys=tenant, env, location

Custom Application Names

You can set up custom application names for Jaeger and Zipkin. The process differs depending on your source, as follows.

Custom Application Names for Jaeger

You can specify custom application names at the level you need, like this:

• Span-level tag: Add the application tag to all spans.
• Process-level tag: Add the application tag as a Jaeger tracer tag, that is, a process tag.
• Proxy-level tag: Add traceJaegerApplicationName=<application-name> in the proxy configuration at /etc/wavefront/wavefront-proxy/wavefront.conf. See Proxy Configuration Paths for details on the config file location.

The order of precedence is span level > process level > proxy level.

Custom Application Names for Zipkin

You can specify custom application names at the level you need, like this:

• Span-level tag: Add tag application to all spans.
• Proxy-level tag: Add traceZipkinApplicationName=<application-name> in the proxy configuration at /etc/wavefront/wavefront-proxy/wavefront.conf. See Proxy Configuration Paths for details on the config file location.

The order of precedence is span level > proxy level.

Visualizing Trace Data from an Integration

You view trace data from an integration using the Application Status, Service Dashboard. and Traces Browser.

If you want context for identifying problem traces, you can start by viewing the derived RED metrics:

1. From the toolbar, select Applications > Application Status, and find the application (by default, Jaeger or Zipkin).
2. Click the application name, and find the service whose RED metrics you want to see.
3. Click the Details link for the service.
4. Select an operation from one of the charts to examine the traces for that operation.

If you want to view trace data directly, you can start by submitting a trace query:

1. From the toolbar, select Applications > Traces.
2. In the Traces Browser, submit a trace query by selecting the operations and filters that describe the spans of interest.
3. Click Search.

Trace Sampling and RED Metrics from an Integration

When you use a 3rd party distributed tracing system, you normally configure it to perform sampling. Doing so means that sampling occurs first, before the Wavefront proxy derives the RED metrics. The Wavefront proxy receives only a subset of the generated spans, and the RED metrics will reflect just that subset.

For more accurate RED metrics, you can disable the 3rd party sampling, and set up sampling through the Wavefront proxy.

The Wavefront proxy will auto-derive the RED metrics first, and then perform the sampling.

Enable Logs

You can enable logging for your Jaeger and Zipkin integrations.

Enable Logs for Jaeger Integrations

Follow these steps:

1. Open the <wavefront_config_path>/log4j2.xml file.

2. Add the configurations to enable and manage logs under <Appenders>.
   Example:

    <Appenders>
       <RollingFile name="[Enter_Your_File_Name]" fileName="${log-path}/wavefront-jaeger-data.log" filePattern="${log-path}/wavefront-jaeger-data-%d{yyyy-MM-dd}-%i.log">
          <PatternLayout>
             <pattern>%m%n</pattern>
          </PatternLayout>
          <Policies>
             <TimeBasedTriggeringPolicy interval="1" />
             <SizeBasedTriggeringPolicy size="1024 MB" />
          </Policies>
          <DefaultRolloverStrategy max="10">
             <Delete basePath="/var/log/wavefront" maxDepth="1">
                <IfFileName glob="wavefront-jaeger*.log" />
                <IfLastModified age="7d" />
             </Delete>
          </DefaultRolloverStrategy>
       </RollingFile>
    </Appenders>
   

   Note: See the log4j2 documentation for information on each parameter.

3. Add the logger name for JaegerDataLogger inside <Loggers>.
   Example:

      <!-- Set level="ALL" to log Jeager data to a file. -->
      <Loggers>
         <AsyncLogger name="JaegerDataLogger" level="DEBUG" additivity="false">
            <AppenderRef ref="[Enter_Your_File_Name]" />
         </AsyncLogger>
      </Loggers>
   

4. Save the file.

Enable Logs for Zipkin Integrations

Follow these steps:

1. Open the <wavefront_config_path>/log4j2.xml file.

2. Add the configurations to enable and manage logs under <Appenders>.
   Example:

    <Appenders>
       <RollingFile name="[Enter_Your_File_Name]" fileName="${log-path}/wavefront-zipkin-data.log" filePattern="${log-path}/wavefront-zipkin-data-%d{yyyy-MM-dd}-%i.log">
          <PatternLayout>
             <pattern>%m%n</pattern>
          </PatternLayout>
          <Policies>
             <TimeBasedTriggeringPolicy interval="1" />
             <SizeBasedTriggeringPolicy size="1024 MB" />
          </Policies>
          <DefaultRolloverStrategy max="10">
             <Delete basePath="${log-path}/var/log/wavefront" maxDepth="1">
                <IfFileName glob="wavefront-zipkin*.log" />
                <IfLastModified age="7d" />
             </Delete>
          </DefaultRolloverStrategy>
       </RollingFile>
    </Appenders>
   

   Note: See the log4j2 documentation for information on each parameter.
3. Add the logger name for ZipkinDataLogger under <Loggers>.
   Example:

    <!-- Set level="ALL" to log Zipkin data to a file. -->
    <Loggers>
       <AsyncLogger name="ZipkinDataLogger" level="ALL" additivity="false">
          <AppenderRef ref="[Enter_Your_File_Name]" />
       </AsyncLogger>
    </Loggers>
   

4. Save the file.

Alternatives to Integrations

If using the Jaeger or Zipkin integration doesn’t make sense in your environment, you can still use Operations for Applications with those systems.

Send Raw Trace Data

We don’t support an integration for your distributed tracing system, or if you are using your own proprietary tracing system, you can use a sender SDK to send raw trace data to Operations for Applications. With a sender SDK, you can write code that obtains the component values from your spans, and assembles those values into the Operations for Applications span format. The sender SDK also lets you configure your application to send the trace data to a Wavefront proxy or directly to Operations for Applications.

For SDK setup details, see the Sender SDK for your programming language.

Use the Wavefront OpenCensus Go Exporter

If you have instrumented your Go application with OpenCensus, you can use the Wavefront OpenCensus Go Exporter to push metrics, histograms, and traces into Operations for Applications. This exporter is built on the Sender SDK for Go.