Improve error reporting for Next.js app

Fixed

• Report errors in AppSignal for exception events in Next.js render spans.

This release can be installed through our collector packages and Docker image.

Improved histograms and logs, report all child spans, and CORS support

Added

• Improve histogram support. When processing OpenTelemetry histograms, allow for histograms that do not have min, max or sum values. Use the exemplars provided alongside the histogram to improve the accuracy of the histogram.
• Return CORS headers from the export endpoints to make it possible to export OpenTelemetry data to these endpoints without the need of a proxy server. The Access-Control-Allow-Origin header value can be configured in the collector with the cors_origin config option or APPSIGNAL_CORS_ORIGIN environment variable.
• Support key-value list attributes in log lines. When an attribute sent in a log line is a key-value list, it will be converted into multiple attributes, one for each key-value pair in the list, prefixed with the key of the list attribute.

Changed

• Disable the default AppSignal request and response headers filter. By default, the OpenTelemetry SDK in the applications do not send any request and response headers and require configuration of to record these headers. This required double configuration in applications for AppSignal and OpenTelemetry. The appsignal.config.request_headers and appsignal.config.response_headers configuration options can still be used for additional header filtering if needed. Consult your application's language OpenTelemetry SDK for the configuration option to record request and response headers.
• Log a warning when making a GET request to a POST endpoint. The error page when making a GET request will also explain to make a POST request instead.
• Read the app language from the OpenTelemetry telemetry.sdk.language resource attribute if set. This attribute is set by the OpenTelemetry SDK for some languages, like Go, Java and PHP. If your application's OpenTelemetry SDK sets the telemetry.sdk.language resource attribute, you will not need to set the appsignal.config.language_integration attribute. If the appsignal.config.language_integration attribute is set, it will take precedence over the telemetry.sdk.language resource attribute.
• Report log lines with unspecified severity. When a log line does not have a severity, report it as a log line with info level severity.

Fixed

• Fix child spans not always being reported. This was due to the collector not properly traversing all the levels the trace span hierarchy.

This release can be installed through our collector packages and Docker image.

Ignore logs and improved trace processing

Added

• Add an ignore_logs config option, which can be set using the appsignal.config.ignore_logs resource attribute. When this resource attribute is set, logs whose message matches any of the values in the ignore_logs list of values will be discarded.

  The values on the ignore_logs configuration option support the ^, $ and .* meta-characters, with their expected meanings in regular expressions. When neither ^ or $ are specified at the beginning or ending of the value respectively, ignore_logs will match a log message if its value is present at any point in the message.

  For example, to ignore logs starting with "success" or containing the word "OK", when setting the resource attributes in a Go application:

  Go

  res := resource.NewWithAttributes(
  	attribute.StringSlice("appsignal.config.ignore_logs", []string{"^success", "OK"}),
  	// And other resource attributes
  )

  The list of patterns can also be provided as a comma-separated string -- this is useful when using the OTEL_RESOURCE_ATTRIBUTES environment variable, such as in a PHP application:

  Shell

  function encode() {
    echo -n "$@" | sed 's/,/%2C/g'
  }
   
  export OTEL_RESOURCE_ATTRIBUTES="\
    appsignal.config.ignore_errors=$(encode "^success,OK"),\
    ..."

Changed

• Improve AppSignal compatibility with traces across multiple services.

This release can be installed through our collector packages and Docker image.

Collector v0.6.2

Added

• When reporting a boolean resource attribute configuration option, such as send_function_parameters, send_request_payload, send_request_query_parameters and send_request_session_data, allow it to be provided as a string containing the case-insensitive values "true" or "false".

  This makes it possible to set these configuration options using the OTEL_RESOURCE_ATTRIBUTES environment variable.

Changed

• Allow overriding automatically set namespaces. When a namespace is automatically by AppSignal, set it only if a different namespace has not been set in the trace using the appsignal.namespace attribute.
• Improve communication from the collector's exporter server to the client by returning the expected OpenTelemetry export response messages from the collector's HTTP server. This includes updating the tracing, metrics and logs endpoints by returning their respective response messages.

Fixed

• Truncate the keys of attributes and the overall size of the attribute list for spans, events, scopes, resources and links.

This release can be installed through our collector packages and Docker image.

Ignore errors, report Express errors and logging improvements

Added

• Add ignore_errors option. Configure this option using the appsignal.config.ignore_errors resource attribute. The errors are matched by class/type.

  The value can be set both as an array or a comma-separated string. The following examples are equivalent:

  Shell

  appsignal.config.ignore_errors=["RuntimeError", "NoMethodError"]
  appsignal.config.ignore_errors="RuntimeError,NoMethodError"
  

• Allow log records with non-string values as bodies.

  When a log record does not have a string as its body, but it has attributes, serialise those attributes as JSON and use that as the log message instead of the body.

  When a log record has a key-value list as its body, use those key-value pairs as attributes.

• Extract attributes in Laravel logs, encoded into the context attribute, into separate attributes that can be explored individually in AppSignal.

Fixed

• Support Express instrumentation version 0.200.0, by ensuring that exception events from Express request handler spans are always reported.

This release can be installed through our collector packages and Docker image.

Breaking change: update required, internal protocol update

Added

• Add support for OpenTelemetry logs. Logs sent to the collector's /v1/logs endpoint in OTLP/HTTP format will be delivered to AppSignal's logging feature.
• Enrich Sidekiq events by setting the action name. Now Sidekiq jobs are reported as <WorkerClass>#perform, e.g. MyWorker#perform.

Changed

• BREAKING CHANGE: Use a better optimized protocol to send data to our API. This change is backwards incompatible and we recommend you upgrade to the latest version of the collector. Older versions of the collector will not be able to send data to our API. Please refer to the documentation for more information on how to upgrade your collector.

This release can be installed through our collector packages and Docker image.

Update error incident generation

Added

• Update our support OpenTelemetry Semantic Conventions for code attributes used in unique span detection.

Changed

• The shutdown behavior on internal errors is improved, and now logs what crashed and why.

• Limit appsignal.tag.<key> tag values to 256 characters.

• Rename an internal span event attribute for metadata distributions. Older collectors will no longer report metadata distributions for error incidents.

  This will create new incidents for existing errors. This is intentional.

Fixed

• Fix a crash on truncating Unicode values.

This release can be installed through our collector packages and Docker image.

Configuration via request headers

Added

• Configure AppSignal via HTTP request headers on export requests. If it's not possible to configure resource attributes in OpenTelemetry export data, configure AppSignal via request headers set on the export requests made to the collector's HTTP endpoints.

  Supported headers are:

  AppSignal-Config-Name: app name (required) AppSignal-Config-Environment: app environment (required) AppSignal-Config-PushApiKey: push API key (required) AppSignal-Config-Revision: app revision (required) AppSignal-Config-LanguageIntegration: language integration like ruby, elixir, php, etc. (required) OpenTelemetry-ServiceName: service name of the app OpenTelemetry-HostName: name of the app's host (required)

  For more information, see our configuration documentation.

Changed

• Close HTTP connections from OpenTelemetry exports when we have successfully received the request. This fixes an issue with the server not shutting down when the connection is kept open.

This release can be installed through our collector packages and Docker image.

Ignore empty environment variables for config options

Added

• Add the appsignal.config.otp_app resource attribute config option. When set for Elixir applications, we will use the OTP app name to detect which stacktrace lines originate from the app, and which lines are from dependencies.

Changed

• Consider empty environment variables as unset. From now on, an environment variable that's set to an empty string is considered unset when reading the collector configuration from system environment variables.

  In the example below, the environment variable will be considered unset. It will not fail on parsing an empty string as a number and set the default HTTP port instead.

  Shell

  APPSIGNAL_HTTP_PORT=""
  

  This behavior is changed for the following config options:

  • APPSIGNAL_CPU_COUNT
  • APPSIGNAL_ENABLE_HOST_METRICS
  • APPSIGNAL_FILES_WORLD_ACCESSIBLE
  • APPSIGNAL_HOSTNAME
  • APPSIGNAL_HTTP_PORT
  • APPSIGNAL_LOG_LEVEL
  • APPSIGNAL_PUSH_API_ENDPOINT
  • APPSIGNAL_PUSH_API_KEY
  • APPSIGNAL_RUNNING_IN_CONTAINER

• In the appsignal/collector Docker image we will now listen to the PORT environment variable if it's set, and use it to configure the HTTP server's port.

  If the APPSIGNAL_HTTP_PORT environment variable is also set in addition to the PORT environment variable, it will read from the APPSIGNAL_HTTP_PORT environment variable.

This release can be installed through our collector packages and Docker image.

Support gzipped payloads

Added

• Gzipped payloads for the collector requests are now accepted if specified in the Content-Encoding header.

This release can be installed through our collector packages and Docker image.

Container host metrics and bug fixes

Changed

• Log debug messages and warnings from the config initialization. Previously, any debug messages or warnings that should have been logged upon config initialization where not logged. Now, all config log messages are logged, including debug messages even if the log level is configured as info. This may be improved in the future.
• Shut down the collector in case of an internal error. We've seen problems where internal channels close and it causes the collector to stop reporting data. In this case, log what happens and shut down.

Fixed

• Disable the host metrics only on the collector container, by default. When the collector is installed on a container alongside the app, it will collect container host metrics.

This release can be installed through our collector packages and Docker image.

Filter OpenTelemetry query parameters span attributes

Added

• Filter query parameters in the url.query span attribute. When an url.query attribute contains an URL encoded query parameter we will filter out the keys configured in the appsignal.config.filter_request_query_parameters resource attribute.

  In the situation that a url.query span attribute has the value password=secret&some_param=some_value, and the appsignal.config.filter_request_query_parameters resource attribute is configured with ["password"], the url.query span attribute will be stored as password=[FILTERED]&some_param=some_value.

  If no appsignal.request.query_parameters span attribute is set, we will store the query parameters as a JSON map on this attribute. For example:

  JSON

  {
    "password": "[FILTERED]",
    "some_param": "some_value"
  }

Changed

• Rename the working_dir_path config option to working_directory_path. This names matches with the naming in our other tools and integrations.

  This is a breaking change: With this change the working_dir_path config option will no longer work.

• When the appsignal.config.send_request_query_parameters resource attribute config option value is set to false, we will remove the url.query span attribute from spans.

This release can be installed through our collector packages and Docker image.

Cumulative histogram support and improved query sanitization

Added

• Sanitize database query attributes following the OpenTelemetry Semantic Conventions 1.30 database specification. We now sanitize SQL queries in the db.query.text attribute and Redis queries in the db.operation.name attribute.
• Add support for cumulative histograms. The previous known state of a cumulative histogram will be stored and its delta will be calculated, in order to send it to AppSignal as a delta histogram.

This release can be installed through our collector packages and Docker image.

Add data filtering options and host metrics

Added

• Add the appsignal.config.app_path resource attribute config option. In the future this will be used to sanitize backtrace/stacktrace lines and detect which lines originate from the instrumented app and which lines originate from frameworks/libraries/dependencies/etc.

• Add the hostname config option to allow customization of the hostname reported with the host metrics. By default, it will try to determine the hostname from the host itself. On Heroku it will read from the DYNO environment variable.

• Add CPU count cpu_count config option. Use it to override the auto-detected, cgroups-provided, number of CPUs that is used to calculate CPU usage percentages.

  To set it, use the APPSIGNAL_CPU_COUNT environment variable or the cpu_count configuration option in the configuration file.

  TOML

  # /etc/appsignal-collector.conf
  # These value formats are supported:
  cpu_count = 1
  cpu_count = 2.5
  cpu_count = 4

• Collect host metrics from the machine where the collector is running. This is disabled by default when the collector is running in a container. If it's not, it's enabled by default.

  Use the APPSIGNAL_ENABLE_HOST_METRICS env var or enable_host_metrics option to true to enable it and to false to disable it. If this config option is set, it overrides the defaults.

• Add resource attributes for config options to not send request query parameters, request session data and function parameters at all. If these options are set to false, their respective metadata attributes are removed from the span attributes.

  • appsignal.config.send_request_query_parameters: default is true, if set to false, the appsignal.request.query_parameters attribute is removed.
  • appsignal.config.send_request_payload: default is true, if set to false, the appsignal.request.payload attribute is removed.
  • appsignal.config.send_request_session_data: default is true, if set to false, the appsignal.request.session_data attribute is removed.
  • appsignal.config.send_function_parameters: default is true, if set to false, the appsignal.function.parameters attribute is removed.

  These options are for when the deny list appsignal.config.filter_<type> options are not enough and all data needs to be removed.

• Add the running_in_container config option. Use it to override the automatic detection of containers. This determines the method of reading host metrics. Only set this when the reported host metrics seem wrong.

  To set it, use the APPSIGNAL_RUNNING_IN_CONTAINER environment variable or the running_in_container configuration option in the configuration file.

  Shell

  # These value formats are supported:
  export APPSIGNAL_RUNNING_IN_CONTAINER="true"
  export APPSIGNAL_RUNNING_IN_CONTAINER="false"

  TOML

  # /etc/appsignal-collector.conf
  # These value formats are supported:
  running_in_container = true
  running_in_container = false

• Add a resource attribute for a config option to filter function parameters. Configure the appsignal.config.filter_function_parameters resource attribute deny list in OpenTelemetry with a list function parameter keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_function_parameters = ["email", "secret"]
  

  The function parameters, set in the appsignal.function.parameters span attribute, are filtered. The appsignal.function.parameters span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter request query parameters. Configure the appsignal.config.filter_request_query_parameters resource attribute deny list in OpenTelemetry with a list request query parameter keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_query_parameters = ["password", "secret"]
  

  The request query parameters, set in the appsignal.request.query_parameters span attribute, are filtered. The appsignal.request.query_parameters span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter request headers. Configure the appsignal.config.request_headers resource attribute allow list in OpenTelemetry to a list of header names to keep. All other headers are removed.

  Shell

  # Given this config:
  appsignal.config.request_headers = ["accept", "date"]
  
  # Results in these headers as span attributes to be stored on the span
  http.request.header.accept
  http.request.header.date
  

  By default the headers listed below are included. Setting the appsignal.config.request_headers resource attribute overrides this default. If you want to include them, you need to list them again. To include no headers, set the resource attribute value to an empty array.

  • content-length
  • accept
  • accept-charset
  • accept-encoding
  • accept-language
  • cache-control
  • connection
  • range
  • host

• Add a resource attribute for a config option to filter request payloads. Configure the appsignal.config.filter_request_payload resource attribute deny list in OpenTelemetry with a list request payload keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_payload = ["password", "secret"]
  

  The request payload, set in the appsignal.request.payload span attribute, are filtered. The appsignal.request.payload span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter response headers. Configure the appsignal.config.response_headers resource attribute allow list in OpenTelemetry to a list of header names to keep. All other headers stored by HTTP client instrumentations are removed.

  Shell

  # Given this config:
  appsignal.config.request_headers = ["accept", "date"]
  
  # Results in these headers as span attributes to be stored on the span
  http.request.header.accept
  http.request.header.date
  

  By default the headers listed below are included. Setting the appsignal.config.response_headers resource attribute overrides this default. If you want to include them, you need to list them again. To include no headers, set the resource attribute value to an empty array.

  • content-length
  • accept
  • accept-charset
  • accept-encoding
  • accept-language
  • cache-control
  • connection
  • range
  • host

• Add a resource attribute for a config option to filter request session data. Configure the appsignal.config.filter_request_session_data resource attribute deny list in OpenTelemetry with a list request session data keys to filter out. The session data is recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_session_data = ["token", "secret"]
  

  The request session data, set in the appsignal.request.session_data span attribute, are filtered. The appsignal.request.session_data span attribute value needs to be a valid JSON object for it to be filtered.

Fixed

• Fix an issue with the collector not shutting down when the HTTP server's port is in use and it cannot claim it. It would get stuck retrying to claim the port. It will now shut down properly when receiving a quit signal.
• Fix value reporting of cumulative counter metrics. The conversion did not take into account metric attributes, so all metrics with the same name would be merged.
• Fix an issue with the HTTP server not shutting down after it has started and received a quit signal. It would continue to receive requests and prevent the collector from shutting down.

This release can be installed through our collector packages and Docker image.

Support for OpenTelemetry metrics

Added

• Add OpenTelemetry metrics endpoint. The collector can now receive OpenTelemetry metrics exports n the 0.0.0.0:8099/v1/metrics endpoint. The following metric types are supported: Gauge, Sum and Histogram. Configure your OpenTelemetry SDK to export to this application to track OpenTelemetry metrics in AppSignal.

Changed

• Rename the 'port' config option to 'http_port'. This better describes which port the configuration option is for.
• Use default transmission interval when invalid transmission interval values are set.
• Retry binding HTTP port for 5 times if it's in use. This gives the collector a longer time to connect to the port if it becomes available within a 4 minute window.
• Show all config errors on collector start failure. This will help resolve all errors at once rather than fix one at a time.
• Show all errors on OpenTelemetry's export request's resource validation. When one or more resource attributes are missing from the export request's resources, all errors are listed.

Fixed

• Report metrics for not sampled traces when actions reached the sampling rate. Now metrics are reported for all traces.

This release can be installed through our collector packages and Docker image.

Initial release

Added

• Initial collector release for public beta.