Add data filtering options and host metrics

Added

• Add the appsignal.config.app_path resource attribute config option. In the future this will be used to sanitize backtrace/stacktrace lines and detect which lines originate from the instrumented app and which lines originate from frameworks/libraries/dependencies/etc.

• Add the hostname config option to allow customization of the hostname reported with the host metrics. By default, it will try to determine the hostname from the host itself. On Heroku it will read from the DYNO environment variable.

• Add CPU count cpu_count config option. Use it to override the auto-detected, cgroups-provided, number of CPUs that is used to calculate CPU usage percentages.

  To set it, use the APPSIGNAL_CPU_COUNT environment variable or the cpu_count configuration option in the configuration file.

  TOML

  # /etc/appsignal-collector.conf
  # These value formats are supported:
  cpu_count = 1
  cpu_count = 2.5
  cpu_count = 4

• Collect host metrics from the machine where the collector is running. This is disabled by default when the collector is running in a container. If it's not, it's enabled by default.

  Use the APPSIGNAL_ENABLE_HOST_METRICS env var or enable_host_metrics option to true to enable it and to false to disable it. If this config option is set, it overrides the defaults.

• Add resource attributes for config options to not send request query parameters, request session data and function parameters at all. If these options are set to false, their respective metadata attributes are removed from the span attributes.

  • appsignal.config.send_request_query_parameters: default is true, if set to false, the appsignal.request.query_parameters attribute is removed.
  • appsignal.config.send_request_payload: default is true, if set to false, the appsignal.request.payload attribute is removed.
  • appsignal.config.send_request_session_data: default is true, if set to false, the appsignal.request.session_data attribute is removed.
  • appsignal.config.send_function_parameters: default is true, if set to false, the appsignal.function.parameters attribute is removed.

  These options are for when the deny list appsignal.config.filter_<type> options are not enough and all data needs to be removed.

• Add the running_in_container config option. Use it to override the automatic detection of containers. This determines the method of reading host metrics. Only set this when the reported host metrics seem wrong.

  To set it, use the APPSIGNAL_RUNNING_IN_CONTAINER environment variable or the running_in_container configuration option in the configuration file.

  Shell

  # These value formats are supported:
  export APPSIGNAL_RUNNING_IN_CONTAINER="true"
  export APPSIGNAL_RUNNING_IN_CONTAINER="false"

  TOML

  # /etc/appsignal-collector.conf
  # These value formats are supported:
  running_in_container = true
  running_in_container = false

• Add a resource attribute for a config option to filter function parameters. Configure the appsignal.config.filter_function_parameters resource attribute deny list in OpenTelemetry with a list function parameter keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_function_parameters = ["email", "secret"]
  

  The function parameters, set in the appsignal.function.parameters span attribute, are filtered. The appsignal.function.parameters span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter request query parameters. Configure the appsignal.config.filter_request_query_parameters resource attribute deny list in OpenTelemetry with a list request query parameter keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_query_parameters = ["password", "secret"]
  

  The request query parameters, set in the appsignal.request.query_parameters span attribute, are filtered. The appsignal.request.query_parameters span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter request headers. Configure the appsignal.config.request_headers resource attribute allow list in OpenTelemetry to a list of header names to keep. All other headers are removed.

  Shell

  # Given this config:
  appsignal.config.request_headers = ["accept", "date"]
  
  # Results in these headers as span attributes to be stored on the span
  http.request.header.accept
  http.request.header.date
  

  By default the headers listed below are included. Setting the appsignal.config.request_headers resource attribute overrides this default. If you want to include them, you need to list them again. To include no headers, set the resource attribute value to an empty array.

  • content-length
  • accept
  • accept-charset
  • accept-encoding
  • accept-language
  • cache-control
  • connection
  • range
  • host

• Add a resource attribute for a config option to filter request payloads. Configure the appsignal.config.filter_request_payload resource attribute deny list in OpenTelemetry with a list request payload keys to filter out. The parameters are recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_payload = ["password", "secret"]
  

  The request payload, set in the appsignal.request.payload span attribute, are filtered. The appsignal.request.payload span attribute value needs to be a valid JSON object for it to be filtered.

• Add a resource attribute for a config option to filter response headers. Configure the appsignal.config.response_headers resource attribute allow list in OpenTelemetry to a list of header names to keep. All other headers stored by HTTP client instrumentations are removed.

  Shell

  # Given this config:
  appsignal.config.request_headers = ["accept", "date"]
  
  # Results in these headers as span attributes to be stored on the span
  http.request.header.accept
  http.request.header.date
  

  By default the headers listed below are included. Setting the appsignal.config.response_headers resource attribute overrides this default. If you want to include them, you need to list them again. To include no headers, set the resource attribute value to an empty array.

  • content-length
  • accept
  • accept-charset
  • accept-encoding
  • accept-language
  • cache-control
  • connection
  • range
  • host

• Add a resource attribute for a config option to filter request session data. Configure the appsignal.config.filter_request_session_data resource attribute deny list in OpenTelemetry with a list request session data keys to filter out. The session data is recursively filtered, so any nested objects will also be filtered.

  With this config:

  Shell

  appsignal.config.filter_request_session_data = ["token", "secret"]
  

  The request session data, set in the appsignal.request.session_data span attribute, are filtered. The appsignal.request.session_data span attribute value needs to be a valid JSON object for it to be filtered.

Fixed

• Fix an issue with the collector not shutting down when the HTTP server's port is in use and it cannot claim it. It would get stuck retrying to claim the port. It will now shut down properly when receiving a quit signal.
• Fix value reporting of cumulative counter metrics. The conversion did not take into account metric attributes, so all metrics with the same name would be merged.
• Fix an issue with the HTTP server not shutting down after it has started and received a quit signal. It would continue to receive requests and prevent the collector from shutting down.

This release can be installed through our collector packages and Docker image.