Menu

← Learning center

What logs should I store?

What logs should I store?

Logs are a fantastic resource for additional "behind the scenes" information on your application's behavior. However, with so many different logs emitted by an application across various logging levels, which ones should you be storing?

The answer to what logs you should store will vary depending on your application and intentions; however, this article will provide examples of when logs can be valuable and worth storing and which logs are better suited to local debugging.

What to store

The logs you store should be useful in helping you gain more significant insights into the behavior of your application and should provide valuable information. When deciding if a log is worth storing, consider:

  • Is the log meaningful?
  • Does the log provide helpful context?
  • Is the log readable?

To ensure your logs meet the above criteria, we recommend:

Process start and end markers

Logs that describe critical flows in your application are very useful when debugging an issue in your application.

You can format your logs in a logical and descriptive manner that allows you to quickly understand where in your application a log is being generated from:

bash
FlowName - UUID action Function/MethodName
bash
FlowName - UUID action Function/MethodName

Let's imagine we want to log user actions within an application's order flow:

bash
INFO -- Main: Order - User 1293 started CreateOrder
INFO -- Main: Order - User 1293 started OrderPayment
INFO -- Main: Order - User 1293 finished OrderPayment
INFO -- Main: Order - User 1293 started SendOrder
INFO -- Main: Order - User 1293 finished SendOrder
bash
INFO -- Main: Order - User 1293 started CreateOrder
INFO -- Main: Order - User 1293 started OrderPayment
INFO -- Main: Order - User 1293 finished OrderPayment
INFO -- Main: Order - User 1293 started SendOrder
INFO -- Main: Order - User 1293 finished SendOrder

In the above example, the logs tell us that a user has started and ended a process flow within our application.CreateOrder, OrderPayment, and SendOrder refer to modules in our code base, as they are defined in our code base, which allows us to locate the code generating the logs easily.

Important contextual information

When writing logs, providing additional context, such as external requests, response data, and outcomes of critical steps, can often be invaluable.

For example:

bash
INFO -- Main: Order - User 1293 started SendOrder
INFO -- Main: Order - User 1293 sent order 5302 to PizzaAPI
INFO -- Main: Order - User 1293 received response 201 status Created from PizzaAPI
INFO -- Main: Order - User 1293 finished SendOrder
bash
INFO -- Main: Order - User 1293 started SendOrder
INFO -- Main: Order - User 1293 sent order 5302 to PizzaAPI
INFO -- Main: Order - User 1293 received response 201 status Created from PizzaAPI
INFO -- Main: Order - User 1293 finished SendOrder

The above log lines give us more significant insights into the user's flow and allow us to filter on the user's UUID.

Errors

As we often look at our logs to help us understand why something went wrong, it's invaluable to store error logs.

When writing error logs, include as much information as possible to help deduce the reason for an error, such as response codes and messages.

bash
ERROR -- Main: Order - User 1293 received response 400 status Bad request from PizzaAPI with message "Invalid data type for 'pizza_topping_ids'. Expected an array of integers."
bash
ERROR -- Main: Order - User 1293 received response 400 status Bad request from PizzaAPI with message "Invalid data type for 'pizza_topping_ids'. Expected an array of integers."

What not to store

While your application may generate dozens of logs a minute, not all are worth storing. Some may only be useful in specific use cases, such as locally debugging your application.

Storing logs that are not needed can generate a lot of noise, making log filters and queries less effective not to mention over time, these logs can take up a lot of space which may cost money or prevent you from storing more useful logs.

To avoid storing logs that provide no to little added value to your application monitoring, we advise:

Debug and trace logs

Regarding logging levels, debug and trace level logs are for development and debugging purposes, often contain verbose information, and should not be stored in your log management. You can read more about log levels in our "What is logging?" article.

Non-descriptive logs

Logs that aren't descriptive should ideally be omitted. For example, imagine you have an application that writes an info log every time a user logs in.

bash
INFO -- Main: User 2313 logged in
INFO -- Main: User 4476 logged in
INFO -- Main: User 5968 logged in
INFO -- Main: User 4092 logged in
bash
INFO -- Main: User 2313 logged in
INFO -- Main: User 4476 logged in
INFO -- Main: User 5968 logged in
INFO -- Main: User 4092 logged in

While this information may be helpful when debugging your application locally, on a production application with thousands of unique sign-ins per day, it's noise that can potentially distract from more important and valuable logs - and not to mention, if you are using a log management solution you'll be paying to store these logs too!

It can be advised to remove such logs or write them as debug logs so that they can assist you in debugging while not being stored by your log manager.

Custom Metrics

Suppose you log information, like the above example, to see if a user is logged in. In that case, you can use alternative methods like AppSignal's Custom Metrics to track an active user count rather than rely on logs.

Verbose logs

It can be tempting to log everything because if you log everything, you can re-create every scenario and understand everything happening in your application.

However, logging too much information can be as detrimental as logging too little; flooding your logs with large amounts of verbose data can make it more difficult to query and filter your logs for useful information and slow down your ability to debug issues:

bash
INFO -- Main: User 87235 logged in: {"user_id"=>87235, "last_log_in"=>"2023-07-27 09:15:23", "language"=>"English", "country"=>"Canada", "notifications_enabled"=>false, "organization_id"=>567, "profile_description"=>"Travel enthusiast.", "avatar_image"=>"avatar_default.jpg", "updated_at"=>"2023-07-27 10:30:05", "created_at"=>"2023-07-27 08:40:12"}
INFO -- Main: User 46579 logged in: {"user_id"=>46579, "last_log_in"=>"2023-07-27 17:55:10", "language"=>"Spanish", "country"=>"Spain", "notifications_enabled"=>true, "organization_id"=>987, "profile_description"=>"Tech lover and gamer.", "avatar_image"=>"avatar_46579.jpg", "updated_at"=>"2023-07-27 18:10:20", "created_at"=>"2023-07-27 16:20:30"}
INFO -- Main: User 23107 logged in: {"user_id"=>23107, "last_log_in"=>"2023-07-27 06:25:48", "language"=>"German", "country"=>"Germany", "notifications_enabled"=>true, "organization_id"=>321, "profile_description"=>"Fitness freak and nature enthusiast.", "avatar_image"=>"avatar_23107.jpg", "updated_at"=>"2023-07-27 07:40:15", "created_at"=>"2023-07-27 05:10:25"}
INFO -- Main: User 90876 logged in: {"user_id"=>90876, "last_log_in"=>"2023-07-27 12:45:33", "language"=>"Japanese", "country"=>"Japan", "notifications_enabled"=>false, "organization_id"=>123, "profile_description"=>"Foodie and photography enthusiast.", "avatar_image"=>"avatar_90876.jpg", "updated_at"=>"2023-07-27 13:20:55", "created_at"=>"2023-07-27 11:30:40"}
INFO -- Main: User 78903 logged in: {"user_id"=>78903, "last_log_in"=>"2023-07-27 23:05:15", "language"=>"English", "country"=>"Belgium", "notifications_enabled"=>true, "organization_id"=>456, "profile_description"=>"Bookworm and writer.", "avatar_image"=>"avatar_78903.jpg", "updated_at"=>"2023-07-27 23:45:30", "created_at"=>"2023-07-27 21:50:10"}
bash
INFO -- Main: User 87235 logged in: {"user_id"=>87235, "last_log_in"=>"2023-07-27 09:15:23", "language"=>"English", "country"=>"Canada", "notifications_enabled"=>false, "organization_id"=>567, "profile_description"=>"Travel enthusiast.", "avatar_image"=>"avatar_default.jpg", "updated_at"=>"2023-07-27 10:30:05", "created_at"=>"2023-07-27 08:40:12"}
INFO -- Main: User 46579 logged in: {"user_id"=>46579, "last_log_in"=>"2023-07-27 17:55:10", "language"=>"Spanish", "country"=>"Spain", "notifications_enabled"=>true, "organization_id"=>987, "profile_description"=>"Tech lover and gamer.", "avatar_image"=>"avatar_46579.jpg", "updated_at"=>"2023-07-27 18:10:20", "created_at"=>"2023-07-27 16:20:30"}
INFO -- Main: User 23107 logged in: {"user_id"=>23107, "last_log_in"=>"2023-07-27 06:25:48", "language"=>"German", "country"=>"Germany", "notifications_enabled"=>true, "organization_id"=>321, "profile_description"=>"Fitness freak and nature enthusiast.", "avatar_image"=>"avatar_23107.jpg", "updated_at"=>"2023-07-27 07:40:15", "created_at"=>"2023-07-27 05:10:25"}
INFO -- Main: User 90876 logged in: {"user_id"=>90876, "last_log_in"=>"2023-07-27 12:45:33", "language"=>"Japanese", "country"=>"Japan", "notifications_enabled"=>false, "organization_id"=>123, "profile_description"=>"Foodie and photography enthusiast.", "avatar_image"=>"avatar_90876.jpg", "updated_at"=>"2023-07-27 13:20:55", "created_at"=>"2023-07-27 11:30:40"}
INFO -- Main: User 78903 logged in: {"user_id"=>78903, "last_log_in"=>"2023-07-27 23:05:15", "language"=>"English", "country"=>"Belgium", "notifications_enabled"=>true, "organization_id"=>456, "profile_description"=>"Bookworm and writer.", "avatar_image"=>"avatar_78903.jpg", "updated_at"=>"2023-07-27 23:45:30", "created_at"=>"2023-07-27 21:50:10"}

For this reason, just like we advise against storing debug and trace logs, we also advise against including hashes in your log message and advocate instead of using custom log attributes instead. These attributes can be queried and filtered; you can log valuable attributes instead of all of an object's attributes.

Personal Identifiable Information (PII)

Regardless of what logs you ultimately decide to store in your log manager, you must refrain from logging Personal Identifiable Information, or PII for short. Your log data must not contain any personal data, such as names, email addresses, etc. You must filter and remove this data before writing a log.

If you need to identify specific users in your logs, you must use alternative forms of identification, such as a user ID or pseudonym.

Custom log attributes

AppSignal's Custom log attributes allow you to add additional, filterable data to your log lines. These attributes are helpful when logging API interactions, allowing you to contextualize better how the API and your application behave.

Let's imagine we have an app that allows users to order pizza, an inefficient way to capture the API request information would be to include it in the log message:

bash
INFO -- Main: Order - User 9421 sent order to PizzaAPI params: {"timestamp": "2022-06-02T04:17:25.783Z", "group": "organizations", "severity": "info", "message": "Order sent successfully to pizzeria", "shop_id": "23910", "order_id": "5302", "pizza_base_id": "12", "pizza_topping_ids": ["9", "12", "17", "24"], "user_id": "9421", "delivery_method": "collect"}
bash
INFO -- Main: Order - User 9421 sent order to PizzaAPI params: {"timestamp": "2022-06-02T04:17:25.783Z", "group": "organizations", "severity": "info", "message": "Order sent successfully to pizzeria", "shop_id": "23910", "order_id": "5302", "pizza_base_id": "12", "pizza_topping_ids": ["9", "12", "17", "24"], "user_id": "9421", "delivery_method": "collect"}

However, as previously discussed, sending large hashes of data is not optimal and can be counterproductive. Instead, with AppSignal, we can create a log with custom attributes of the information we need, like in this JSON example:

json
{
  "timestamp": "2022-06-02T04:17:25.783Z",
  "group": "organizations",
  "severity": "info",
  "message": "Order sent successfully to pizzeria",
  "attributes": {
    "shop_id": "23910",
    "order_id": "5302",
    "user_id": "9421"
  }
}
json
{
  "timestamp": "2022-06-02T04:17:25.783Z",
  "group": "organizations",
  "severity": "info",
  "message": "Order sent successfully to pizzeria",
  "attributes": {
    "shop_id": "23910",
    "order_id": "5302",
    "user_id": "9421"
  }
}

In AppSignal, we'd be able to filter our logs based on the log message or any of the custom attributes we added to the log, like shop_id, helping us quickly access the contextual information needed should an incident occur.

Further Reading

If you're interested in AppSignal log management, you may be interested in the following:

Start logging with AppSignal

You can start managing your application's logs in as little as five minutes with AppSignal if you are not yet a customer sign up for a free trial. If you get stuck or have a question about AppSignal, our dev-to-dev support is always on hand to help!

You can learn more about AppSignal Logging on our Log management page.

Start your free trial

Don’t let the bad bugs bite. Try AppSignal for free.

AppSignal offers a 30-day free trial, no credit card is required. All features are available in all plans. Start monitoring your application in just a few clicks!

Start free trialView pricing