About logging, auditing and monitoring

Once our application goes to production, the end user (either a person or a script) starts interacting with it. For various reasons, it’s important to be aware of what is going on in our application at all times, especially to detect and do forensics on security issues. This is why it’s crucial to have concrete evidence to allow us to understand what is going on in our app (to keep us alert) at a certain moment with a certain user, endpoint, or resource.

We have different alternatives to put this into practice:

• Logging: the term ‘log’ is used to refer to the register of events that run in a system or an application. Generally speaking, the events are divided into informational, debugging, warning, error, and alert. For example, an event that indicates that a user made a payment goes into the informational category. Instead, an event that notifies us of an unexpected input could be considered a warning or an alert of a possible threat.
• Auditing: a set of records that show the activities that affected a resource or event over time. For instance, the set of records that show an item’s creation, modification, and deletion.
• Monitoring: a diagnostic tool used to know an application status. It can be oriented towards performance, transactions, or security, monitoring acceptable uses of a functionality and warning us in case of a diversion.

Implementation in companies

Services such as Datadog, Sentry, and Kibana are commonly used (the choice varies depending on the company’s infrastructure). Here we can monitor our app, count events, and generate alerts to track any events (CPU and RAM use, number of 400 or 500 errors, event occurrences of a specific event per unit of time).

The maintenance team is responsible for deciding what events to register, which type of register should be, and how it has to be done. The following steps should may be considered to implement them correctly during development.

Implementation

Implementation stages

1. Define the mechanism (logging, auditing, or monitoring)
2. Define the event
3. Define event information

Register methodology

• Logging is recommended to register events that we won’t need to audit in the future for/before security events. In this case, the logs can be helpful in the case of an attack, where we would need an incident response in real-time.
• However, for events we could need to revise in the future, audit trails are suggested.
• Monitoring is used to generate alerts when certain actions occur, such as the tracking of the number of accesses to an entity per unit of time.

It is necessary to highlight that logging and auditing are not exclusive to the security monitoring of the app. They are complementary strategies.

How to define which events to register?

It is necessary to define what to log in the design phase, this will depend on the type of events and the activities of the app. If we take a static landing, as the users are not allowed to do more than just visit, it does not make sense to do a security logging, except for the traffic information on WAF.

As a general rule, we can do logging in the cases of:

• Failing input validations.
• Failing authorization validations.
• Authentication errors.
• Logical security controls that are not complied with.

and keep as part of the auditing register:

• Events performed by users on the resources managed by the application.
• Use of high-risk or critical functionalities within the application.

When implementing metrics using Datadog, we must take into account the metrics will be added, we can’t send metrics with tags of high cardinality. As a consequence, according to the types of events, not all metrics will have enough information to do a forensic analysis in the future.

To define which events to log, it is recommended to start with the regulatory requirements.

What information should a log have?

At the moment of registering events, it is crucial not only to do the registration but also to attach the necessary information to understand the context and reflect on whom, what, how, and when a specific event took place. Let’s say we are interested in registering modifications to user data by employees. If we only register the changes, but not the time or the employee who executed the action, we will lack that information when we do a forensic analysis.

In the case of our web applications, the app itself carries the context of the user who carries out the action, when and why we need to do the logging of the event. This information will have to be added to the logs/audit to have all the necessary context to understand the events or act on them.

As an example, if a user wants to have access to a payment that does not correspond to him/her, we need to know when, which user and which payment was involved to analyse it.

As a whole, the event and contextual data provide the following information:

• What? Event data.
• Who? Employee or authenticated user ID
• When? Event timestamp.
• Result? For instance, data modification.

As a rule, we can contextualize an event by adding the following information:

• Timestamp
• Request-ID
• Authenticated user ID: either an employee or a platform user.
• Affected application component.
• IP
• URL

How to do the logging?

The fact that our applications register events using logs does not mean these logs can be easy to access. This is needed in auditing or forensic analysis. If our logs cannot be accessed, not only they can’t answer the questions we need them to, but they also do not correctly meet its requirements.

An easy way of making our events easy to access is using structured logging. This allows us to find, filter, process, and visualize logs in a simpler way. The log format changes depending on the stack used. For example, in DataDog/ElasticSearch, the optimal log format is:

1[tag:valor] [tag2:valor2] …

You can add the tags you need in each case, for instance, by filling the log context with different tags.

For further information on structure logging, check out the following resources:

• What is structured logging and why developers need it
• Structured logging (Kartar.net)

Monitoring services use

Once we define the events which will be measured, we can send them to services such as Datadog or Kibana to summarize the information and get a global status of the application. These events which will be monitored could be the same as the ones which are being logged, but we are going to see them globally here.

Having the metrics already created in the corresponding service, we need to create alerts to get notifications on diversions in the status of the application. This is vital, otherwise, we would never be aware of the diversions in the global state of the application. Here are two possible cases:

• We know in advance which thresholds we should be concerned about.
• We don’t know the thresholds of benign behavior of our application. This can occur on new apps or functionalities where we are unsure of how a benign user would behave. In these cases, we recommend starting by establishing wide thresholds, studying alerts, and then narrowing them down accordingly.

Caveats

Auditing registers without enough information

We oftentimes generate an auditing register without enough details to understand which was the event and its context. This could be a problem when we do the forensics of an incident. However, it can be avoided by answering the questions explained in the ‘What information should a log have?’ section.

Logging of sensitive data

Sometimes our applications work with sensitive data such as users’ personal information or secrets we use for a specific functionality. We usually apply special security measures for sensitive data, which differ from the measures used in the logging service.

The logging of sensitive data, personal data, secrets, or confidential information must be avoided.

Thanks for reading, and If you have any suggestions, please leave a comment, hope to see you in my next article!