Server Logs: What to Keep and for How Long

Logs are the only evidence of what a system did. They are also a growing
storage cost, a disk-space hazard and, because they routinely contain personal
data, a compliance obligation. Retention is therefore a decision rather than a
default, and leaving it to the default usually produces both too much and too
little.

The categories worth separating

Access logs record requests: time, path, status, response
time, client address, user agent. Useful for diagnosing problems, understanding
traffic and investigating abuse. Also the highest volume.

Application logs are what your code emits: errors, warnings,
and whatever you chose to record. The most valuable category for debugging and
the one most likely to contain sensitive data, because developers log request
bodies while investigating something and forget to remove it.

System logs cover the operating system: authentication
attempts, service restarts, kernel messages. The first place to look for a
compromise.

Audit logs from the cloud provider record who did what to
your infrastructure. Different in kind from the rest, and the most important to
keep.

Retention that makes sense

There is no universal correct answer, but these are defensible starting
points:

• Access logs: two to four weeks on disk, longer in cheap
  storage if you analyse them. Most diagnostic questions concern the last few
  days.
• Application logs: two to four weeks for everything, with
  errors kept longer. Error history is useful for spotting recurring issues.
• System logs: around thirty days. Security investigations
  often start well after the event.
• Audit logs: a year or more. These are small, infrequent,
  and the ones you will want when answering a difficult question.

Two things override these. A regulatory requirement specifying a period wins.
And a legal hold means stop deleting immediately, including automated
deletion.

Logs are personal data

This is routinely overlooked. An IP address is personal data under the GDPR.
So is a user identifier, an email address in an error message, or a session
token.

Three consequences follow. Retention has to be justifiable rather than
indefinite. Logs fall within a subject access request, which means you need to
be able to find an individual’s records in them. And they need the same access
control as any other store of personal data. See
data residency and GDPR.

The practical mitigations are to avoid logging sensitive values in the first
place, to truncate or hash client addresses where you only need them in
aggregate, and to set and automate a retention period.

Stop them filling the disk

Unrotated logs are among the most common causes of a small server failing,
and the failure is total: a full disk stops the database writing and often
prevents you logging in to fix it.

Log rotation is standard on every Linux distribution and usually configured
by default for system logs. The gaps are application logs written to custom
paths, which are frequently not covered. Check that every log file your
application writes is included, compress rotated files, and set a maximum age or
size.

Then alert on disk usage anyway, because the one file nobody configured is
the one that fills the volume.

Shipping logs elsewhere

Keeping logs only on the server has a specific weakness: if the machine is
lost or compromised, so are the logs, and a competent intruder’s first act is to
edit them.

Sending logs to object storage or a logging service solves both and makes
searching across several servers possible. Object storage with lifecycle rules
is the cheap version and handles retention automatically. Hosted logging
platforms are more capable and priced by volume, which means controlling what
you send is the main cost lever.

If you ship logs to a hosted service, check which region processes them,
since this is a common way for personal data to leave a jurisdiction
unnoticed.

Make them useful

• Log in a structured format, so fields can be filtered rather than
  pattern-matched.
• Include a request identifier and carry it across services, so one
  transaction can be followed.
• Use severity levels consistently, so errors can be separated from
  noise.
• Log the context needed to understand the event, not the entire object.
• Never log passwords, tokens, keys or full payment details.

A smaller volume of well-structured logs is more useful than a large volume
of unstructured ones, and considerably cheaper to keep.