Skip to content

Conversation

@TimidRobot
Copy link
Member

Fixes #31

Description

This add two custom NGINX access logs:

  1. JSON formatted long term archive access log with privacy (access_archive.log)
    • rotated daily for 5 years
    • truncates last part of IP to preserve user privacy (remote_addr_anon)
  2. JSON formatted short term debug access log (access_debug.log)
    • rotated daily for 7 days
    • includes full client IP (remote_addr)
    • also includes and Cloudflare IP (realip_remote_addr)

This PR also includes support for setting the real client IP (instead of the Cloudflare IP) and a script to allow us to update the Cloudflare IPs.

Please also evaluate the decisions made. For example:

  • Should archive log have more or less info?
  • Should archive logs be kept for a shorter or longer time period?
  • Should debug log have more or less info?
  • Should debug logs be kept for a shorter or longer time period?
  • Should user agent be kept in full?
  • JSON?

Other information

access_archive.log

Entry example:

{"time_iso8601":"2020-02-27T21:16:49+00:00", "remote_addr_anon":"104.35.248.0", "request_method":"GET", "request_uri":"/licenses/by/4.0/?1582838209", "status":"200", "http_referer":"", "http_user_agent":"HTTPie/0.9.4"}

Formatted/linted:

{
  "time_iso8601": "2020-02-27T21:16:49+00:00",
  "remote_addr_anon": "104.35.248.0",
  "request_method": "GET",
  "request_uri": "/licenses/by/4.0/?1582838209",
  "status": "200",
  "http_referer": "",
  "http_user_agent": "HTTPie/0.9.4"
}

access_debug.log

{"time_iso8601":"2020-02-27T21:16:49+00:00", "msec":"1582838209.620", "request_time":"0.011", "request_method":"GET", "request_uri":"/licenses/by/4.0/?1582838209", "status":"200", "body_bytes_sent":"10944", "http_referer":"", "proxy_host":"10.22.11.12", "proxy_port":"80", "upstream_http_server":"Apache", "upstream_connect_time":"0.000", "upstream_header_time":"0.008", "upstream_response_time":"0.008", "realip_remote_addr":"172.69.33.45", "remote_addr":"[REDACTED]", "http_user_agent":"HTTPie/0.9.4"}

Formatted/linted:

{
  "time_iso8601": "2020-02-27T21:16:49+00:00",
  "msec": "1582838209.620",
  "request_time": "0.011",
  "request_method": "GET",
  "request_uri": "/licenses/by/4.0/?1582838209",
  "status": "200",
  "body_bytes_sent": "10944",
  "http_referer": "",
  "proxy_host": "10.22.11.12",
  "proxy_port": "80",
  "upstream_http_server": "Apache",
  "upstream_connect_time": "0.000",
  "upstream_header_time": "0.008",
  "upstream_response_time": "0.008",
  "realip_remote_addr": "172.69.33.45",
  "remote_addr": "[REDACTED]",
  "http_user_agent": "HTTPie/0.9.4"
}

Checklist

  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the master branch of the repository.
  • My commit messages follow best practices.
  • I added tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@TimidRobot TimidRobot requested a review from a team as a code owner February 27, 2020 22:14
@TimidRobot TimidRobot self-assigned this Feb 27, 2020
Copy link

@mathemancer mathemancer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the exception of the missing space comment, these are all minor quibbles (so is that one, arguably). I leave it to @TimidRobot to decide which, if any, to resolve with further commits.

@TimidRobot TimidRobot merged commit dba22de into master Mar 2, 2020
@TimidRobot TimidRobot deleted the custom-nginx-logs branch March 2, 2020 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update dispatch host to support long term logging

4 participants