Process nginx error logs by GoAccess

With GoAccess, we can monitor/analyze almost any logs and view them nicely in a terminal on the fly and it can also output as HTML, json or csv.

Most GoAccess examples we see out there are for web access logs. What if we need to analyze error logs for attack logs like:

  
2016/04/27 02:18:05 [error] 21235#21235: *326 limiting requests, excess: 5.297 by zone "one", client: 91.214.169.44, server: www.webfoobar.com, request: "GET /node/8 HTTP/1.1", host: "www.webfoobar.com", referrer: "https://www.webfoobar.com/archive/201502"
  

In this article, we will use that nginx sample error log to make GoAccess process that pattern.

But first, install GoAccess:

  
yum install goaccess
  

Lets create a custom GoAccess configuration that is suitable for this use case.

  
vi /opt/goaccess_nginx_errorlog.conf
  

Have the following script as its content:

  
######################################
# Time Format Options (required)
######################################
#
# The hour (24-hour clock) [00,23]; leading zeros are permitted but not required.
# The minute [00,59]; leading zeros are permitted but not required.
# The seconds [00,60]; leading zeros are permitted but not required.
# See `man strftime` for more details
#
time-format %H:%M:%S

######################################
# Date Format Options (required)
######################################
#
# The date-format variable followed by a space, specifies
# the log format date containing any combination of regular
# characters and special format specifiers. They all begin with a
# percentage (%) sign. See `man strftime`
#
date-format %Y/%m/%d

######################################
# Log Format Options (required)
######################################
#
# The log-format variable followed by a space or \t for
# tab-delimited, specifies the log format string.
#
# NOTE: If the time/date is a timestamp in seconds or microseconds
# %x must be used instead of %d & %t to represent the date & time.
#
log-format %d %t %^, %^, client: %h, server: %^, request: "%r", host: "%v", referrer: "%R"

######################################
# UI Options
######################################

# Prompt log/date configuration window on program start.
#
config-dialog false

# Choose among color schemes
# 1 : Default grey scheme
# 2 : Green scheme
#
color-scheme 1

# Color highlight active panel.
#
hl-header true

# Set HTML report page title and header.
#
html-report-title Nginx error log statistics

# Turn off colored output. This is the  default output on
# terminals that do not support colors.
# true  : for no color output
# false : use color-scheme
#
no-color false

# Don't write column names in the terminal output. By default, it displays
# column names for each available metric in every panel.
#
no-column-names false

# Disable progress metrics.
#
no-progress false

# Enable mouse support on main dashboard.
#
with-mouse true

# Disable summary metrics on the CSV output..
#
no-csv-summary false

# Custom colors for the terminal output
# Tailor GoAccess to suit your own tastes.
#
# Color Syntax:
# DEFINITION space/tab colorFG#:colorBG# [[attributes,] PANEL]
#
# FG# = foreground color number [-1...255] (-1 = default terminal color)
# BG# = background color number [-1...255] (-1 = default terminal color)
#
# Optionally:
#
# It is possible to apply color attributes, such as:
# bold,underline,normal,reverse,blink.
# Multiple attributes are comma separated
#
# If desired, it is possible to apply custom colors per panel, that is, a
# metric in the REQUESTS panel can be of color A, while the same metric in the
# BROWSERS panel can be of color B.
#
# The following is a 256 color scheme (Monokai palette)
#
#color COLOR_MTRC_HITS     color197:color-1
#color COLOR_MTRC_VISITORS color148:color-1
#color COLOR_MTRC_DATA     color7:color-1
#color COLOR_MTRC_BW       color81:color-1
#color COLOR_MTRC_AVGTS    color247:color-1
#color COLOR_MTRC_CUMTS    color95:color-1
#color COLOR_MTRC_MAXTS    color186:color-1
#color COLOR_MTRC_PROT     color141:color-1
#color COLOR_MTRC_MTHD     color81:color-1
#color COLOR_MTRC_PERC     color186:color-1
#color COLOR_MTRC_PERC     color186:color-1 VISITORS
#color COLOR_MTRC_PERC     color186:color-1 OS
#color COLOR_MTRC_PERC     color186:color-1 BROWSERS
#color COLOR_MTRC_PERC     color186:color-1 VISIT_TIMES
#color COLOR_MTRC_PERC_MAX color208:color-1
#color COLOR_MTRC_PERC_MAX color208:color-1 VISITORS
#color COLOR_MTRC_PERC_MAX color208:color-1 OS
#color COLOR_MTRC_PERC_MAX color208:color-1 BROWSERS
#color COLOR_MTRC_PERC_MAX color208:color-1 VISIT_TIMES
#color COLOR_PANEL_COLS    color242:color-1
#color COLOR_BARS          color186:color-1
#color COLOR_ERROR         color231:color197
#color COLOR_SELECTED      color0:color215
#color COLOR_PANEL_ACTIVE  color7:color240
#color COLOR_PANEL_HEADER  color7:color237
#color COLOR_PANEL_DESC    color242:color-1
#color COLOR_OVERALL_LBLS  color251:color-1
#color COLOR_OVERALL_VALS  color148:color-1
#color COLOR_OVERALL_PATH  color186:color-1
#color COLOR_ACTIVE_LABEL  color7:color237
#color COLOR_BG            color7:color-1
#color COLOR_DEFAULT       color7:color-1
#color COLOR_PROGRESS      color7:color141

######################################
# File Options
######################################

# Specify the path to the input log file. If set, it will take
# priority over -f from the command line.
#
#log-file /var/log/apache2/access.log

# Send all debug messages to the specified file.
#
#debug-file debug.log

# Specify a custom configuration file to use. If set, it will take
# priority over the global configuration file (if any).
#
#config-file 

# Log invalid requests to the specified file.
#
#invalid-requests 

# Do not load the global configuration file.
#
#no-global-config false

######################################
# Parse Options
######################################

# Include static files that contain a query string in the static files
# panel.
# e.g., /fonts/fontawesome-webfont.woff?v=4.0.3
#
all-static-files false

# Consider the following extensions as static files
# The actual '.' is required and extensions are case sensitive
#
static-file .css
static-file .CSS
static-file .dae
static-file .DAE
static-file .eot
static-file .EOT
static-file .gif
static-file .GIF
static-file .ico
static-file .ICO
static-file .jpeg
static-file .JPEG
static-file .jpg
static-file .JPG
static-file .js
static-file .JS
static-file .map
static-file .MAP
static-file .mp3
static-file .MP3
static-file .pdf
static-file .PDF
static-file .png
static-file .PNG
static-file .svg
static-file .SVG
static-file .swf
static-file .SWF
static-file .ttf
static-file .TTF
static-file .txt
static-file .TXT
static-file .woff
static-file .WOFF

# Exclude an IPv4 or IPv6 from being counted.
# Ranges can be included as well using a dash in between
# the IPs (start-end).
#
#exclude-ip 127.0.0.1
#exclude-ip 192.168.0.1-192.168.0.100
#exclude-ip ::1
#exclude-ip 0:0:0:0:0:ffff:808:804-0:0:0:0:0:ffff:808:808

# Enable a list of user-agents by host. For faster parsing, do not
# enable this flag.
#
agent-list false

# Include HTTP request method if found. This will create a
# request key containing the request method + the actual request.
#
http-method true

# Include HTTP request protocol if found. This will create a
# request key containing the request protocol + the actual request.
#
http-protocol false

# Ignore request's query string.
# i.e.,  www.google.com/page.htm?query => www.google.com/page.htm
#
# Note: Removing the query string can greatly decrease memory
# consumption, especially on timestamped requests.
#
no-query-string false

# Disable IP resolver on terminal output.
#
no-term-resolver false

# Write output to stdout given one of the following formats:
# csv  : A comma-separated values (CSV)
# json : JSON (JavaScript Object Notation)
# html : HTML report
#
#output-format json

# Display real OS names. e.g, Windows XP, Snow Leopard.
#
real-os false

#  Enable IP resolver on HTML|JSON output.
#
with-output-resolver false

# Treat non-standard status code 444 as 404.
#
444-as-404 false

# Add 4xx client errors to the unique visitors count.
#
4xx-to-unique-count false

# Decode double-encoded values.
#
double-decode false

# Ignore parsing and displaying one or multiple status code(s)
#
#ignore-status 400
#ignore-status 502

# Ignore crawlers from being counted.
# This will ignore robots listed under browsers.c
# Note that it will count them towards the total
# number of requests, but excluded from any of the panels.
#
ignore-crawlers false

# Ignore parsing and displaying the given panel.
#
#ignore-panel VISITORS
#ignore-panel REQUESTS
ignore-panel REQUESTS_STATIC
ignore-panel NOT_FOUND
#ignore-panel HOSTS
ignore-panel OS
ignore-panel BROWSERS
#ignore-panel VISIT_TIMES
#ignore-panel VIRTUAL_HOSTS
#ignore-panel REFERRERS
ignore-panel REFERRING_SITES
ignore-panel KEYPHRASES
#ignore-panel GEO_LOCATION
ignore-panel STATUS_CODES

# Ignore referers from being counted.
# This supports wild cards. For instance,
# '*' matches 0 or more characters (including spaces)
# '?' matches exactly one character
#
#ignore-referer *.domain.com
#ignore-referer ww?.domain.*

# Sort panel on initial load.
# Sort options are separated by comma.
# Options are in the form: PANEL,METRIC,ORDER
#
# Available metrics:
#  BY_HITS
#  BY_VISITORS
#  BY_DATA
#  BY_BW
#  BY_AVGTS
#  BY_CUMTS
#  BY_MAXTS
#  BY_PROT
#  BY_MTHD
# Available orders:
#  ASC
#  DESC
#
#sort-panel VISITORS,BY_DATA,ASC
sort-panel VISITORS,BY_HITS,DESC
#sort-panel REQUESTS,BY_HITS,ASC
sort-panel REQUESTS,BY_HITS,DESC
#sort-panel REQUESTS_STATIC,BY_HITS,ASC
#sort-panel NOT_FOUND,BY_HITS,ASC
#sort-panel HOSTS,BY_HITS,ASC
sort-panel HOSTS,BY_HITS,DESC
#sort-panel OS,BY_HITS,ASC
#sort-panel BROWSERS,BY_HITS,ASC
#sort-panel VISIT_TIMES,BY_DATA,DESC
sort-panel VISIT_TIMES,BY_HITS,DESC
#sort-panel VIRTUAL_HOSTS,BY_HITS,ASC
sort-panel VIRTUAL_HOSTS,BY_HITS,DESC
#sort-panel REFERRERS,BY_HITS,ASC
sort-panel REFERRERS,BY_HITS,DESC
#sort-panel REFERRING_SITES,BY_HITS,ASC
#sort-panel KEYPHRASES,BY_HITS,ASC
#sort-panel GEO_LOCATION,BY_HITS,ASC
sort-panel GEO_LOCATION,BY_HITS,DESC
#sort-panel STATUS_CODES,BY_HITS,ASC

######################################
# GeoIP Options
# Only if configured with --enable-geoip
######################################

# Standard GeoIP database for less memory usage.
#
#std-geoip false

# Specify path to GeoIP database file. i.e., GeoLiteCity.dat
# .dat file needs to be downloaded from maxmind.com.
#
# For IPv4 City database:
# wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
# gunzip GeoLiteCity.dat.gz
#
# For IPv6 City database:
# wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
# gunzip GeoLiteCityv6.dat.gz
#
# For IPv6 Country database:
# wget -N http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz
# gunzip GeoIPv6.dat.gz
#
# Note: `geoip-city-data` is an alias of `geoip-database`
#
#geoip-database /usr/local/share/GeoIP/GeoLiteCity.dat

######################################
# Tokyo Cabinet Options
# Only if configured with --enable-tcb=btree
######################################

# GoAccess has the ability to process logs incrementally through the on-disk
# B+Tree database.
#
# It works in the following way:
# - A data set must be persisted first with --keep-db-files, then the same data
#   set can be loaded with --load-from-disk.
# - If new data is passed (piped or through a log file), it will append it to
#   the original data set.
# - To preserve the data at all times, --keep-db-files must be used.
# - If --load-from-disk is used without --keep-db-files, database files will be
#   deleted upon closing the program.

# On-disk B+ Tree
# Persist parsed data into disk. This should be set to
# the first dataset prior to use `load-from-disk`.
# Setting it to false will delete all database files
# when exiting the program.
#keep-db-files true

# On-disk B+ Tree
# Load previously stored data from disk.
# Database files need to exist. See `keep-db-files`.
#load-from-disk false

# On-disk B+ Tree
# Path where the on-disk database files are stored.
# The default value is the /tmp directory.
#
#db-path /tmp

# On-disk B+ Tree
# Set the size in bytes of the extra mapped memory.
# The default value is 0.
#
#xmmap 0

# On-disk B+ Tree
# Max number of leaf nodes to be cached.
# Specifies the maximum number of leaf nodes to be cached.
# If it is not more than 0, the default value is specified.
# The default value is 1024.
#
#cache-lcnum 1024

# On-disk B+ Tree
# Specifies the maximum number of non-leaf nodes to be cached.
# If it is not more than 0, the default value is specified.
# The default value is 512.
#
#cache-ncnum 512

# On-disk B+ Tree
# Specifies the number of members in each leaf page.
# If it is not more than 0, the default value is specified.
# The default value is 128.
#
#tune-lmemb 128

# On-disk B+ Tree
# Specifies the number of members in each non-leaf page.
# If it is not more than 0, the default value is specified.
# The default value is 256.
#
#tune-nmemb 256

# On-disk B+ Tree
# Specifies the number of elements of the bucket array.
# If it is not more than 0, the default value is specified.
# The default value is 32749.
# Suggested size of the bucket array is about from 1 to 4
# times of the number of all pages to be stored.
#
#tune-bnum 32749

# On-disk B+ Tree
# Specifies that each page is compressed with ZLIB|BZ2 encoding.
# Disabled by default.
#
#compression zlib
  

The key lines on the script above are the date, time and log format. These lines tailored to parse our sample nginx log above:

  
time-format %H:%M:%S

date-format %Y/%m/%d

log-format %d %t %^, %^, client: %h, server: %^, request: "%r", host: "%v", referrer: "%R"
  

Now, to use this in action execute:

  
zcat -f /var/log/nginx/error_log* | goaccess -p /opt/goaccess_nginx_errorlog.conf
  

To output as HTML:

  
zcat -f /var/log/virtualmin/*nginx_error_log* | goaccess -p /opt/goaccess_nginx_errorlog.conf -a > /var/www/html/nginx_error.html
  

We can use grep to narrow the results to a specific date:

  
zcat -f /var/log/virtualmin/*nginx_error_log* | grep "2016/04/27" | goaccess -p /opt/goaccess_nginx_errorlog.conf -a > /var/www/html/nginx_error.html
  

Comments

I am trying to parse Nginx error log with goaccess, but its giving following error:-

Parsing... [0] [0/s]
GoAccess - version 1.0.2 - Oct 9 2016 06:13:21
Config file: /opt/script/goaccess-errorlog.conf
Fatal error has occurred
Error occured at: src/goaccess.c - main - 1230
Nothing valid to process. Verify your date/time/log format.

I have followed your steps. My query is:--zcat -f /var/log/nginx/error.log*.gz | grep "04/27/2017" | goaccess -p /opt/goaccess-errorlog.conf -a > /tmp/fe1_error.html

please help.

Try if the following will output something:

  
zcat -f /var/log/nginx/error.log*.gz | grep "2017/04/27"
  

If there's output, then try:

  
zcat -f /var/log/nginx/error.log*.gz | grep "2017/04/27" | goaccess -p /opt/goaccess-errorlog.conf -a > /tmp/fe1_error.html
  

If ever there's still no output, try adjusting the date (your cron script might already deleted logs dated you are trying to access).

I gett he following...

sh-4.1# goaccess -f /virtual/domain.com/logs/error.log -o /virtual/domain.com/htdocs/report.html --real-time-html -p /usr/etc/goaccess_nginx_errorlog.conf
/virtual/jedds.com/logs/error.log
Parsed 10 lines producing the following errors:

Token for '%h' specifier is NULL.
Token 'PHP' doesn't match specifier '%d'
Token for '%h' specifier is NULL.
Token 'PHP' doesn't match specifier '%d'
Token for '%h' specifier is NULL.
Token 'PHP' doesn't match specifier '%d'
Token for '%h' specifier is NULL.
Token 'PHP' doesn't match specifier '%d'
Token for '%h' specifier is NULL.
Token 'PHP' doesn't match specifier '%d'

Format Errors - Verify your log/date/time format

Here is a sample of my log file...
2017/11/14 09:55:40 [error] 16070#16070: *4745789 FastCGI sent in stderr: "PHP message: PHP Warning: Unknown: open(/var/lib/php/session/sess_3pf7qkf0u7fqfcj9scerae1b91, O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP message: PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0" while reading upstream, client: 192.88.134.8, server: www.domain.com, request: "GET /product-category/supplements/?add-to-cart=95530&add_to_wishlist=95492 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm/php-fpm.sock:", host: "www.domains.com"

(I fixed this error btw)

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.