Matteo Mattei

Hello, my name is Matteo Mattei and this is my personal website. I am computer engineer with a long experience in Linux system administration and web software development.

linkedin rss twitter google+ github facebook

Install GoAccess to monitor web server statistics

GoAccess is a nice tool that parses Apache logs and create a report in various format extracting lot of interesting data and statistics. This guide has been tested on CentOS 6.x but it should be very similar also for other distributions.

First of all install some dependencies given we are going to compile the sources:

yum install glib2 glib2-devel glibc make

Download the goaccess source code, copile and install it:

cd /usr/loca/src
wget http://tar.goaccess.io/goaccess-1.0.2.tar.gz
tar xzf goaccess-1.0.2.tar.gz
cd goaccess-1.0.2
./configure
make
make install

Now run goaccess and select the format of the Apache log file from the list it proposes. In case you already know how the Apache output file is generated, you can edit the configuration file /usr/local/etc/goaccess.conf with the appropriate patterns for time-format, date-format and log-format. In my case I have the following:

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b

Create a goaccess folder inside a virtualhost document root (so that it is accessible from the web):

mkdir /var/www/vhosts/myhost.com/public_html/goaccess
chown myuser.myuser /var/www/vhosts/myhost.com/public_html/goaccess

Now edit /etc/crontab and add a cronjob for goaccess that runs every 10 minutes:

*/10 * * * *    myuser    /usr/local/bin/goaccess -f /usr/local/apache/logs/access_log -a -d -o /var/www/vhosts/myhost.com/public_html/goaccess/index.html &> /dev/null 

Generally is a good idea to protect the goaccess folder with a password so that nobody except you can access and see the statistics of the web server.

cat << EOF > /var/www/vhosts/myhost.com/public_html/goaccess/.htaccess
AuthType Basic
AuthName "GoAccess"
AuthUserFile /home/myuser/goaccess_htpasswd
Require valid-user
EOF

htpasswd -c /home/myuser/goaccess_htpasswd myuser

Now every 10 minutes the statistics of your Apache (or Nginx) web server are correctly parsed and served in a nice HTML web interface!


Enforce Apache security and performance

Production Apache web servers need to be well configured for what regards security and performance. Here below a quick tips to make your servers more secure and performant.

First of all you need to verify if you are using prefork module:

apachectl -V | grep -i mpm

If prefork is enabled, you should see a line like this:

Server MPM:     prefork

If it is, I wrote a simple script to calculate the number of MaxClients your server can support:

Basically this number is calculated with this formula:

(TOTAL_RAM - MYSQL_RAM - 50MB) / APACHE_RAM

So, edit /etc/apache2/apache2.conf on Debian/Ubuntu and /etc/httpd/conf/httpd.conf on RedHat/CentOS and set the prefork section like this:

<IfModule prefork.c>
    StartServer 5
    MinSpareServers 5
    MaxSpareServers 10
    MaxClients 300            # value calculated
    MaxRequestPerChild 3000   # 3000 is a good number, avoid to leave it at 0
</IfModule>

Set now some parameters that affects security and performances. Depending on your distribution they can be already set in the following files:

Debian/Ubuntu:

  • /etc/apache2/apache2.conf
  • /etc/apache2/conf.d/security

RedHat/CentOS:

  • /etc/httpd/conf/httpd.conf
  • /etc/httpd/conf/extra/httpd-default.conf
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
Timeout 45
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

Now test apache configuration and if all goes well, restart the web server:

apachectl configtest

/etc/init.d/httpd restart    # RedHat/CentOS
/etc/init.d/apache2 restart  # Debian/Ubuntu

Enable SSH authentication using RSA key-pair without password

If you have to manage multiple servers, if you want to enforce the security of your servers, if you want to run remote script using SSH in crontab, or simply if you don’t want to remember the SSH password everytime, this is the guide for you!

First of all you need to generate a RSA keypair in your PC/Mac:

ssh-keygen -t rsa -b 2048 -C "your-email@example.com" -f ~/.ssh/id_rsa
  • -t is the type of algorithm to use (RSA)
  • -b is the length of the key to generate (2048 is sufficient)
  • -C is the comment/identification of the key (you can use your email address)
  • -f is the path of the private key to generate (the public will be stored in the same folder with .pub suffix)

When you are asked for a passphrase just press Enter to not input any passphrase. At the end a couple of keys will be stored in ~/.ssh folder with the correct permissions and they will be called respectively id_rsa (the private key) and id_rsa.pub (the public key).

In case you are copying the keys from somehow to your ~/.ssh folder make sure the permissions are correct:

-rw-------  1 matteo matteo  1679 Aug 15  2015 id_rsa
-rw-r--r--  1 matteo matteo   398 Aug 15  2015 id_rsa.pub

Now from your PC/Mac copy the private key to the remote server:

ssh-copy-id remoteuser@remoteserver-ip

This time you will need to provide the password because the remote server is still not aware of your key. Even if the best approach is the this, the same operation could also be done manually using scp:

scp ~/.ssh/id_rsa.pub remoteuser@remoteserver-ip:/tmp/
[ENTER IN THE REMOTE SERVER]
ssh remoteuser@remoteserver-ip
mkdir ~/.ssh
cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
rm /tmp/id_rsa.pub
exit

Now try to connect to the remote server via SSH:

ssh remoteuser@remoteserver-ip

If all goes well, the password should not be asked and you can access to the server directly. But it is not finished… now we want to block the password authentication for all users and allow root login, so login to the server as root and change /etc/ssh/sshd_config in this way:

PermitRootLogin without-password
RSAAuthentication yes
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

Restart ssh daemon (/etc/init.d/ssh restart) and from another shell try to connect again. You should be able to access to the server without enterning any password. I suggest to use another shell because if something went wrong you can always recover the issue using the first shell.

Remember to copy the public key in the authorized_keys file of every remote user that can accept remote connections via ssh.


Enable MySQL slow query and query not using indexs logs

Sometimes, expecially in production, is important to monitor how your database is performing and in general, when you see the websites are loading slow and/or there is high picks of CPU/RAM on MySQL, a good idea is to enable slow queries and queries not using indexes log. To do it, edit /etc/mysql/my.cnf on Debian (and derivates) or /etc/my.cnf on RedHat (and derivates) and add the following lines:

slow_query_log_file = /var/log/mysql/mysql-slow.log
slow_query_log = 1
long_query_time = 2
log_queries_not_using_indexes

Before restarting MySQL server, create the log file and set the correct permission on it:

touch /var/log/mysql/mysql-slow.log
chown mysql.mysql /var/log/mysql/mysql-slow.log

Now you can restart MySQL server and check that the new log file (/var/log/mysql/mysql-slow.log) is correctly populated:

/etc/init.d/mysql restart

NOTE: I suggest to keep the slow query log enabled only on debugging because it consumes lot of resources and, depending on your application code, the log file might become huge in just few days.


HTTP/HTTPS GET and POST requests in Python3 including file upload with builtin modules

The following script performs GET and POST requests to [httpbin.org] using only builtin python 3 modules. There is also a class to support file encoding for upload.


Cross compile wget statically for ARM

The following script can be used to statically cross compile wget for ARM.

Requirements:

  • You need openssl and zlib already present in the current $ROOTPATH directory with related libraries and included respectively inside libs and include folders.
  • You need a glibc compiled with --enable-static-nss flag so that getaddrinfo and gethostbyname cannot complain at link time.

The resulting binary will be placed into the build folder.