Matteo Mattei

Hello, my name is Matteo Mattei and this is my personal website. I am computer engineer with a long experience in Linux system administration and web software development. My preferred computer languages and technologies are PHP, Python, Bash, HTML, HTML5, CSS3, jQuery, MySQL, Node.js, MongoDB, etc...

© 2006-2015 Matteo Mattei - all rights reserved. About my blog code

linkedin rss twitter google+ github facebook

How to create self contained executables in Python

This is a very quick guide on how to create self contained Python executables (for all platforms). First of all install PyInstaller (I am using pip3 because I work with Python 3.x):

sudo pip3 install pyinstaller

Now install upx for a better compression:

sudo apt-get install upx

Now you are ready to create your self-contained executable:

pyinstaller \
    --onefile \
    --noconfirm \
    --noconsole \
    --clean \
    --log-level=WARN \
    --key=MySuperSecretPassword \
    --strip \
    myscript.py

Resulting executable will be placed inside dist folder and it will be called myscript. This is what each parameter does:

  • --onefile allows to create a single self contained binary.
  • --noconfirm replaces output directory without asking for confirmation.
  • --noconsole should be used in GUI application with no console. --console should be used otherwise.
  • --clean cleans PyInstaller cache and remove temporary files.
  • --log-level=WARN shows only warnings and errors during build.
  • --key=yourkey uses the given key to encrypt the Python bytecode (yes it's secure!).
  • --strip removes debug information to executable and shared libraries.

For windows, if you want to add also the icon to the resulting exe file you can add this additional parameter:

--icon-file=myapplication.ico

The application have only to be recompiled on every platform you want to release your application for. I know, the resulting binary will be a little heavy (~24MB for a PySide GUI application on Linux) but we have to consider that it contains the interpreter itself and all the needed libraries!!!


Configure production web sever with Debian 8 Jessie

This is the list of the services we are going to install and configure:

  • Apache
  • PHP-FPM
  • MySQL
  • Postfix
  • PhpMyAdmin
  • Shorewall
  • Nodejs
  • Couchdb

First of all configure ssh key access to your server. Make sure to have a rsa key-pair in ~/.ssh/ folder in your pc. Assuming xxx.xxx.xxx.xxx is the public ip address of your server, execute the following command to copy the public key:

ssh-copy-id root@xxx.xxx.xxx.xxx

Setup root shell configuration and update the system:

cp /etc/skel/.bashrc /root/.bashrc
apt-get update
apt-get upgrade
apt-get dist-upgrade

Configure a fully qualified domain name (FQDN) in /etc/hosts:

127.0.0.1       localhost.localdomain localhost
xxx.xxx.xxx.xxx srv1.mycompany.com srv1

where xxx.xxx.xxx.xxx is the public IP address assigned by your provider.

Install MySQL server and client:

apt-get install mysql-server mysql-client

New password for the MySQL "root" user: xxx
Repeat password for the MySQL "root" user: xxx

[...]

[Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.

Correct the above MySQL configuration warning:

sed -i "{s/^key_buffer/key_buffer_size/g}" /etc/mysql/my.cnf

Enable MySQL slow query logging:

sed -i "{s/^#slow_query_log_file /slow_query_log_file /g}" /etc/mysql/my.cnf
sed -i "{s/^#slow_query_log /slow_query_log /g}" /etc/mysql/my.cnf
sed -i "{s/^#long_query_time /long_query_time /g}" /etc/mysql/my.cnf
sed -i "{s/^#log_queries_not_using_indexes/log_queries_not_using_indexes/g}" /etc/mysql/my.cnf

Restart MySQL server:

systemctl restart mysql

Install Apache (mpm-worker is needed for using PHP-FPM):

apt-get install apache2-mpm-worker

Install PHP-FPM and related PHP extensions:

apt-get install libapache2-mod-fastcgi php5-fpm php5 php-pear php5-readline php5-gd php5-mysql php5-mcrypt mcrypt php5-imagick imagemagick php5-curl

Configure Apache server properly:

sed -i "{s#^Timeout 300#Timeout 45#g}" /etc/apache2/apache2.conf
sed -i "{s#^KeepAliveTimeout 5#KeepAliveTimeout 15#g}" /etc/apache2/apache2.conf
sed -i "{s#^ServerTokens OS#ServerTokens Minimal#g}" /etc/apache2/conf-enabled/security.conf
sed -i "{s#^ServerSignature On#ServerSignature Off#g}" /etc/apache2/conf-enabled/security.conf

Enable needed Apache modules and restart Apache server:

a2enmod actions fastcgi alias proxy prxy_http ssl
service apache2 restart

Configure correct umask for PHP-FPM and restart the daemon:

sed -i "{s#^\[Service\]#[Service]\nUMask=0022#g}" /etc/systemd/system/multi-user.target.wants/php5-fpm.service

systemctl daemon-reload
systemctl restart php5-fpm.service

Install and configure PhpMyAdmin:

apt-get install phpmyadmin

Web server to reconfigure automatically: [apache2]
Configure database for phpmyadmin with dbconfig-common? [Yes]
Password of the database's administrative user: [xxx]
MySQL application password for phpmyadmin: [xxx]
Password confirmation: [xxx]

Make MySQL more secure:

mysql_secure_installation

Enter current password for root (enter for none): [xxx]
Change the root password? [Y/n] [n]
Remove anonymous users? [Y/n] [Y]
Disallow root login remotely? [Y/n] [Y]
Remove test database and access to it? [Y/n] [Y]
Reload privilege tables now? [Y/n] [Y]

Create a group for SFTP and configure SFTP in a secure manner with chroot directory:

addgroup sftponly

cat <<EOF >> /etc/ssh/sshd_config
Match Group sftponly
   ChrootDirectory %h
   ForceCommand internal-sftp -u 0022
   AllowTcpForwarding no
   AllowAgentForwarding no
   PermitTunnel no
   X11Forwarding no
EOF

service ssh restart

Install and configure firewall:

apt-get install shorewall

cd /usr/share/doc/shorewall/examples/one-interface
cp interfaces /etc/shorewall/interfaces
cp policy /etc/shorewall/policy
cp rules /etc/shorewall/rules
cp zones /etc/shorewall/zones

echo "HTTP/ACCEPT net \$FW" >> /etc/shorewall/rules
echo "HTTPS/ACCEPT net \$FW" >> /etc/shorewall/rules
echo "SSH/ACCEPT  net \$FW" >> /etc/shorewall/rules

sed -i "{s#^startup=0#startup=1#g}" /etc/default/shorewall

systemctl restart shorewall

Install and configure Postfix:

apt-get install postfix heirloom-mailx

General type of mail configuration: [Internet Site]
System mail name: [your FQDN as reported by hostname -f]

Install CouchDb dependencies:

apt-get install libmozjs185-1.0 libmozjs185-dev build-essential curl erlang-nox erlang-dev libnspr4 libnspr4-0d libnspr4-dev libcurl4-openssl-dev curl libicu-dev

Now create CouchDb account:

useradd -d /var/lib/couchdb couchdb
mkdir -p /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb /var/lib/couchdb
chown -R couchdb:couchdb /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb
chmod -R g+rw /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb

Download and install CouchDb:

cd /root
wget http://apache.panu.it/couchdb/source/1.6.1/apache-couchdb-1.6.1.tar.gz
tar xzf apache-couchdb-1.6.1.tar.gz
cd apache-couchdb-1.6.1
./configure --prefix=/usr/local --with-js-lib=/usr/lib --with-js-include=/usr/include/js --enable-init
make && make install

Configure startup script for CouchDB and run it:

chown couchdb:couchdb /usr/local/etc/couchdb/local.ini
ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
/etc/init.d/couchdb start
update-rc.d couchdb defaults

Configure Apache Proxy to forward all requests on a particular domain to localhost:5984 (CouchDb) over SSL. To do it, create a new virtual host for your db URL creating /etc/apache2/site-available/db.mysite.com.conf with the following content:

<VirtualHost *:443>
    ServerAdmin info@matteomattei.com
    ServerName db.mysite.com

    ErrorLog /var/www/vhosts/db.mysite.com/logs/error.log
    CustomLog /var/www/vhosts/db.mysite.com/logs/access.log combined

    SSLEngine On
    SSLCertificateFile      /etc/ssl/certs/ssl-cert-snakeoil.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

    <FilesMatch "\.(cgi|shtml|phtml|php)$">
        SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory /usr/lib/cgi-bin>
        SSLOptions +StdEnvVars
    </Directory>
    BrowserMatch "MSIE [2-6]" \
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0
    BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

    AllowEncodedSlashes On
    ProxyRequests Off
    ProxyPreserveHost On
    KeepAlive On
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyPass / http://localhost:5984/ nocanon
    ProxyPassReverse / http://localhost:5984/
</VirtualHost>

Create the folder /var/www/vhosts/db.mysite.com/logs for storing logs and restart Apache server:

mkdir -p /var/www/vhosts/db.mysite.com/logs
service apache2 restart

Now your couchdb futon interface should be available at https://db.mysite.com/_utils. Remember however to setup authentication in /usr/local/etc/couchdb/local.ini in order to limit the db access to only authenticated users.

Install now the latest version of nodejs:

apt-get install -y curl
curl -sL https://deb.nodesource.com/setup_5.x | bash -
apt-get install -y nodejs

Install pm2 node process manager globally:

npm install pm2 -g

HTML5 Ajax file upload

Very often I have to write HTML forms for uploading files and since I would like to have something responsive without a full page reload I decided to write a little script as example for the next time I will need something similar. The following example should be compatible with every browser but the actual dynamic part is only available on new browsers that support FormData object from XMLHttpRequest2. Old browsers should be able to upload the files using the old (standard) way with just few changes in the server side part.

The following example creates a form for uploading PDF files (checks at line 34) and show the list of uploaded files inside the

element with class result.

The main components of the script are the following:

  • FormData object from XMLHttpRequest2.
  • multiple attribute of file input HTML field.
  • FileReader object from the new File API.

Remember to apply all needed checks in the PHP server side script.

Please note the TransferCompleteCallback() at line 10 that can be used to get the content of the files once they are transferred. This might be useful for example to directly render uploaded images to the screen without an additional request to the server to get them back.


Mount NFS share using Virtualbox under NAT

I am using Ubuntu 14.04 (but it is valid also with other Linux distributions) that runs under Virtualbox. My network interface is set as NAT and I am running nfs version 4 (NFSv4).

Every time I try to mount a remote share I always obtain the following error:

matteo@vm:~$ sudo mount -t nfs xxx.xxx.xxx.xxx:/opt/share /mnt/remote
mount.nfs: access denied by server while mounting xxx.xxx.xxx.xxx:/opt/share

And in the kernel log (dmesg) I see this:

[ 1351.443078] RPC: server xxx.xxx.xxx.xxx requires stronger authentication.

The issue here is that the NFS client is trying to use UDP protocol to access the remote host. You can verify it by capturing the network traffic with wireshark (or tcpdump). In this case is sufficient to force the usage of the TCP protocol when you mount the share:

matteo@vm:~$ sudo mount -t nfs xxx.xxx.xxx.xxx:/opt/share /mnt/remote -o proto=tcp
matteo@vm:~$

Another option is to change your network adapter in Virtualbox from NAT to BRIDGE.

That's all for the moment.


Use Python to print labels with Dymo LabelWriter 450 Turbo

Dymo is a leading company for label printing. Altought several things can be done using their DYMO Label v.8 Software, I found that there was no way to set a specific date in a label layout so that every time I need to print a label I have the current date plus 30 days. I read the manual and I found that to do it we have to use a variable field but this is something that has to be handled using an external program.

So I wrote a little python tool that runs on Windows 7 and that does exactly what I need. Make sure to install pywin32 since we are going to use COM apis.

This is my example label (my.label) and the related Python code:

As you can see I have two fields in the XML with IsVariable set to True and I have assigned to them the name TEXT1 and TEXT2. Now, all the logic happens between line 30 and line 41 of the Python script.

  • I initialize the Dispatch for the printer, configure the label I want to use (mylabel) and then I select the printer [lines 30-34].
  • Then I apply my rules to the variables TEXT1 and TEXT2 [lines 36-37].
  • And finally I launch the print job [lines 39-41].

That's all! With only few lines of Python code we are able to print a label using our Dymo LabelWriter 450 Turbo printer.

Note: the full code is available on GitHub including the scripts to build a self-contained executable.


Backup your server on mega.co.nz using megatools

Mega is a wonderful and secure online cloud service that offers 50GB of free storage. So I thought that it would be great using it as additional backup service for my Linux servers. Fortunately there is a good application library and tools to access Mega storage from Linux command line. This tools are called megatools and it is released under GPLv2 license.

In this post I will show you how to compile, install and configure a full backup system for your server using Mega and MegaTools.

First of all register an account with Mega at http://mega.co.nz then follow these instructions to compile and install megatools.

#!/bin/bash

VERSION="1.9.95"

apt-get install -y build-essentials pkg-config libglib2.0-dev libssl-dev libcurl4-openssl-dev libfuse-dev glib-networking

wget http://megatools.megous.com/builds/megatools-${VERSION}.tar.gz
tar xzf megatools-${VERSION}.tar.gz
cd megatools-${VERSION}
./configure && make && make install && ldconfig

Note: the above instructions are valid for Debian 7 Wheezy and Debian 8 Jessie. In case you have a different Linux distribution please install the required dependencies.

Now that you have MegaTools installed in /usr/local/bin create a configuration file with your credentials in /root/.megarc:

[Login]
Username = Your_Mega_Username
Password = Your_Mega_Password

Since the password is in clear, it is important to protect the file:

chmod 640 /root/.megarc

Test now your mega installation and login credentials:

root@debian:~# megals 
/Contacts
/Inbox
/Root
/Trash

If all goes well you are ready to prepare your backup script. Create a new file called megabackup.sh and place it in /root:

Make it executable and accessible only to root:

chmod 750 /root/megabackup.sh

You only need to set a cron-job now to execute the backup every day:

04 04 * * * root /root/megabackup.sh

Update 2015-02-14: MegaFuse

I recently found a better tool to accomplish my needings, it is called MegaFuse and it works like a Linux mountpoint. First of all download all needed dependencies:

apt-get install libcrypto++-dev libcurl4-openssl-dev libdb5.1++-dev libfreeimage-dev libreadline-dev libfuse-dev make

Then download MegaFuse and compile it:

git clone https://github.com/matteoserva/MegaFuse
cd MegaFuse
make

Now create a configuration file /root/.megafuse.conf with your Mega credentials:

USERNAME = your_mega_email
PASSWORD = your_mega_password
MOUNTPOINT = /mnt

The above configuration assume you will use /mnt as your mountpoint (change it if you want).

Protect it in the same way:

chmod 640 /root/.megafuse.conf

And this is the revisited script to do the backup:

You can now configure the cronjob:

04 04 * * * root /root/megafusebackup.sh