13 Feb 2015 
Matteo Mattei 
linux bash backup megatools 
Mega is a wonderful and secure online cloud service that offers 50GB of free storage. So I thought that it would be great using it as additional backup service for my Linux servers. Fortunately there is a good application library and tools to access Mega storage from Linux command line. This tools are called megatools and it is released under GPLv2 license.
In this post I will show you how to compile, install and configure a full backup system for your server using Mega and MegaTools.
First of all register an account with Mega at http://mega.co.nz then follow these instructions to compile and install megatools.
#!/bin/bash
VERSION="1.9.95"
apt-get install -y build-essentials pkg-config libglib2.0-dev libssl-dev libcurl4-openssl-dev libfuse-dev glib-networking
wget http://megatools.megous.com/builds/megatools-${VERSION}.tar.gz
tar xzf megatools-${VERSION}.tar.gz
cd megatools-${VERSION}
./configure && make && make install && ldconfig
Note: the above instructions are valid for Debian 7 Wheezy and Debian 8 Jessie. In case you have a different Linux distribution please install the required dependencies.
Now that you have MegaTools installed in /usr/local/bin create a configuration file with your credentials in /root/.megarc:
[Login]
Username = Your_Mega_Username
Password = Your_Mega_Password
Since the password is in clear, it is important to protect the file:
Test now your mega installation and login credentials:
root@debian:~# megals
/Contacts
/Inbox
/Root
/Trash
If all goes well you are ready to prepare your backup script. Create a new file called megabackup.sh and place it in /root:
Make it executable and accessible only to root:
chmod 750 /root/megabackup.sh
You only need to set a cron-job now to execute the backup every day:
04 04 * * * root /root/megabackup.sh
Update 2015-02-14: MegaFuse
I recently found a better tool to accomplish my needings, it is called MegaFuse and it works like a Linux mountpoint. First of all download all needed dependencies:
apt-get install libcrypto++-dev libcurl4-openssl-dev libdb5.1++-dev libfreeimage-dev libreadline-dev libfuse-dev make
Then download MegaFuse and compile it:
git clone https://github.com/matteoserva/MegaFuse
cd MegaFuse
make
Now create a configuration file /root/.megafuse.conf with your Mega credentials:
USERNAME = your_mega_email
PASSWORD = your_mega_password
MOUNTPOINT = /mnt
The above configuration assume you will use /mnt as your mountpoint (change it if you want).
Protect it in the same way:
chmod 640 /root/.megafuse.conf
And this is the revisited script to do the backup:
You can now configure the cronjob:
04 04 * * * root /root/megafusebackup.sh
12 Feb 2015 Matteo Mattei linux bash installer In this post I will show you how to develop a self contained Linux command line installer in Bash that will decompress an archive and perform some tasks.
Installer content
Our installer that is basically a self-extracting archive with some logic around, consists in three parts:
- A bash script that performs the extraction of the archive and applies some logic.
- A marker to separate the bash script and the archive.
- An archive containing the actual data to install.
Start now!
Create a new bash script called installer.sh with the following content:
#!/bin/bash
echo ""
echo "My Command Line Installer"
echo ""
# Create destination folder
DESTINATION="/opt/my_application"
mkdir -p ${DESTINATION}
# Find __ARCHIVE__ maker, read archive content and decompress it
ARCHIVE=$(awk '/^__ARCHIVE__/ {print NR + 1; exit 0; }' "${0}")
tail -n+${ARCHIVE} "${0}" | tar xpJv -C ${DESTINATION}
# Put your logic here (if you need)
echo ""
echo "Installation complete."
echo ""
# Exit from the script with success (0)
exit 0
__ARCHIVE__
This script is self-explain but I will try to describe the steps:
- Create a destination folder ${DESTINATION}.
- Find __ARCHIVE__ marker and put the tarball content into ${ARCHIVE} variable.
- Decompress the tarball into the destination folder.
- Eventually apply your installation logic (copy some files, change some others, etc…).
- Exit from the script (this step is mandatory otherwise bash will try to interpret the tarball and will exit with error).
- Add __ARCHIVE__ marker at the bottom of the script. This marker will be used to separate the actual bash script with the tarball content.
Now generate a compressed tarball of your application (I used .tar.xz in the above example):
tar cJf myarchive.tar.xz /folder/to/archive
OK, now append it to the installer bash script and make it executable:
cat myarchive.tar.xz >> installer.sh
chmod +x installer.sh
That’s all! You can now distribute your installer.
Execute your installer
The users will execute your installer simply running:
09 Jan 2015 Matteo Mattei linux server couchdb debian erlang Setup repository and install all dependencies
echo "deb http://packages.erlang-solutions.com/debian wheezy contrib" >> /etc/apt/sources.list
wget -qO - http://packages.erlang-solutions.com/debian/erlang_solutions.asc | apt-key add -
apt-get update
apt-get install -y build-essential curl erlang-nox erlang-dev libmozjs185-1.0 libmozjs185-dev libcurl4-openssl-dev libicu-dev
Create CouchDB account
useradd -d /var/lib/couchdb couchdb
mkdir -p /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb /var/lib/couchdb
chown -R couchdb:couchdb /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb
chmod -R g+rw /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb
Install CouchDB
wget http://apache.panu.it/couchdb/source/1.6.1/apache-couchdb-1.6.1.tar.gz
tar xzf apache-couchdb-1.6.1.tar.gz
cd apache-couchdb-1.6.1
./configure --prefix=/usr/local --with-js-lib=/usr/lib --with-js-include=/usr/include/js --enable-init
make && make install
Create symlinks and start the database
chown couchdb:couchdb /usr/local/etc/couchdb/local.ini
ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
/etc/init.d/couchdb start
update-rc.d couchdb defaults
Verify that all is working fine
curl http://127.0.0.1:5984/
The output should be like this:
{"couchdb":"Welcome","uuid":"5da242ff50cecec904d6caf36be34194","version":"1.6.1","vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}}
Finalize setup
In order to connect from remote edit /usr/local/etc/couchdb/local.ini and change
to:
Restart the database:
And from a web browser visit the CouchDB Futon:
08 Dec 2014 Matteo Mattei linux server mysql varnish debian php iptables postfix Setup bash and update the system
cp /etc/skel/.bashrc /root/.bashrc
apt-get update
apt-get dist-upgrade
Make sure to have the following two lines (with the same format) at the top of your /etc/hosts file
127.0.0.1 localhost.localdomain localhost
xxx.xxx.xxx.xxx web1.myserver.com web1
Note: xxx.xxx.xxx.xxx is the public IP address assigned to your server.
Install all needed packages
apt-get install php5 mysql-server mysql-client apache2 iptables phpmyadmin varnish shorewall vsftpd php5-cli php5-curl php5-dev php5-gd php5-idn php5-imagick php5-imap php5-memcache php5-ming php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xcache php5-xmlrpc php-apc php-pear php-xml-rpc postfix apg ca-certificates heirloom-mailx
MySQL/PhpMyAdmin:
- mysql root password: xxx
- repeat mysql root password: xxx
- web server to reconfigure automatically: apache2
- configure database for phpmyadmin with dbconfig-common? Yes
- Password of the database’s administrative user: xxx
- Password for phpmyadmin: xxx
- Password confirmation: xxx
Postfix:
- Select Internet Site
- System mail name: (insert here the FQDN, for example web1.myserver.com)
Setup FTP
Stop VSFTP server:
Create backup configuration:
mv /etc/vsftpd.conf /etc/vsftpd.conf.backup
Add new configuration:
listen=YES
listen_port=21
anonymous_enable=NO
local_enable=YES
guest_enable=YES
guest_username=nobody
user_sub_token=$USER
local_root=/var/www/vhosts/$USER
virtual_use_local_privs=YES
user_config_dir=/etc/vsftpd/users
pam_service_name=vsftpd_local_and_virtual
chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list
ftpd_banner=Welcome to my ftp server
write_enable=YES
download_enable=YES
dirlist_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
xferlog_file=/var/log/xferlog
connect_from_port_20=YES
connect_timeout=60
data_connection_timeout=300
idle_session_timeout=300
local_max_rate=0
max_clients=0
max_per_ip=3
Create an empty chroot_list file:
touch /etc/vsftpd/chroot_list
Start VSFTP server:
Setup Apache
Stop Apache web server:
Backup Apache configuration:
cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.backup
Edit the following lines in /etc/apache2/apache2.conf
- From Timeout 300 to Timeout 45
- From KeepAliveTimeout 5 to KeepAliveTimeout 15
- Change the mpm_prefork_module section like the following:
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 10000
</IfModule>
Edit /etc/apache2/ports.conf and change the port 80 with 8080 since we are going to use Varnish:
NameVirtualHost *:8080
Listen 8080
Change the port (from 80 to 8080) also in the default virtual host /etc/apache2/sites-enabled/000-default Now restart Apache:
/etc/init.d/apache2 restart
Setup Varnish
Stop Varnish daemon:
Open /etc/varnish/default.vcl and make sure the backend section is like this:
backend default {
.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
Now edit /etc/default/varnish and set the DAEMON_OPTS variable like this:
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-p thread_pools=4 \
-p thread_pool_max=1500 \
-p listen_depth=2048 \
-p lru_interval=1800 \
-h classic,169313 \
-p max_restarts=6 \
-p connect_timeout=600 \
-p send_timeout=2000 \
-s malloc,2G"
Restart Varnish:
/etc/init.d/varnish restart
Setup MySQL
MySQL is already configured. You only need to log slow queries (that is often usefult during slow load page investigation). Todo it, open /etc/mysql/my.cnf and decomment the following two lines:
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 2
Copy the default configuration for one interface:
cp /usr/share/doc/shorewall/examples/one-interface/interfaces /etc/shorewall/interfaces
cp /usr/share/doc/shorewall/examples/one-interface/policy /etc/shorewall/policy
cp /usr/share/doc/shorewall/examples/one-interface/rules /etc/shorewall/rules
cp /usr/share/doc/shorewall/examples/one-interface/zones /etc/shorewall/zones
Now open /etc/shorewall/policy file and change the line:
removing info directive given it fills the system logs:
Now open /etc/shorewall/rules and add the following rules at the bottom of the file:
HTTP/ACCEPT net $FW
SSH/ACCEPT net $FW
FTP/ACCEPT net $FW
# real apache since varnish listens on port 80
#ACCEPT net $FW tcp 8080
NOTE: in case you want to allow ICMP (Ping) traffic from a specific remote hosts you need to add a rule similar to the following where xxx.xxx.xxx.xxx is the remote IP address, before the Ping(DROP) rule:
Ping(ACCEPT) net:xxx.xxx.xxx.xxx $FW
Now edit /etc/default/shorewall and change startup=0 to startup=1 You are now ready to start the firewall:
/etc/init.d/shorewall start
Setup Postfix
Stop postfix server:
Edit /etc/mailname and set your server domain name, for example:
Then, in order to monitor mail traffic coming from PHP you need to edit /etc/php5/apache2/php.ini. Go to [mail function] section and set the following two options:
sendmail_path = /usr/local/bin/sendmail-wrapper
auto_prepend_file = /usr/local/bin/env.php
Now create the two files above:
sendmail-wrapper:
#!/bin/sh
logger -p mail.info sendmail-wrapper.sh: site=${HTTP_HOST}, client=${REMOTE_ADDR}, script=${SCRIPT_NAME}, pwd=${PWD}, uid=${UID}, user=$(whoami)
/usr/sbin/sendmail -t -i $*
env.php:
<?php
putenv("HTTP_HOST=".@$_SERVER["HTTP_HOST"]);
putenv("SCRIPT_NAME=".@$_SERVER["SCRIPT_NAME"]);
putenv("SCRIPT_FILENAME=".@$_SERVER["SCRIPT_FILENAME"]);
putenv("DOCUMENT_ROOT=".@$_SERVER["DOCUMENT_ROOT"]);
putenv("REMOTE_ADDR=".@$_SERVER["REMOTE_ADDR"]);
?>
Now make they both have executable flag:
chmod +x /usr/local/bin/sendmail-wrapper
chmod +x /usr/local/bin/env.php
Add also /usr/local/bin/ to the open_basedir php list in /etc/apache2/conf.d/phpmyadmin.conf
php_admin_value open_basedir /usr/share/phpmyadmin/:/etc/phpmyadmin/:/var/lib/phpmyadmin/:/usr/local/bin/
Restart Postfix:
/etc/init.d/postfix start
Prepare environment
Create all needed directories and files
mkdir /root/cron_scripts
mkdir -p /var/www/vhosts
mkdir -p /etc/vsftpd/users
touch /etc/vsftpd/passwd
Now download all tools to manage the server locally:
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_DOMAIN.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_FTP_VIRTUAL_USER.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ALIAS_LIST.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_DOMAIN.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_FTP_VIRTUAL_USER.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DOMAIN_LIST.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/MYSQL_CREATE.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/UPDATE_ALL_FTP_PASSWORD.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/UPDATE_FTP_PASSWORD.sh
chmod 770 *.sh
Download also the tools that will be used with cron:
cd /root/cron_scripts
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/cron_scripts/backup_mysql.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/cron_scripts/mysql_optimize.sh
chmod 770 *.sh
- Edit /root/ADD_DOMAIN.sh and change ADMIN_EMAIL variable with your email address.
- Edit /root/MYSQL_CREATE.sh and change the variable MYSQL_ROOT_PASSWORD with your MySQL root password.
- Edit /root/cron_scripts/backup_mysql.sh and change the variable DB_PASSWORD with your MySQL root password and MAIL_NOTIFICATION with your email address.
- Edit /root/cron_scripts/mysql_optimize.sh and change the variable MYSQL_ROOT_PASSWORD with your MySQL root password.
Edit /etc/crontab and add the following lines at the bottom:
# mysql optimize tables
3 4 * * 7 root /root/mysql_optimize.sh
# mysql backup
32 4 * * * root /root/backup_mysql.sh
03 Dec 2014 Matteo Mattei linux cron mysql backup Given I did it dozens of times and everytime I have rewritten the code form scratch, I decided to write a simple script to backup all MySQL databases separately in order to avoid to always reinvent the wheel.
The following script must be configured with:
- the MySQL root user.
- the password of MySQL root user.
- the email address to receive the notifications in case of failures.
- the destination folder of the backups.
- the number of copies to keep before overwriting the old backup.
NOTE: given the MySQL root password is in clear is important to limit the access to the script:
chown root.root mysql_backup.sh
chmod 770 mysql_backup.sh
Then, to set a cronjob to do it automatically every day, open /etc/crontab and add the following line at the bottom:
# mysql backup
32 4 * * * root /root/mysql_backup.sh
21 Aug 2014 Matteo Mattei linux embedded curl Every time I have to cross compile a new application or library it is always painful and I often have to spend several minutes (hours) to figure out how to build it. In this case I am going to show you how to cross compile CURL library (and application) with SSL and ZLIB support for an embedded system based on ARM.
First of all I suppose you already have all includes and libraries (static and dynamic) of openSSL and Zlib somewhere in your system (how to cross-compile openssl and zlib is out of the scope of this post).
In my case I have this structure:
.
├── openssl
│ ├── apps
│ │ └── openssl
│ ├── include
│ │ └── openssl
│ │ ├── aes.h
│ │ ├── asn1.h
│ │ ├── asn1_mac.h
│ │ ├── asn1t.h
│ │ ├── bio.h
│ │ ├── blowfish.h
│ │ ├── bn.h
│ │ ├── buffer.h
│ │ ├── camellia.h
│ │ ├── cast.h
│ │ ├── cmac.h
│ │ ├── cms.h
│ │ ├── comp.h
│ │ ├── conf_api.h
│ │ ├── conf.h
│ │ ├── crypto.h
│ │ ├── des.h
│ │ ├── des_old.h
│ │ ├── dh.h
│ │ ├── dsa.h
│ │ ├── dso.h
│ │ ├── dtls1.h
│ │ ├── ebcdic.h
│ │ ├── ecdh.h
│ │ ├── ecdsa.h
│ │ ├── ec.h
│ │ ├── engine.h
│ │ ├── e_os2.h
│ │ ├── err.h
│ │ ├── evp.h
│ │ ├── hmac.h
│ │ ├── idea.h
│ │ ├── krb5_asn.h
│ │ ├── kssl.h
│ │ ├── lhash.h
│ │ ├── md4.h
│ │ ├── md5.h
│ │ ├── mdc2.h
│ │ ├── modes.h
│ │ ├── objects.h
│ │ ├── obj_mac.h
│ │ ├── ocsp.h
│ │ ├── opensslconf.h
│ │ ├── opensslv.h
│ │ ├── ossl_typ.h
│ │ ├── pem2.h
│ │ ├── pem.h
│ │ ├── pkcs12.h
│ │ ├── pkcs7.h
│ │ ├── pqueue.h
│ │ ├── rand.h
│ │ ├── rc2.h
│ │ ├── rc4.h
│ │ ├── ripemd.h
│ │ ├── rsa.h
│ │ ├── safestack.h
│ │ ├── seed.h
│ │ ├── sha.h
│ │ ├── srp.h
│ │ ├── srtp.h
│ │ ├── ssl23.h
│ │ ├── ssl2.h
│ │ ├── ssl3.h
│ │ ├── ssl.h
│ │ ├── stack.h
│ │ ├── symhacks.h
│ │ ├── tls1.h
│ │ ├── ts.h
│ │ ├── txt_db.h
│ │ ├── ui_compat.h
│ │ ├── ui.h
│ │ ├── whrlpool.h
│ │ ├── x509.h
│ │ ├── x509v3.h
│ │ └── x509_vfy.h
│ └── libs
│ ├── lib4758cca.so
│ ├── libaep.so
│ ├── libatalla.so
│ ├── libcapi.so
│ ├── libchil.so
│ ├── libcrypto.a
│ ├── libcrypto.so
│ ├── libcrypto.so.1.0.0
│ ├── libcswift.so
│ ├── libgmp.so
│ ├── libgost.so
│ ├── libnuron.so
│ ├── libpadlock.so
│ ├── libssl.a
│ ├── libssl.so
│ ├── libssl.so.1.0.0
│ ├── libsureware.so
│ └── libubsec.so
└── zlib
├── include
│ ├── zconf.h
│ └── zlib.h
└── libs
├── libz.a
├── libz.so -> libz.so.1.2.8
├── libz.so.1 -> libz.so.1.2.8
└── libz.so.1.2.8
Now download the last version of CURL, decompress and configure it:
$ wget http://curl.haxx.se/download/curl-7.37.1.tar.gz
$ tar xzf curl-7.37.1.tar.gz
$ export ROOTDIR="${PWD}"
$ cd curl-7.37.1/
$ export CROSS_COMPILE="arm-none-linux-gnueabi"
$ export CPPFLAGS="-I${ROOTDIR}/openssl/include -I${ROOTDIR}/zlib/include"
$ export LDFLAGS="-L${ROOTDIR}/openssl/libs -L${ROOTDIR}/zlib/libs"
$ export AR=${CROSS_COMPILE}-ar
$ export AS=${CROSS_COMPILE}-as
$ export LD=${CROSS_COMPILE}-ld
$ export RANLIB=${CROSS_COMPILE}-ranlib
$ export CC=${CROSS_COMPILE}-gcc
$ export NM=${CROSS_COMPILE}-nm
$ export LIBS="-lssl -lcrypto"
$ ./configure --prefix=${ROOTDIR}/build --target=${CROSS_COMPILE} --host=${CROSS_COMPILE} --build=i586-pc-linux-gnu --with-ssl --with-zlib
At the end of the configure you should see a configuration resume similar to the following:
curl version: 7.37.1
Host setup: arm-none-linux-gnueabi
Install prefix: /tmp/working_copy/build
Compiler: arm-none-linux-gnueabi-gcc
SSL support: enabled (OpenSSL)
SSH support: no (--with-libssh2)
zlib support: enabled
GSS-API support: no (--with-gssapi)
SPNEGO support: no (--with-spnego)
TLS-SRP support: enabled
resolver: default (--enable-ares / --enable-threaded-resolver)
ipv6 support: no (--enable-ipv6)
IDN support: no (--with-{libidn,winidn})
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
SSPI support: no (--enable-sspi)
ca cert bundle: no
ca cert path: no
LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support: no (--enable-ldaps)
RTSP support: enabled
RTMP support: no (--with-librtmp)
metalink support: no (--with-libmetalink)
HTTP2 support: disabled (--with-nghttp2)
Protocols: DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SMTP SMTPS TELNET TFTP
Now compile and install it in the previously configured build directory:
Now in the parent directory you should have a new folder called build with the new curl library and application with OpenSSL and Zlib support cross compiled for ARM:
build/
├── bin
│ ├── curl
│ └── curl-config
├── include
│ └── curl
│ ├── curlbuild.h
│ ├── curl.h
│ ├── curlrules.h
│ ├── curlver.h
│ ├── easy.h
│ ├── mprintf.h
│ ├── multi.h
│ ├── stdcheaders.h
│ └── typecheck-gcc.h
└── lib
├── libcurl.a
├── libcurl.la
├── libcurl.so -> libcurl.so.4.3.0
├── libcurl.so.4 -> libcurl.so.4.3.0
├── libcurl.so.4.3.0
└── pkgconfig
└── libcurl.pc
(I voluntary omitted the share folder in the tree exploded).
