Matteo Mattei

Hello, my name is Matteo Mattei and this is my personal website. I am computer engineer with a long experience in Linux system administration and web software development. My preferred computer languages and technologies are PHP, Python, Bash, HTML, HTML5, CSS3, jQuery, MySQL, Node.js, MongoDB, etc...

© 2006-2015 Matteo Mattei - all rights reserved. About my blog code

linkedin rss twitter google+ github facebook

Backup your server on mega.co.nz using megatools

Mega is a wonderful and secure online cloud service that offers 50GB of free storage. So I thought that it would be great using it as additional backup service for my Linux servers. Fortunately there is a good application library and tools to access Mega storage from Linux command line. This tools are called megatools and it is released under GPLv2 license.

In this post I will show you how to compile, install and configure a full backup system for your server using Mega and MegaTools.

First of all register an account with Mega at http://mega.co.nz then follow these instructions to compile and install megatools.

#!/bin/bash

VERSION="1.9.94"

apt-get install -y pkg-config libglib2.0-dev libssl-dev libcurl4-openssl-dev libfuse-dev glib-networking

wget http://megatools.megous.com/builds/megatools-${VERSION}.tar.gz
tar xzf megatools-${VERSION}.tar.gz
cd megatools-${VERSION}
./configure && make && make install && ldconfig

Note: the above instructions are valid for Debian 7 Wheezy. In case you have a different Linux distribution please install the required dependencies.

Now that you have MegaTools installed in /usr/local/bin create a configuration file with your credentials in /root/.megarc:

[Login]
Username = Your_Mega_Username
Password = Your_Mega_Password

Since the password is in clear, it is important to protect the file:

chmod 640 /root/.megarc

Test now your mega installation and login credentials:

root@debian:~# megals 
/Contacts
/Inbox
/Root
/Trash

If all goes well you are ready to prepare your backup script. Create a new file called megabackup.sh and place it in /root:

Make it executable and accessible only to root:

chmod 750 /root/megabackup.sh

You only need to set a cron-job now to execute the backup every day:

04 04 * * * root /root/megabackup.sh

Update 2015-02-14: MegaFuse

I recently found a better tool to accomplish my needings, it is called MegaFuse and it works like a Linux mountpoint. First of all download all needed dependencies:

apt-get install libcrypto++-dev libcurl4-openssl-dev libdb5.1++-dev libfreeimage-dev libreadline-dev libfuse-dev make

Then download MegaFuse and compile it:

git clone https://github.com/matteoserva/MegaFuse
cd MegaFuse
make

Now create a configuration file /root/.megafuse.conf with your Mega credentials:

USERNAME = your_mega_email
PASSWORD = your_mega_password
MOUNTPOINT = /mnt

The above configuration assume you will use /mnt as your mountpoint (change it if you want).

Protect it in the same way:

chmod 640 /root/.megafuse.conf

And this is the revisited script to do the backup:

You can now configure the cronjob:

04 04 * * * root /root/megafusebackup.sh

Create a self-contained installer in Bash

In this post I will show you how to develop a self contained Linux command line installer in Bash that will decompress an archive and perform some tasks.

Installer content

Our installer that is basically a self-extracting archive with some logic around, consists in three parts:

  • A bash script that performs the extraction of the archive and applies some logic.
  • A marker to separate the bash script and the archive.
  • An archive containing the actual data to install.

Start now!

Create a new bash script called installer.sh with the following content:

#!/bin/bash

echo ""
echo "My Command Line Installer"
echo ""

# Create destination folder
DESTINATION="/opt/my_application"
mkdir -p ${DESTINATION}

# Find __ARCHIVE__ maker, read archive content and decompress it
ARCHIVE=$(awk '/^__ARCHIVE__/ {print NR + 1; exit 0; }' "${0}")
tail -n+${ARCHIVE} "${0}" | tar xpJv -C ${DESTINATION}

# Put your logic here (if you need)

echo ""
echo "Installation complete."
echo ""

# Exit from the script with success (0)
exit 0

__ARCHIVE__

This script is self-explain but I will try to describe the steps:

  1. Create a destination folder ${DESTINATION}.
  2. Find ARCHIVE marker and put the tarball content into ${ARCHIVE} variable.
  3. Decompress the tarball into the destination folder.
  4. Eventually apply your installation logic (copy some files, change some others, etc...).
  5. Exit from the script (this step is mandatory otherwise bash will try to interpret the tarball and will exit with error).
  6. Add ARCHIVE marker at the bottom of the script. This marker will be used to separate the actual bash script with the tarball content.

Now generate a compressed tarball of your application (I used .tar.xz in the above example):

tar cJf myarchive.tar.xz /folder/to/archive

OK, now append it to the installer bash script and make it executable:

cat myarchive.tar.xz >> installer.sh
chmod +x installer.sh

That's all! You can now distribute your installer.

Execute your installer

The users will execute your installer simply running:

./installer.sh

Install CouchDB 1.6.x on Debian 7 (Wheezy)

Setup repository and install all dependencies

echo "deb http://packages.erlang-solutions.com/debian wheezy contrib" >> /etc/apt/sources.list
wget -qO - http://packages.erlang-solutions.com/debian/erlang_solutions.asc | apt-key add -
apt-get update
apt-get install -y build-essential curl erlang-nox erlang-dev libmozjs185-1.0 libmozjs185-dev libcurl4-openssl-dev libicu-dev

Create CouchDB account

useradd -d /var/lib/couchdb couchdb
mkdir -p /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb /var/lib/couchdb
chown -R couchdb:couchdb /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb
chmod -R g+rw /usr/local/{lib,etc}/couchdb /usr/local/var/{lib,log,run}/couchdb

Install CouchDB

wget http://apache.panu.it/couchdb/source/1.6.1/apache-couchdb-1.6.1.tar.gz
tar xzf apache-couchdb-1.6.1.tar.gz
cd apache-couchdb-1.6.1
./configure --prefix=/usr/local --with-js-lib=/usr/lib --with-js-include=/usr/include/js --enable-init
make && make install

Create symlinks and start the database

chown couchdb:couchdb /usr/local/etc/couchdb/local.ini
ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
/etc/init.d/couchdb start
update-rc.d couchdb defaults

Verify that all is working fine

curl http://127.0.0.1:5984/

The output should be like this:

{"couchdb":"Welcome","uuid":"5da242ff50cecec904d6caf36be34194","version":"1.6.1","vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}}

Finalize setup

In order to connect from remote edit /usr/local/etc/couchdb/local.ini and change

bind_address 127.0.0.1

to:

bind_address to 0.0.0.0

Restart the database:

service couchdb restart

And from a web browser visit the CouchDB Futon:

http://HOST:5984/_utils

Full web server setup with Debian 7 (Wheezy)

Setup bash and update the system

cp /etc/skel/.bashrc /root/.bashrc
apt-get update
apt-get dist-upgrade

Configure hostname correctly

Make sure to have the following two lines (with the same format) at the top of your /etc/hosts file

127.0.0.1       localhost.localdomain localhost
xxx.xxx.xxx.xxx web1.myserver.com web1

Note: xxx.xxx.xxx.xxx is the public IP address assigned to your server.

Install all needed packages

apt-get install php5 mysql-server mysql-client apache2 iptables phpmyadmin varnish shorewall vsftpd php5-cli php5-curl php5-dev php5-gd php5-idn php5-imagick php5-imap php5-memcache php5-ming php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xcache php5-xmlrpc php-apc php-pear php-xml-rpc postfix apg ca-certificates heirloom-mailx

MySQL/PhpMyAdmin:

  • mysql root password: xxx
  • repeat mysql root password: xxx
  • web server to reconfigure automatically: apache2
  • configure database for phpmyadmin with dbconfig-common? Yes
  • Password of the database's administrative user: xxx
  • Password for phpmyadmin: xxx
  • Password confirmation: xxx

Postfix:

  • Select Internet Site
  • System mail name: (insert here the FQDN, for example web1.myserver.com)

Setup FTP

Stop VSFTP server:

/etc/init.d/vsftpd stop

Create backup configuration:

mv /etc/vsftpd.conf /etc/vsftpd.conf.backup

Add new configuration:

listen=YES
listen_port=21
anonymous_enable=NO
local_enable=YES
guest_enable=YES
guest_username=nobody
user_sub_token=$USER
local_root=/var/www/vhosts/$USER
virtual_use_local_privs=YES
user_config_dir=/etc/vsftpd/users
pam_service_name=vsftpd_local_and_virtual
chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list
ftpd_banner=Welcome to my ftp server
write_enable=YES
download_enable=YES
dirlist_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
xferlog_file=/var/log/xferlog
connect_from_port_20=YES
connect_timeout=60
data_connection_timeout=300
idle_session_timeout=300
local_max_rate=0
max_clients=0
max_per_ip=3

Create an empty chroot_list file:

touch /etc/vsftpd/chroot_list

Start VSFTP server:

/etc/init.d/vsftpd start

Setup Apache

Stop Apache web server:

/etc/init.d/apache2 stop

Backup Apache configuration:

cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.backup

Edit the following lines in /etc/apache2/apache2.conf

  • From Timeout 300 to Timeout 45
  • From KeepAliveTimeout 5 to KeepAliveTimeout 15
  • Change the mpm_prefork_module section like the following:
  <IfModule mpm_prefork_module>
      StartServers         5
      MinSpareServers      5
      MaxSpareServers      10
      ServerLimit          400
      MaxClients           400
      MaxRequestsPerChild  10000
  </IfModule>

Edit /etc/apache2/ports.conf and change the port 80 with 8080 since we are going to use Varnish:

NameVirtualHost *:8080
Listen 8080

Change the port (from 80 to 8080) also in the default virtual host /etc/apache2/sites-enabled/000-default Now restart Apache:

/etc/init.d/apache2 restart

Setup Varnish

Stop Varnish daemon:

/etc/init.d/varnish stop

Open /etc/varnish/default.vcl and make sure the backend section is like this:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
}

Now edit /etc/default/varnish and set the DAEMON_OPTS variable like this:

DAEMON_OPTS="-a :80 \
                -T localhost:6082 \
                -f /etc/varnish/default.vcl \
                -p thread_pools=4 \
                -p thread_pool_max=1500 \
                -p listen_depth=2048 \
                -p lru_interval=1800 \
                -h classic,169313 \
                -p max_restarts=6 \
                -p connect_timeout=600 \
                -p send_timeout=2000 \
                -s malloc,2G"

Restart Varnish:

/etc/init.d/varnish restart

Setup MySQL

MySQL is already configured. You only need to log slow queries (that is often usefult during slow load page investigation). Todo it, open /etc/mysql/my.cnf and decomment the following two lines:

log_slow_queries       = /var/log/mysql/mysql-slow.log
long_query_time = 2

Configure Shorewall firewall rules

Copy the default configuration for one interface:

cp /usr/share/doc/shorewall/examples/one-interface/interfaces /etc/shorewall/interfaces
cp /usr/share/doc/shorewall/examples/one-interface/policy /etc/shorewall/policy
cp /usr/share/doc/shorewall/examples/one-interface/rules /etc/shorewall/rules
cp /usr/share/doc/shorewall/examples/one-interface/zones /etc/shorewall/zones

Now open /etc/shorewall/policy file and change the line:

net             all             DROP            info

removing info directive given it fills the system logs:

net             all             DROP

Now open /etc/shorewall/rules and add the following rules at the bottom of the file:

HTTP/ACCEPT     net             $FW
SSH/ACCEPT      net             $FW
FTP/ACCEPT      net             $FW

# real apache since varnish listens on port 80
#ACCEPT         net             $FW             tcp             8080

NOTE: in case you want to allow ICMP (Ping) traffic from a specific remote hosts you need to add a rule similar to the following where xxx.xxx.xxx.xxx is the remote IP address, before the Ping(DROP) rule:

Ping(ACCEPT)    net:xxx.xxx.xxx.xxx       $FW

Now edit /etc/default/shorewall and change startup=0 to startup=1 You are now ready to start the firewall:

/etc/init.d/shorewall start

Setup Postfix

Stop postfix server:

/etc/init.d/postfix stop

Edit /etc/mailname and set your server domain name, for example:

server1.mycompany.com

Then, in order to monitor mail traffic coming from PHP you need to edit /etc/php5/apache2/php.ini. Go to [mail function] section and set the following two options:

sendmail_path = /usr/local/bin/sendmail-wrapper
auto_prepend_file = /usr/local/bin/env.php

Now create the two files above:

sendmail-wrapper:

#!/bin/sh
logger -p mail.info sendmail-wrapper.sh: site=${HTTP_HOST}, client=${REMOTE_ADDR}, script=${SCRIPT_NAME}, pwd=${PWD}, uid=${UID}, user=$(whoami)
/usr/sbin/sendmail -t -i $*

env.php:

<?php
putenv("HTTP_HOST=".@$_SERVER["HTTP_HOST"]);
putenv("SCRIPT_NAME=".@$_SERVER["SCRIPT_NAME"]);
putenv("SCRIPT_FILENAME=".@$_SERVER["SCRIPT_FILENAME"]);
putenv("DOCUMENT_ROOT=".@$_SERVER["DOCUMENT_ROOT"]);
putenv("REMOTE_ADDR=".@$_SERVER["REMOTE_ADDR"]);
?>

Now make they both have executable flag:

chmod +x /usr/local/bin/sendmail-wrapper
chmod +x /usr/local/bin/env.php

Add also /usr/local/bin/ to the open_basedir php list in /etc/apache2/conf.d/phpmyadmin.conf

php_admin_value open_basedir /usr/share/phpmyadmin/:/etc/phpmyadmin/:/var/lib/phpmyadmin/:/usr/local/bin/

Restart Postfix:

/etc/init.d/postfix start

Prepare environment

Create all needed directories and files

mkdir /root/cron_scripts
mkdir -p /var/www/vhosts
mkdir -p /etc/vsftpd/users
touch /etc/vsftpd/passwd

Now download all tools to manage the server locally:

wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_DOMAIN.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ADD_FTP_VIRTUAL_USER.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/ALIAS_LIST.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_ALIAS.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_DOMAIN.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DEL_FTP_VIRTUAL_USER.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/DOMAIN_LIST.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/MYSQL_CREATE.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/UPDATE_ALL_FTP_PASSWORD.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/UPDATE_FTP_PASSWORD.sh
chmod 770 *.sh

Download also the tools that will be used with cron:

cd /root/cron_scripts
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/cron_scripts/backup_mysql.sh
wget https://raw.githubusercontent.com/matteomattei/servermaintenance/master/LAMP/cron_scripts/mysql_optimize.sh
chmod 770 *.sh
  • Edit /root/ADD_DOMAIN.sh and change ADMIN_EMAIL variable with your email address.
  • Edit /root/MYSQL_CREATE.sh and change the variable MYSQL_ROOT_PASSWORD with your MySQL root password.
  • Edit /root/cron_scripts/backup_mysql.sh and change the variable DB_PASSWORD with your MySQL root password and MAIL_NOTIFICATION with your email address.
  • Edit /root/cron_scripts/mysql_optimize.sh and change the variable MYSQL_ROOT_PASSWORD with your MySQL root password.

Configure CRON

Edit /etc/crontab and add the following lines at the bottom:

# mysql optimize tables
3  4  *  *  7   root    /root/mysql_optimize.sh

# mysql backup
32 4  *  *  *   root    /root/backup_mysql.sh

How to backup MySQL database using shell and cron

Given I did it dozens of times and everytime I have rewritten the code form scratch, I decided to write a simple script to backup all MySQL databases separately in order to avoid to always reinvent the wheel.

The following script must be configured with:

  • the MySQL root user.
  • the password of MySQL root user.
  • the email address to receive the notifications in case of failures.
  • the destination folder of the backups.
  • the number of copies to keep before overwriting the old backup.

NOTE: given the MySQL root password is in clear is important to limit the access to the script:

chown root.root mysql_backup.sh
chmod 660 mysql_backup.sh

Then, to set a cronjob to do it automatically every day, open /etc/crontab and add the following line at the bottom:

# mysql backup
32 4  *  *  *   root    /root/backup_mysql.sh

How to cross compile CURL library with SSL and ZLIB support

post_image

Every time I have to cross compile a new application or library it is always painful and I often have to spend several minutes (hours) to figure out how to build it. In this case I am going to show you how to cross compile CURL library (and application) with SSL and ZLIB support for an embedded system based on ARM.

First of all I suppose you already have all includes and libraries (static and dynamic) of openSSL and Zlib somewhere in your system (how to cross-compile openssl and zlib is out of the scope of this post).

In my case I have this structure:

.
├── openssl
│   ├── apps
│   │   └── openssl
│   ├── include
│   │   └── openssl
│   │       ├── aes.h
│   │       ├── asn1.h
│   │       ├── asn1_mac.h
│   │       ├── asn1t.h
│   │       ├── bio.h
│   │       ├── blowfish.h
│   │       ├── bn.h
│   │       ├── buffer.h
│   │       ├── camellia.h
│   │       ├── cast.h
│   │       ├── cmac.h
│   │       ├── cms.h
│   │       ├── comp.h
│   │       ├── conf_api.h
│   │       ├── conf.h
│   │       ├── crypto.h
│   │       ├── des.h
│   │       ├── des_old.h
│   │       ├── dh.h
│   │       ├── dsa.h
│   │       ├── dso.h
│   │       ├── dtls1.h
│   │       ├── ebcdic.h
│   │       ├── ecdh.h
│   │       ├── ecdsa.h
│   │       ├── ec.h
│   │       ├── engine.h
│   │       ├── e_os2.h
│   │       ├── err.h
│   │       ├── evp.h
│   │       ├── hmac.h
│   │       ├── idea.h
│   │       ├── krb5_asn.h
│   │       ├── kssl.h
│   │       ├── lhash.h
│   │       ├── md4.h
│   │       ├── md5.h
│   │       ├── mdc2.h
│   │       ├── modes.h
│   │       ├── objects.h
│   │       ├── obj_mac.h
│   │       ├── ocsp.h
│   │       ├── opensslconf.h
│   │       ├── opensslv.h
│   │       ├── ossl_typ.h
│   │       ├── pem2.h
│   │       ├── pem.h
│   │       ├── pkcs12.h
│   │       ├── pkcs7.h
│   │       ├── pqueue.h
│   │       ├── rand.h
│   │       ├── rc2.h
│   │       ├── rc4.h
│   │       ├── ripemd.h
│   │       ├── rsa.h
│   │       ├── safestack.h
│   │       ├── seed.h
│   │       ├── sha.h
│   │       ├── srp.h
│   │       ├── srtp.h
│   │       ├── ssl23.h
│   │       ├── ssl2.h
│   │       ├── ssl3.h
│   │       ├── ssl.h
│   │       ├── stack.h
│   │       ├── symhacks.h
│   │       ├── tls1.h
│   │       ├── ts.h
│   │       ├── txt_db.h
│   │       ├── ui_compat.h
│   │       ├── ui.h
│   │       ├── whrlpool.h
│   │       ├── x509.h
│   │       ├── x509v3.h
│   │       └── x509_vfy.h
│   └── libs
│       ├── lib4758cca.so
│       ├── libaep.so
│       ├── libatalla.so
│       ├── libcapi.so
│       ├── libchil.so
│       ├── libcrypto.a
│       ├── libcrypto.so
│       ├── libcrypto.so.1.0.0
│       ├── libcswift.so
│       ├── libgmp.so
│       ├── libgost.so
│       ├── libnuron.so
│       ├── libpadlock.so
│       ├── libssl.a
│       ├── libssl.so
│       ├── libssl.so.1.0.0
│       ├── libsureware.so
│       └── libubsec.so
└── zlib
    ├── include
    │   ├── zconf.h
    │   └── zlib.h
    └── libs
        ├── libz.a
        ├── libz.so -> libz.so.1.2.8
        ├── libz.so.1 -> libz.so.1.2.8
        └── libz.so.1.2.8

Now download the last version of CURL, decompress and configure it:

$ wget http://curl.haxx.se/download/curl-7.37.1.tar.gz
$ tar xzf curl-7.37.1.tar.gz
$ export ROOTDIR="${PWD}"
$ cd curl-7.37.1/
$ export CROSS_COMPILE="arm-none-linux-gnueabi"
$ export CPPFLAGS="-I${ROOTDIR}/openssl/include -I${ROOTDIR}/zlib/include"
$ export LDFLAGS="-L${ROOTDIR}/openssl/libs -L${ROOTDIR}/zlib/libs"
$ export AR=${CROSS_COMPILE}-ar
$ export AS=${CROSS_COMPILE}-as
$ export LD=${CROSS_COMPILE}-ld
$ export RANLIB=${CROSS_COMPILE}-ranlib
$ export CC=${CROSS_COMPILE}-gcc
$ export NM=${CROSS_COMPILE}-nm
$ export LIBS="-lssl -lcrypto"
$ ./configure --prefix=${ROOTDIR}/build --target=${CROSS_COMPILE} --host=${CROSS_COMPILE} --build=i586-pc-linux-gnu --with-ssl --with-zlib

At the end of the configure you should see a configuration resume similar to the following:

  curl version:     7.37.1
  Host setup:       arm-none-linux-gnueabi
  Install prefix:   /tmp/working_copy/build
  Compiler:         arm-none-linux-gnueabi-gcc
  SSL support:      enabled (OpenSSL)
  SSH support:      no      (--with-libssh2)
  zlib support:     enabled
  GSS-API support:  no      (--with-gssapi)
  SPNEGO support:   no      (--with-spnego)
  TLS-SRP support:  enabled
  resolver:         default (--enable-ares / --enable-threaded-resolver)
  ipv6 support:     no      (--enable-ipv6)
  IDN support:      no      (--with-{libidn,winidn})
  Build libcurl:    Shared=yes, Static=yes
  Built-in manual:  enabled
  --libcurl option: enabled (--disable-libcurl-option)
  Verbose errors:   enabled (--disable-verbose)
  SSPI support:     no      (--enable-sspi)
  ca cert bundle:   no
  ca cert path:     no
  LDAP support:     no      (--enable-ldap / --with-ldap-lib / --with-lber-lib)
  LDAPS support:    no      (--enable-ldaps)
  RTSP support:     enabled
  RTMP support:     no      (--with-librtmp)
  metalink support: no      (--with-libmetalink)
  HTTP2 support:    disabled (--with-nghttp2)
  Protocols:        DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SMTP SMTPS TELNET TFTP

Now compile and install it in the previously configured build directory:

$ make
$ make install

Now in the parent directory you should have a new folder called build with the new curl library and application with OpenSSL and Zlib support cross compiled for ARM:

build/
├── bin
│   ├── curl
│   └── curl-config
├── include
│   └── curl
│       ├── curlbuild.h
│       ├── curl.h
│       ├── curlrules.h
│       ├── curlver.h
│       ├── easy.h
│       ├── mprintf.h
│       ├── multi.h
│       ├── stdcheaders.h
│       └── typecheck-gcc.h
└── lib
    ├── libcurl.a
    ├── libcurl.la
    ├── libcurl.so -> libcurl.so.4.3.0
    ├── libcurl.so.4 -> libcurl.so.4.3.0
    ├── libcurl.so.4.3.0
    └── pkgconfig
        └── libcurl.pc

(I voluntary omitted the share folder in the tree exploded).