Installing Drupal with CIVIcrm On Debian 8 and using AWS S3 backup

The following guide is for installing Drupal 7 wuth CIVIcrm on Debian Jessie. I will be using Nginx as the web server and I will show you how to generate letsencrypt certificates and autorenew them. As an added bonus i will also show teh steps necessary to use the backup module of Drupal to backup on Amazon S3.

Install dependecies

Note I installed php5 since i tried with php7 and was getting some errors I also red in the documentation that there are some known bugs, especially in contributed modules (see the discussion for more details)
https://www.drupal.org/blog/drupal-7-50

 apt-get update
 apt-get dist-upgrade
 apt -y install php-apc php-pear php5-cli php5-common php5-curl php5-fpm php5-gd php5-mysql php-soap nginx mariadb-server git drush

You will add password from mysql root user

Create user

useradd -s /bin/bash -m drupal

Get drupal using drush inside user drupal home directory

drush dl drupal-7
mv drupal-7.54/ drupal

Modify php-fpm to listen to port 127.0.0.1:9000

nano /etc/php5/fpm/pool.d/www.conf

Change

listen = /var/run/php5-fpm.sock

listen = 127.0.0.1:9000

restart the service

systemctl restart php5-fpm

Configure Nginx

The following is the config file i am using for nginx i have tested the settings on SSL LABs and get an A+ . I will explain the letsencrypt setup on the next step

server {


listen         80;
       server_name    www.compucorp.milidonis.com compucorp.milidonis.com;
       return         301 https://$server_name$request_uri;
}



server {


   listen 443 ssl;
       add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
   server_name www.drupal.milidonis.com drupal.milidonis.com;
   ssl_certificate   /etc/letsencrypt/live/compucorp.milidonis.com/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/compucorp.milidonis.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'kEECDH+ECDSA+AES128 kEECDH+ECDSA+AES256 kEECDH+AES128 kEECDH+AES256 kEDH+AES128 kEDH+AES256 DES-CBC3-SHA +SHA !aNULL !eNULL !LOW !kECDH !DSS !MD5 !EXP !PSK !SRP !CAMELLIA !SEED';

    ssl_prefer_server_ciphers   on;

    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_stapling on;
    ssl_session_cache builtin:1000 shared:SSL:10m;

    
    access_log  /var/log/nginx/localhost.access.log;
    error_log /var/log/nginx/error.log;
 
    root /home/drupal/drupal;
    index index.php;
 
## serve imagecache files directly or redirect to drupal if they do not exist.
    location ~* files/styles {
      access_log off;
      expires 30d;
      try_files $uri @drupal;
    }
 
## serve imagecache files directly or redirect to drupal if they do not exist.
    location ~* ^.+.(xsl|xml)$ {
      access_log off;
      expires 1d;
      try_files $uri @drupal;
    }
 
## Default location
    location / {
        try_files $uri $uri/ @drupal;
        index  index.php;
    }
 
    location @drupal {
        rewrite ^/(.*)$ /index.php?q=$1 last;
    }
 
## Images and static content is treated different
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml)$ {
      access_log        off;
      expires           30d;
    }
 
## Parse all .php file in the /var/www directory
    location ~ .php$ {
        include fastcgi_params;
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass   backend;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  SCRIPT_NAME      $fastcgi_script_name;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;
        fastcgi_param  CONTENT_LENGTH   $content_length;
        fastcgi_intercept_errors        on;
        fastcgi_ignore_client_abort     off;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 180;
        fastcgi_read_timeout 180;
        fastcgi_buffer_size 128k;
        fastcgi_buffers 4 256k;
        fastcgi_busy_buffers_size 256k;
        fastcgi_temp_file_write_size 256k;
    }

##block access to certain civicrm files
 location ~ ^/sites/.*/files/civicrm/(ConfigAndLog|upload|templates_c) {
    deny all;
  }

## Letsencrypt path
location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
    }

 
## Disable viewing .htaccess & .htpassword
    location ~ /\.ht {
        deny  all;
    }
}
 
upstream backend {
        server 127.0.0.1:9000;
}

Certificates generation using Letsencrypt and automatic renewal.

There is also a method using certbot however i prefer doing it this way

mkdir /opt/letsencrypt
/usr/bin/git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Create a configuration directory for the domains you intend to use

mkdir -p /srv/letsencrypt/config/
nano /srv/letsencrypt/config/domains.conf

domains=compucorp.milidonis.com,www.compucorp.milidonis.com
# increase key size
rsa-key-size = 4096

# the current closed beta (as of 2015-Nov-07) is using this server
server = https://acme-v01.api.letsencrypt.org/directory

# this address will receive renewal reminders, IIRC
email = stelios@milidonis.com

# turn off the ncurses UI, we want this to be run as a cronjob
text = True

# authenticate by placing a file in the webroot (under .well-known/acme-challenge/) and then letting
# LE fetch it
authenticator = webroot
webroot-path = /var/www/letsencrypt

Create the directory that will be called by nginx

mkdir /var/www/letsencrypt

Run letencrypt
*** Note you will have a separate nginx configuration to initially run this the first time not the nginx conf above or just modify the header oart of nginx configuration to look like bellow

server {


listen         80;
       server_name    www.compucorp.milidonis.com compucorp.milidonis.com;
      # return         301 https://$server_name$request_uri;
location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
    }

Thne generatte the certificates

/opt/letsencrypt/letsencrypt-auto certonly --config /srv/letsencrypt/config/domains.conf --agree-tos

Create a cron job to run weekly checking if the certificates are about to expire and renew them

nano /etc/cron.weekly/letsenrypt

#!/bin/bash

web_service='nginx'
config_file="/srv/letsencrypt/config/domains.conf"

le_path='/opt/letsencrypt'
exp_limit=30;

if [ ! -f $config_file ]; then
        echo "[ERROR] config file does not exist: $config_file"
        exit 1;
fi

domain=`grep "^\s*domains" $config_file | sed "s/^\s*domains\s*=\s*//" | sed 's/(\s*)\|,.*$//'`
cert_file="/etc/letsencrypt/live/$domain/fullchain.pem"

if [ ! -f $cert_file ]; then
        echo "[ERROR] certificate file not found for domain $domain."
fi

exp=$(date -d "`openssl x509 -in $cert_file -text -noout|grep "Not After"|cut -c 25-`" +%s)
datenow=$(date -d "now" +%s)
days_exp=$(echo \( $exp - $datenow \) / 86400 |bc)

echo "Checking expiration date for $domain..."

if [ "$days_exp" -gt "$exp_limit" ] ; then
        echo "The certificate is up to date, no need for renewal ($days_exp days left)."
        exit 0;
else
        echo "The certificate for $domain is about to expire soon. Starting webroot renewal script..."
        $le_path/letsencrypt-auto certonly -a webroot --agree-tos --renew-by-default --config $config_file
        echo "Reloading $web_service"
        /usr/sbin/service $web_service reload
        echo "Renewal process finished for domain $domain"
        exit 0;
fi

Make it exacutable

chmod +x /etc/cron.weekly/letsenrypt

Generate ssl_dhparam /etc/nginx/ssl/dhparam2048.pem;

to generate your dhparam.pem file, run in the terminal

mkdir /etc/nginx/ssl

openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

Setup Drupal and CiviCRM Databases

Connect to mysql

sudo mysql -p

Create the databases the users and their access

CREATE DATABASE drupaldb CHARACTER SET = utf8_general_ci COLLATE = utf8_general_ci;

create user drupaluser@localhost identified by 'password';

grant all privileges on drupaldb.* to drupaluser@localhost identified by 'password';

create database civicdb  CHARACTER SET = utf8_general_ci COLLATE = utf8_general_ci  ;

create user civicuser@localhost identified by 'password';

grant all privileges on civicdb.* to civicuser@localhost identified by 'password';

grant SELECT on civicdb.* to drupaluser@localhost identified by 'password';

Drupal Setup

First create the sites/default/files directory (as the admin user):

 cd /home/drupal/drupal/
 sudo mkdir sites/default/files
 sudo chown www-data:drupal sites/default/files

the copy the settings.php file

cp sites/default/default.settings.php sites/default/settings.php
chown www-data: drupal sites/default/settings.php

The rest of the setup is done over the browser inetrface..

Configure Drupal to have a private folder

mkdir /home/drupal/drupal/sites/default/private
chown drupal:drupal /home/drupal/drupal/sites/default/private

Administration -> Configuration -> Media

Then use these setting:

Private file system path: sites/default/private
Default download method: Private local files served by Drupal

Installing CiviCRM

First it is recomended that PHP memory_limit set to between 256 and 512 megabytes.

So we can need to modiy /etc/php5/fpm/php.ini

change memory_limit=128 to memory_limit =256

sed -i -e 's/memory_limit\ =\ 128M/memory_limit\ =\ 256M/' /etc/php5/fpm/php.ini

systemctl restart php5-fpm

Download CiviCRM and move to extensions

 sudo su - drupal
 cd ~drupal/drupal/sites/all/modules/
 wget https://download.civicrm.org/civicrm-4.7.17-drupal.tar.gz
 tar xzf civicrm-4.7.17-drupal.tar.gz 
 rm *.gz
 chown www-data /home/drupal/drupal/sites/default

You can visit the website for the next step

change the domain name for yours.

https://compucorp.milidonis.com/sites/all/modules/civicrm/install/index.php

Here you must enter the civicrm user, password and the drupal database user and password

then click on "" Check requiremnts and install CiviCRM"

After less than a minute is should complete.

To secure CiviCRM the folowing was already added to nginx

##block access to certain civicrm files
 location ~ ^/sites/.*/files/civicrm/(ConfigAndLog|upload|templates_c) {
    deny all;
  }

Configure Drupal so that Drupal Views can use the CiviCRM database

In my case i pointer my browser to the following url

https://compucorp.milidonis.com/civicrm/admin/setting/uf?reset=1

Install postfix

You can install postfix do that emails can be sent

apt-get install postfix

Installing Backup and Migrate Drupal Module

Navigate to "/home/drupal/drupal/sites/all/modules" as user drupal

and

wget https://ftp.drupal.org/files/projects/backup_migrate-7.x-3.1.tar.gz
tar xzf backup_migrate-7.x-3.1.tar.gz
 rm backup_migrate-7.x-3.1.tar.gz

You have to then enable the module from

Home » Administration >> Modules

Configure Backup and Migrate to S3

Create your backups bucket on S3

From the AWS console, Find the S3 section.
"Create Bucket" and name it something like ** compucorp **. This slug will be used in many placed as a resource identifier. Make sure it's lowercase and simple.
For best results, use the US-Default region According to some unsubstantiated reports, as of 2015-04, the stable version of the PHP-S3 library does not support any other regions quite as well.

Create a backup user on IAM

You MUST NOT use your admin credentials to get your backup script to connect for you. Instead we must set up a limited access account.

From the AWS console, find the Identity and Access Management (IAM) section.
Under "Users", "Create new Users". I called mine backup_daemon.
I also set Permissions so that the user only has access to S3
When it asks to "Generate an access key for each user", do so and be sure to download the credentials it gives you as you can't recover them later and have to regenerate if you lose it.

Grant the backup user access to the S3 Bucket

This starts to get tricky now. The most direct method I've found is:

Edit your backup user. Under "Create Polisys" : "Create Your own  Policy".
You can use the Policy Generator, though it's tricky to grok if you are unfamiliar with AWS. Instead, choose "Custom Policy" and paste the following:

Policy name: can-store-website-backups
Policy Document:

{
    "Version": "2017-3-21",
    "Statement": [
        {
            "Sid": "AllowUserToFindBucket",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListObjects"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "AllowUserToStoreFiles",
            "Action": [
                "s3:DeleteObject",
                "s3:GetBucketAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:ListObjects",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::compucorpaws/*"
        }
    ]
}

Replace compucorpaws with your bucket identifier
Validate and save.

At this point, you should be able to return to your Drupal screen and fill in the details.

Destination name (your label, enter anything)
Host (defaults to s3.amazonaws.com) ... which works best if you are using region "US-standard" If using a different region, you may also need to change this hostname
S3 Bucket (Your variation of mycompany-websitebackups)
Subdirectory (Should probably be required and be unique. It does not need to already exist, and may be any depth)
Access Key ID (From the credentials.csv you downloaded for the backup-daemon)
Secret Access Key (From the credentials.csv you downloaded for the backup-daemon)

Navigate toyour equivalent url

https://compucorp.milidonis.com/#overlay=admin/config/system/backup_migrate/settings/destination/add

and "Add Destination", choosing Amazon S3 Bucket as the type.

Due to drupal.org code hosting policies, the S3 library needed to use an S3 destination is no longer distributed with this module

wget https://github.com/tpyo/amazon-s3-php-class/tarball/master
mv master legacy.tar.gz
tar xfs legacy.tar.gz 
rm legacy.tar.gz 
mv tpyo-amazon-s3-php-class-9cf2eec s3-php5-curl

You can then create a daily backup schedule that will backup daily to S3 all your databases and files

Also you must add the CIVICRM database to the settings so that it also gets backed up:
navigate to the following equivalent url and add under sources--->create new sourc--->mysql database and add the civicrm database details(database name, user and password)

https://compucorp.milidonis.com/#overlay=admin/config/system/backup_migrate/settings