This high performance Drupal system is based on the work of Project Mercury. Project Mercury aims to increase the performance and ability of Drupal powered websites to respond to increased load. This document leverages the same collection of popular open source tools including the Varnish Accelerating Proxy, the APC bytecode cache for PHP, Memcached for accelerating database operations, and the Solr search engine. These instructions modify the Drupal community instructions for Project Mercury for use with the Linode Platform and the 10.04 release of the Ubuntu Distribution.
You may also choose to deploy your high performance Drupal stack with Linode's StackScript. Simply click on the "Deploy using StackScripts" link on the "Deploy a Linux Distribution" page. When the new Linode boots your system will be running Pantheon Project Mercury. Note that both the StackScript edition and the version of Mercury described in this document require substantial system resources; ensure your system has at least 512 megabytes of RAM. Additionally, at the time of initial publication, be aware that many Drupal Modules may not be fully compatible with PHP version 5.3 that ships with Lucid. Prior to beginning this guide for installing the high performance stack based on Project Mercury, we assume that you have completed our getting started guide. If you're new to Linux systems administration we recommend engaging the guides in our using Linux guide series, particularly the administration basics guide.
Before you begin installing and configuring the components described in this guide, please make sure you've followed our instructions for setting your hostname. Issue the following commands to make sure it is set properly:
hostname hostname -f
The first command should show your short hostname, and the second should show your fully qualified domain name (FQDN).
Before installing this software, you must enable the universe, multiverse, and restricted repositories. Your /etc/apt/sources.list file will resemble the following:
## main & restricted repositories deb http://us.archive.ubuntu.com/ubuntu/ lucid main restricted deb-src http://us.archive.ubuntu.com/ubuntu/ lucid main restricted deb http://security.ubuntu.com/ubuntu lucid-updates main restricted deb-src http://security.ubuntu.com/ubuntu lucid-updates main restricted deb http://security.ubuntu.com/ubuntu lucid-security main restricted deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted ## universe repositories - uncomment to enable deb http://us.archive.ubuntu.com/ubuntu/ lucid universe deb-src http://us.archive.ubuntu.com/ubuntu/ lucid universe deb http://us.archive.ubuntu.com/ubuntu/ lucid-updates universe deb-src http://us.archive.ubuntu.com/ubuntu/ lucid-updates universe deb http://security.ubuntu.com/ubuntu lucid-security universe deb-src http://security.ubuntu.com/ubuntu lucid-security universe ## multiverse repositories deb http://us.archive.ubuntu.com/ubuntu/ lucid multiverse deb-src http://us.archive.ubuntu.com/ubuntu/ lucid multiverse deb http://us.archive.ubuntu.com/ubuntu/ lucid-updates multiverse deb-src http://us.archive.ubuntu.com/ubuntu/ lucid-updates multiverse deb http://security.ubuntu.com/ubuntu lucid-security multiverse deb-src http://security.ubuntu.com/ubuntu lucid-security multiverse
Issue the following commands to ensure that your system's package database is brought up to date and that all installed software is running the latest released version:
apt-get update apt-get upgrade
Issue the following command to install all required software:
apt-get install apache2 tomcat6 apache2-mpm-prefork apache2-utils apache2.2-common autoconf memcached automake automake1.4 autotools-dev bzr defoma fontconfig-config libapache2-mod-php5 libapache2-mod-rpaf libapr1 libaprutil1 libdbd-mysql-perl libdbi-perl libfontconfig1 libfreetype6 libgd2-xpm libhtml-template-perl libjpeg62 libltdl7 libltdl7-dev libmysqlclient16 libnet-daemon-perl libplrpc-perl libpng12-0 libpq5 libssl-dev libt1-5 libtool libvarnish1 libxpm4 m4 mysql-common mysql-server mysql-client php-apc php5 php5-cli php5-curl php5-common php5-dev php5-gd php5-mysql php5-xmlrpc php-pear postfix shtool ssl-cert subversion ttf-dejavu ttf-dejavu-core ttf-dejavu-extra varnish zlib1g-dev bzr bzrtools php5-memcached
As your package management tool installs the required software, you will be prompted to answer two questions. First, if you have not already installed the MySQL database system, you will be asked for a MySQL root password. This is only the root password for the database and should be different from your system's root password. Second, you you will be asked to configure the Postfix email server. You want to select the "Internet Site" option and then insert the publicly resolvable unique name of the server. You may then proceed with the configuration of the remainder of this system.
Project Mercury isn't a single piece of software that you can install and configure with a single apt-get invocation, but rather a collection of a number of different tools that combine to provide the increases in performance and capability for Drupal sites. The following sections provide instructions for configuring these components.
For the purposes of this document we assume that your Drupal site will be hosted at the domain, example.com and that your virtual hosting file points to a DocumentRoot located at /srv/www/example.com/public_html/.
Drush is a command based interface for manipulating Drupal deployments on your system. Download and install the package by issuing the following commands:
cd /opt/ wget http://ftp.drupal.org/files/projects/drush-All-versions-3.0-beta1.tar.gz tar -xvzf drush-All-versions-3.0-beta1.tar.gz chmod +x /opt/drush/drush ln -s /opt/drush/drush /usr/bin/drush
This system uses the Apache HTTP server as an application server rather than as front end web server. Rather than responding to incoming requests on port 80, Varnish will respond to requests on port 80 and proxy these requests back to Apache, which will listen on port 9880 in this case. We need to reconfigure Apache to behave in this manner. First in /etc/apache2/ports.conf modify the Listen and the NameVirtualHost directives to specify port 9880 rather than 80, as in the following example:
NameVirtualHost *:9880 Listen 9880
Edit the /etc/apache2/sites-available/default file to modify the default Virtual Host to correspond to the virtual host specified in the NameVirtualHost directive. The first line of this file should resemble:
Issue the following command to enable the mod_rewrite module:
a2enmod rewrite /etc/init.d/apache2 restart
When you configure or reconfigure the virtual host for the domain where your Drupal site will be hosted, ensure that this virtual host is pointed to the proper port, i.e. 9880. You may not run any services on port 80 of the IP address where Varnish will be running.
Create a virtual host entry in your Apache configuration tree for the site you have created. Consider the following example:
File excerpt:Apache Virtual Host Configuration
<VirtualHost *:9880> ServerName example.com ServerAlias www.example.com ServerAdmin email@example.com DocumentRoot /srv/www/example.com/public_html ErrorLog /srv/www/example.com/logs/error.log CustomLog /srv/www/example.com/logs/access.log combined </VirtualHost>
Issue the following command to restart the web server so that all changes can take effect:
Because all external requests for your website in this system are handled by Varnish, we need to reconfigure Varnish to listen for requests on port 80. Begin by changing the value of the INSTANCE variable in the /etc/default/varnish file:
Edit the DAEMON_OPTS settings in the same file to respond to requests under port 80, and to modify the amount of RAM varnish will use. Consider the following example:
DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/pressflow.vcl \ -S /etc/varnish/secret \ -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,135MB"
Above, Varnish will use up to 135 MB of RAM for cache. Project Mercury recommends using 25% of available RAM for Varnish. Modify this number to reflect the amount of RAM you want to dedicate to Varnish. In general, you should give Varnish as much RAM as possible without impinging upon your system's ability to provide other services, including the database and other application services. As this balance may depend on a multitude of factors including load and use patterns, start with a conservative memory usage value and adjust as needed based on system load.
Additionally you will need to create pressflow.vcl file to allow Varnish and Drupal to be able to communicate properly.
Additionally, issue the following command to ensure that the required directory exists for Varnish's use and to restart the varnish web server:
mkdir -p /var/lib/varnish/pressflow/ /etc/init.d/varnish restart
Apache Solr is a fast search tool for supplying search services to applications like Drupal. We'll download the 1.4.0 release of Solr. Check the upstream to ensure that this is the latest release and update as needed.
cd /opt wget http://apache.osuosl.org/lucene/solr/1.4.0/apache-solr-1.4.0.tgz tar xvzf apache-solr-1.4.0.tgz mv apache-solr-1.4.0/example/solr /opt/ mv apache-solr-1.4.0/dist/apache-solr-1.4.0.war /opt/solr/solr.war chown -R tomcat6:root /opt/solr/
Connect Solr with Apache Tomcat 6 by creating the /etc/tomcat6/Catalina/localhost/solr.xml file with the following content:
<Context docBase="/opt/solr/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true"> <Environment name="solr/home" type="java.lang.String" value="/opt/solr" override="true" /> </Context>
APC, the "Alternative PHP Cache," is a bytecode cache for PHP that speeds the execution of PHP code by storing a byte-compiled version of your application in memory for faster execution. Add the following lines to your /etc/php5/conf.d/apc.ini file:
extension=apc.so apc.shm_size=256 apc.include_once_override = 1
APC can also be configured to not recheck already cached code, for faster implementation. This can provide a significant performance boost in production environments , but is not desirable for systems where the PHP code is changing regularly. If you wish to enable this setting, add the following value to the /etc/php5/conf.d/apc.ini file:
apc.apc.stat = 0
Project Mercury is based around a distribution of the Drupal content management system called "Pressflow," which is optimized to provide a Drupal system that is more resilient to higher demand situations. Issue the following sequence of commands. This will remove the entire contents of your DocumentRoot. Only issue this command sequence if there is nothing of worth in the /srv/www/example.com/public_html directory.
rm -rf /srv/www/example.com/public_html bzr branch lp:pressflow /srv/www/example.com/public_html echo "create database pressflow;" | mysql -u root -p
These commands clear the contents of the DocumentRoot, download a copy of Pressflow into the DocumentRoot using the Bazaar version control system, and create a database for the Pressflow site. Issue the following commands to create the default Pressflow site profile:
cd /srv/www/example.com/public_html/ mkdir /srv/www/example.com/public_html/sites/default/files mv /srv/www/example.com/public_html/profiles/default ./ rmdir /srv/www/example.com/public_html/profiles bzr branch lp:pantheon/profiles /srv/www/example.com/public_html/profiles/ mv default /srv/www/example.com/public_html/profiles
Use the drush tool to install the Varnish, Memcache, and related modules for Drupal, which will allow your Drupal instance to take full advantage of these services:
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules memcache-6.x-1.x-dev drush dl --destination=/srv/www/example.com/public_html/sites/all/modules varnish drush dl --destination=/srv/www/example.com/public_html/sites/all/modules cacherouter drush dl --destination=/srv/www/example.com/public_html/sites/all/modules apachesolr
Issue the following sequence of commands to install the Apache Solr module and ensure that the proper permissions have been set on your files:
cp /srv/www/example.com/public_html/sites/default/default.settings.php /srv/www/example.com/public_html/sites/default/settings.php mkdir -p /srv/www/example.com/public_html/sites/all/modules/ cd /srv/www/example.com/public_html/sites/all/modules/apachesolr/ svn checkout -r22 http://solr-php-client.googlecode.com/svn/trunk/ SolrPhpClient mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/schema.xml /opt/solr/conf/ mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/solrconfig.xml /opt/solr/conf/ chown -R root:www-data /srv/www/example.com/public_html/* chmod -R 775 /srv/www/example.com/public_html/sites chmod 755 /srv/www/example.com/public_html/sites/all/modules/
You may now visit your Drupal instance in your web browser at the address http://example.com:9880. Select the "Mercury" installation profile, and enter your database credentials at this juncture. When this process is complete, issue the following commands to modify the permissions of your Drupal files as required by the Drupal system:
chmod -R 755 /srv/www/example.com/public_html/sites/default chmod g+w /srv/www/example.com/public_html/sites/default/files
Once the database is configured, return to this document before beginning to use this Drupal instance as various components of Drupal and Mercury require additional configuration.
Add or edit the following values in the default settings.php for your Pressflow/Drupal instance, which will make it possible for your site to work with memcache and varnish.
# Varnish reverse proxy on localhost $conf['reverse_proxy'] = TRUE; $conf['reverse_proxy_addresses'] = array('127.0.0.1'); # Memcached configuration $conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc'; $conf['memcache_servers'] = array( '127.0.0.1:11211' => 'default', '127.0.0.1:11212' => 'menu', '127.0.0.1:11213' => 'filter', '127.0.0.1:11214' => 'form', '127.0.0.1:11215' => 'block', '127.0.0.1:11216' => 'update', '127.0.0.1:11217' => 'views', '127.0.0.1:11218' => 'content', '127.0.0.1:11219' => 'apachesolr', ); $conf['memcache_bins'] = array( 'cache' => 'default', 'cache_menu' => 'menu', 'cache_filter' => 'filter', 'cache_form' => 'form', 'cache_block' => 'block', 'cache_update' => 'update', 'cache_views' => 'views', 'cache_views_data' => 'views', 'cache_content' => 'content', 'cache_apachesolr' => 'apachesolr', );
You may enable all of the modules installed above using drush by issuing the following command:
cd /srv/www/example.com/public_html/ drush en apachesolr apachesolr_search search cacherouter
Issue the following drush commands to search settings for Apache Solr:
cd /srv/www/example.com/public_html/ drush vset --yes apachesolr_port 8080; drush vset --yes apachesolr_cron_limit 50; drush vset --yes apachesolr_search_make_default 1; drush vset --yes apachesolr_search_spellcheck 1; drush vset --yes search_cron_limit 50
Issue the following set of drush commands to optimize Drupal's caching and to allow it to interoperate with the Varnish cache:
cd /srv/www/example.com/public_html/ drush vset –yes cache 3; drush vset –yes cache_lifetime 0; drush vset –yes page_cache_max_age 600; drush vset –yes block_cache 1; drush vset –yes preprocess_css 1; drush vset –yes preprocess_js 1; drush vset –yes page_compression 0;
Finally, create the following cron job by placing the following line in a /etc/cron.d/drupal file:
0 * * * * root /usr/bin/wget -O - -q -t 1 http://example.com:9880/cron.php
This will allow Drupal to run a set of regular tasks every hour for search indexing and other important Drupal maintenance. You will want to restart Tomcat to make it possible for Drupal to communicate with the Solr instance by issuing the following command:
Congratulations! You have successfully installed and configured all of the components of a high performance Drupal setup! You may now visit your site at http://example.com to view and modify your site.
You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.
- Project Mercury
- BFCG2 Configuration Management Tool
- Project Mercury Launchapd Files
- Drupal Compatibility with PHP 5.3