Mercury on Ubuntu 10.04 (Lucid)

Install Drush, the Drupal Shell

Drush is a command based interface for manipulating Drupal deployments on your system. Download and install the package by issuing the following commands:

cd /opt/
wget http://ftp.drupal.org/files/projects/drush-All-versions-3.0-beta1.tar.gz
tar -xvzf drush-All-versions-3.0-beta1.tar.gz
chmod +x /opt/drush/drush
ln -s /opt/drush/drush /usr/bin/drush

Configure Apache

This system uses the Apache HTTP server as an application server rather than as front end web server. Rather than responding to incoming requests on port 80, Varnish will respond to requests on port 80 and proxy these requests back to Apache, which will listen on port 9880 in this case. We need to reconfigure Apache to behave in this manner. First in /etc/apache2/ports.conf modify the Listen and the NameVirtualHost directives to specify port 9880 rather than 80, as in the following example:

File excerpt:/etc/apache2/ports.conf

NameVirtualHost *:9880
Listen 9880

Edit the /etc/apache2/sites-available/default file to modify the default Virtual Host to correspond to the virtual host specified in the NameVirtualHost directive. The first line of this file should resemble:

File excerpt:/etc/apache2/site-available/default

<VirtualHost *:9880>

Issue the following command to enable the mod_rewrite module:

a2enmod rewrite
/etc/init.d/apache2 restart

When you configure or reconfigure the virtual host for the domain where your Drupal site will be hosted, ensure that this virtual host is pointed to the proper port, i.e. 9880. You may not run any services on port 80 of the IP address where Varnish will be running.

Create Apache Virtual Host

Create a virtual host entry in your Apache configuration tree for the site you have created. Consider the following example:

File excerpt:Apache Virtual Host Configuration

<VirtualHost *:9880>
   ServerName example.com
   ServerAlias www.example.com
   ServerAdmin webmaster@example.com

   DocumentRoot /srv/www/example.com/public_html

   ErrorLog /srv/www/example.com/logs/error.log
   CustomLog /srv/www/example.com/logs/access.log combined
</VirtualHost>

Issue the following command to restart the web server so that all changes can take effect:

/etc/init.d/apache2 restart

Configure Varnish

Because all external requests for your website in this system are handled by Varnish, we need to reconfigure Varnish to listen for requests on port 80. Begin by changing the value of the INSTANCE variable in the /etc/default/varnish file:

File excerpt:/etc/default/varnish-cache.org

INSTANCE=pressflow

Edit the DAEMON_OPTS settings in the same file to respond to requests under port 80, and to modify the amount of RAM varnish will use. Consider the following example:

File excerpt:/etc/default/varnish

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/pressflow.vcl \
             -S /etc/varnish/secret \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,135MB"

Above, Varnish will use up to 135 MB of RAM for cache. Project Mercury recommends using 25% of available RAM for Varnish. Modify this number to reflect the amount of RAM you want to dedicate to Varnish. In general, you should give Varnish as much RAM as possible without impinging upon your system's ability to provide other services, including the database and other application services. As this balance may depend on a multitude of factors including load and use patterns, start with a conservative memory usage value and adjust as needed based on system load.

Additionally you will need to create pressflow.vcl file to allow Varnish and Drupal to be able to communicate properly.

File:/etc/varnish/pressflow.vcl

backend default {
.host = "127.0.0.1";
.port = "9880";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}

sub vcl_recv {
  // Remove has_js and Google Analytics cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*",
"");

  // To users: if you have additional cookies being set by your system (e.g.
  // from a javascript analytics file or similar) you will need to add VCL
  // at this point to strip these cookies from the req object, otherwise
  // Varnish will not cache the response. This is safe for cookies that your
  // backed (Drupal) doesn't process.
  //
  // Again, the common example is an analytics or other Javascript add-on.
  // You should do this here, before the other cookie stuff, or by adding
  // to the regular-expression above.

  // Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
  // Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
  }

  // No varnish for install or update.php
  if (req.url ~ "install\.php|update\.php") {
    return (pass);
  }

  // Normalize the Accept-Encoding header
  // as per: http://varnish-cache.org/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
   } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;
    }
  }

  // Let's have a little grace
  set req.grace = 30s;

}

// Strip any cookies before an image/js/css is inserted into cache.
// Also: future-support for ESI.
// Disabled pending error resolution
// sub vcl_fetch {
//  if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {
//    unset obj.http.set-cookie;
//  }
//  esi;
// }

sub vcl_hash {
  if (req.http.Cookie) {
    set req.hash += req.http.Cookie;
  }
}

sub vcl_error {
  // Let's deliver a slightly more friedly error page.
  // You can customize this as you wish.
  set obj.http.Content-Type = "text/html; charset=utf-8";
  synthetic {"
  <?xml version="1.0" encoding="utf-8"?>
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
  <html>
    <head>
      <title>"} obj.status " " obj.response {"</title>
      <style type="text/css">
      #page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black;
background-color: #FFF;}
      p {margin-left:20px;}
      body {background-color: #DDD; margin: auto;}
      </style>
    </head>
    <body>
    <div id="page">
    <h1>Page Could Not Be Loaded</h1>
    <p>We're very sorry, but the page could not be loaded properly. This should be
fixed very soon, and we apologize for any inconvenience.</p>
    <hr />
    <h4>Debug Info:</h4>
    <pre>
Status: "} obj.status {"
Response: "} obj.response {"
XID: "} req.xid {"
</pre>
      <address><a href="http://www.varnish-cache.org/">Varnish</a></address>
      </div>
    </body>
   </html>
   "};
   return(deliver);
}

Additionally, issue the following command to ensure that the required directory exists for Varnish's use and to restart the varnish web server:

mkdir -p /var/lib/varnish/pressflow/
/etc/init.d/varnish restart

Install Apache Solr Search

Apache Solr is a fast search tool for supplying search services to applications like Drupal. We'll download the 1.4.0 release of Solr. Check the upstream to ensure that this is the latest release and update as needed.

cd /opt
wget http://apache.osuosl.org/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
tar xvzf apache-solr-1.4.0.tgz
mv apache-solr-1.4.0/example/solr /opt/
mv apache-solr-1.4.0/dist/apache-solr-1.4.0.war /opt/solr/solr.war
chown -R tomcat6:root /opt/solr/

Connect Solr with Apache Tomcat 6 by creating the /etc/tomcat6/Catalina/localhost/solr.xml file with the following content:

File:/etc/tomcat6/Catalina/localhost/solr.xml

<Context docBase="/opt/solr/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
        <Environment name="solr/home" type="java.lang.String" value="/opt/solr" override="true" />
</Context>

Download Drupal Modules with Drush

Use the drush tool to install the Varnish, Memcache, and related modules for Drupal, which will allow your Drupal instance to take full advantage of these services:

drush dl --destination=/srv/www/example.com/public_html/sites/all/modules memcache-6.x-1.x-dev
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules varnish
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules cacherouter
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules apachesolr

Issue the following sequence of commands to install the Apache Solr module and ensure that the proper permissions have been set on your files:

cp /srv/www/example.com/public_html/sites/default/default.settings.php /srv/www/example.com/public_html/sites/default/settings.php
mkdir -p /srv/www/example.com/public_html/sites/all/modules/
cd /srv/www/example.com/public_html/sites/all/modules/apachesolr/
svn checkout -r22 http://solr-php-client.googlecode.com/svn/trunk/ SolrPhpClient
mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/schema.xml /opt/solr/conf/
mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/solrconfig.xml /opt/solr/conf/
chown -R root:www-data /srv/www/example.com/public_html/*
chmod -R 775 /srv/www/example.com/public_html/sites
chmod 755 /srv/www/example.com/public_html/sites/all/modules/

Complete Drupal Installation Process

You may now visit your Drupal instance in your web browser at the address http://example.com:9880. Select the "Mercury" installation profile, and enter your database credentials at this juncture. When this process is complete, issue the following commands to modify the permissions of your Drupal files as required by the Drupal system:

chmod -R 755 /srv/www/example.com/public_html/sites/default
chmod g+w /srv/www/example.com/public_html/sites/default/files

Once the database is configured, return to this document before beginning to use this Drupal instance as various components of Drupal and Mercury require additional configuration.

Edit Drupal Settings File

Add or edit the following values in the default settings.php for your Pressflow/Drupal instance, which will make it possible for your site to work with memcache and varnish.

File excerpt:/srv/www/example.com/public_html/sites/default/settings.php

# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

# Memcached configuration
$conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
$conf['memcache_servers'] = array(
         '127.0.0.1:11211' => 'default',
         '127.0.0.1:11212' => 'menu',
         '127.0.0.1:11213' => 'filter',
         '127.0.0.1:11214' => 'form',
         '127.0.0.1:11215' => 'block',
         '127.0.0.1:11216' => 'update',
         '127.0.0.1:11217' => 'views',
         '127.0.0.1:11218' => 'content',
         '127.0.0.1:11219' => 'apachesolr',
       );
$conf['memcache_bins'] = array(
          'cache'        => 'default',
          'cache_menu'   => 'menu',
          'cache_filter' => 'filter',
          'cache_form'   => 'form',
          'cache_block'  => 'block',
          'cache_update' => 'update',
          'cache_views'  => 'views',
          'cache_views_data'  => 'views',
          'cache_content'  => 'content',
          'cache_apachesolr'  => 'apachesolr',
       );

Enable and Configure Drupal Modules with Drush

You may enable all of the modules installed above using drush by issuing the following command:

cd /srv/www/example.com/public_html/
drush en apachesolr apachesolr_search search cacherouter

Issue the following drush commands to search settings for Apache Solr:

cd /srv/www/example.com/public_html/
drush vset --yes apachesolr_port 8080;
drush vset --yes apachesolr_cron_limit 50;
drush vset --yes apachesolr_search_make_default 1;
drush vset --yes apachesolr_search_spellcheck 1;
drush vset --yes search_cron_limit 50

Issue the following set of drush commands to optimize Drupal's caching and to allow it to interoperate with the Varnish cache:

cd /srv/www/example.com/public_html/ drush vset –yes cache 3; drush vset –yes cache_lifetime 0; drush vset –yes page_cache_max_age 600; drush vset –yes block_cache 1; drush vset –yes preprocess_css 1; drush vset –yes preprocess_js 1; drush vset –yes page_compression 0;

Finally, create the following cron job by placing the following line in a /etc/cron.d/drupal file:

File:/etc/cron.d/drupal

0 * * * * root /usr/bin/wget -O - -q -t 1 http://example.com:9880/cron.php

This will allow Drupal to run a set of regular tasks every hour for search indexing and other important Drupal maintenance. You will want to restart Tomcat to make it possible for Drupal to communicate with the Solr instance by issuing the following command:

/etc/init.d/tomcat6 restart

Congratulations! You have successfully installed and configured all of the components of a high performance Drupal setup! You may now visit your site at http://example.com to view and modify your site.