High availability and scalability of fedora commons

HA and Scalability of Fedora Commons.

If you trying to run a media rich web site , it is better to separate serving media and static content such as audio,video, images ,css,js etc to a server other than your application server.

CDN is the solution for these kind of scaling issues .But cost of CDN is not cheap. if you do not need scaling across the globe/different locations, it is much better and cheaper to have a separate media server farm doing this work for you.

Fedora commons is a open source digital object repository server which we can use as the media server , the main uses of it being

  • Store and manage digital assets (audio,video files and images).
  • Serve images,css and other static content to web clients.

Unfortunately Fedora commons does not yet have a working clustering solution although it is a web application running inside a tomcat server.

So we built a scalable solution using a load balancer fronting a group of fedora tomcat servers which use the file system exposed through GlusterFS, an open source clustered storage solution from http://www.gluster.org/

GlusterFS server is started on 2 separate amazon ec2 instances to achieve high availability using separate volumes mounted on each server. Then this file system is exposed and mounted through GlusterFS client on each of the instances where Fedora Tomcat servers are running. Writing to the filesystem exposed through this GlusterFS client will replicate the data in both the GlusterFS Server volumes.

Scaling Fedora Commons using Tomcat cluster and GlusterFS involves

  • Set up GlusterFS server and clients
  • Set up fedora tomcat servers.
  • Update the Fedora code to support clustering.

1.GlusterFS setup

Environment : Ubuntu LTS, GlusterFS 3.0.2 – 2 GlusterFS servers for availability , n number of GlusterFS clients.

Refer to the following links to install GlusterFS

http://www.gluster.com/community/documentation/index.php/GlusterFS_on_Scalr/EC2

http://www.howtoforge.com/high-availability-storage-cluster-with-glusterfs-on-ubuntu-p2

http://www.gluster.com/community/documentation/index.php/Storage_Server_Installation_and_Configuration

After installation , log into the GlusterFS server instance.

Edit the glusterfs config file.

vi /etc/glusterfs/glusterfsd.vol

make sure you have similar entries as below

create and mount an EBS volume on each of the GlusterFS server instances.

ec2-create-volume –size 500

if you need 500 GB of data to be stored and served through GFS , for replication you need 2 volumes of 500 GB each.

Then attach and mount the newly created volume

ec2-attach-volume vol-id -i <your instance ID> -d /dev/sd<x>

/bin/mount -a

Start the GFS server on both the servers

/etc/init.d/glusterfs-server start

Log into glusterfs client instances .

Edit the glusterfs client config file

vi /etc/glusterfs/glusterfs.vol

make sure you have similar entries as below

volume gfsserver1

type protocol/client

option transport-type tcp

option remote-host ip-gfsserver-1

option transport.socket.nodelay on

option transport.remote-port 6996

option remote-subvolume brick1

option username user

option password pass

end-volume

volume gfsserver2

type protocol/client

option transport-type tcp

option remote-host ip-gfsserver-2

option transport.socket.nodelay on

option transport.remote-port 6996

option remote-subvolume brick1

option username user

option password pass

end-volume

volume mirror-0

type cluster/replicate

subvolumes gfsserver1 gfsserver2

end-volume

volume writebehind

type performance/write-behind

option cache-size 2048MB

subvolumes mirror-0

end-volume

volume readahead

type performance/read-ahead

option page-count 4

subvolumes writebehind

end-volume

volume iocache

type performance/io-cache

option cache-size `grep ‘MemTotal’ /proc/meminfo | awk ‘{print $2 * 0.2 / 1024}’ | cut -f1 -d.`MB

option cache-timeout 1

subvolumes readahead

end-volume

start the GFS client on all your Fedora Tomcat server instances.

/usr/sbin/glusterfs <fedora install dir>/data/datastreams (Start this after the next 2 steps , look into the next step on why this is needed)

2. Fedora setup

Read through http://www.fedora-commons.org/documentation/3.0b1/userdocs/distribution/installation.html for basic installation .

3. Update the fedora config and code

Edit config files to change the data store directory since we will be mounting a single replicated folder which will have both datastreams and objects folder in it .If you try to share the datasource directory as a whole , fedora complains of sharing violation during execution .so just share the data folders and everything else is left for the individual fedora instances to maintain.

vi /<fedora install dir>/server/config/fedora.fcfg

make sure you have similar settings as below.

<param name=”object_store_base” value=”data/datastreams/objects” isFilePath=”true”></param>

<param name=”datastream_store_base” value=”data/datastreams” isFilePath=”true”></param>

Now remove the folder datastreams and objects under /<fedora install dir>/data/ folder .

Start the glusterfs client to start sharing the folder

/usr/sbin/glusterfs <fedora install dir>/data/datastreams

Fedora also needs a temp directory when digital objects are uploaded into it. This folder needs to be on the share too , other wise load balanced requests will not be able to find the tmp uploaded file.

So remove the folder / <fedora install dir>/server/management/upload

Create a folder at /<fedora install dir>/data/datastreams/upload

create a link named upload to this shared folder

ln -s /<fedora install dir>/data/datastreams/upload/ upload

edit and compile DefaultManagement.java in fedora commons webapp in tomcat (you may need to download fedora sources if you do not already have)

/<fedora tomcat installation>/webapps/fedora/WEB-INF/classes/fedora/server/management/ DefaultManagement.class

In the getTempStream( method , we need to disable in memory checking for file exists .This is nto the best solution but works well. Comment out the foll line as shown

//if (m_uploadStartTime.get(internalId) != null)

if (1 == 1) {

// found… return inputstream

try {

return new FileInputStream(new File(m_tempDir, internalId));

} catch (Exception e) {

throw new StreamReadException(e.getMessage());

}

} else {

..

….

Do the same on all tomcat instances hosting fedora webapp and restart them , now whenever you create a digital object through the load balancer , no matter which server the request goes to , fedora will work fine and glusterfs will take care of replication and availability.

We can fire up as many fedora tomcats with glusterfs clients as we want to enable in auto scale.

About these ads

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: