Step by Step: Setting up an AWS ELB based load balanced Web Farm with RDS in a VPC

This post will cover how to set up an ELB based load balancer with multiple web servers and RDS within AWS and how to avoid the few gotchas along the way.

For keeping the basic steps simple, it is assumed that you have chosen the same availability zone in all the places where it is relevant. Setting up multi-availability zone farms will be the subject of another topic.

Setup the VPC

This setup will be done in the VPC section of AWS. 

a) Create a unique VPC for your website. Do not use an existing VPC unless you know for sure how it fits in with the existing VPC setup. Use a CIDR Block of say 10.0.0.0/16 which gives about 65K addresses for your VPC.
     Make sure you have the DNS Resolution and DNS Hostnames set to true, so that EC2 instances that get created with hostnames such as ip-10-0-1-56 get resolved to their correct IP.

b) Create the Internet Gateway for use by the VPC. This allows your VPC to connect to the Internet. Associate your Internet Gateway to the VPC created in (a).

c) Create a Security Group - this one will be for communications to the VPC and within the VPC. Name it appropriately such as MyWebsite-VPC-SecGroup. We will later create another security group for the Internet side of things. Assuming this is a LAMP/LEMP stack, you will probably have to allow

•  port 22 (from your IP(s) only)
•  80, 8080 (or similar if you are running a reverse proxy such as Varnish and Apache/Nginx on 8080)  from within the same security group only. Enter the Security Group name in the Source field.
•  3306 (assuming the Mysql/Aurora/MariaDB setup will use this port) from within the same security group only.

d) Create a Subnet. If you are going to use the Amazon RDS service you will need two subnets. Amazon currently forces you to use a different separate subnet for the RDS instance.

Note: - Now, when creating these subnets you have to be careful with the CIDR block chosen. If you create the first subnet with the same CIDR of 10.0.0.0/16 that you used for your VPC, you will have exhausted all the IPs for that VPC and will not be able to create another subnet that will need to be within the IP address range of your VPC but that doesn't overlap your first subnet. You will receive error messages similar to "CIDR block 10.0.1.0/24 overlaps with pre-existing CIDR block 10.0.0.0/16 from subnet-6afacb2c (10.0.0/16)" OR "10.0.0.2/16 is not within the range of 10.0.0.1/16". To avoid those,

• Subnet 1 - Create this subnet with a CIDR block of say 10.0.1.0/24 which gives you about 250 IPs.  This subnet will be used to launch the Web servers .
• Subnet 2 - Create this subnet with a CIDR block of say 10.0.2.0/24 which again gives you about 250 IPs. Make sure you create this subnet in a different availability zone (e.g. if the first one used ap-southeast-2a, then the second one would be ap-southeast-2b). This subnet will be required to launch the RDS instance even if you are putting your RDS instance in the same availability zone of Subnet 1.

The networking setup should now be complete. We can now head to the EC2 section of AWS.

Create the Web servers and Load Balancer (ELB)

In the EC2 section of AWS, now 

e) Create the first web server as per your server sizing with the whole stack and the website code setup. Now create an AMI image from this web server. Lets calls this WEB01. Add this as part of the VPC , Subnet 1 and MyWebsite-VPC-SecGroup.

f) Create the second web server from the AMI image created earlier. This is a quick way to get both the servers to be exactly alike. Lets calls this WEB02. Add this as part of the VPC,  Subnet 1 and MyWebsite-VPC-SecGroup. 

g) Spawn more web servers as per your sizing requirements essentially as all clones of the original.

h) Create a new security group. This security group will be for the internet facing traffic, lets call this MyWebsite-ELB-SecGroup. You will probably have to allow,

• Port 80 from the whole world
• Port 443 from the whole world , everything else can remain blocked.

i) Create the load balancer (ELB). 

• Setup port forwarding for HTTP and (if your site supports HTTPS) for HTTPS. Its probably a good idea for simple sites to terminate the HTTPS traffic at the load balancer and have the ELB communicate with the instances using HTTP only. In that case, you can have the 443 traffic forwarded to the port 80.
• Stickiness. Unless its required by the application (in cases where sessions are stored on file locally on the web server), its best to leave this turned off. This allows ELB to distribute the load evenly between the configured instances. 
• Setup the instances that will be part of this ELB. These will be the web servers created earlier.
• Make the ELB part of the same subnet (Subnet 1) as the web servers and the VPC.
• Make the ELB part of two security groups, the MyWebsite-VPC-SecGroup and the MyWebsite-ELB-SecGroup. The latter allows traffic into the ELB from the whole world, and the former allows the ELB to communicate with the instances in the VPC-SecGroup. Without this, you will not be able to have the ELB communicate with the VPC and secure the VPC at the same time.

The EC2 part of the setup is now done. The last remaining step is RDS.

Create the RDS instance

j) Create the RDS Instance.

• Based on your server sizing, pick an RDS instance size, database type (MariaDB/Aurora/Mysql) and version.
• Add to the VPC
• Select Publicly Accessible to No (most cases don't need the database to be publicly accessible)
• Make sure you select the MyWebsite-VPC-SecGroup and not the ELB security group
• Select the Subnet 2. 
• Make sure you select the same availability zone as your web servers and ELB (assuming you want them all in one).
• Deselect Multi-AZ (unless you need it, it costs more but gives you better availability). Note that the Subnet in a different availability zone is still needed if you don't need Multi-AZ and even if you don't plan to place your RDS instance in that different availability zone. 
• Your RDS instance is now created with two subnets. 

k) Now update your web server configuration to point to shiny new RDS instance and load up the database on RDS.

Your AWS setup with ELB, Web servers and RDS all in a VPC should now be ready for use.

In another post, we will cover multiple availability zone setups which are slightly more complicated and more expensive too.

Deploying a drupal site - Considerations and a handy checklist

If you are deploying a drupal site to production for the first time OR even doing it after a really long time, here are a few things you should probably consider:

Note:-
1. This page mentions Apache, MySQL and Linux but the principles apply to Nginx, PostGreSQL and other databases,  and other operating systems too.
2. Some of the advice is not Drupal specific but more to do with the environment in which Drupal runs. However, it is important to consider them all holistically to get the best setup for your site.

Security:

1. The first thing to do after deploying your drupal site is to change the $drupal_hash_salt in your settings.php. Never keep this the same as your other environments. In addition, it is best if you keep this in a file system outside of your docroot.

2. Check the status report. The status report is an excellent guide of mistakes in your deployment. A good module will also report here of any mis-configurations. Fix all the problems listed here especially updating modules that have security fixes.

3.  Make sure the permissions setup for the different roles in the site are appropriate. Since this list can be quite long, its best to scrutinize this carefully line by line. Especially, make sure you aren't granting more than is necessary for anonymous and other untrusted roles.

4. Remove all generic user accounts you may have created during testing. Make sure all active users on the site are named users you can track and audit actions of on the site.

5. Add a brute force module (such as login_security) to make sure your users logins aren't compromised. You may also consider TFA (albeit this module is still in Beta).

6. Set Error Message display to None. You wouldn't want to inadvertently disclose additional information of how your website has been built.

7. Ensure the docroot file/folder ownership (recursively) is setup to something other than the webserver user account. E.g. devops-admin:devops-users. Also change the permissions to something like 755 so the webuser can traverse through the folder structure. The two exceptions to this are the settings.php (and any files included within this file) which should be set to 444 and the sites/default/files (or equivalent as setup in the File System settings) where drupal stores files created and uploaded that require the webserver user write permissions (the same goes for the Private file system if you are using one).

8. If using the Private file system of drupal, then make sure its outside of the docroot.

9. If you are using Git on the production server, make sure you keep this outside of the docroot (by using the GIT_WORK_TREE environment variable) OR atleast block access to the folder and its contents through htaccess OR in the Apache config itself.

10. Ensure you have a firewall configured (such as by using tools such as ufw OR directly using iptables) to block everything except traffic you must allow in . Also make sure services such as mysql, redis aren't directly exposed to the internet and aren't listening on all ports if they don't need to.  In addition to the firewall on the server, make sure you are also using the firewall of your service provider such as Amazon EC2 Security Groups  effectively.

11. Setup SSL for the site. This is both good for security as well as SEO.

12. Update the server with the latest updates for the operating system, services such as mysql.

13. Create an application specific mysql user with a strong password and assign appropriate rights to the drupal db to this user only. Use this user in settings. php for your site.

Performance:

1. Assuming you have done everything you can to optimize your code itself (and adding appropriate indexes to your tables), turn on Drupal Page and Block caching if that is appropriate for your setup. Set the Minimum Cache Lifetime to a high enough number (depending on how often your site's content is refreshed) and the Expiration of Cached Pages setting (which tells proxy servers for example how long they can serve up stale pages).
In addition, turn on Aggregate/Compress CSS and Aggregate JS files too. Make sure you have tested your site with all these settings well in advance, because some of them may have unintended consequences and require you to rejig your code.

2. Turn off all modules that you aren't likely to need on production such as Devel, Views UI etc.

3. Turn off unnecessary extensions in PHP and Apache.

4. Consider the use of Varnish and one of  Redis, MongoDB or Memcached. These can seriously improve your site's performance if you get or expect a lot of traffic and your pages don't have to rebuilt afresh for every page request. Drupal has contributed modules for each of these that work fairly well with these software.

5. Turn on any additional drupal modules that you can benefit from based on the functionality available on your site. Some examples are Boost (to enable file based caching as opposed to Database caching of drupal), Entity cache, Varnish , Memcache API, AuthCache etc. A full list can be found at https://groups.drupal.org/node/21897. As always, test thoroughly before enabling on production.

6.  Make sure you are serving minified JS and CSS (some contrib modules help here but can be done without them too).

7. Make sure the images on your site are optimised to be the least size possible.

8. Test your site on https://developers.google.com/speed/pagespeed/insights/ to make sure Google sees your site as performant. Ideally, conduct this test while you have a close to actual load simulated on the site using tools such as Apache JMeter.

SEO: 

1. Make sure your robots.txt isn't blocking any css, javascript or images that Google and other search engines should be accessing. Check what Google can and cannot see based on your robots.txt configuration by testing it in the Google Webmaster Tools under Fetch as Google.

2. For the below points use tools to detect these such as the Moz Bar.

a. Make sure you don't have duplicate pages (pages with different urls but the same title). If you need to have them for some reason make sure you set up the canonical url to the one common url on both the duplicate pages. The drupal metatag module is excellent and useful in setting these up.

b. Ensure you have no duplicate page titles. This can happen especially when you have pages that are in a paginated series. Set the titles up to display the page no. too in such cases using drupal tokens. Also, if you can (this would likely involve writing code),  setup the meta tag rel="prev" and rel="next to the next and previous pages in the series. For paginated series, ensure you don't have a canonical url setup so google doesn't think all the pages have the same content. Also, if you have facets being used to filter content, make sure you aren't creating several paths for google to index that essentially don't add any further new pages with new content. This can be controlled in the robots.txt by disallowing certain page parameters so google doesn't needlessly traverse through every facet.

3. Setup meta descriptions, snippets, social tags (such a og:) appropriately for your pages.

4. Ensure if you are using both www. and the no www. version (naked domain) for your site,  you have the canonical url setup for one or the other consistently across all pages.

5. Use the excellent XML Sitemap module to generate a sitemap for your site. Ensure you only have pages in there that need to be indexed by Google. Check the URLs on the sitemap to make sure that you aren't pointing to default pages that Drupal generates (such as for taxonomy, all content items of any content type etc) that you haven't themed or don't want visitors to see directly. Add a 404 redirect in addition to such pages or add a noindex meta tag and ensure they aren't linked within any of your pages. A good tool to scan your website for such links is Xenu that can crawl your website's links.

6. Ensure that you have got all the 301 Redirects setup correctly if you are migrating over from an old site.

Monitoring:

1. Setup something like NewRelic or Pingdom to ping your website periodically with real user monitoring (RUM) setup. These give you useful alerts should your site go down for some reason OR is giving users a slow experience.

2. Setup server level monitoring using your cloud services provider's tools (such as Amazon EC2) or tools such as NewRelic agents, Munin, Nagios etc. to monitor disk space, disk i/o, cpu , network i/o, memory utilisation, service uptime etc. These can be useful to give you alerts of impending risks on your site.

Miscellaneous:

1. Ensure Site Information in Drupal is setup correctly with the Email address, 404 and 403 pages setup correctly.

2. Set Cron to run at an interval suitable for your website

3. Create SPF records for emails that you are going to use if your site sends out emails.

4. Add favicons for all devices. A useful tool for this is http://www.favicon.cc/?

5. Ensure you are using the correct SMTP server, third party service logins (e.g for AddThis, Google Tag Manager, 3rd party APIs used on the site etc) .

When you have run through this checklist, you should be quite confident of launching your new Drupal website.

 

6 skills of a highly effective digital developer

Digital transformation of the world has gathered pace and changed significantly in shape, size and character over the years. So have the skills required of a digital developer to be highly effective in this digital age changed? Let's take a look at the six key skills that a digital developer must have today.

1) Ready for change - Gone are the days when we built things as per spec and stuck to it. The world is changing so fast, competition is constantly gnawing at our heels , digital disruption by unknown startups are constantly shaking up the status quo and defining new behaviors for our customers. So the first skill that a developer building digital products must have is a preparedness for change. That feature you built yesterday is too old already. Your customer will need it changed constantly throughout the day, week, month and year to keep pace with competition and stay ahead. How can you be prepared for change?

First of, mentally to have the attitude to embrace constant change. After that, there are a few habits that should be part of your daily grind.

a) Writing tests and beautiful code. All your code have unit tests written, you practice test driven development, behavior driven development and other variations so when you make those changes there is a scientific way to know if your code still passes all the tests without breaking things and that you delivered what was expected. Since your code is changing thick and fast you are writing beautiful code so those changes can be made fast and confidently. You also measure test coverage, adherence to best practices and standards and technical debt through the use of tools such as SonarQube .
b) Continuous Integration and Deployment. You use tools such as Jenkins, Bamboo , Go , Docker, Ansible to ensure changes to the digital product are made and delivered fast and safely even several times a day to multiple servers and environments.
c) Agile and Lean mindset -  You have a strong agile and lean mindset. This means that you truly believe in their principles, but also you are a master at doing things such as A/B Testing,  using Kanban boards effectively,  comfortable pair programming,  participating and conducting team retrospectives, capturing and using process metrics, planning sprints and releases, using version control effectively, practicing lean startup principles or asking of your product managers that they do.

2) Full stack hacker - You are a full stack developer. What this means really is that you are:

a) very comfortable with HTML5/CSS/(LESS or SASS)/(AMP?)
b) a guru in Javascript/jQuery/AngularJS,
c) a master of atleast three mature languages used at the server side (such as Java, C#, PHP, C#, Ruby, Python, Objective C) ,
d) good with atleast one modern programming language (such as Scala, Go) and
e) a guru with SQL.
You are also comfortable with
f) a few different IDEs,
g) a couple of version control systems and
h) atleast a couple of different operating environments (iOS, Linux, Windows).

In a nutshell, you can build a digital product all on your own from scratch be it a responsive website or an app or a data crunching algorithm that delivers superior analysis (more on this later) to your customer.  Why must you be a full stack developer and not just a master of one or two languages? This has a lot to do with #1 above. When you are comfortable with the full stack used in a solution, you can design better solutions, make changes faster and thus respond to change faster and better and be more effective in a team.

3) Data technologies - The era of data is here like we have never seen before. Banks, Retailers, Social Networks and apps and everyone else these days crunch enormous amounts of data to hone their products and services. So processing enormous amounts of data and deriving intelligence from them is a skill that you should probably possess to build great digital products. What does this mean in terms of a skillset?

a) Domain knowledge. You know a couple of industries really well. To understand data you need to understand the business. Apart from that, of the list below from ETL Tools to DW/OLAP you have worked with at least one category of tools/technologies.

b) ETL Tools. You are comfortable using tools such as Pentaho Data Integration or Oracle Data Integrator. Data almost never is in a ready to use form or all in one place or nice and clean, so you need these to put them together.
c) Data science. You know a few well-known classification, class probability estimation, regression, clustering, profiling etc. machine learning algorithms and how to use them, when to use which one, how to tune them, building models, how to choose attributes, how to preprocess data and have an intelligent conversation with a machine learning expert. You also have the necessary mathematics and statistics knowledge to understand how they work. This may not necessarily mean that you are an expert here, but you should be able to straddle the worlds of machine learning, programming and industry knowledge to derive useful insights.
d) Big Data. You have worked with "Big data" with tools such as Hadoop, Spark or the Elastic stack.
e) Visualisation. You are comfortable with the use of visualisation tools (such as Tableau, MatPlotLib) to show the results of your data processing.
f) NoSQL. You can go beyond RDBMS solutions and use products such as Redis, MongoDB, Neo4j in storing and asking questions in an efficient way of your data.
g) DW and OLAP. You can build data warehouses, are comforable with star schemas and OLAP tools.

4) Social, SEO and SEM - You are familiar with a range of social APIs. Facebook, Twitter, Instagram, Pinterest, Google social APIs, apps and networks are of second nature to you. You use good SEO practices while developing websites. Not only do you have an excellent understanding of how to measure user behavior, but you can use various tools such as Google Tag Manager, Google Analytics, Facebook Analytics, Twitter Analytics comfortably to measure everything about your digital product and change strategies to improve outcomes.

5)  Cloud -  You are at least comfortable with one cloud infrastructure provider such as Amazon AWS or Microsoft Azure and have a reasonably good understanding of their full service offering , the differences with similar such on-premise services and have hands-on experience with a few of their cloud services. Apart from that there are several other providers of various services. Their services include A/B Testing, Real User Monitoring, CDNs, Reverse Proxy services, Security services and so on. Familarity with a few popular ones such as CloudFlare, Incapsula, Optimizely, Pingdom etc. will definitely be useful.

6)  Code -  You write secure, performant, scalable, accessibility compliant and beautiful code. Most digital products/services these days store store some kind of personal data and its very important to protect it with sound secure coding practices. If you ended up building a popular digital product, then performance and scalability of  your code is paramount. Its important to write accessibility standards compliant code (levels depend on what you are building and for whom) so we don't neglect or ignore the less able. Finally, your code is beautiful and can be maintained well by people after you.

Too much to ask for? Missed something? Please leave your pithy comments.

Setting up openLDAP on Red Hat Linux Enterprise 5

1. Install the openldap, openldap-servers and openldap-clients packages by running "yum install openldap", "yum install openldap-servers" "yum install openldap-clients"

2. Follow instructions at http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Deployment_Guide/s2-ldap-files-slapd-conf.html

3. Run "/sbin/service ldap start"

4. If you get an error message similar to Starting slapd: /bin/bash: /tmp/start-slapd.h Permission denied then do the following:

In the /etc/init.d/ldap file on the line that reads:

wrapper=`mktemp ${TMP:-/tmp}/start-slapd.XXXXXX`

change the name of the directory to /var/tmp or /opt/openldap/tmp

i.e. wrapper=`mktemp ${TMP:-/var/tmp}/start-slapd.XXXXXX`

The reason this change is necessary is because on the machine where this error occurs the /tmp directory has been barred from running scripts. This may happen to /var/tmp also in which case you may need to set it up elsewhere like I did on /opt/

How to setup: Remote desktop of Red Hat Linux on Windows via SSH and Xming

I was attempting to install Autonomy IDOL Server 7 on a Red Hat Enterprise Linux (RHEL) box through SSH (Putty) when the installer asked for an X11 window to launch the installer. This server was setup in a hosted environment and it didn't have VNC Server setup and the only access I had to that server was through SSH.

So I decided to get a remote desktop (the X11 window of the RHEL server) onto my Windows XP desktop. With some help from various folks on the internet, I got it setup using the following steps.

Note:- You need to do most of the steps on the RHEL server as root.

1. I needed a X11 Windowing client for Windows XP. I found XMing in the open source world and installed Xming 6.9.0.31 from http://sourceforge.net/project/showfiles.php?group_id=156984

2. I opened the ssh_config file on the RHEL server and made changes to have the following entries and saved the file:
a. "Host *"
b. "ForwardAgent yes"
c. "ForwardX11 yes"
d. "ForwardX11Trusted yes"

3. Opened sshd_config file on the RHEL server and made changes to have the following entries and saved the file:

a. "X11Forwarding yes"

4. Installed xauth by running the "yum install xauth" command in the RHEL server. This asked if it could download and install a particular package for which I said yes.

5. I restarted the sshd service by running "service sshd --full-restart" on the RHEL server.

The server configuration for X11 Window forwarding was completed.

6. At putty on the Windows box, I made the following changes:
For that particular server in Putty, I went to Connection> SSH>Tunnels>
a. Checked the "Enable X11 Forwarding"
b. Entered "localhost:0" as the X Display location
c. The MIT-Magic-Cookie-1 was left selected by default

Saved the configuration for that server and connected to the server through Putty. This time when I launched the Autonomy IDOL server installer and it in turn launched a GUI based screen it was shown in my XMing window (you need to uncheck Hide Root Window in the Xming icon in the taskbar).

Contents of the ASP.NET Reference Architecture (or Framework)

We have completed building the ASP.NET Reference Architecture or Framework and here are the details of what version 1.0 contains.

The figure below depicts the various components of the framework.

Framework Architecture
The framework has been styled as an n-layer design.

1. The Presentation Layer – This layer uses base classes for the page, control and master page. The base classes implement features such as localization and themes. The presentation layer also includes:
• Anti-cross site scripting libraries
• URL rewriting (“pretty URLs”) libraries
• Dirty word filtering libraries
• Navigation libraries
• Custom error handling pages
• Javascript libraries
• The framework’s user interface pages (including pages for the tools available in the framework, administration etc)
• CAPTCHA for preventing bots
• Custom user controls
• Service references to call into the service layer
2. The Business Layer and Business Layer Host – The business layer, modeled completely as a service layer uses WCF/.Net 3.0. The Business Layer Host hosts all the services in the business layer in IIS as web services. The framework allows changing the service host and service type via WCF. The business layer provides the following:
• A Business Rules Engine (BRE) that allows authorized business users to add/modify/delete business rules in the system using a friendly user interface (the BRE editor). The BRE allows for writing complex business logic outside of code using the BRE editor. Thus, when business rules change business users with little or no assistance from developers could modify them in the system.
• Base Classes for creating lists, entity objects etc.
• Authentication/Authorization Classes available in the framework make setting up the application’s security simpler.

3. The Data Access Layer (DAL) - This layer isolates database access and any data related activity such as reading, writing, deleting from the web application and business layer. This layer provides classes to create database connections, access stored procedures, run queries etc. The DAL currently supports MS SQL Server data access, but can be extended to support Oracle, MySql etc

Tools
The framework provides various tools useful to different stakeholders of the application including administrators, project managers, developers, QA engineers addressing some common supportability requirements of most applications. Tools included in the framework include:

• SQL Data Population utility – This web-based utility allows the project team to insert large amounts of meaningful data into the application’s database(s). Doing this on day one will ensure that the project team detects any load related performance problems early in the project. This is also useful to QA teams who can use this utility to generate test data.

• Performance Monitors – Two tools monitor performance in the framework. The first tool displays various useful statistics of the web server including CPU usage, memory usage, no. of requests, queued requests etc. This tool is useful in production environments to administrators and developers, who do not have access to the server directly, yet are able to monitor the server’s behavior. The second tool displays useful statistics about the application such as average no. of calls made to a method per second, total no. of calls made to a method, total no. of exceptions, no. of exceptions per second, the duration of calls made to a method etc. This tool is useful in production environments to developers in diagnosing problem areas in the application.

• Application Dependency Checker – This web-based tool displays the health of all the various dependencies of the application such as web services, FTP servers, DB servers, HTTP servers. It shows whether the dependent service/server is up or not. It also displays the version numbers of assemblies (DLLs) used by the application. This utility is useful to developers and administrators to debug problems in the application quickly.

• Exception Report – This report displays the exceptions generated by the application in a given time period. This tool is useful to administrators and developers in analyzing errors that are occurring in the application.

• Health Monitor – The health monitor tool displays useful information about the application for a particular day such as heartbeat events (including when the application was stopped/started), application request events (the requests the application received), audit events (when authorization/authentication succeeded/failed) and management events.

• Site Usage Report – This tool is useful to business users and developers in understanding the usage of the web application. This report provides information such as No. of requests received in an hour, No. of bytes sent in a hour, Most frequently used pages/sections, clients that have been using the application, The types of browsers that are using the application and HTTP errors that were generated by the application.

• Secure Config – This tool can be used to encrypt certain sections of the web.config file by administrators in order to prevent anyone with access to the file from gathering critical information such as userids/passwords, server names etc.

• User/Group/Permissions Admin Tool – This tool allows authorized personnel to add new users to the application, add users to certain groups, and assign permissions to the user or group.

• Database Tools – There are two tools in the framework to monitor the database(s) used by the application. A Database Growth Report shows the percentage growth in the application’s database and specific tables that are growing at a rapid rate. This report is useful at predicting and preventing future performance problems. The Database Details Report shows various details about a database such as no. of SPs in the database, no. of tables, the tables in the database and their size and other useful information. This report is useful to developers who do not have access to production database environments and need this information to diagnose problems.

• Business Rules Editor – The Business Rules Editor is the user interface to the Business Rules Engine. The Editor is useful to business users who wish to change business rules and logic applied in the application.


Developer Utilities
The framework contains numerous classes useful for carrying out various common tasks during development of the application. These include:

• Caching - The caching classes provide developers with the ability to cache frequently used data objects and refresh the cache. This reduces redundant calls to the database, improving the performance of the application.
• Exception Handling - The exception handling classes provide developers with a standardized way in which to manage exceptions.
• Logging – The logging classes provide developers with a standardized way in which to log various types of messages in the application.
• Security (Encryption, Password) - The Security classes provide developers with various encryption algorithms, password hash creation methods and random password generators.
• Health Monitoring - The heath monitoring classes provide methods to create various performance counters related to the system (such as CPU usage) and the application (such as no. of calls made to a method).
• Xml Data Utility - The XML data utility class provides easy to use methods for reading and manipulating XML data.
• Others - FTP classes, Date Time formatters, Enum creators, various formatter classes, Config readers, Reflection classes, a generic HTML class, Zip libraries etc. are also included as utility classes.

Visual Studio Enhancements

The framework provides its own
• Templates - Custom versions of frequently used templates in Visual Studio such as the user control, ASPX page, Master Page and the project template. These custom templates provide several benefits to developers including standardization in the way code is written and lesser repetitive work for developers.
• Code Snippets – Code snippets are provided for frequently written code fragments, to hasten the development process
• Setup/Install Package – The framework includes a project that allows for creation of an installer and changing certain configuration settings of the framework depending on the environment (QA, production etc.) installed in. This makes for a smoother deployment of applications built with the framework.

Framework Building Blocks
The framework uses the following key components (among others):
• .NET 2.0 framework – This provides the execution runtime for the framework and class libraries for various functions based on which the framework is built
• ASP.NET 2.0 – This provides the core infrastructure and libraries for running the website along with IIS.
• ASP.NET Role & Membership providers – The Authentication, Authorization, and Role Based Access Control components in the framework use these providers.
• ASP.NET AJAX – The framework uses ASP.NET AJAX to provide a rich user experience, where appropriate.
• ASP.Net Health Monitoring – This provides the core infrastructure for the health-monitoring component in the framework.
• Microsoft Enterprise Library – The framework uses the Logging, Exception, Data Access and Policy Injection blocks to build various components of the framework.
• ADO.NET – The data access layer of the framework uses ADO.NET.
• WCF and .NET 3.0 – The framework’s service layer use these two components.
• Silverlight and Visifire – The Business Rules Engine’s user interface uses Silverlight, whereas some of the reports in the framework use Visifire.
• Microsoft Anti-XSS library – The framework includes the Microsoft Anti-XSS library that provides methods to prevent Cross-Site Scripting attacks.
• SharpZipLib – The framework includes the GPL licensed SharpZipLib to provide methods for compression and decompression.
• Visual Studio 2005/2008 – The framework uses Visual Studio 2005 and 2008 editions as the IDE tools.

Version 2.0 of the framework will look at inclusion of Workflow Foundation, ADO.NET Entity Framework, ASP.NET MVC and others as appropriate.

WSS 3.0 Search Web Service returning fewer results

It appears that the WSS (Windows Sharepoint Services 3.0) Search Index/Database doesn't update rights/permissions given to new users of documents already indexed.

A colleague presented a strange problem where he was attempting to use the search web service (spsearch.asmx) to search certain terms. He was providing the service, credentials of a site collection administrator because he needed all the search results without any permission restrictions. The site was operational for a few days and the search working fine and then he added a new site collection administrator. This administrator was also a local server administrator on the server where the service was installed (probably doesn't make a difference, but anyway). Strangely, to this new administrator's credentials the search web service was returning results as if the user was a reader.

We looked around the internet not finding any solutions. However, I suddenly remembered something about IFILTERs. If you install an IFILTER (say for PDF) after you have uploaded some 100 pdfs, those 100 pdfs are not indexed ever and the quick and dirty solution was to recreated the index. I just thought this might be a similar situation. The older documents in the index were not updated of the fact that this new user was a site collection administrator with access to those documents. So we did the following to try and solve the issue:

1. Goto Operations in Central Administration
2. Goto Services On Server
3. Stop the WSS Search service. This will delete all the index files
4. Now setup a database by clicking the WSS Search Service link
5. Restart the search service.

It rebuilt the index and this time the search web service was able to receive results uploaded before the new administrator was added.

Reference Architecture (or framework) for building ASP.NET applications

The software services industry and IT shops in general frequently grapple with three recurring problems

1. Similar applications are built repeatedly without getting the full benefit of the learnings, best practices, tools from other projects within the company/industry.

2. Enforcing good practices on developers who are increasingly inexperienced but tasked with important projects has become difficult.

3. Customers need and expect many supportability features (see the supportability blog post) to be standard from an application and are often unsaid in requirements documents.

To solve these three problems - reuse/knowledge sharing, enforcing good practices and providing a set of standard out-of-the-box features, a reference architecture or a framework could be built for different types of applications - web applications, mobile applications, office applications, windows forms applications etc.

Once such a framework is built future applications built by the company can utilize this framework to get applications that are guaranteed with good quality code (best practices are followed), guaranteed with a standard set of useful features regardless of the domain and allows tools, utilities, techniques of other projects within the company/industry to be utilizied by the project team. Such a framework has many benefits including reduction in development time, reduction in maintenance effort expended on the application after it has gone live (because this follows good practices and hence is robust) and provides useful features that come free.

Of course, the framework itself must be made very easy to use (with the help of wizards, add-ins, VS 2005 item templates and project templates). The ultimate vision for such a framework is to have developers strictly only build UI, their business classes, the DB schema and queries. Everything else should be taken care of by the framework.

An example of what such a framework may contain for an ASP.NET application is provided in the mind map shown here. Will explain this mind map in another post.

Solving the Change Request problem

Problem Statement:

1) How do we minimize the no. of change requests during development of an application?

2) How do we reduce the friction between software service vendors and customers on what constitutes a change?

Two ways to solve the problem

Solution 1: We will give you as many change requests as the budget permits – Change requests are a contentious issue because the vendor thinks he is spending from his budget and the customer from his. Planning a change request budget right at the beginning and making it transparent will assuage this issue. That way the money, schedule and effort variance is accounted for upfront. The plan is no longer a fixed one, but a fixed one + anticipated changes. This gives breathing room for both the vendor as well as the customer during the execution of the project.

Solution 2: We will show you the working application once every week – Change requests are a contentious issue because we deliver much too late in the game, when the business has changed or better ideas have evolved after looking at the actual software. Frequently showing a working demo of the application will go a long way in minimizing change requests. Even changes that come are easily accommodated during to the still evolving nature of the application.

How do we implement Solution 1?

Arrive at the requirement change index. A collaborative
process where Vendor assigns a requirement clarity index (VC Index – Vendor Confidence Index) to each requirement. The higher the clarity/confidence index, the clearer it is for Vendor to implement it. This is Vendor’s confidence in the way the requirement is defined/stated.

1. The customer assigns a requirement importance index (RI
   Index). The higher the index, the more important the requirement is.


2. The customer assigns a “I Know How To Implement It” index
   (IKHTII Index) to each requirement. This is the customer saying that they have already visualized how this requirement should be implemented. The higher this index, the better the feature’s visualization in the customer’s mind.


3. The customer assigns “Likely To Change” index (LTNC Index)
   to each requirement. This is the customer assigning a probability of some requirement not changing in the future after its features have been defined. The higher the index, the more likely the customer thinks this feature will go through changes after the first implementation.

All these indices are calculated on a scale of 1 to 5.

At the RFP stage, only (a) is used to arrive at the change
request budget, if the customer refuses to do (b), (c) and (d). So, at the RFP stage the vendor gives a project estimate (fixed bid) + (change request budget) that the customer should account for right upfront. The change request budget may or may not be used; it is a reserve budget for the project.

At the contract stage, the change request budget becomes part of the contract. The vendor does not put any buffers (most of the times these are gut-feel arbitrary ones which is why the problems start in the first place) in the estimate because of this safety net that the customer has accounted for.

During PDW, (b), (c) and (d) must be completed. Based on (a), (b), (c) and (d), we arrive at a change index (CI). This will determine the change request budget assigned at the beginning. This budget will be in addition to the initial project estimate made by Vendor and accepted by the customer.

An illustration of this concept is provided in the table below:

The Total CI is calculated using the formula:

((6-VC)/5)*(RI/5)*((6-IKHTII)/5)*(LTC/5)

The change budget is calculated using the formula:

Effort Estimate to Implement Feature X Total CI

Thus, we are now getting into the project knowing or anticipating a certain degree of change that is planned. The factor of change in a project is now a scientific guess and planned.

How do we implement Solution 2?

The most common reason given by development teams to refuse starting work until there is a concrete functional specification and/or technical design is that the application needs a framework to build on. The other is of course that the requirements are not clear and the fear of requirement change without a proper document (we have addressed this second part in Solution 1). This framework is frequently required to define a standard way of exception handling, logging, authorization, authentication, auditing, data access/ORM, handling business rules and objects etc. However, these are most frequently repetitive in similar projects. If there was a readily available framework (or a reference architecture) for building a particular
class of application (say web-based applications), then the development team could directly get down to implementing features for the requirements rather than waiting for some functional specification being built and the technical design being completed.

Sure enough, this may lead to some problems later on because
we are using a generic framework that may or may not have fully addressed the unique requirements of the application being built. For this, we need to have a refactoring budget again planned.

Again, we could calculate a refactoring index right at the beginning of the project. This is the vendor saying given a requirement, how much of the implementation is already existent in the framework. The higher this index, the more the refactoring required to implement the requirement.

An illustration is given below:

The Refactoring Budget is calculated using the formula:

(RI/5)*Effort Estimate

Note that the refactoring budget is necessary to be calculated when we are using aggressive iterative methodologies that do minimal requirements planning/functional specification writing, but instead relying heavily on directly getting to develop the application in week 1 or 2 of the project. This refactoring budget may not be necessary for traditional waterfall models.

Ways to integrate with iSeries applications and data

I started researching methods to integrate an ASP.NET application with iSeries applications and data and found the following ways/products. This list is still in its infancy and this is not a complete evaluation just some initial observations.

1 Send CSV files and receive CSV files from iSeries
- Minimal cost
- Cumbersome
- Lot of hand-written code, hence difficult to maintain
- Takes time to implement

2 ASNA DataGate record-level access
- Has a trial offer
- iSeries native method of accessing data - record-level access

3 LookSoftware's soarchitect Product
- Claims to create web services from existing iSeries programs
- Has a trial offer

4 IBM's iSeries Access for Windows
- Doesn't support .NET 2.0 as yet.

5 Jacada HostFuse
- J2EE/JCA solution only.

6 Attunity Connect & Stream
- Connect allows querying DB2/400 data
- Allows creating web services from existing RPG programs
- Stream monitors changes and propagates it

7 RJS Software Systems iSeries data access provider
- Not looked at yet.

8 Arbutus Software
-Arbutus does not have write back capabilities. Only one way supply into ASP.NET from iSeries

Outcome based contracts and Pricing in Software Services

Outcome based contracts in software services are a little difficult to implement. One way that outcome based contracts can possibly be arrived at that benefits both the customer and the vendor is as follows:

Let the contract have two components - A fixed component and a variable component. The fixed component will guarantee payment of costs incurred (manpower, hw/sw, standard operations costs per head etc) no matter what the outcome. Here the vendor shouldn't add any margins. Thus, the vendor's expenditure is atleast guaranteed to be reimbursed for services delivered. This is the extent of risk (a no profit, no loss risk) that the vendor should pledge to take .

The variable component can be the outcome based contract. The outcomes that we would like to measure will be different for different types of projects and should be arrived at in conjunction with the customer. Again here, there can be two parts. Part 1 will contain a set of outcomes that the vendor will guarantee to deliver and if delivered will receive certain compensation. Part 2 will contain outcomes that are directly related to the business of the customer and will most likely be defined by the customer.

Part 1 outcomes will be in the realm of good software development. These should be defined in terms of features that the vendor will give as part of the application regardless of whether these features were defined in the requirements or not. These will be of the nature mentioned in the post http://sharathshenoy.blogspot.com/2008/02/supportability-transform-making-non.html
. This is the vendor guaranteeing to build good quality software.

Part 2 outcomes will typically be defined by the customer. These could be things like:

1. %age increase in sales (for an e-commerce site),
   no. of times a utility built is downloaded (such as the Vista widget that was built for Move),
2. time taken (this is not page response time, but the ease of use/lead time for business users to see new copy they created to go live) to upload content and get it live vs. how long it used to take earlier (for CMS projects),
3. reduction in application support costs, improvement in SLAs to fix issues, reduction in rate of bugs arrived (for application management projects) etc.

This is just the beginning of a thought, obviously this has to be detailed well and remove as much ambiguity as possible.

Software Oriented Services

Before we begin
Throughout this note, the term “service” or “services” are NOT used as executable pieces of business logic (as used in SOA), but used in the economics sense – a set of benefits delivered by the service provider to the customer by supplying some level of skill, ingenuity and experience. Services and service offerings are used interchangeably in the note.

IT Challenges
IT Departments and CTOs (the “customer”) face two common challenges today when executing software projects, application and software maintenance services via Systems Integrators (SI). These are:

• Lack of predictability in costs – This has various reasons when executing an application development project including scope creep, poor quality (leading to rework and hence higher costs), improper planning etc. When it comes to application maintenance services it could be due to peaks and troughs in the application’s defects lifecycle (leading to over utilized/underutilized resources, delays in implementation etc.)
• Lack of predictability in quality of the software – This is another major concern that is usually the result of lack of skilled resources, poor defect detection and prevention during early phases of the lifecycle, lack of clear definition of requirements/acceptance criteria etc.

What are Software Oriented Services?
Software Oriented Services (SOS) are services that adhere to an execution framework that attempt to assuage these two problems and bring in better predictability in executing application development projects and provide better application and software maintenance services to the end customer.

At the heart of it, the fundamental principle behind SOS is simple. The SI first needs to identify what it is really good at based on factors such as past experience, knowledge base built on, skills available and ready-made software components available. It then defines a set of services based on what has been identified that the SI can confidently claim to be repeatable (in terms of quality and cost) and hence predictable. These can then be introduced as specialized services that the SI introduces to the market with the assurance of predictability backed by guarantees to the customer in the way of Service Level Agreements (SLA), result-oriented contracts and other such means. The reason why they are called SOS and not just services is to emphasise the fact that there is a software component to the service that requires specialized knowledge/skill of the software and that they are not purely business process oriented thus differentiating them from traditional BPO services.

The idea is to provide the kind of predictability that traditional BPOs and IT Infrastructure management services have been able to offer in terms of quality and cost to the service provided but in the software realm.

While SIs offer many services, the SOS offerings will be always be the ones that the SI can guarantee a degree of predictability.

To explain this better, let us take an example from the consumer electronics products industry. It is like the difference in buying a Samsung LCD TV and buying a Samsung Home Theatre in a Box (HTIB). It is common knowledge that Samsung makes great LCD TVs and there is an assurance of predictable and guaranteed quality when you buy one (costs are always predictable because they are products, so these are not mentioned). However, when it comes to HTIBs Samsung hasn’t made its mark yet (as compared to say an Onkyo) and you may find some chinks in the products under this range. Yet, Samsung’s goal would be to quickly achieve the kind of name it has been in LCD TVs, with HTIBs too and because it is in the same consumer electronics area in which it has vast experience and knowledge, eventually this should be possible. In this case, Samsung’s LCD TVs can now be classified as a SOS offering (ie if LCD TVs were a software service offering) and HTIBs as another service offering but one that hasn’t yet reached the SOS stage.

Just because a service hasn’t been classified as an SOS offering, doesn’t mean it isn’t good enough for consumption. It just means it hasn’t reached the level of predictability that say the customer would like. The service offering may still be a viable one such as the HTIB case in Samsung otherwise the SI wouldn’t bother spending time and money in introducing that service to the market.

Salient Features of SOS
Now that we have looked at the definition of SOS, let us look at what characterizes SOS.

• The first and foremost feature of a SOS offering is that its cost and quality are predictable to a high degree of accuracy. This definition may take the nature of a clearly defined SLA, a result-oriented contract or clearly defined acceptance criteria.
• The second feature of a SOS offering is that it could be offered to the customer by the SI in one of two ways
  

1. As a dedicated service – In this model, the hardware/software infrastructure and the personnel are dedicated to the customer and cannot work on or used for other customers. This model increases the cost for the end customer, but offers better security of data for instance.
2. As a shared service – In this model, some or all of the same hardware, software, and personnel will be used to provide similar services to multiple customers. This model reduces the cost for the end customer and should be used when security of data is less of a concern.

• The third feature of an SOS offering is that it has some software component attached with it. This could be software that the SI built (its intellectual property) or some software tools and products that the SI has gained mastery on and can confidently offer services around them as an SOS offering.

Benefits of SOS
Because an SOS offering bring in an increased degree of predictability in cost/quality and penalties/rewards for wavering beyond what is defined, customers are allowed to better plan IT budgets and are also delighted with the services provided.

SOA vs. SOS
Finally, since what has been written here also starts with a “Software-Oriented”, it is worthwhile differentiating it from SOA. SOS offerings are an execution philosophy, whereas SOA is an architecture philosophy. Service in SOA means an executable piece of business logic; service in SOS is the definition of a service given at the beginning of the note. SOS maybe applied where SOA is used, but SOA need not be applied when SOS is used.

When to use SOS?

SIs that define SOS offerings cannot do so for every service offering because the software landscape is forever changing, there are many complex components in a software service offering and thus does not lend itself to predictability always. However, an SI usually has some services that have stabilized, where the company has vast experience, knowledge and skills, has built up a sizeable software component library etc. that can be defined within a predictability framework.

Best Practices or Recommendations for using the Logging Application Block in your .NET code

Log Message Categorization

Decide on how each message will be categorized. Once the categories are decided add them to the Category Sources node under Logging Application Block in the web.config of the application.

Examples of categories could be:

• Performance Monitoring messages (ex:- logging the start time of a SP call and the end time of a SP call with the SP input/output parameters OR logging the start time of a page load and end time of a page load etc.)
• Security messages (ex: - Authorization, Authentication and Auditing messages)
• UI Event messages (ex:- user submitted form with values a, b, c etc
• Business messages (ex:- the result of a method that does a lot of calculations , its input parameters and periodic outputs of each step in the calculation)
• Data Access messages (ex:- creation of a database connection, closing the connection, calling an SP etc.)
• Exception messages
  
  Then every message must necessarily be categorized as one and only one of the categories decided for the application before logging.

Severity Levels of Log Messages

• Decide on the levels of severity for each message within a category. While the application block provides for 10 levels of severity (Critical, Error, Information, Resume, Start, Stop, Suspend, Transfer, Verbose, Warning), to keep things simple, it is recommended that only three severity levels are used.
• Critical – Messages that indicate a serious systemic error that needs immediate manual intervention such as a database inaccessible OR web service inaccessible OR access to a folder that the application requires access to is denied etc.
• Warning – Messages that indicate a potential problem in the system such as a long running SP OR a disk close to filling up etc
• Information - Messages in this level contain contextual information. They are mostly used for problem diagnosis.
  
  When logging a message, apart from adding the category to the log message, also add the severity level to the log message.

Trace Listeners for the log message (or where to store the log messages)

Decide Trace Listeners to be used in the application. Typical listeners are:

• Email – Use this for all Error severity level messages to notify someone.
• Event Log – Use this for Warning severity level messages.
• Flat File – Use this (or the Rolling Flat File) for Information severity level messages. Alternately, Database can be used for this severity level, especially if easy querying of the messages is required.

Decide filters (what gets logged and what doesn't)
Decide the filters that will determine what gets logged and what doesn’t. It is recommended that the following filters are added:

• LogEnabled Filter – Set this to True
• Category – You may set this up with two different types of filters based on the environment.
• For production - Deny all except: Critical Messages, Warning Messages, Exceptions Generated, Performance Monitoring, Security Events is recommended.
• For QA, staging & development - Allow all

The Supportability transform - Making non-functional requirements functional

Introduction

We frequently hear the term RASP being bandied about when anyone speaks about architecture or non-functional requirements. Frequently, the S means Scalability (and R- Reliability, A – Availability, P –Performance).

An interesting take on the “S” however has been made by Nagaraju Pappu and Satish Sukumar (of Canopus Consulting) to replace the Scalability with Supportability. It is interesting because S for scalability restricts itself to how an application copes with large no. of users, data growth etc. whereas the term Supportability brings into its ambit everything related to how a good application should be designed including the attribute of Scalability.

Before we delve into what supportability brings into its ambit, let us first understand what problems applications typically face. Right from the beginning of developing an application the focus is overwhelmingly on what the end-users want or what business functions the application must provide – the functional requirements. Depending on the company that has sponsored the application development and/or the company developing the application, important other aspects of an application such as ease of maintenance, ease of management, ease of modification, ease of debugging, ease of integration, ease of testing, ease of determining usage patterns etc. are given short shrift.

This is also the reason why most companies when they float an RFP for application development also mention that the application should have non-functional features such as modifiability, manageability, maintainability, scalability etc. However, these terms remain undefined in the RFP, remain undefined in a software specification, undefined in an architecture document and will remain undefined till the end of the application development project. This results in these non-functional requirements being untestable. Contrast these with other non-functional requirements such as Availability and Performance. These can be tested clearly most times.

Being untestable makes it a point of contention between the project sponsor and the development team on whether these were implemented in the application or not.

This post discusses a way in which this problem can be solved.
Supportability
Supportability is a collection of all those non-functional requirements of an application that have been hitherto considered untestable. These are all those requirements that support the job of the various stakeholders involved in defining, creating, deploying and running the application.

So this would include everything from usability (“must be easy to use”) , scalability (“the application should support growth in concurrent users”), modifiability (“the application should be easily modifiable to add new features”), maintainability, manageability, configurability, upgradability, compatibility, testability and all the numerous “-ility”s that adorn the non-functional requirements section of a requirements/specification document.


If each of these supportability requirements is examined closely a common thread amongst all of these emerges - Each of these features actually enhances the productivity of some stakeholder or the other. For instance, manageability enhances the productivity of remote management personnel, system administrators etc. Maintainability enhances the productivity of the application management team that is charged with maintenance of the application. Testability allows the QA team on a project to test the application better.

The problem remains however, of arriving at a common definition that everyone can agree on whether these requirements were met in the application built or not. The way to solve this is to apply a “supportability transform” which is discussed in the next section.

The supportability transform
Simply put, the supportability transform is a way to define these requirements as functional requirements. Going by the definition of each of these requirements enhancing the productivity of various stakeholders, we can start asking the question of each of the stakeholders.

For instance, if modifiability is the requirement, the most important stakeholders here are the developer who modifies the code to change functionality or add new features and the business user who needs these changes to be implemented with minimal effort and with maximum ease. So if we start with the functions/features the application supports, we can start by asking the business user and the developer of what is it that they want from the application that will help make this easier. The answer a business user may give is that he/she needs a form to be able to modify the business rules associated with the application. The developer might say that the application must be written in a particular manner in order for the application to be easily modifiable. Once we start asking the right stakeholders what this requirement means to them, the answers start emerging about what should be implemented in the application in order for it to be defined clearly. Once these requirements are defined clearly as part of the functionality of the application, these requirements no longer become non-functional requirements. Moreover, once these are defined as functional requirements to be implemented as part of the application, these also become testable.

Another way to look at it is to clearly identify all the stakeholders in an application. The stakeholders frequently forgotten by most requirements definition workshops are the system administrator, the remote management team, the product manager, the CEO, the CFO, the CIO, the developer on the project, the test engineer on the project, the architect, the project manager etc. As we ask these stakeholders what is it that the application should support as features for these stakeholders, the non-functional requirements gradually transform into functional requirements.

Standard supportability features for every application

This section attempts to compile a standard set of features that most engineering/operations stakeholders require out of any application. For each feature mentioned here, the stakeholder whose productivity is benefitted is mentioned and the common supportability requirement bucket that it usually belongs to is also mentioned. Attaching these as images, since I didn't want to muck with HTML tables to put the table structure here.

Setting up CruiseControl.Net with VSS, MSBuild, Nunit, Email etc.

I couldn't find simple step-by-step instructions to setup CruiseControl.Net, so thought I will provide what I followed here.

The practice of continuous integration represents a fundamental shift in the process of building software. It takes integration, commonly an infrequent and painful exercise, and makes it a simple, core part of a developer's daily activities. Integrating continuously makes integration a part of the natural rhythm of coding, an integral part of the test-code-refactor cycle. Continuous integration is about progressing steadily forward by taking small steps.

CruiseControl.NET from Thoughtworks makes continuous integration fully automated. CruiseControl.NET comprises of three components:

1. The server - CruiseControl.NET Server a tool that automates the integration process by monitoring the team's source control repository directly. Every time a developer commits a new set of modifications, the server will automatically launch an integration build to validate the changes. When the build is complete, the server notifies the developer whether the changes that they committed integrated successfully or not. Effectively, integration becomes as easy as checking in code. Using an automated integration server not only makes integration easy, it also guarantees that an integration build will happen. There is no danger of developers forgetting to validate their changes after checking in.




2. Web Dashboard - A .NET web application that gives you a status and detailed view of the CruiseControl.NET projects and servers in your organization.




3. CCTray - A client System Tray application that enables you to see 'at a glance' the state of your projects from any machine on your network.

Installing and Configuring the CruiseControl.NET Server

1. CruiseControl.Net Server installer (with web dashboard) is available at: http://downloads.sourceforge.net/ccnet/CruiseControl.NET-1.3-Setup.exe?modtime=1182463056&big_mirror=0

• Install the CruiseControl.NET server on an integration server such as a Development server. The server must have .NET framework 2.0




• Install the Web Dashboard also on the Development server. The server must have IIS 6.0 (with a website running on port 80) to install the web dashboard.




• Download the file http://ccnetlive.thoughtworks.com/MSBuildXmlLogger-Builds/ThoughtWorks.CruiseControl.MSBuild.dll and copy the file to the server directory in the CruiseControl.NET installation (usually C:\Program Files\CruiseControl.NET\server)

1. Ensure that the CruiseControl.Net server Windows service has been started




2. Locate the ccnet.config file (usually under C:\Program Files\CruiseControl.NET\server).




3. Modify the ccnet.config file and change the following key settings:

• Projects to be monitored (such as ClubPlanning, FolderMonitor, TransmitSKU etc.)




• VSS settings (VSS branches to monitor for check-ins, userid/pwd, location of VSS DB, Working Directory etc.) for the projects.




• Visual Studio Solutions that should be compiled when check-ins occur for the projects.




• Email settings for the projects.

A sample is shown below of how a ccnet.config will look like when these are setup.

Installing and Configuring the CruiseControl.Net CCTray Desktop Application

1. CruiseControl.Net CCTray is available at:
   http://downloads.sourceforge.net/ccnet/CruiseControl.NET-CCTray-1.3-Setup.exe?modtime=1182463076&big_mirror=0


2. Install the CCTray application on every developer desktop. The desktop must have .NET framework 2.0.


3. After installation, run CCTray and goto File->Settings


4. In the Build Projects tab, click Add and in the ensuing screen click Add Server.


5. In the Build Server screen, select Via the CruiseControl.NET dashboard and type in the URL of the web bashboard (such as http://myserver/ccnet ) and click OK on the Build Server screen.


6. On the Project screen that follows, select the Available Projects that appear in the right hand side and click OK.


7. This should now show the projects list and their build status in the CCTray application.

How to choose a Commercial Off The Shelf (COTS) Product

Here are some guidelines in choosing a product for integration in an application or product or selecting a product for an independent implementation. The following key factors must be considered when evaluating such Commercial Off The Shelf (COTS) products in the order in which it has been listed (the most important considerations are higher up on the list):

1 Fit-for-purpose

This evaluation is to ensure fitness for purpose. Make a list of all the must-have requirements and nice-to-have requirements in a table format and check the product for coverage of all these requirements and features. Some of the common features that all products must have include a good management instrumentation support, good alerting facilities, good health monitoring support etc.

2 Technology Risks

Carefully evaluate the product for various technology risks. Some of these could be in the following areas:
- Product built in an older technology – Check whether the product has been built in an older/obsolete technology.
- Product built in the latest technology – Check whether the product has been built in the very latest technology. Technology typically evolves through a couple of versions before stabilizing. Consider for instance the advances made by Java from JDK 1.0 to JDK 1.5. If a company had built a product in JDK 1.0 when it was just released, today the product would be missing some of the enhancements and advances made by the latest version of the JDK.
- Product built in an obscure technology – Check whether the product has been build using an obscure technology such as PERL where resources are hard to find.
- Lack of standards-based support – This is related to the Integration section mentioned earlier. A product with minimal or no standards-based API support is a big risk.

3 Scalability and Performance

Evaluate the product for scalability considerations (how many users, how many object calls etc. can the product support? Also, how well does the product perform when additional load is applied on it in terms of no. of users, amount of data to be processed etc.

4 Product Strength

This parameter is to evaluate how well the product is doing in general. A product that isn’t doing well is probably not a good candidate for evaluation or purchase. Some of the ways in which this can be evaluated are:
- No. of implementations done – This indicates the various scenarios that the product would have factored in in its feature list and hence would be resilient to most feature and requirement requests
- No. of good reviews received – This can be gauged by the no. of independent industry journals, whitepapers, websites, blogs etc where the product has received a favorable rating or review.
- No. of forums available – Check the no. of forums available for discussing issues with the product. Few or none is not a good indication.
- Quality of documentation – Read through the product data sheets, technical documentation, user guides, deployment guides to determine the quality of documentation.

5 Company Strength

This parameter is to evaluate the strength of the company making the product. Some of the ways in which this parameter can be evaluated are:

- No. of years in business –This indicates the stability of the company and the success of the product. It is important that a product being purchased have a financially sound pedigree to ensure adequate support and new versions that will be required as and when there are technology-advances in the area in which the product was built.
- Worldwide presence – This indicates the financial health of the company and availability of a wide support network. It also indicates location sensitiveness built in to the product which is very important for language/culture support.
- Clientele list – A clientele list that has quite a few Fortune 1000 companies will indicate that that product is well adopted and accepted in the community. Since larger companies typically have a very good due diligence process, this is a good indication of the product’s viability. Especially, see if there is a client that is mentioned in a similar industry. For instance, if the product is being evaluated for a bank, look for other competitor banks in the clientele list.
- Evaluation Support – This is one of the most important factors for choosing a product. When evaluating a product, verify the responsiveness, appropriateness of answers, willingness to help etc. from the support team of the product to get a good indication of how support will be post purchase.

6 Product Costs

This parameter is to evaluate how much will the product cost - obvious and hidden. Some of the ways in which this can be evaluated are:
- License Costs – Carefully evaluate the license costs for the product. Is it per-CPU, per-client access, per-machine etc. This would be the biggest contributor to the overall cost of the product.
- Implementation Costs – Ask about how much it would cost for implementation of the product. A lot of products do require the product company’s personnel to perform the implementation and their costs would be in addition to the license costs.
- Training Costs – Most proprietary products require extensive training for using, developing and administration. Check what the training costs are for these and how many people would require this training.
- Hardware Costs – Ask about what kind of hardware is required to run the product and find the cost of the hardware.

7 Integration Considerations

Almost every product will require to be integrated with other software products or applications. A product being purchased off-the-shelf must be evaluated for integration issues. Some of the ways in which this can be evaluated are:
- Integration Interfaces available – Ask about the APIs available especially for J2EE, .NET and web services. Ensure that all major web service standards are largely implemented.
- SDKs available – Ask about any Software Development Kits that will allow the company buying the product to enhance the product by itself.

8 Availability of Resources

Availability of resources for the following is a key consideration when deciding a product
- Availability of Administrators – Products such as Oracle 10g, Business Objects Data Integrator etc. require slightly specialized skills to manage in a production environment.
- Availability of Developers – Ensure the product has good APIs that support J2EE, .NET and web services. A product without this support will require specialized developer skills and the cost of such resources is higher than a J2EE/. NET resource.

Conducting performance appraisals for software engineers

Conducting performance appraisals for software engineers

Some parameters you should consider when appraising a software engineer on your quarterly/half-yearly/annual appraisal. Ideally set goals based on these parameters.

Technical Ability - Level of knowledge, skills required to perform current job. Good aptitude to learn new technologies? Fast learner (smart?).

Technical Hygiene - Documents work properly, quality of work is good, follows standards established regarding coding, technical documentation, follows good SCM principles (labeling, adding comments to checked-in files).

Potential - What the individual given proper mentoring, training can achieve? Can fill shoes of his/her boss on short notice?Attitude -
· Enthusiasm - How enthused is the individual about the work assigned?
· Customer Focus - How much is the individual thinking about the best interests of the customer?
· Earnestness - How seriously does the individual take work assigned?
· Proactive-ness - How good is the individual anticipating problems, future proofing whatever is designed/developed? Alerting boss early on about problem situations.
· Professionalism –General hygiene, mannerisms, grooming. No profanity, political incorrectness (race, gender, nationality and other biases) in speech, writing, behavior. Replying to emails promptly, attending meetings on time, general courtesy, keeping promises, organized and subordination. Respect for Co., policies. Is where expected in the office always, puts in 8 hrs diligently everyday.
· Teamwork - How well does the individual gel with the team, friendliness, and willingness to help others?

Leadership - Ability to manage, inspire a team. Mentor subordinates. Ability to resolve conflicts. When there was a problem, lifted his/her hand and got the job done? Respected by others?

Communication Ability - Ability to communicate orally, by email/documents. Ability to understand oral, written communications. English language skills. Presentation skills.

Independence - Can work independently without much supervision, picks up work without specifically being assigned?