I have been working on an update of my Bamboo ruby plugin which uses bundler to install all the gems for a given project within the working copy of the project and then run rake using these gems.

The aim of this post is to illustrate how this is done and how to craft an environment to run ruby once gems are “staged” within a working copy.

The aim of this post is to illustrate how a rails project is staged using bundler without installing any gems in the base ruby installation.

Firstly I will create a new rails project in an environment similar to that used when developing rails applications. I will be using an installation of ruby with bundler, rake and rails already installed, note I am passing the -T switch as I want to setup an alternate test framework.

rails new somenewrailsproj -T

Once created I navigate into the project and setup the test framework I am intending to use which is Rspec-2 for rails.

Add the following code to the end of the Gemfile.

group :test, :development do
    gem "rspec-rails", "~> 2.0"
end

Run bundle install.

bundle install

Run the Rspec-2 generator to plugin into the rails project.

rails generate rspec:install

Scaffold a sample model.

rails generate scaffold post title body:text published:boolean

Migrate this to the database.

rake db:migrate

Remove the pending specs as their not important

rm ./spec/helpers/posts_helper_spec.rb ./spec/models/post_spec.rb

Run Rspec and we should be all green.

bundle exec rake spec

Now to complete isolate our test environment we need a clean bash shell, to do this we run the following.

env -i bash

Next we export the PATH variable, in my case I only want my specific ruby version, and indeed the only thing I want is this versions bins.

export PATH=/Users/markw/.rbenv/versions/1.9.2-p320/bin

This version ruby has just been installed and therefore only has the base set of gems

$ gem list
minitest (1.6.0)
rake (0.8.7)
rdoc (2.5.8)

To simulate a build server the only gem we will add to this installation is bundler.

gem install bundler

Now we run gem adding the vendor/bundle/ruby/1.9.1 directory to the GEM_PATH this shows the gems we had previously bundled.

GEM_PATH=vendor/bundle/ruby/1.9.1 gem list

Now to illustrate how this will work inside a build environment we need augment our path a little, this will ensure gem install can find tools like compilers and such if required.

export PATH=/Users/markw/.rbenv/versions/1.9.2-p320/bin:/bin:/sbin:/usr/bin:/usr/sbin

Now run bundle install to recreate vendor/bundle and bin

bundle install --path vendor/bundle --binstubs

Now run our specs with the augmented GEM_PATH.

$ GEM_PATH=vendor/bundle/ruby/1.9.1 bundle exec rake spec                             
...
Finished in 0.32614 seconds
28 examples, 0 failures

So I have illustrated how I can stage gems for a rails application and run it’s tests without installing anything in the base ruby. This should work for any gem or project which uses bundler.

Some points to consider about this approach are:

• One should note that ruby has a notion of STD Library API compatibility which is reflected in the ruby/1.9.1 section of the path, this may vary for each release.
• For commercial projects I would recommend using a frozen set of packaged and quality assured gems and running tests with just this set.

That said for things like Octopress projects and gems / projects used internally this is quite a flexible way to run a CI build.

I am currently working on new design for my site using HTML5, CSS and a sprinkling of JavaScript. Once I started building my basic design I was re-acquainted with a process that really, really annoys me; the constant tweak refresh loop associated with developing a new site layout. So in true yak shaving sysadmin fashion I got side tracked looking for a solution to this problem.

After considering the issue for a bit it dawned on me that this may be a good excuse to try out LessCSS JavaScript library. When incorporated into a website this library enables the developer to use a CSS like markup which significantly reduce the amount of duplication and redundancy in the style sheet. The markup is processed on the client using JavaScript and has an API to mess around with how the styles are loaded.

Initially I planned to invoke the reload function in less library using a timer, however after reading the docs I found this feature was already built into LessCSS. Simply add #!watch to the URL in the browser and LessCSS will poll the style sheet for updates.

On a wide screen monitor this means I can tweak the CSS and watch the changes appear, which in turn removes quite a bit of keyboard gymnastics while working on a design.

To take advantage of these features in your site simply add the following fragment to your html page.

<link rel="stylesheet/less" type="text/css" href="css/styles.less">
<script src="js/libs/less.js" type="text/javascript"></script>

Then move all your styles to css/styles.less within your mockup/site and reload the page. To enable auto reload of the styles.less file append #!watch to the URL and refresh the page.

One thing to note is you will need to serve the site using a web server of some sort otherwise you will get XHR issues, to do this on OSX I use a python one liner.

$ python -m SimpleHTTPServer

To illustrate this feature I have created a sample project up on github lesscss_watch_example_site.

I am currently working on a site which will also live reload page fragments using Ember and Handlebars.

Having recently developed a plugin (Ruby Rake Plugin) for Atlassian’s Bamboo continuous integration (CI) server I thought I would put together a list of tips for those looking to do the same. As there are some great documents provided by Atlassian on how to get started with plugin development I am not going to go into a lot of detail in this area, it is assumed you already messed around a bit with the Plugin SDK.

So to start with there are some things you should learn before beginning:

Maven

The entire plugin development kit revolves around it so you need to understand it, have a read over the Maven Reference and add it to your bookmarks.

The first thing I do when coming back to maven is practice the release process, for most developers this is one of the most frustrating and complicated areas of maven so practice it.

Generate a test Java project using the basic archetype, and push it up to your version control site of choice, either bitbucket or github is fine, and work through the development cycle. Make a few changes check them in and then perform a release, this process normally takes me a few goes to get all the settings right in your maven project.

I recommend you use this approach to re-familiarise yourself with the release process after any long breaks as well, this will ensure maven hasn’t change since you last did it, and you don’t make a mess of your plugin project.

Once you have created your plugin project ensure you fill out all the relevant meta information in your maven pom file, as seen in the sample below. In addition to it being a good practice to do so this information can be used by maven plugins you may include in your project in the future.

    <description>This is the ruby rake plugin for Atlassian Bamboo.</description>

    <organization>
        <name>Mark Wolfe</name>
        <url>http://www.wolfe.id.au/</url>
    </organization>

    <developers>
        <developer>
            <name>Mark Wolfe</name>
            <email>mark@wolfe.id.au</email>
        </developer>
    </developers>

    <licenses>
        <license>
            <name>Apache 2</name>
            <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
            <comments>A business-friendly OSS license</comments>
        </license>
    </licenses>

The Java Ecosystem

A small part of this ecosystem is downloaded to your system when you run maven to build your plugin, so I recommend you do a little bit of reading on some of them, the ones I have listed below are a few of my favourites. I myself are a big proponent of the old saying “When in Rome, do as the Romans do”, for this reason I will always try and use the libraries which are already in the SDK.

• Google Collections, in my view one of the core libraries which a Java developer should know.
• SLF4J, one of the many logging abstractions which are used in Java projects but the one I tend to prefer.
• Apache Commons Lang, this library has quite a few utility classes for manipulating strings as well as builders for toString and equals methods in classes.
• Spring Framework, most of the Atlassian products are built using this dependency injection framework so it is handy to understand a bit of how this works.
• JUnit, this unit testing framework has been around for a long time for good reason, learn how to use it.
• Mockito, because mocking is a BIG must when building something in a large application so learn this API and ensure it is included in your plugin project from the start.

Development process

This is one of the areas which is often left up to the developers themselves to manage so for this reason I typically follow a simple process, especially when I am working on open source projects.

1. Before you start write down what you want to achieve, keep things simple and don’t plan world domination at this stage.
2. Build a first release focusing on the goals more than the method, the goal is to prove the concept and most important ship it.
3. Do some research now that you know what you looking for, read other peoples code and hack on your initial release a bit.
4. Delete your code, start the whole thing again, this sounds nuts but your proof of concept code is probably best left behind (see Corey Haines Code Retreat).
5. Build a new release from scratch with more of a focus on structure, testing and extensibility, and again ship it.

Currently I do quite a bit of work in and around the Java virtual machine (JVM), most of the time on Linux. When things go awry and I am trying to establish why, I reach for the Java performance analysis tools. These tools come in two forms, the wonderful GUI known as visualvm, which I use when I am working on my local machine, and the cli tools packaged with the Java Development Kit (JDK), which I use when working remotely.

The CLI tools I am referring to are:

• jps - JVM Process Status Tool
• jstat - JVM Statistics Monitoring Tool
• jhat - Java Heap Analysis Tool
• jstack - Java Stack Trace Tool

The tools I use most commonly are jps, jstat and jstack, the jhat tool is also very handy but really needs an entire blog post to itself as it crazy what you can do with it. In this post I have put together some tips, observations and sample outputs to illustrate how I use them.

As I am using ubuntu 11.10, which only installs the Java runtime environment (JRE) I will need to install the JDK. In my case I decided to give openjdk 7 a shot, but version 6 would work just fine.

root@oneric:~# apt-get install openjdk-7-jdk

To try out these commands I have installed tomcat7 this can be done through apt on ubuntu, again the previous version being tomcat 6 would be fine.

root@oneric:~# apt-get install tomcat7

Now that I have tomcat installed I want to list the Java processes, note that it is best to assume the same user account as the service when doing this. On ubuntu I would su to the user account, as the tomcat7 user is a system account I have to override the shell as it is /bin/nologin by default, I can then run jps as this user.

The jps command outputs the PID of the java process along with the main class name and the argument(s) passed to it on startup.

root@oneric:~# su - tomcat7 -s /bin/bash 
tomcat7@oneric:~$ jps -ml
12728 org.apache.catalina.startup.Bootstrap start
13926 sun.tools.jps.Jps -ml
tomcat7@oneric:~$ 

Now that we have the PID of these processes we can run jstat, the first switch I use is -gcutil this gives me an overview of the heap use within the jvm. In cases where there are pauses or performance degradation I will look at the last two columns. These contain the garbage collection time (GCT) and full garbage collection time (FGCT). If the FGCT column is increasing every second then it is likely we have an issue.

The following example I am running jstat against the PID of tomcat. I have also instructed the command to display the table headers every 20 rows and print the statistics continuously with an interval of 1000 milliseconds, as normal control C with end the output.

This sample shows a newly started tomcat 7 with very little happening, this is clear from the values in the full garbage collection time(FGCT) and garbage collection time(GCT) columns.

Also of note is the permgen space (P) which is currently sitting at 70%. The permgen space is an important area of the heap as it holds user classes, method names and internal jvm objects. If you have used tomcat for a while you will have seen the java.lang.OutOfMemoryError: PermGen space error wich indicates when this space fills up and cannot be garbage collected. This frequently happened when redeploying large web applications.

Also in the sample we can see that the Survivor 0 (S0), Survivor 1 (S1), Eden and Old spaces have quite a bit of free space which is good.

tomcat7@oneric:~$ jstat -gcutil -h20 12728 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032

To illustrate what a tomcat under load looks like in comparison we can install a tool called Apache bench.

root@oneric:~# apt-get install apache2-utils

And run the following command to hit the base page with a large number of requests concurrently.

markw@oneric:~$ ab -n 1000000 -c 100 http://localhost:8080/

Below is the output after this test was run for a bit, as we can see there has been considerable growth of the survivor 1, eden and old space, however the server hasn’t spent a lot of time doing full garbage collects as indicated by the value of the full garbage collection count(FGC) which is only 10, most of the work is in the young generation as seen by the increase in the young generation collection count (YGC).

Also to note here is that there wasn’t a lot of change in the permgen space, it actually went down, this was due to an increase in size of heap.

tomcat7@oneric:~$ jstat -gcutil -h20 12728 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250

To look deeper into the cause of garbage collection we use the jstat command with the -gccause option, this displays the same columns as the previous command but with two extras which supply the reasons for GC.

In the following example we can see an example of an allocation failure, this indicates that a full gc is being performed because the heap is too small.

tomcat7@oneric:~$ jstat -gccause -h20 12728 1000
100.00   0.00   0.00  78.91  59.67    168    1.680    14    0.083    1.763 unknown GCCause      No GC               
100.00   0.00  72.61  83.73  59.67    170    1.698    14    0.083    1.781 unknown GCCause      No GC               
  0.00 100.00  46.24  91.83  59.67    173    1.729    14    0.083    1.811 unknown GCCause      No GC               
100.00   0.00  11.39  29.80  59.67    176    1.759    16    0.086    1.846 unknown GCCause      No GC               
100.00   0.00  92.41  35.30  59.67    179    1.777    16    0.086    1.864 unknown GCCause      Allocation Failure  
  0.00 100.00  62.58  43.05  59.67    181    1.803    16    0.086    1.889 unknown GCCause      No GC               

Another area which I like to look into when diagnosing performance issues is the threads running in the vm. This can help me undertand if any component is overloaded and therefore operating a lot of threads trying to catch up. This is mostly only applicable to async processes like messaging, or scheduling routines.

To dump a list of threads and their current stack us the jstack command as illustrated by the sample below, again I normally run this as the owner of the process.

tomcat7@oneric:~$ jstack 12728
2011-10-16 14:53:58
Full thread dump OpenJDK 64-Bit Server VM (20.0-b11 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00000000015be800 nid=0x4004 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"http-bio-8080-exec-182" daemon prio=10 tid=0x00007f9d84274800 nid=0x3cd3 waiting on condition [0x00007f9d7a0df000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ef16da38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
...

I plan on doing a bit of work on some visualisation tools, in jruby of course, however that can be the focus of my next post. In the process of writing this post I located some interesting articles, these are linked below:

• Chuk-Munn Lee of Sun Microsystems Troubleshoots Java SE 6 Deployment
• Explaining java.lang.OutOfMemoryError: PermGen space

The following post is a review of the new installer added in 4.4 of Atlassian JIRA. It details my experience with this installer and provides some advice on how to improve it.

Since it’s recent release I have seen quite a few posts praising the new JIRA installer so I decided to give it a try at work. First thing that struck me when I went to download it was the linux version was a single file, no deb or RPM(s).

I copied the file to the Linux server and logged in as the jira user I wanted to run the service under, and ran the installer. The first thing the installer did was inform me I wasn’t running as an administrator, and as such it wouldn’t install a startup file for the JIRA. As this was a VM I took a snapshot and decided to try running it as root. When I did it asked me a couple of questions about where I would to put parts of the installation and then went off and did it’s thing. Once it completed I examined what it had done, these were the results:

1. Created a jira1 user.
2. Installed a startup script in /etc/init.d.
3. Installed a version of tomcat.
4. Installed a version of Java.

The first point is quite amusing as I already had a user called jira on the system and rather than ask me whether it should use it, it went and created jira1. Now most installers I have run recently at least had the decency of telling me before adding a user to my system, especially when they discover their preferred user id is currently in use. I also noted rather than set the user’s home directory to the location of the services working data it had just used the default add user leaving a pretty much unused home directory in /home. Overall user creation and use could be much better.

The second point was fine until I opened the script, when I did I was astounded to find a very brief script as follows:

#!/bin/bash

# JIRA Linux service controller script
cd "/opt/atlassian/jira/bin"

case "$1" in
    start)
        ./start-jira.sh
        ;;
    stop)
        ./stop-jira.sh
        ;;
    *)
        echo "Usage: $0 {start|stop}"
        exit 1
        ;;
esac

Now there are a numerous issues with this init script, firstly changing directory within a startup script and then running a script in the current working directory is a big no no, not only is it a potential security hole waiting to happen, it is also just plain bad form. In the case of tomcat itself there are a couple of system variables which instruct it where it’s base files are and where it’s working data is located. Using these variables removes the need for change directory (CD) in scripts and is much safer.

This script also not using any of the nice shell functions present in most linux distributions, for instances the linux standard base (LSB) functions which identify the distribution of Linux this script is running on, or the daemon functions which help run your service. Even worse is the fact this init script also calls another script which changes user context, then calls another script which starts the JVM. The tragedy of this multilayered shell abomination is further compounded by the fact the last script in the chain is just the default tomcat catalina.sh script.

The catalina.sh this script is a good script, however it is quite a generalised script primarily designed for use during development. All that is really required to start tomcat correctly is a small piece at the end of this script with paths configured based on the installation, along with some tuning for the application and the user it is to run under. As previously mentioned this is all typically available in shell functions within the init system.

The third point is a one of the ones I really have an issue with, the installer has put an unknown version of tomcat onto my system. As tomcat is a web server it is unfortunately targeted for exploitation, and does have the odd security issue. This would be ok if you ensured I had a method of updating said tomcat but alas you just dump it in your location of choice and leave me holding the baby. Note upon searching I noted tomcat bundled with JIRA, which is version 6.0.32, does indeed have a few security advisories, not a good start.

And lastly this installer has gone and dumped another copy of java on my system, again with no information on what version this is, and again no way of upgrading it if there is any security exploit for it. So overall not very impressed at all with this new installer.

Overall I think what astounds me most is that a company of this size is completely ignoring recognised best practices when packing software for Linux. Considering the customers who know least about linux are most likely to use this installer blissfully unaware of the traps it has dragged them into I am very disappointed.

Now as I am continually reminded by my boss, if I am going to take the time to present issues I should always accompany it with some suggested solutions to them. So my suggestions are as follows:

1. Work out what linux distributions your customers are running.
2. Review how each of these distributions package their software, in general be a good citizen in these operating systems.
3. Set up a couple of repositories for packages tuned for ease of deployment on the main couple of distributions. Essentially make it as easy as possible for a novice user to deploy and update your products using these distribution points.
4. Write a nice init script following the best practices promoted by each distribution.
5. Break your system into it’s distinct components, in a similar way to how you modularise your software, pull the system down to a few discreet packages, ideally with upstream version numbers.
6. Offer updates to these packages to ensure your customers data is secure.

One of things I respect about Microsoft and Apple is they have a keen focus on ensuring people who deploy their products look like wizards, a few clicks and everything is up and running. In my opinion the user experience for the person installing and maintaining this software is as important as the end user. In a lot of cases this person has the potential to be one of Atlassian’s greatest assets, so please don’t mess them around.

As Atlassian has had some quite high profile compromises, for example the Apache foundations JIRA incident I would have assumed that providing a secure method to install and operate their software would be at the top of their priority list.

Lastly I am always happy to sit down and have a chat with anyone from Atlassian, preferably over a cold beer. I have been using their products for 8 or 9 years, yes I even had a support request responded to by one of the founders. Atlassian’s products have served me and my customers well so this is the least I could do.