Creating A Maven/Ivy Repository For Your Organization

In my previous blog entry, i gave an overview of how to setup a componentized application architecture in my blog post JEE Appplication Architecture For Success.  The main foundational element that article is built upon is having a functioning Ivy/Maven repository working that will pass an audit by release management and bank regulators.  In this blog entry I will describe the specifics of how to set up an Ivy/Maven repository and integrate its use into your packaging and release strategy.

When we first devised the setup for implementing our repository there were several requirements that we considered and addressed.  The first requirement is that we have to tightly control all libraries that are present in repositories used for our release builds, at all times, and the introduction of any new libraries into the mix had to be a tightly controlled event.  A second requirement is that for libraries that aren't 3rd party libraries the source code that generated that repository artifact must be available and mapped directly to that artifact.  A third requirement is that we need multiple repositories including snapshot, development, and release repositories to adequately address our development, test, and production implementation processes.  The final requirement is that the artifacts should be downloadable through a web based URL and that end development staff should be able to browse the contents of this repository in a browser.

When considering these requirements as whole, implementing this as a filesystem based repository backed up Clearcase by project view really jumped out at me.  Clearcase would not be my first choice as it relates to versioning system.  I would prefer to use SVN or Git as my versioning system to implement this.  That being said, the standard versioning system we are required to use is Clearcase.  And despite the holy wars that go on over what the best versioning system is, Clearcase will implement this just fine as it provides the functionality to address three of the four requirements above.

The first requirement is quite obviously addressed by using a Clearcase view to back a file system based repository on a server where only release management has access.  The italicized bold part is a key qualifier, as you'll come to see when I get to the part about addressing http based download and navigation.  Implementing your repository, or repositories as i'll call out soon, in this manner very clearly address the issue of tightly controlling what libraries go into the repository and thus tightly controlling what libraries can be consumed by a build.

The second requirement we mentioned is needing to map the exact set of source files that produced the artifact to the artifact itself.  This is accomplished by bundling the source files at build time and associating these source files as a source artifact in the repository.  As part of the artifact check in, the source jar is checked in along with it, and voila ... the source files that spawned this artifact are directly mapped to it without painful searching through Clearcase.

While tightly controlling what is going in and out of the repository and mapping the source files to artifact they created, using Clearcase views to implement filesystem based repositories also addresses the issue of being able to manage multiple repositories in an auditable way.  We mentioned needing three repositories, a release repository, a development repository, and a snapshot repository.  Clearcase can easily handle the development and release repository cases by having a release repository branch and a development repository branch.

The process of an artifact moving into the release repository starts with it first being installed into the development repository.  It is then consumed in an integration build.  This integration build is then passed off to test and validation is done.  Once the testing is complete and the end product is validated, the libraries can be promoted to the release repository.  This is accomplished by merging these artifacts from the development repository Clearcase branch into the release repository Clearcase branch.  Once these artifacts are merged into the release repository branch, and the release build can be done in preparation for production acceptance testing and validation.

While Clearcase handles all of the above well, it does not handle a few things which needed to be done separately.  These consisted of defining a snapshot repository, keeping the snapshot views up to date, and exposing their content through http.  Fortunately, all were easily solvable.

As it relates to providing a snapshot repository, this is intended to be a latest and greatest repository that is not versioned and only holds the custom application libraries we developed.  This is a filesystem repository like the others, the key difference is it's not backed by Clearcase.  The libraries are built via continuous integration and published to this filesystem, and developers can pull these latest versions at their leisure.  There is nothing major here.

While continuous integration is used to generate the libraries for the snapshot repositories, it is also used to solve the problem of keeping the filesystem repositories backed by Clearcase views up to date.  We have continuous integration update the repository view when it detects changes, and thus that problem goes away.  We opted to not implement this with Clearcase Dynamic Views, as they tend to be painful and difficult at times ... namely if the connection to the Clearcase server goes down your entire repository is hosed.

Finally, to expose this via http we simply setup an Apache instance in front of each of the repositories.  We set the appropriate mappings to the appropriate filesystem locations, and all works great.  You can use URL based resolution that is typically used when resolving Maven/Ivy artifacts, and everything works as it should.  Your end state ends as follows....

This is a high level of how to setup a Maven/Ivy repository for your organization.  I will be posting follow up articles on how the build process plays into this, how continuous integration works into this, and how to specifically build your artifacts and push them to these various different repositories and consume them in your builds.

JEE Appplication Architecture For Success

Introduction

How you choose to segment any application that you build from scratch or inherit is absolutely critical to the success, mediocrity, or failure of your organization and its business goals surrounding this functionality.  I am adamant that choosing to build your application as a set of components is imperative to unlocking the full potential of it.  This article will cover how such an application is composed, and the direct efficiency and cost benefits that you will derive from it. 

Background

I was first introduced to a component based application architecture as a contractor when I had the great fortune to be sent to a client with a very gifted organization.  This was in late 2007 and they already had a solid web presence, but were completely absent a mobile application view into their system.  I along with their staff were being tasked with developing a mobile html application that provided their customers a view into their accounts and related information.

Considering i had just joined their organization, i knew little to nothing about the system(s) in which the account information was stored and how to extract that information.  I expressed this concern with staff and asked where i could find this information so i could come up to speed about this.  I was told not to worry myself with that, I was to be the user interface developer and I had access to all of the routines to do what i needed.  Where there were holes, client staff would provide me with what i'd need to fulfill the requirements.

Architecture

Compared to what you're used to seeing as a contractor, their architecture was stunning.  Their organization was already responsible for an existing suite of online applications that back office, call center, and their end customers used.  There were separate components encapsulating the vast majority of the business logic, and these components were directly associated with Clearcase source code components.  Additionally they had other non business logic components along the lines of custom tag libraries and system resources (datasources, etc) which were also managed as separate Clearcase projects.  The mobile application I would be developing with them would fall in as just one more participant in the grand scheme of things, as depicted below.

By now you might be thinking that separating your business logic modules into a separate layer is far from unique, and while it's great they did that there's nothing new here.  An important note to take away from the document is this is a virtual separation, this is not a physical separation of their business logic and common modules.  I would like to elaborate on this point precisely.

Many organizations who have implemented such a scheme will actually host the business logic modules on a separate server that exposes these business modules as web services.  This is not the case here.  The business logic modules and common modules are co-located within the same war/ear.  Since they were a Spring based shop all of the modules were Spring based and each business logic module provided its own Spring configuration file in a standardized location and naming convention.  The applications would scan the classpath for Spring contexts fitting this standard (using the ContextLoaderListener) and thus any of the modules that had been consumed by the end application would be loaded into the Spring container that the end applications had instantiated.

What has been accomplished is the ability of each application to execute different versions of various services and be developed on a separate release schedule.  They can consume the exact version of the libraries that they want, and execute that version of the code within the application server they are running on. This provides many advantages from an operational standpoint in that there are no multiple deployments when it comes time to deploy this application.

The end applications can operate on different release schedules, thus having different versions of the libraries pulled in, and all of this without having to manage multiple versions of remote web services running if we wanted to accomplish that same feat. Additionally, it allows new applications to be built by new developers who can simply pull these in and not be concerned with how the business logic underneath works.  This allows an expensive contractor like myself to be efficient immediately making such an expenditure a lot easier to justify, or it can allow cheaper players to over achieve by insulating them from complexity.  In short, it makes anyone responsible for building this new application a far more efficient option than they would have been.

Consuming the modules and writing the app

This all sounds good theoretically, but how in principal does this work.  A possible choice would be that you're writing and publishing your own modules to a maven repository, and i have made that work well in projects aside from this.  However, the client I was at was an ant based shop and in no way wanted to give up the scripting control ant gives you over maven.  So, i found that they had implemented this via an Ivy repository.  Conceptually it's very similar to a maven repository, there are eclipse plugins for it, so the fact that they chose Ivy over Maven to accomplish this is of little consequence in the grand scheme of things.

So I sat down, setup a workspace per client standards so i didn't have to write a build script, and consumed all of the business logic modules and common modules that i would need to write this application.  I bound my user interface models either directly to the domain model objects the business logic modules returned or a close approximation thereof.  I was able to put up the scaffolding that our user experience design teams provided us, get it working on the appropriate models that would be compatible with the API i was writing to, and shortly thereafter get the application up and running with some level of integration.  All the while, I had no idea how this data was being extracted from downstream systems and formatted into the view of the data i was getting back from the API.  And for the purposes of my contract, i did not care and neither did they.  Which is precisely the point.

Conclusion

This client had separated their business logic and common modules from the end applications and facilitated this using Ivy/Maven.  This provided an astounding level of efficiency for personnel not familiar with their business domain translating into a faster time to market and labor cost savings.  This is not a one hit wonder and can be replicated, as i have introduced this process from the ground up in a subsequent role as architect at my current position.  I will provide the details of how to implement each piece of the scheme mentioned above in subsequent blog entries, so that you may implement this type of architecture and derive the same benefits that I and those I learned this from have.