Headlines

Headlines

Build Management Allows Developers to Do What They Do Best

Follow these build practices to optimize a developer's workflow while transitioning to a more automated process.

By Matt Gabor, Senior Developer, Catalyst Systems Corporation

Whether writing code for a small, single-platform environment or contributing to a sophisticated cross-platform, multi-language application, one truth remains consistent: A disorganized or poorly implemented build-management strategy will adversely affect a developer's workflow. Because developers possess an intimate working knowledge of their code's structure, they're often called upon to write and maintain build scripts. When integration-build problems occur during QA, their phones ring. When build problems prevent applications from rolling into production, their inboxes are flooded. These kinds of intrusions on a developer's daily coding practices may start small. Over time, however--as applications grow in complexity--they can bring a company's entire application-development process to a grinding halt. This result can be avoided by implementing a well-thought-out build-management system that empowers the developer while meeting enterprise-level requirements.

In small- and medium-sized environments, using a properly implemented in-house build system can mitigate many of the risks and challenges associated with builds. With a few simple steps, the burdensome task of developing and maintaining build scripts also can be significantly reduced. Developers can therefore focus on writing code rather than debugging builds. As development efforts grow in size and complexity, however, there comes a point at which significant benefits can be realized by moving to an automated system. Whether using a manually scripted or automated build solution, every enterprise should strive for build consistency, portability, and repeatability.

Build-Management Evolution
Application builds are traditionally managed using a rules-based program that's derived from Make--the world's oldest, best-known build tool. Make controls the generation of executables and other non-source files from a program's source files. There are many Make versions. Each one has a unique file syntax that precludes portability between development tools, operating systems, or even compilers within the same operating environment.

Java development required a new, platform-independent build tool, which lead to the creation of Apache Software Foundation's Java-based Ant (so-named by developer James Duncan Davidson because, it is a little thing that can build big things). Ant eliminates Make's platform- dependent wrinkles. It is extended with Java classes rather than shell-based commands. Configuration files are XML-based, calling out a target tree where various tasks get executed. Each task is run by a Java class that implements a particular task interface.

Although Ant is powerful, the XML configuration scripts can create limitations. XML doesn't handle conditional logic effectively. It's therefore difficult to use Ant to write "intelligent" build scripts that support robust batch processing. In addition, many development projects include Java and non-Java components that require both Ant and Make, as neither handles both languages.

Scripting for the two is very different. Make scripts are the input to Make programs. They dictate how to build each software component. A Make file tells the Make program which binaries are to be created from which source modules. Make rules are then "fired" based on out-of-date conditions between source and object. In contrast, Ant/XML scripting uses serial batch processing (see the Figure). The rules for creating Java binaries, such as .jar, .war, and .ear, are handled statically for each step or "task" in the XML script. For either approach, the programmer must know how the application is constructed. Yet he or she also must understand the specific syntax requirements of the build-scripting language being used. Additionally, Make and Ant/XML scripts aren't re-usable because static application information is coded into the script.


Make And Ant/XML Challenges
As client/server and Java development have evolved, so has build complexity--especially with Make. When Make is used recursively, there is one Make file per final target executable or binary--the most common method of managing large build processes. In this usage, an application with 50 binaries would require 50 Make files plus a "driver" Make file. The system build is completed by calling the Make program repeatedly and passing a different Make file each time. Because dependency checking between the individual Make files is impossible, large application Make files can't be managed by a single Make file. Developers get around Make challenges through clever file-ordering to track dependencies. In addition, object-borrowing and multi-system parallel building techniques reduce the associated long system-build times. The process is still difficult and cumbersome--especially when there are multiple circular dependencies between executables and libraries. These dependencies require multiple builds to achieve the desired results.

Most Ant build systems don't appear to be as complex as many of the Make-based build systems. Yet it is only a matter of time. Because Ant is a relatively young technology, Ant-based build systems haven't been subjected to the years of additions and changes that can corrupt and obscure what originally appears to be an elegant build-scripting solution. Over the years, Ant scripts will suffer from being passed through different developer's hands. In addition, new technology will emerge that affects the way Ant scripts are coded or used. Plus, applications will grow more complex. As a result of all of these trends, Ant will encounter many of the problems associated with a Make-based system.

Scripted build processes also introduce other challenges that devour developers' time. In fact, build scripting can be the most time-consuming and resource-intensive component faced by development teams. Problems scale with an environment's size and complexity. The key is to implement best practices for manual scripting starting with an in-house build system. At the same time, be sure to monitor the factors that would signal the need to move to an automated, non-scripting approach.

Solving Typical Scripting Problems
Scripting challenges are easier to solve in small, homogeneous development environments that are confined to a single language and target operating system. Yet most companies strive to expand product reach and capabilities. As scope grows, development complexity follows. A company that lays the groundwork for an organized build process early on will better overcome future growth challenges. The first step is to shift from a developer-centric view of builds to a team-centric view. Also, it's important to move from the notion of scripting "my build solution" to scripting "our build solution."

Build inconsistency is the toughest problem. If developers use their own build scripts in the language or tool of choice, it can be difficult to know whether problems result from bad code or a bad build. Build administrators must standardize on a single scripting approach that best suits the language being used. The first step to reducing build inconsistency between individual developer's build scripts is to create build-script templates, which can be used as a basis for all build scripts. All builds require the same basic information: source code, compiler, and final target. Individual developers can populate the build-script templates with their own build specifics (i.e., the source-code location, compiler location, and a final-target description). To ease the process of populating the template with build-specific information, build-script templates should offer adequate comments and be clearly organized.

A major contributing factor to build inconsistency is a lack of compiler and third-party library standardization. In a disparate build environment, developers frequently build against different versions of compilers and third-party libraries. This makes it difficult to re-create builds and diagnose problems. To promote standardization, all compilers should be centralized on a network drive that can be accessed by all developers. That drive should be on a clean and "locked-down" machine. The build-script templates should specifically reference the standard compiler versions on the mapped network drives, thereby ensuring that all builds occur against consistent compiler versions. All third-party libraries should similarly be consolidated on a shared network drive so that the latest, approved versions are used.

Another commonly faced problem is lack of build portability. Builds often work only on an individual developer's machine, which by default becomes the "production" build machine. The philosophy is, "Build it where it builds and get it out the door." Obviously, this approach to builds can cause severe problems when trying to track down bugs that are discovered once an application has been released to production. To solve this problem, development teams should standardize their directory structure. All developers should work on code in the same directory structure. If a versioning or CM tool is used, pull the directory structure from it. If not, enforce strong directory conventions for all developers.

Portability problems also can be mitigated by using global variables in the build-script templates. Such variables will identify the root location for all source code, compilers, and common libraries. By setting environment variables, such as SRC_HOME, COMPILER_HOME, and COMMON_HOME, the same build scripts should work on all machines. Using global variables in the build-script templates also reduces the amount of template editing that's required by developers.

Finally, isolate the build scripts to just that: builds. Too often, "build" scripts include substantial pre- and post-build logic that's unrelated to the build. Pre- and post-build logic can be extremely complex-- especially as an application matures and development is being performed on multiple versions simultaneously. Pre- and post-build logic shouldn't be written within a build script, where the functionality is often limited by the scripting language or tool. Instead, place the non-build logic in external scripts. These external scripts should be written in a scalable, lightweight, and cross-platform language like PERL or PYTHON. Tightly focused build scripts can then have built-in hooks to the external build utility.

By partitioning the build scripts in this way, developers (or build masters) who encounter build problems can drill down to the root cause very rapidly. As development grows in complexity and new languages or target operating systems are added, the in-house build utility also can scale more effectively. For example, consider a C and C++ development shop that uses an entirely Make-based build system. All of the pre- and post-build logic is written in the Make scripts. When the development shop decides to add a Java component to its application, that shop is faced with writing an Ant component that's equivalent to its existing Make scripts. This component manages all of the Java-related pre-build, build, and post-build logic. However, say the development shop has a build utility written in PERL that executes Make scripts limited to build execution. In this scenario, the shop would only have to write Ant scripts that handle the Java builds. The shop can use the existing PERL framework as a basis for all of the non-build functionality.

Through careful build-management planning and implementation, developers can be free to focus on the development that they were hired to perform. By optimizing a developer's workflow, companies are more capable of meeting market pressures with increasingly sophisticated and robust applications. The practices outlined here can be used to develop a build system that handles current build requirements. At the same time, these practices pave the way for an orderly transition to automated build strategies. Such strategies will enable significantly more flexible and manageable development environments.





Matt Gabor is the Senior Developer at Catalyst Systems Corp. He has extensive background in the use and deployment of Eclipse Plug-ins. His technical skills include J2EE, J2SE, and .Net development.