A Guide to Effective Version Control

We have come a long way since 2008 when I wrote this introductory article about version control systems. Their rising popularity in the past few years is a notable sign of maturity in the software development community. Whereas they were once a bastion of only “serious” development operations, now even app developers working alone use them regularly.

In this article, I talk about how a team can use a version control system to its advantage by describing a preferred way to store the files in the repository and some processes to complement the structure.

There are myriad ways of organizing a version control repository. It is a sophisticated file system, after all. A well-structured repository can be a valuable asset to a software team by creating a clearly defined workspaces for programmers, opening up the opportunity to create a continuous integration pipeline, and standardizing deployment.

Repository Layout

In my preferred layout, the trunk is used for the main development line. In addition, branches are created for releases and feature sandboxes. This style of organization is loosely based on trunk-based development.

The Trunk

All the primary development of the project occurs in this directory. The trunk is further organized into sub-directories for every project that constitutes the final product. For example, a product might consist of separate web and desktop components. Or it might have a dependency on a separate static or dynamic library. The exact arrangement of this is determined by the programmers and release engineers working together. All these projects reside side-by-side in the trunk.

A feature or bug fix is considered open until the code has not been committed to the trunk. Hence, all developers have access to this directory and must make it a regular part of their daily work.

trunk

Maintenance Branches

Once the product reaches a milestone and is deemed ready for release, the release engineer is responsible for generating a binary and deploying the product. But the first step is to make a maintenance branch. This branch is a snapshot of the trunk that can be used by developers to provide bug-fixes to an already deployed product without accidentally releasing new features.

It is named according to whatever convention the team decides (I prefer a numeric x.y scheme).

branch

If bugs are discovered in the product post release, they are patched in the trunk and merged into the active maintenance branch. Other developers, meanwhile, can safely continue adding new features on the trunk without inadvertently bringing them into the current deployed versions.

A bug-fix release is made out of the maintenance branch after a bug is patched.

It is important that bug fixes should originate in the trunk and later be merged into the branch rather than the other way around. If the fix is made in the branch and the developer forgets to merge it back into the trunk, it can cause a regression later down the line. By patching in the trunk, the developer uses the process to automatically ensure that patches exist in both places.

Tags

Release binaries should always be made out of a clean export from the repository. But which revision should one export? If you’re creating a maintenance branch, you could use its HEAD revision. But you’re out of luck if you need to generate a previously released build again after subsequent edits have been made to the branch. Unless you tagged the branch before generating a binary.

A tag is a branch that everyone agrees to never edit. It is given a unique name from the rest of its siblings. My preference is to go the x.y.z scheme, where the x and y components of the name come from the maintenance branch from which it derives its lineage.

tag

The z component of the tag name is set to 0 when the first maintenance branch is made. This number is incremented whenever subsequent tags are made out of the maintenance branch.

Using the Repository

Once the structure is ready, some processes need to be put in place in order to utilize it. The typical developer role has it easy – keep committing error-free code into the trunk, and make sure the working directory is updated from the trunk as often as possible.

Feature Releases

A new feature release begins when one of the managers green-lights it. The repository is locked down temporarily and a maintenance branch and a tag are created from the head revision of the trunk. These two copies are named according to the version convention in place for the project.

This maintenance branch supersedes all previously generated branches, which may even be deleted from the repository. The repository is then unlocked and developers can go back to making commits in the trunk.

Maintenance

When bugs are discovered in a released product, the developers investigate in the trunk and attempt to fix it there. If found and fixed, they notify the release engineer who then cherry-picks the relevant commits and merges them into the current maintenance branch.

The maintenance branch is tagged after it is deemed ready for release. The binary generation and deployment process remains the same – clean export from the newly created tag, followed by compilation and deployment.

If a bug cannot be located in the trunk, the developer may have to look for it in the maintenance branch and fix it if it is found there. In this case, the bug fix may be merged back into the trunk if deemed necessary.

Release Generation

The release engineer’s work is not done yet. The release binaries are yet to be created from the newly created tag. For this, the tag is exported from the repository into an empty directory. This is important so that no unnecessary files are accidentally added into the binaries. This step cannot be emphasized enough.

All binaries must be generated from a clean and complete export from the repository.

Generating binaries from partially exported directories runs a high risk of including incorrect or outdated files into the binary. This inaccuracy can be the source of impossible-to-reproduce bugs or regressions. At the minimum, it is a sign of a poorly managed process.

Once the binary is generated, it is packaged to its final consumption form – an archive or installer for distribution – then deployed to wherever it is needed (such as a file server or web server).

The binaries and installers may be added into the repository for posterity.

Asynchronous AIR Native Extensions

What could be worse than attempting to build an image processing application in ActionScript? While the language has its positives, handling large data sets quickly is not one of them. This was the dilemma we were up against in a recent assignment. And the images themselves were expected to be hot off the camera sensors. At a time when even tiny point-and-shoot cameras regularly reach 12 megapixels, our target audiences of photographers and studios were expected to hit us with a minimum of 24 megapixels.

That works out to 72 megabytes of data per image. Now multiply that by a few hundred images and you know why we were very concerned. The task in question was not very complicated – just resizing images. But the volume of data ensured that even this mundane operation took forever to complete, without any feedback to the user because ActionScript runs on a single thread.

We flirted a bit with ActionScript Workers, but they were an incomplete solution. Our UI thread became available again, but the processing still remained unacceptably slow.

Our fallback came in the form of Adobe’s Native Extension API, that allows ActionScript applications running in the Adobe Integrated Runtime to access libraries written in C or similar languages.

Well, that’s that then. This task was easy enough. Bang out some functions to resize and encode images and make a call from ActionScript. Since it was native, it would be fast and the user would never even notice the pause.

Unfortunately, it didn’t turn out that straightforward.

While this operation became faster by an order of magnitude, it still stuttered when loading really high resolution images. And when multiplied by a hundred or so photos to be batch processed, the frequent freezing of the UI was very obvious.

So it was back to the drawing board.

Asynchronous Processing

The native extension API offers the ability to dispatch events from the native code back into the runtime. This has been designed precisely for such situations. The extension spawns a new thread that handles the processing and returns control back to the runtime on the UI thread. After the computations are complete, the native extension dispatches an event that the runtime listens for.

To illustrate, let us implement a simple native extension that runs a timer and notifies the runtime when the allocated time is up. For simplicity, this timer will run for ten seconds. It can be made to run for arbitrary durations by passing the interval value as a parameter to the native side. We’ll call this extension the Ten Second Timer. Its ActionScript class definition is as follows.

namespace com.notadesigner.utils
{
    public class TenSecondTimer extends flash.events.EventDispatcher
    {
        public function start():void {}

        private function context_statusHandler(event:Event):void
    }
}

This class extends EventDispatcher. Collectively, the client, TenSecondTimer and the native code set up a chain. The client listens for events from TenSecondTimer, which in turn subscribes for events from the native code. When the native code dispatches an event, TenSecondTimer creates its own Event instance and dispatches it. The client thus receives an indirect notification from the native code through the TenSecondTimer class.

this.m_context = ExtensionContext.createExtensionContext("com.notadesigner.TenSecondDelay", "");
this.m_context.addEventListener(StatusEvent.STATUS, this.context_statusHandler);
this.m_context.call("start");

On the native side, the start function is implemented with the function signature required of all native API functions.

FREObject start(FREContext, void*, uint32_t, FREObject[]);

When this function is invoked by the runtime, it spawns a new thread (using the pthread API in this case) and immediately returns control back to the runtime. A reference to the waitForDuration function is passed to the pthread_create function. The newly created thread executes that function.

FREObject start(FREContext context, void* functionData, uint32_t argc, FREObject argv[])
{
    pthread_t thread;
    pthread_create(&thread, NULL, (void *)&waitForDuration, NULL);

    return NULL;
}

The waitForDuration function calls the usleep API that suspends the thread for 10 seconds. The CPU wakes this thread again after this duration has elapsed, and the function dispatches an event through FREDispatchStatusEventAsync.

void* waitForDuration(void* arg)
{
    usleep(10000000);

    FREDispatchStatusEventAsync(_context, (const uint8_t*) "complete", (const uint8_t*) "done");

    return NULL;
}

In order to notify the runtime, the native coded needs to maintain a reference to the context. The context is passed as a parameter by the native extension API to the context initializer function. This function must store the reference somewhere that is accessible to the thread. A global variable works nicely in a pinch.

The runtime then kicks back into action, and passes on the event notification to the TenSecondTimer class. The context_statusHandler method is triggered, which in turn dispatches a complete event for the client to handle.

private function context_statusHandler(event:StatusEvent):void
{
    var event2:Event = new Event(Event.COMPLETE);
    this.dispatchEvent(event2);
}

This pattern of triggering actions in the runtime from the native code can be used for a variety of other tasks that require asynchronous execution. The function invoked by the thread can perform whatever task may be required of it in place of the usleep call.

In our case, we implemented the native extension method to resize images asynchronously. Since loading the original image was taking the most amount of time, it went straight into the separate thread. The thread also took care of resizing the image after it was loaded and saving the resized file back to disk.

Packaging AIR Native Extensions

Native extensions are a handy improvement to the AIR ecosystem. There are numerous benefits of being able to to drop into some C code for processor-intensive tasks or extending the Flash API into domains outside of the standard library provided by Adobe.

But anybody who has had to deal with developing their own extensions knows that the packaging process is less than straightforward. It requires maintaining at least two different code trees – one for the native portion of the extension, and the other for the ActionScript library that glues the client application to the native portion. The two projects have to be compiled separately – the ActionScript library through the compc tool, and the native portion with its relevant compiler. Finally, both binary outputs have to be packaged into a single ANE file, which is equivalent in structure and function to a SWC.

In addition to all these, the ADT tool which packages the ANE has its own quirks with relative paths (learned at the school of hard knocks), which are best dealt with by placing all the files required for the extension into a single directory rather than scattered across different directories on the hard drive. Which means copying all the various output binaries and XML files into a single directory every time they are changed.

Running all these activities by hand is time consuming and error prone. However, they are mechanical tasks that computers are well-suited to handle. The following batch script is designed for a Windows build chain that compiles both source trees and packages them into an ANE for consumption into the client application.

  1. Delete the previously generated binaries to begin with a thoroughly clean slate.
    del build/*.*
    
    rmdir build /Q
  2. Create a new designated build directory. This is the directory where your binaries are copied into and packaged into an ANE.
    md build
  3. Compile the ActionScript library using the compc tool. It becomes much easier if you pass a flex-config.xml file as a parameter to the tool rather than passing every setting as a parameter to the command line.
    compc -load-config bridge/src/flex-config-lib.xml
  4. Copy the output SWC into the build directory. Alternatively, the compc tool has an -output parameter that can be used to specify the location and filename of the generated binary.
    copy bridge.swc build
  5. Extract the library.swf file from the SWC generated in the previous step into the build directory. Unfortunately, Windows does not ship with a native command line utility to extract files from an archive. Alternatives such as 7-zip help fill up that gap.
    7z.exe x build/bridge.swc
  6. Compile the native code using your compiler toolchain of choice.
    MSBuild.exe NativeLibrary/NativeLibrary.vcxproj
  7. Copy the native runtime library along with a previously authored descriptor.xml into the build directory.
    copy NativeLibrary/Release/NativeLibrary.dll build
    
    copy descriptor.xml build
  8. Package all these files into an ANE using the ADT tool. Invoke the ADT command from inside the build directory.
    cd build
    
    adt.exe -package -target ane NativeLibrary.ane descriptor.xml -swc bridge.swc -platform Windows-x86 NativeLibrary.dll library.swf

Runtime Shared Libraries with Plain ActionScript

Using RSLs is still a bit of a black art in ActionScript. This is partly due to the fact that documentation on it is sparse, partly because of the complex ActionScript code that mxmlc generates when compiling MXML files, and partly because the process itself is a bit unintuitive. These notes are the result of a few days of research on the topic.

A runtime shared library is simply a SWF file that is loaded into the application at launch. The code and assets in the library are then available for the application to use. This makes it handy when using third-party code libraries, or even your own libraries which are built to a published specification and are not going to change frequently. By making them external to the application, you can also share them between several different applications.

The Flex Way

ActionScript developers working with mxmlc directly outside the Flash Builder IDE may be familiar with the following warning.

/Projects/rsl-demo/Main.as: Warning: This compilation unit did not have a factoryClass specified in Frame metadata to load the configured runtime shared libraries. To compile without runtime shared libraries either set the -static-link-runtime-shared-libraries option to true or remove the -runtime-shared-libraries option.

This warning indicates that the programmer may have overlooked the task of loading external libraries at runtime. The Flash Player will throw a runtime exception and halt further execution of the application if attempts are made to execute code which is not yet loaded.

The solution is to add a Frame metatag at the top of the application class and set its factoryClass attribute to point to a class that will be responsible to load external libraries.

If you were to inspect the output of the mxmlc compiler on a Flex application, you will see hard-coded references to all the RSLs that the application is using in the factoryClass designate. These references come from the configuration options passed on to the compiler through either its command-line parameters or the compiler configuration file.

The class that contains these references is usually a child of the mx.managers.SystemManager class and generated automatically by mxmlc. The SystemManager class provides the infrastructure to load these files along with error handling and progress feedback to the user.

This generated class also contains a reference to the entry point class – the one that extends from the Application class. When all the library files are loaded, the framework instantiates this class and adds it to the display list of the factoryClass designate. This makes the factoryClass the root document class, while the Application-derived class is actually a child of the loader in the display graph.

The programmer still provides the Application-inheriting class as the compiler target. But when the compiler encounters the Frame metatag, it automatically associates the Preloader class with the document root and makes the Application a child of the Preloader.

An ActionScript Implementation

When using ActionScript directly instead of the Flex framework, the developer must manually add the Frame metatag to the top of the application entry point class.

[Frame(factoryClass="Preloader")]
public class Main extends Sprite
{

}

The Preloader Class

The preloading is a straightforward consumption of the flash.display.Loader API. It is implemented here using the Preloader class. This class must fetch every external library file needed by the application. The paths to the libraries are supplied to the Preloader class. When using the Flex framework, the mxmlc compiler bakes in the references to the library files into the code that it generates. The example below also uses the same technique. However, the URLs of library files can also be supplied from any other source such as a web service or external text file. All standard Flash Player APIs are already available to the Preloader class.

public function Preloader()
{
	var loader:Loader = new Loader();
	loader.contentLoaderInfo.addEventListener(Event.COMPLETE, this.loader_completeHandler);
	loader.contentLoaderInfo.addEventListener(IOErrorEvent.IO_ERROR, this.loader_ioErrorHandler);
	var request:URLRequest = new URLRequest("math.swf");
	var context:LoaderContext = new LoaderContext(false, ApplicationDomain.currentDomain);
	loader.load(request, context);
}

It is important to note that the library has to be loaded into the same application domain as the main application. Otherwise, it will not have access to the classes in the library and any attempt to instantiate them will trigger a VerifyError at runtime.

VerifyError: Error #1014: Class com.notadesigner.math::IntegerArithmetic could not be found.

Using the Runtime Shared Library API

After the library has been downloaded, the Preloader class instantiates the application class and adds it to the display list. The developer must use the flash.system.getDefinitionByName API to get a reference to the application class. This is necessary because application class contains a reference to the IntegerArithmetic class. If the Preloader references Main directly, the compiler will pick up the complete chain of references and statically link the IntegerArithmetic class into the application SWF. By deferring to reference the application class until runtime, the compiler is prevented from scanning the dependency chain and statically linking the library classes into the application SWF.

private function loader_completeHandler(event:Event):void
{
	var mainClass:Class = getDefinitionByName("Main") as Class;
	var mainInstance:Main = new mainClass();
	this.addChild(mainInstance);
}

The Main class then continues with its business as normal. In this case, it is instantiating a type declared in the library and calling its method.

public function Main()
{
	var integer:IntegerArithmetic = new IntegerArithmetic(); // Type declared in math.swf
	var operand1:int = 10;
	var operand2:int = 10;
	var result:int = integer.add(operand1, operand2);
}

Deploying Runtime Shared Libraries

The confusing bit about using a runtime shared library is realizing that the SWF has to be extracted from the SWC at the time of deploying the application. This was not immediately obvious and I ended up spending days placing a compiled SWC file in various locations and wondering why the application was unable to load it at runtime. An obscure article on the Adobe website made explicit this particular step and set things straight.

Again, when placing the files, standard path rules apply. The Preloader can refer to relative or absolute paths. If the files are on external domains, the Flash Player attempts to fetch a crossdomain policy file before attempting to download the SWF. The policy file is to be specified as an additional value to the -runtime-shared-library-path parameter to mxmlc.

Sharing Code – Static and Dynamic Libraries in ActionScript

When I was in school, practically nobody had access to the internet. As a result, every student would have two essential items in his toolkit – an encyclopedia and a membership to a library. Homework, projects and supplementary studies were highly dependent on both of them.

Encyclopedias ranged from simple one-book units that specialized in a single subject, to multi-volume tomes spanning practically every topic imaginable (or at least deemed allowably imaginable by its publishers) by a school student. In both cases, the research was already done by someone else. When we were assigned a project on the solar system, nobody expected us to discover the planets in it. The planets remained discovered. All we had to do was read about them and share the information when making a presentation to the class.

Code Libraries

It is heartening to know that somebody creating computer languages took this to heart and invented the concept of code libraries – units of code written by people who are really good in the domain, then shared with the also rans who just wanted to implement their business applications without knowing the mathematics of relational databases or the bit-juggling that network libraries perform.

Libraries are compiled separately from the application that ends up eventually using them. They are linked to the application through a process that maps function calls in the application code with function addresses in the library. This process, predictably, is called linking.

Code stored in statically linked libraries is copied into the application executable at compile time, resulting an individual copy being created for every application that uses the library. Static libraries become an integral part of the application that uses them and cannot be shared with other applications. If the library is modified in any manner, the application must be recompiled and relinked with the new version in order to utilize those changes.

Dynamic libraries are stored in a shared location, and are linked to the application at runtime by the operating system or other environment controller such as a virtual machine. The metaphor falls apart a bit when it comes to sharing dynamic libraries – in a physical library, only one member can borrow a book at a time, whereas any number of programs can concurrently use the code stored in a dynamic library.

With static linking, it is enough to include those parts of the library that are directly and indirectly referenced by the target executable (or target library). With dynamic libraries, the entire library is loaded, as it is not known in advance which functions will be invoked by applications. Whether this advantage is significant in practice depends on the structure of the library.

Other than the obvious benefit of being able to compile units of code separately, either type of libraries offer their own individual benefits also. Statically linked code guarantees that all required code units are present and compatible with the application. Dynamically linked libraries may be of a different version than what the application requires, or not be present at all. Either case either causes a runtime error, or requires defensive code and reduced functionality in the application. Static linking also simplifies product distribution by reducing the number of files to be shipped and installed.

All said and done, both types of libraries are popular and well-worn concepts in computer programming. Many modern languages support either method, usually both.

Code Libraries in the Flash Ecosystem

Adobe’s MXML compiler offers several parameters for programmers to employ libraries – both static and dynamic – in their code.

Static linking is almost identical to how it is used in conventional languages. Dynamic linking works slightly differently from Windows’ Dynamically Linked Libraries and Unix’s Shared Objects due to the browser-based operation of the Flash Player. But these differences are at an implementation level only. The concepts remain the same.

Static Linking

To statically link a library into an application, the compiler offers the library-path and include-libraries directives.

library-path

When a library is specified using the library-path directive, the compiler includes only those assets and classes that are referenced in the application. For example, if the Math library contains IntegerArithmetic and FloatArithmetic for performing arithmetic operations on two separate numeric data types, but the client application only uses integers, the FloatArithmetic class is excluded from the output. This reduces the file size of the application SWF.

mxmlc Main.mxml -output=Main.swf -library-path=math.swc
include-libraries

The include-libraries directive packages the entire library, irrespective of what the compiler thinks is used in the application. This comes in handy in a dynamic language like ActionScript because all method calls do not necessarily get tested for linkages by the compiler.

var classRef:Class = getDefinitionByName("com.notadesigner.math.FloatArithmetic") as Class;
var instance:Object = new classRef();
instance["add"](10.5, 2.5); // Linkage not tested by the compiler

The include-libraries directive is used like this.

mxmlc Main.mxml -output=Main.swf -include-libraries=math.swc
static-link-runtime-shared-libraries

This directive is a convenient shortcut to convert runtime shared libraries into static libraries without significant changes to the compiler configuration files. Simply setting its value to true causes the compiler to embed all RSLs into the application. This is a handy way to investigate library-related problems when debugging the application.

mxml Main.mxml -output=Main.swf -static-link-runtime-shared-libraries=true

Dynamic Linking

A Runtime Shared Library (commonly abbreviated to RSL) is loaded by the application before it begins execution. Loading a RSL is a complicated task if done in plain ActionScript, but is taken care of automatically if the application builds on the Spark or MX Application class.

Adobe-provided framework libraries are the most obvious use case for using RSLs. All Flex-based applications depend upon these files. By caching them after the first time they are downloaded, future applications can be started much faster as they do not need to download the same code again. Custom libraries can also take advantage of the same technique.

Flash libraries are compiled using the compc utility, that ships as part of the Flex SDK. It generates a SWC file which is a compressed archive containing a SWF (named library.swf) and an XML (catalog.xml). To use this library, the developer must manually extract the SWF from the SWC using an archival utility (such as PKZip) and place it where the application can download it at runtime. As a good practice, the SWF is also usually renamed from library.swf to something more meaningful.

runtime-shared-library-path

This directive is used to specify the location of RSL files for the compiler. The compiler requires the names of both, the SWC as well as the extracted SWF, separated by a comma.

mxmlc -o=main.swf -runtime-shared-library-path=libs/math.swc,bin/math.swf Main.mxml

Related Directives

The Flex compiler provides two other directives to externalize the application’s assets and code fragments. The compiler tests linkages against these assets at compile-time, but leaves them out of the application binary. Libraries that contain these assets or classes are required at runtime, and the Flash Player throws a runtime error if they are not available.

These directives are useful when creating modules which are loaded dynamically into an application that they share code with. The application is linked to the requisite RSL, and the module does not need to fetch it again. However, the compiler still needs to test the linkages against the symbols – either against the original source code or a compiled binary. These directives assist in that task.

external-library-path

The compiler uses SWC files specified at these paths to test linkages with the application code, but does not compile the binaries into the application itself.

externs

This directive points to source files containing assets or classes which will be available at runtime. The compiler uses the source files for link testing, but does not compile them into the application binary.