This post is part of a series about creating a custom game engine.

In every job I have had I have become involved with the system of how the software is built, and this is also something I spent time making my graduate students work on in their game engine class. Every time, and in all of my personal projects, I have wished that I had my own system that worked the way that I wanted it to, and while I currently have the time to create a custom game engine I have taken the opportunity to try and make such a build system.

The term “build system” may mean different things to different people, but my usage is to refer to the process of taking any source files (code and assets) and transforming them into the final generated files that can be distributed in order to run the software. In some cases the process can be quite simple conceptually (in order to create an executable application using C++, for example, the source files are compiled into object files and then those object files are linked into an executable), but when creating a game the process becomes much more complex: Not only is there traditional code to deal with but there are also assets (input authored data that is not code) as well as interdependencies between code and assets where the content of one can influence how the other is built.

Wishlist

The following is a non-exhaustive list of what I would want in a build system, which contains elements both of the build system tool and of the implementation of how a piece of software uses a build system:

• The build system is simple to use
  • There should be exactly one step required to go from only the authored files (the ones contained in source control) to the final product that is ready to be distributed
  • (If there is more than one step that should be the fault of the human and not the build system, and the build system should make it possible to automate any extra steps so that there is only one)
• Builds are deterministic
  • Building the software should always have the same results
    • If an attempt to build has an error and then a subsequent attempt to build succeeds this is a problem
    • If an attempt to build has an error there is never a question of whether it is a real problem or not
• Nothing is done if a build is requested and nothing needs to be done
• It is fast to request a build when nothing needs to be done
  • When I am developing the common case should be to request the entire software to be built (it is exceptional when I want to build some parts but explicitly don’t want other parts to change) and it should be fast for a “no-op” build so that it doesn’t interfere with iteration
• Dependencies are easy to work with
  • It is easy for a human to specify important dependencies
  • As much as possible, however, the system should figure out the dependencies itself without requiring a human to specify them
  • If an attempt to build has an error there is never a question of whether it is because a dependency was missing
  • When an attempt to build something is made there is never a question of whether the result might not be valid because one of its dependencies was not in a correct state (this is restating the deterministic requirement)
• Assets (i.e. things that aren’t code) are easy to build
• Generated code is easy to build
• It is easy to build abstractions and to document and comment
  • When I am creating the system that builds software I want to be able to use good programming practices

When I am developing software with a build system that behaves as I describe above then I have the confidence to iterate and make changes and know that I am seeing the correct results of the changes I made, without worrying about extra steps or that something might be wrong.

Philosophy

I had to come up with some name and I settled on “jpmake”, a silly variation on the common naming scheme of “make”, “nmake”, “cmake”, etc., for build system software. It is based on some of the principles I have thought about for years, although its development is focused on allowing me to work on the custom engine project and so there are features that I know that I would like that I will intentionally not work on until/unless they are needed.

Despite using the term “build” and even though the motivation behind creating this is in order to build software I am designing things with the mental model that there is a “project” which is a collection of tasks that must be done (and these tasks may or may not have dependencies on each other). This means conceptually that a jpmake project could be used the same way e.g. a BAT file in Windows often is, to automate a series of tasks. Although a requirement is to be able to easily specify and build C++ programs the design goal is to have that be just one kind of task that can be done, with the hope that approaching it this way will make it easy to add other as-yet unanticipated kinds of tasks. The user should be able to focus on defining which tasks must be done and what the inputs and outputs of each task are, and then jpmake should be able to figure out the rest.

Lua

I am a big fan of the Lua scripting language and use it when I want to create a text-based interface. I have in fact used it multiple times in the past for smaller build systems, focusing on being able to build assets as part of a larger code-based project, and so I already have some experience with what I am trying to currently accomplish with jpmake.

My favorite thing about Lua is the ability to create interfaces that are easy to understand and use. It has enough flexibility that I can create files that are understandable even by someone who doesn’t know Lua (often understandable enough to make small changes), but it also has the power of a full programming language behind it so that I don’t feel limited when I want to use what I consider good programming practices.

As an example, observe the following lines from a jpmake project file:

local myTask = CreateNamedTask("myTask")
myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")

Without knowing any details you could probably guess what line 2 does, and even without knowing the correct syntax you could probably copy line 2, modify it, and get the correct result that you wanted and expected.

One issue I have run into with previous large-scale Lua projects that I’ve done, however, is that although I am completely satisfied with using the finished product in terms of the interface it has been very difficult to remember the details of the inner workings and to debug when things go wrong. For this current project I am taking a different approach to try and mitigate this, where I use Lua script files as the project files that specify what tasks to execute and how, but otherwise everything is implemented in C++. In the case of how I anticipate using jpmake this is probably what I would have done anyway because it means that there is a single executable application that can do everything, but it also has the advantage of easier maintainability because of static typing and making it easy to debug. (Additionally, of course, it can be more efficient. I am trying to manage all of the strings that exist in a program like this (i.e. because there are so many file paths) in an efficient way to keep things fast that wouldn’t be possible if the implementation were in pure Lua.)

Example

Details will have to wait for a part 2, but below is an example from the test project file that I have been using while developing:

-- Set Up
--=======

-- Define the platforms that tasks can be executed for for
local platform_windows = DefinePlatform("Windows", "win64")
-- Get the currently-specified (from the command arguments) platform
local platform_current = GetTargetPlatform()
local platformName_current = platform_current:GetName()
-- Define the configurations that can be used when executing tasks
local configuration_debug = "Debug"
local configuration_optimized = "Optimized"
DefineConfigurations{configuration_debug, configuration_optimized}
-- Get the currently-specified (from the command arguments) configuration
local configuration_current = GetConfiguration()

-- Define environment variables that can be used in paths
SetEnvironmentVariable("TempDir", table.concat{"temp/", platformName_current, "/", configuration_current, "/"})
SetEnvironmentVariable("IntermediateDir", "$(TempDir)intermediate/")
SetEnvironmentVariable("OutputDir", "$(TempDir)output/")

-- Define the C++ info that determines how C++ is built
local cppInfo_common = CreateCppInfo()
do
	-- Libraries can be disabled for situations that only use custom code
	do
		cppInfo_common.shouldStandardLibrariesBeAvailable = false
		cppInfo_common.shouldPlatformLibrariesBeAvailable = false
	end
	cppInfo_common.shouldExceptionsBeEnabled = false
	-- Platform-specific configuration
	if (platform_current == platform_windows) then
		-- A #define can be set for platform-specific conditional compilation
		cppInfo_common:AddPreprocessorDefine("PLATFORM_WINDOWS")
		-- Preprocessor symbols can also have values
		cppInfo_common:AddPreprocessorDefine("PLATFORM_NAME", platformName_current)
		-- Use different VC run-times based on the current configuration
		cppInfo_common.VisualStudio.shouldCRunTimeBeDebug = configuration_current == configuration_debug
	end
end
-- Alternate C++ infos can be created from the base configuration and then have targeted changes
local cppInfo_withStandardLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withStandardLibraries.shouldStandardLibrariesBeAvailable = true
end
local cppInfo_withPlatformLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withPlatformLibraries.shouldPlatformLibrariesBeAvailable = true
end

-- C++ Tasks
--==========

do
	-- A "named task" is just for human organization:
	-- It allows only a subset of sub tasks to be executed (rather than the entire project) by specifying a name
	local cppNamedTask = CreateNamedTask("BuildMyCppProgram")
	local staticLibrary_usingPlatformLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesPlatformLibraries.lib",
			compile = {
				"MyClass.cpp",
			},
			-- Any libraries specified as needing to be linked when defining static library tasks
			-- don't actually influence the creation of the static libraries themselves,
			-- but instead are stored as dependencies
			-- that eventually are linked when an application is created that uses the static libraries
			link = {
				platform_current == platform_windows and CalculateAbsolutePathOfPlatformCppLibrary("Advapi32.lib", cppInfo_withPlatformLibraries) or nil,
			},
			-- This static library looks up a registry value,
			-- and by requesting that platform libraries are available
			-- it allows the #include <windows.h> directive to look in the correct Windows SDK folder
			info = cppInfo_withPlatformLibraries,
		}
	local staticLibrary_usingStandardLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesStandardLibraries.lib",
			compile = {
				"MySource.cpp",
			},
			-- This static library uses std::cout,
			-- and by requesting that standard libraries are available
			-- it allows the #include <iostream> directive to look in the correct Visual Studio folder
			info = cppInfo_withStandardLibraries,
		}
	local myApplication = cppNamedTask:BuildCpp{
			targetType = "consoleApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "windowedApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "sharedLibrary", target = "$(OutputDir)MyLibrary.dll",
			compile = {
				"MyEntryPoint.cpp",
			},
			link = {
				staticLibrary_usingPlatformLibraries,
				staticLibrary_usingStandardLibraries,
			},
			info = cppInfo_common,
		}
end

-- Copy Tasks
--===========

do
	-- Here is an example of a differently-named task that does something other than building C++
	local myTask = CreateNamedTask("myTask")
	myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")
	myTask:Copy("$(OutputDir)someDirectory/someIntermediateFile.txt", "$(OutputDir)someDirectory/someOtherDirectory/someCopiedFile.txt")	
end