Application with an Empty Window

Behold, the main window of an application using my nascent engine:

It does as little as one might expect from the screenshot, but some of the architecture might be of interest.

Build System

This section shows parts of the jpmake project file.

Platforms and Configurations

The different possible platforms and configurations are defined as follows:

-- Platforms
------------

-- Define the platforms by name and type

platform_windows = DefinePlatform("windows", "win64")
local platform_windows = platform_windows

-- Get the specified platform to execute tasks for
platform_target = GetTargetPlatform()
local platform_target = platform_target

-- Configurations
-----------------

-- Define the configurations by name

configuration_unoptimized = DefineConfiguration("unoptimized")
local configuration_unoptimized = configuration_unoptimized

configuration_optimized = DefineConfiguration("optimized")
local configuration_optimized = configuration_optimized

configuration_profile = DefineConfiguration("profile")
local configuration_profile = configuration_profile

configuration_release = DefineConfiguration("release")
local configuration_release = configuration_release

-- Get the specified configuration to execute tasks for
local configuration_current = GetCurrentConfiguration()

I am creating global variables for each platform and configuration so that they are accessible by other Lua files and then immediately assigning them to local variables so that they are cheaper to use in this file. (I currently only have the single Lua project file, but soon I will want to change this and have separate files that can focus on different parts of the project.)

At the moment I am specifying any platform-specific information using a strategy like if platform_target == platform_windows and that works fine (there are several examples later in this post), but I am considering defining something like isWindows = platform_target == platform_windows instead. There won’t be many platforms (only one for the foreseeable future!) and it seems like it would be easier to read and write many platform-specific things with a single boolean rather than with a long comparison. I am doing something similar with the configurations where I define booleans that serve as classification descriptions, and so far it feels nice to me (again, there are examples later in this post).

Directory Structure

The directory structure from the perspective of jpmake is currently defined as follows:

-- Source Files
do
	SetEnvironmentVariable("engineDir", "Engine/")
	SetEnvironmentVariable("appsDir", "Apps/")
end
-- Generated Files
do
	-- Anything in the temp directory should be generated by jpmake executing tasks
	-- and the entire folder should be safely deletable.
	-- Additionally, any files that are not part of the Git repository
	-- should be restricted to this folder.
	SetEnvironmentVariable("tempDir", ConcatenateTable{"temp/", platform_target:GetName(), "/", configuration_current, "/"})
	-- The intermediate directory is for files that must be generated while executing tasks
	-- but which aren't required to run the final applications
	SetEnvironmentVariable("intermediateDir", "$(tempDir)intermediate/")
	SetEnvironmentVariable("intermediateDir_engine", "$(intermediateDir)engine/")
	-- The artifact directory is where jpmake saves files
	-- that it uses to execute tasks
	SetArtifactDirectory("$(intermediateDir)jpmake/")
	-- The staging directory contains the final applications that can be run
	-- independently of the source project and intermediate files
	SetEnvironmentVariable("stagingDir", "$(tempDir)staging/")
end

As a general rule I don’t like abbreviations in variable or function names but I decided to keep the “dir” convention from Visual Studio since these environment variable names will be used so frequently in paths that it seems like a reasonable exception to keep things shorter and more readable. (I did, however, decide to change the first letter to lowercase which fits with my variable naming convention better.)

An issue that I have run into in the past is having trouble deciding how to name directory environment variables to distinguish between source and generated files, and with games where there can be code and assets both for the engine and the application the possible choices are even more complex (and, to make matters worse, with this project I am intending to support multiple applications using the engine and so there is yet a further distinction that must be made). What I have will likely change as time goes on and I write more code, but it feels like a good start. The root repository folder looks like this:

Any files that are generated by the build process are kept quarantined in a single folder (temp/) so that the distinction between source and deletable files is very clear. This is very important to me (as anyone who has worked with me can attest). The temp directory looks like the following, expanded for one platform and configuration:

With such a simple application the only thing in the staging directory is the executable file, but when I develop more complicated applications there will be other files in staging directories (e.g. the assets that the game loads).

One further consideration that is currently missing is what to do with “tools”, those programs that are used during development (either for authoring content or as part of the build process) but that don’t get released to end users. I can imagine that I might want to update some of this directory structure when I start developing tools.

C++ Configuration

The next section in the jpmake project file configures the default settings for how C++ is built for the current target platform and build configuration:

-- C++
------

-- Initialize C++ for the current platform and configuration
cppInfo_common = CreateCppInfo()
local cppInfo_common = cppInfo_common
do
	-- #define VLSH_PLATFORM_SOMENAME for conditional compilation
	do
		local platform_define_suffix
		if (platform_target == platform_windows) then
			platform_define_suffix = "WINDOWS"
		else
			platform_define_suffix = "NONE"
		end
		cppInfo_common:AddPreprocessorDefine(("VLSH_PLATFORM_" .. platform_define_suffix),
			-- There isn't any anticipated reason to check anything other than whether the platform is #defined,
			-- but the name is used as a value because why not?
			platform_target:GetName())
	end
	-- The project directory is used as an $include directory
	-- so that directives like the following can be done to show scope:
	--	#include <Engine/SomeFeature/SomeHeader.hpp>	
	cppInfo_common:AddIncludeDirectory(".")
	local isOptimized = configuration_current ~= configuration_unoptimized
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISOPTIMIZED", isOptimized)
	local isForProfiling = configuration_current == configuration_profile
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISFORPROFILING", isForProfiling)
	local isForRelease = configuration_current == configuration_release
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISFORRELEASE", isForRelease)
	do
		local areAssertsEnabled = not isForRelease and not isForProfiling
		cppInfo_common:AddPreprocessorDefine("VLSH_ASSERT_ISENABLED", areAssertsEnabled)
	end
	cppInfo_common.shouldStandardLibrariesBeAvailable = false
	cppInfo_common.shouldPlatformLibrariesBeAvailable = false
	cppInfo_common.shouldExceptionsBeEnabled = false
	cppInfo_common.shouldDebugSymbolsBeAvailable =
		-- Debug symbols would also have to be available for release in order to debug crashes
		not isForRelease
	if platform_target == platform_windows then
		cppInfo_common.VisualStudio.shouldCRunTimeBeDebug = not isOptimized
		cppInfo_common.VisualStudio.shouldIncrementalLinkingBeEnabled =
			-- Incremental linking speeds up incremental builds at the expense of bigger executable size
			not isForRelease
		-- Warnings
		do
			cppInfo_common.VisualStudio.shouldAllCompilerWarningsBeErrors = true
			cppInfo_common.VisualStudio.shouldAllLibrarianWarningsBeErrors = true
			cppInfo_common.VisualStudio.compilerWarningLevel = 4
		end
	end
end

This shows the general approach I am taking towards configuring things (both from the perspective of the game engine and also from the perspective of jpmake and my personal ideal way of configuring software builds). The named configurations (e.g. unoptimized, optimized, profile, release) that I defined earlier are just arbitrary names from the perspective of jpmake and don’t have any semantics associated with them. Instead it is up to the user to specify how each configuration behaves. I can imagine that this would be seen as a negative for most people, but I have a personal issue where I generally prefer to have full control over things.

This section should not be understood as being complete (most notably there actually aren’t any optimization-related settings except for which C run-time to use!) but that is because I haven’t implemented all of the Visual Studio options in jpmake yet.

Engine Static Library

Below is one example of a static library that I have made, which provides base classes for applications (meaning that an actual application can inherit from the provided framework):

do
	local task_application = CreateNamedTask("Application")
	local cppInfo_application = cppInfo_common:CreateCopy()
	do
		if (platform_target == platform_windows) then
			cppInfo_application.shouldPlatformLibrariesBeAvailable = true
		end
	end
	engineLibrary_application = task_application:BuildCpp{
			target = "$(intermediateDir_engine)Application.lib", targetType = "staticLibrary",
			compile = {
				"$(engineDir)Application/iApplication.cpp",
				"$(engineDir)Application/iApplication_windowed.cpp",
				platform_target == platform_windows and "$(engineDir)Application/iApplication_windowed.win64.cpp" or nil,
			},
			link = {
				engineLibrary_assert,
				platform_target == platform_windows and CalculateAbsolutePathOfPlatformCppLibrary("User32.lib", cppInfo_application) or nil,
			},
			info = cppInfo_application,
		}
end
local engineLibrary_application = engineLibrary_application

My current plan is to have the “engine” consist of a collection of static libraries that all get linked into the single application executable.

This named task shows a file specific to Windows that is only compiled for that platform (iApplication_windowed.win64.cpp, where my convention is to try to put as much platform-specific code in separate platform-specific CPP files as possible and then those files have the platform name as a sub-extension), as well as a Windows library that is only needed for linking on that platform (User32.lib) and another static library (engineLibrary_assert, which was defined earlier but that I don’t show in this blog post) that this static library depends on.

As more files get created that are specific to one platform or another I think my style will have to change to make it less annoying to conditionally specify each one.

Applications

Finally, the two proof-of-concept applications that I have created are defined as follows:

-- Hello World
--============

do
	do
		SetEnvironmentVariable("appDir", "$(appsDir)HelloWorld/")
		SetEnvironmentVariable("stagingDir_app", "$(stagingDir)HelloWorld/")
	end
	local cppInfo_helloWorld = cppInfo_common:CreateCopy()
	do
		-- For std::cout
		cppInfo_helloWorld.shouldStandardLibrariesBeAvailable = true
	end
	do
		local helloWorld_task = CreateNamedTask("HelloWorld")
		local application_subTask = helloWorld_task:BuildCpp{
				target = "$(stagingDir_app)HelloWorld.exe", targetType = "consoleApplication",
				compile = {
					"$(appDir)EntryPoint.cpp",
				},
				info = cppInfo_helloWorld,
			}
		helloWorld_task:SetTargetForIde(application_subTask)
	end
end

-- Empty Window
--=============

do
	do
		SetEnvironmentVariable("appDir", "$(appsDir)EmptyWindow/")
		SetEnvironmentVariable("stagingDir_app", "$(stagingDir)EmptyWindow/")
	end
	local cppInfo_emptyWindow = cppInfo_common:CreateCopy()
	do
		cppInfo_emptyWindow:AddIncludeDirectory("$(appDir)")
	end
	do
		local emptyWindow_task = CreateNamedTask("EmptyWindow")
		local application_subTask = emptyWindow_task:BuildCpp{
				target = "$(stagingDir_app)EmptyWindow.exe", targetType = "windowedApplication",
				compile = {
					"$(appDir)EntryPoint.cpp",
				},
				link = {
					engineLibrary_application,
				},
				info = cppInfo_emptyWindow,
			}
		emptyWindow_task:SetTargetForIde(application_subTask)
	end
end

These show the general approach towards making executable applications that I am envisioning, although these both are as simple as possible.

One idiom that I discovered is reusing the same environment variable names but setting them to different values for different applications. This allowed the names to be shorter and thus more readable (before this I had different versions with _helloWorld and _emptyWindow), but I don’t have enough experience to decide if this will work well long term.

The examples also show calls to SetTargetForIde(), which has no effect when executing tasks but is instead used when generating the solution files so that Visual Studio will correctly have its $(TargetPath) set, which makes setting up debugging easier.

Visual Studio Solution

It is now possible for jpmake to generate Visual Studio solution and project files. I did this work to make it easier to write code and debug in Visual Studio. The Solution Explorer currently looks like the following for the jpmake project that I have been showing in this post:

And the properties of the EmptyWindow project have some things filled in:

I had to spend more time on generating these files and additional jpmake features than I had initially anticipated before working on the engine code because I wasn’t able to debug, which felt like a requirement. With the way it works now, however, I was able to write the empty window application and things worked reasonably well.

I did have one discouraging realization, however, which is that Intellisense doesn’t work yet. I was able to complete the empty window application without it but it was more annoying than I would have anticipated. I think I need to take some more time to improve jpmake so that Intellisense will work at least somewhat because not having it has proven to be an annoying impediment.

Custom Build System (Part 1)

This post is part of a series about creating a custom game engine.

In every job I have had I have become involved with the system of how the software is built, and this is also something I spent time making my graduate students work on in their game engine class. Every time, and in all of my personal projects, I have wished that I had my own system that worked the way that I wanted it to, and while I currently have the time to create a custom game engine I have taken the opportunity to try and make such a build system.

The term “build system” may mean different things to different people, but my usage is to refer to the process of taking any source files (code and assets) and transforming them into the final generated files that can be distributed in order to run the software. In some cases the process can be quite simple conceptually (in order to create an executable application using C++, for example, the source files are compiled into object files and then those object files are linked into an executable), but when creating a game the process becomes much more complex: Not only is there traditional code to deal with but there are also assets (input authored data that is not code) as well as interdependencies between code and assets where the content of one can influence how the other is built.

Wishlist

The following is a non-exhaustive list of what I would want in a build system, which contains elements both of the build system tool and of the implementation of how a piece of software uses a build system:

The build system is simple to use
- There should be exactly one step required to go from only the authored files (the ones contained in source control) to the final product that is ready to be distributed
- (If there is more than one step that should be the fault of the human and not the build system, and the build system should make it possible to automate any extra steps so that there is only one)
Builds are deterministic
- Building the software should always have the same results
  - If an attempt to build has an error and then a subsequent attempt to build succeeds this is a problem
  - If an attempt to build has an error there is never a question of whether it is a real problem or not
Nothing is done if a build is requested and nothing needs to be done
It is fast to request a build when nothing needs to be done
- When I am developing the common case should be to request the entire software to be built (it is exceptional when I want to build some parts but explicitly don’t want other parts to change) and it should be fast for a “no-op” build so that it doesn’t interfere with iteration
Dependencies are easy to work with
- It is easy for a human to specify important dependencies
- As much as possible, however, the system should figure out the dependencies itself without requiring a human to specify them
- If an attempt to build has an error there is never a question of whether it is because a dependency was missing
- When an attempt to build something is made there is never a question of whether the result might not be valid because one of its dependencies was not in a correct state (this is restating the deterministic requirement)
Assets (i.e. things that aren’t code) are easy to build
Generated code is easy to build
It is easy to build abstractions and to document and comment
- When I am creating the system that builds software I want to be able to use good programming practices

When I am developing software with a build system that behaves as I describe above then I have the confidence to iterate and make changes and know that I am seeing the correct results of the changes I made, without worrying about extra steps or that something might be wrong.

Philosophy

I had to come up with some name and I settled on “jpmake”, a silly variation on the common naming scheme of “make”, “nmake”, “cmake”, etc., for build system software. It is based on some of the principles I have thought about for years, although its development is focused on allowing me to work on the custom engine project and so there are features that I know that I would like that I will intentionally not work on until/unless they are needed.

Despite using the term “build” and even though the motivation behind creating this is in order to build software I am designing things with the mental model that there is a “project” which is a collection of tasks that must be done (and these tasks may or may not have dependencies on each other). This means conceptually that a jpmake project could be used the same way e.g. a BAT file in Windows often is, to automate a series of tasks. Although a requirement is to be able to easily specify and build C++ programs the design goal is to have that be just one kind of task that can be done, with the hope that approaching it this way will make it easy to add other as-yet unanticipated kinds of tasks. The user should be able to focus on defining which tasks must be done and what the inputs and outputs of each task are, and then jpmake should be able to figure out the rest.

Lua

I am a big fan of the Lua scripting language and use it when I want to create a text-based interface. I have in fact used it multiple times in the past for smaller build systems, focusing on being able to build assets as part of a larger code-based project, and so I already have some experience with what I am trying to currently accomplish with jpmake.

My favorite thing about Lua is the ability to create interfaces that are easy to understand and use. It has enough flexibility that I can create files that are understandable even by someone who doesn’t know Lua (often understandable enough to make small changes), but it also has the power of a full programming language behind it so that I don’t feel limited when I want to use what I consider good programming practices.

As an example, observe the following lines from a jpmake project file:

local myTask = CreateNamedTask("myTask")
myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")

Without knowing any details you could probably guess what line 2 does, and even without knowing the correct syntax you could probably copy line 2, modify it, and get the correct result that you wanted and expected.

One issue I have run into with previous large-scale Lua projects that I’ve done, however, is that although I am completely satisfied with using the finished product in terms of the interface it has been very difficult to remember the details of the inner workings and to debug when things go wrong. For this current project I am taking a different approach to try and mitigate this, where I use Lua script files as the project files that specify what tasks to execute and how, but otherwise everything is implemented in C++. In the case of how I anticipate using jpmake this is probably what I would have done anyway because it means that there is a single executable application that can do everything, but it also has the advantage of easier maintainability because of static typing and making it easy to debug. (Additionally, of course, it can be more efficient. I am trying to manage all of the strings that exist in a program like this (i.e. because there are so many file paths) in an efficient way to keep things fast that wouldn’t be possible if the implementation were in pure Lua.)

Example

Details will have to wait for a part 2, but below is an example from the test project file that I have been using while developing:

-- Set Up
--=======

-- Define the platforms that tasks can be executed for for
local platform_windows = DefinePlatform("Windows", "win64")
-- Get the currently-specified (from the command arguments) platform
local platform_current = GetTargetPlatform()
local platformName_current = platform_current:GetName()
-- Define the configurations that can be used when executing tasks
local configuration_debug = "Debug"
local configuration_optimized = "Optimized"
DefineConfigurations{configuration_debug, configuration_optimized}
-- Get the currently-specified (from the command arguments) configuration
local configuration_current = GetConfiguration()

-- Define environment variables that can be used in paths
SetEnvironmentVariable("TempDir", table.concat{"temp/", platformName_current, "/", configuration_current, "/"})
SetEnvironmentVariable("IntermediateDir", "$(TempDir)intermediate/")
SetEnvironmentVariable("OutputDir", "$(TempDir)output/")

-- Define the C++ info that determines how C++ is built
local cppInfo_common = CreateCppInfo()
do
	-- Libraries can be disabled for situations that only use custom code
	do
		cppInfo_common.shouldStandardLibrariesBeAvailable = false
		cppInfo_common.shouldPlatformLibrariesBeAvailable = false
	end
	cppInfo_common.shouldExceptionsBeEnabled = false
	-- Platform-specific configuration
	if (platform_current == platform_windows) then
		-- A #define can be set for platform-specific conditional compilation
		cppInfo_common:AddPreprocessorDefine("PLATFORM_WINDOWS")
		-- Preprocessor symbols can also have values
		cppInfo_common:AddPreprocessorDefine("PLATFORM_NAME", platformName_current)
		-- Use different VC run-times based on the current configuration
		cppInfo_common.VisualStudio.shouldCRunTimeBeDebug = configuration_current == configuration_debug
	end
end
-- Alternate C++ infos can be created from the base configuration and then have targeted changes
local cppInfo_withStandardLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withStandardLibraries.shouldStandardLibrariesBeAvailable = true
end
local cppInfo_withPlatformLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withPlatformLibraries.shouldPlatformLibrariesBeAvailable = true
end

-- C++ Tasks
--==========

do
	-- A "named task" is just for human organization:
	-- It allows only a subset of sub tasks to be executed (rather than the entire project) by specifying a name
	local cppNamedTask = CreateNamedTask("BuildMyCppProgram")
	local staticLibrary_usingPlatformLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesPlatformLibraries.lib",
			compile = {
				"MyClass.cpp",
			},
			-- Any libraries specified as needing to be linked when defining static library tasks
			-- don't actually influence the creation of the static libraries themselves,
			-- but instead are stored as dependencies
			-- that eventually are linked when an application is created that uses the static libraries
			link = {
				platform_current == platform_windows and CalculateAbsolutePathOfPlatformCppLibrary("Advapi32.lib", cppInfo_withPlatformLibraries) or nil,
			},
			-- This static library looks up a registry value,
			-- and by requesting that platform libraries are available
			-- it allows the #include <windows.h> directive to look in the correct Windows SDK folder
			info = cppInfo_withPlatformLibraries,
		}
	local staticLibrary_usingStandardLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesStandardLibraries.lib",
			compile = {
				"MySource.cpp",
			},
			-- This static library uses std::cout,
			-- and by requesting that standard libraries are available
			-- it allows the #include <iostream> directive to look in the correct Visual Studio folder
			info = cppInfo_withStandardLibraries,
		}
	local myApplication = cppNamedTask:BuildCpp{
			targetType = "consoleApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "windowedApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "sharedLibrary", target = "$(OutputDir)MyLibrary.dll",
			compile = {
				"MyEntryPoint.cpp",
			},
			link = {
				staticLibrary_usingPlatformLibraries,
				staticLibrary_usingStandardLibraries,
			},
			info = cppInfo_common,
		}
end

-- Copy Tasks
--===========

do
	-- Here is an example of a differently-named task that does something other than building C++
	local myTask = CreateNamedTask("myTask")
	myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")
	myTask:Copy("$(OutputDir)someDirectory/someIntermediateFile.txt", "$(OutputDir)someDirectory/someOtherDirectory/someCopiedFile.txt")	
end