Month: November 2024

Syncing without VSync

I don’t remember how many years ago it was when I first read about doing this, but it was probably the following page which introduced me to the idea: https://blurbusters.com/blur-busters-lagless-raster-follower-algorithm-for-emulator-developers/. I am pleased to be able to say that I have finally spent the time to try and implement it myself:

How it Works

An application can synchronize with a display’s refresh cycle (instead of using vsync to do it) if two pieces of data are known:

  • The length of a single refresh (i.e. the refresh rate)
    • This is a duration
  • The time when a refresh ends and the next one starts (i.e. the vblank)
    • This is a repeating timestamp (a moment in time)

If both of these are known then the application can predict when the next refresh will start and update the texture that the graphics card is sending to the display at the appropriate time. If the graphics card changes the texture at the wrong time then “tearing” is visible, which is a horizontal line that separates the old previous texture above it and the new texture below it. (This Wikipedia article as a simulated example image of tearing.)

The texture that is active that the graphics card is sending to the display is called the “front buffer”. The texture that isn’t visible but that the application can generate before it is activated and being sent to the display is called the “back buffer”. There is different terminology for the act of making a back buffer into a front buffer but this post will call it “swapping”, and conceptually it can be thought of as treating the former back buffer as the new front buffer while what was the front buffer becomes a back buffer.

(What vsync does is take care of swapping front and back buffers at the appropriate time. If the application finishes generating a new texture in the back buffer and submits it in time then the operating system and graphics card will work together to make sure that it becomes the new front buffer for the display during the vblank. If the application doesn’t submit it in time and misses the vblank then nothing changes visually: The display keeps showing the old front buffer (meaning that a frame is repeated) and there is no tearing).

Why does swapping the front and back buffers at the wrong time cause tearing? Because even though swapping the front and back buffers happens instantaneously in computer memory the change from one frame to another on a display doesn’t. Instead the display updates gradually, starting at the top and ending at the bottom. Although this isn’t visible to a human eye the effect can be observed using a slow motion camera:

With that in mind it is possible to understand what I am doing in my video: I wrote a program that is manually swapping front and back buffers four times every refresh period in order to intentionally cause tearing. I am not actually rendering anything but instead just doing a very simple clearing of the entire back buffer to some single color, but by swapping a single color back buffer at the correct time in the middle of a display’s refresh I can change the color somewhere in the middle of the display.

Doing this isn’t particularly new and my results aren’t particularly impressive, but I was happy to finally find the time to try it.

Here is a bonus image where I wasn’t trying to time anything and instead just swapped between alternating red and green as quickly as I was able to:

Implementation Details

The rest of this post contains some additional technical information that I encountered while implementing this. Unlike the preceding section which was aimed at a more general audience the remainder will be for programmers who are interested in specific implementation details.

How to Calculate Time

Calculating the duration of a refresh period isn’t particularly difficult but it’s not sufficient to simply use the reported refresh rate. Although the nominal refresh rate would be close it wouldn’t exactly match the time reported by your CPU clock, and that’s what matters because that’s what you’ll be using to know when to swap buffers. In order to know what the refresh rate is in terms of the CPU clock an average of observed durations must be made. In order to calculate a duration it is necessary to keep track of how much time has elapsed between consistently repeating samples, but it doesn’t actually matter where in the refresh cycle these samples come from as long as they are consistently taken from the same (arbitrary) point in the refresh cycle. So, for example, timing after IDXGISwapChain::Present() returns (with a sync interval of 1 and a full present queue) would work, and timing after IDXGIOutput::WaitForVBlank() returns would also work.

It is more difficult, however, to calculate when the vblank actually happens.

DXGI_FRAME_STATISTICS

I finally settled on using IDXGISwapChain::GetFrameStatistics(). Using this meant that I was relying on DXGI instead of taking my own time measurements, but the big attraction of doing that is that the timestamps were then tied directly to discrete counters. Additionally, as a side benefit, after a bit of empirical testing it seemed like the sampled time in the DXGI frame statistics was always earlier than the time that I could sample myself, and so it seems like it is probably closer to the actual vblank than anything that I knew how to measure.

(The somewhat similar DwmGetCompositionTimingInfo() did not end up being as useful for me as I had initially thought. Alternatively, D3DKMTGetScanLine() seems like it could, in theory, be used for even more accurate results, but it wasn’t tied to discrete frame counters which made it more daunting. If my end goal had been just this particular demo I might have tried using that, but for my actual game engine renderer it seemed like IDXGISwapChain::GetFrameStatistics() would be easier, simpler and more robust.)

The problem that I ran into, however, is that I couldn’t find satisfactory explanations of what the fields of DXGI_FRAME_STATISTICS actually mean. I had to spend a lot of time doing empirical tests to figure it out myself, and I am going to document my findings here. If you found this post using an internet search for any of these DXGI_FRAME_STATISTICS-related terms then I hope this explanation saves you some time. (Alternatively, if you are reading this and find any mistakes in my understanding then please comment with corrections both for me and other readers.)

My Findings

The results of IDXGISwapChain::GetFrameStatistics() are a snapshot in time.

If you call IDXGISwapChain::GetLastPresentCount() immediately after IDXGISwapChain::Present() you will get the correct identifier for the present call that you just barely made, and this is very important to do in order to be able to correctly associate an individual present function call that you made with the information in the DXGI frame statistics (or, at least, it is conceptually important to do; you can also just keep track yourself of how many successful requests to present have been made).

On the other hand, if you call IDXGISwapChain::GetFrameStatistics() immediately after IDXGISwapChain::Present() there is no guarantee that you will get updated statistics (and, in fact, you most likely won’t). Instead, there is some non-deterministic (for you) moment in time after calling IDXGISwapChain::Present() where you would eventually get statistics for that specific request to present in the results of a call to IDXGISwapChain::GetFrameStatistics().

How do you know if the statistics you get are the ones that you want? You know that they are the ones that you want if the PresentCount field matches the value you got from IDXGISwapChain::GetLastPresentCount() after IDXGISwapChain::Present(). Once you call IDXGISwapChain::GetFrameStatistics() and get a PresentCount that matches the one that you’re looking for then you know two things:

  • The statistics that you now have refer to the known state of things when your submitted request to present (made by your call to IDXGISwapChain::Present() ) was actually presented
  • The statistics that you now have will not be updated again for your specific present request. What you now have is the snapshot that was made for your PresentCount , and no more snapshots will be made until another call to IDXGISwapChain::Present() is made (which means that the next time the statistics get updated they will be referring to a different PresentCount from the one that you are currently interested in).

Once you have a DXGI_FRAME_STATISTICS that is a snapshot for your specific PresentCount the important corresponding number is PresentRefreshCount. This tells which refresh of the display your request to present was actually presented during. If vsync is enabled PresentRefreshCount is the refresh of the display when your request to present was actually presented.

Once you have that information you can, incidentally, detect whether your request to present actually happened when you wanted and expected it to. This is described at https://learn.microsoft.com/en-us/windows/win32/direct3ddxgi/dxgi-flip-model#avoiding-detecting-and-recovering-from-glitches, in the “to detect a glitch” section. Although the description of what PresentCount and PresentRefreshCount is confusing to me in that document (and in other official documentation) the description of how to detect a glitch is consistent in my mind with how I have described these fields above, which helps to give me confidence that my understanding is probably correct.

Once you know the information above you can now potentially get timing information. The SyncRefreshCount refers to the same thing as PresentRefreshCount (it is a counter of display refresh cycles), and so it may be confusing why two different fields exist and what the distinction is between the two. PresentRefreshCount is, as described above, a mapping between PresentCount and a display refresh. SyncRefreshCount, on the other hand, is a mapping between the value in SyncQPCTime and a display refresh. The value in SyncQPCTime is a timestamp corresponding to the refresh in SyncRefreshCount. If SyncRefreshCount is the same as PresentRefreshCount then you know (approximately) the time of the vblank when your PresentCount request was actually displayed. It is possible, however, for SyncRefreshCount to be different from PresentRefreshCount, and that is why both fields are in the statistics struct.

To repeat: Information #1 is which display refresh your request was actually displayed in (comparing PresentCount to PresentRefreshCount) and information #2 is what the (approximate) time of a vblank for a specific refresh was (comparing SyncQPCTime to SyncRefreshCount). Derived information #3 is what the (approximate) time of a vblank was for the refresh that your request was actually displayed in.

(Side note: The official documentation here and here is very intentionally vague about when SyncQPCTime is actually measured. The driver documentation here, however, says “CPU time that the vertical retrace started”. I’m not sure if the more accessible user-facing documentation is intentionally vague to not be held accountable for how accurate the timing information is, or if the driver documentation is out-of-date. This post chooses to believe that the time is supposed to refer to the beginning of a refresh, with the caveat that I may be wrong and that even if I’m not wrong the sampled time is clearly not guaranteed to be highly accurate.)

One final thing to mention: A call to IDXGISwapChain::GetFrameStatistics() may return DXGI_ERROR_FRAME_STATISTICS_DISJOINT. One thing to note is that the values in PresentRefreshCount and SyncRefreshCount are monotonically-increasing and, specifically, they don’t reset even when the refresh rate changes. The consequence of this is that the DXGI_ERROR_FRAME_STATISTICS_DISJOINT result is very important for determining timing (like this post is concerned about). If you record the first PresentRefreshCount reported in the first successful call after DXGI_ERROR_FRAME_STATISTICS_DISJOINT was returned then you have a reference point for any future SyncRefreshCounts reported (until the next DXGI_ERROR_FRAME_STATISTICS_DISJOINT). Specifically, you know how many refresh cycles have happened with the current refresh rate.

How to Calculate Refresh Period

Calculating the refresh period using SyncRefreshCount and SyncQPCTime is not difficult: Average the elapsed time between the sampled timestamps of refreshes. I am using the incremental method described here: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford’s_online_algorithm. This is easy to calculate and doesn’t require any storage beyond the current mean and the sample count. It can have problems with outliers or if the duration changes, though, and although I don’t anticipate either of those being issues it remains to be seen.

How to Predict VBlanks

I did some thinking about how to do this, and after some internet searching (I am not a numerical methods expert and so it took me a while to even figure out the correct search terms for what I was thinking) I found a really nice post about how to do exactly what I wanted, an incrementally-updating method for calculating the line that is the least-squares best fit for a bunch of sample points: https://blog.demofox.org/2016/12/22/incremental-least-squares-curve-fitting/. I liked this because it was a match for the incremental average function I was using, and since refresh cycles should happen regularly I figured that I could use SyncRefreshCount as the independent variable and SyncQPCTime as the dependent variable and then have a really computationally cheap way of predicting the time for any given refresh (in the past or in the future).

The good news is that this worked really well after some tweaking of my initial naive approach! The bad news is that the changes that I had to make in order to make it work well made me nervous about whether it would continue to perform well over time.

The big problem was losing precision. The SyncRefreshCount are inherently big numbers, but I already had to do some normalizing anyway so that they started at zero (see discussion above about DXGI_ERROR_FRAME_STATISTICS_DISJOINT) and so that didn’t seem so bad. SyncQPCTime, however, are also big numbers. The same trick of starting at zero can be used, and I also represented them as seconds (instead of the Windows high performance counts) and this helped me to get good results. I was worried about the long-term viability of this, however: Unlike the incremental method for the average this method required squaring numbers and multiplying big numbers together, and these numbers would constantly increase over time.

Even though I was quite happy with finding an algorithm that did what I had thought of, once I had implemented it there was still something that bothered me: I was trying to come up with a line equation, where the coefficients are the slope and the y-intercept. I already knew the slope, though, because I had a very good estimate of the duration of a refresh. In other words, I was solving for two unknowns using a bunch of sample points, but I already knew one of those unknowns! What I really wanted was to start with the slope and then come up with an estimate of the y-intercept only, and so it felt like the method I was using should be constrainable even more.

With that in mind I eventually came up with what I think, in hindsight, is a better solution even aside from precision issues. I know the “exact” duration between every vblank (we will conceptually consider that to be known, even though it’s an estimate), and for each reported sample I know the exact number of refreshes since the initial starting point (which is a nice discrete integer value), and then I know the approximate sampled time, which is the noisy repeated sample data I am getting that I want to improve in order to make predictions. What I can do, then, is to calculate what the initial starting time (for refresh count 0) would be, and incrementally calculate an average of that. This gives me the same cheap way of calculating the prediction (just a slope (refresh period) and a y-intercept (this initial timestamp)), but also a cheap way of updating this estimate (using the same incrementally-updating average method that I discussed above). And, even better, I can update it every frame without worrying about numerical problems. (Eventually with enough sample counts there will be issues with the updated value being too small, but that won’t impact the accuracy of the current average if we assume that it is very accurate by then.)

This means that I don’t have to spend time initially waiting for the averages to converge; instead I can just start with single sample that is already a reasonably good estimate and then proceed with normal rendering, knowing that my two running averages will keep getting more accurate over time with every new DXGI frame statistic that I get with new SyncRefresh information.

Devlog 2024-11-25

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 6.

What I have done

  • Initialized Direct3D 12 and cleared the window
  • Calculate timing of display refresh so that vblanks can be predicted
    • I am able to display horizontal color bars on the display without vsync enabled and with only clearing the entire window by properly timing and predicting when to swap the back buffer
    • This is something that I’ve wanted to do for years after reading about it and so it was fun to finally make it happen. I will make a dedicated post about it once I have cleaned up some remaining problems.
  • Added the ability to specify C++ compile optimizations and update the project configurations accordingly

Next Steps

  • Improve display timing
    • There are still problems that I don’t understand. Rendering without vsync isn’t important because the real engine won’t work that way and so it’s fine if the fun demo of color bars doesn’t work perfectly. What is important, however, is that the underlying timing information is correct for what I want to do with the real engine and I’m concerned with some of the behavior that I occasionally see that I don’t understand because it could lead to hard-to-diagnose problems later.
    • Even though I did the color bars as a fun project they are actually a really good way to visualize whether the timing is correct and so I think it is good to keep working on it before I undo the fun temporary hacky code and make things work the way that they are supposed to.
  • Create a separate thread for the display loop and the game logic
    • I was going to delay doing this until later, but as I’ve been working on the swap chain code I think it would be good to get this more formalized early
  • Draw a triangle
    • I had thought about adding better input next but for now I can just detect key presses using GetAsyncKeyState() in a hacky way and that means I can quickly experiment without having to go through an application layer. So I think I have changed my mind and that it is a better idea to keep working on the graphics system for the moment.

Devlog 2024-11-18

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 5.

What I have done

  • Build system changes:
    • Include directory search paths are used in the dependency hash so that if they change a file is recompiled
    • The use of standard libraries or platform libraries in compiled files that a target depends on now influences its compiled files include directory search paths
      • This is so that if some downstream target #includes a file that #includes <windows.h>, for example, that it will work as expected (meaning that specifying the static library dependency is enough)
    • Added features to help IntelliSense
      • Include directories and preprocessor defines are now added to VCXPROJ files
      • Compile arguments are now added to VCXPROJ files
        • (It turns out that this is important for IntelliSense for things like detecting what version of C++ is used; without it there are lots of false positive red squiggles when using newer features)
      • Files are now added to VCXPROJ files
        • Currently this is done by specifying a file path, along with an optional type (e.g. for C++ compiling or C++ including) and an optional logical display folder (so that files can be organized in a logical file structure in e.g. Solution Explorer).
        • I mostly like this, but one small annoyance is that for C++ files that get compiled it means that each source CPP file has to be listed twice. For now I have decided that the way it is makes sense (rather than e.g. automatically adding these files to be displayed) but it’s something I will have to keep thinking about as I get more experience.
      • Individual files can now be compiled in Visual Studio
        • This involved both adding a pattern for command lines but also adding the ability to jpmake. Previously I had a command line where a file path could be specified to generate just that file (i.e. it was an output file path), but in order to support the Visual Studio functionality the file path must be an input path.
      • Individual C++ files to compile can now have individual C++ build info, rather than always using the C++ build info of the target
        • This was already supported under the hood in jpmake, but I didn’t have a Lua interface for it
  • A new EmptyWindow application exists
    • All this does is create and display a main window, so it is the most basic application imaginable, but it was still exciting to be working on real code rather than the build system
    • I made a post about it
  • I have added many type traits
    • So far these are just clones of standard library ones that I have needed, but it has been fun to learn more about how some of this template “magic” works
    • I made a post about it
  • I added scope guard classes
  • I added custom implementations of move and forward
    • MakeMovable() and ForwardReference() are macros based on https://www.foonathan.net/2020/09/move-forward/
      • I discovered accidentally (there was some reddit or stack overflow comment about that link I think) that the macro version enforces actual moves with no copying fallback, which is something that I have wanted but wasn’t sure if or how it was implementable. After figuring out why the macro had this difference (related to const vs. non-const rvalue references) I have also added a MakeMovableIfPossible() in case I ever want it, which just does the same thing that std::move() does.
  • I added volatile load and store functions, with relaxed, acquire, or release memory ordering
  • I added the ability to allocate and free memory from the operating system
    • (It is also possible to reserve and commit memory in steps if the operating system supports it. This is something I use in jpmake, but don’t currently anticipate using in the game engine itself.)
  • I made array view and memory view classes
    • These don’t have all of the functionality that I would like, but they are a good start of a fundamental building block
    • The cMemoryView class can have a view of both const and mutable memory, and allows slices to be made for partitioning memory
  • I have added memory arena and allocator classes
    • This is just a starting point, but I do have a monotonic allocator and am able to partition and then logically allocate memory from the operating system
    • I have New()/Delete() functions for working with typed memory and Allocate()/Free() functions for working with untyped memory, all of which require an arena
      • The goal is to allocate a single big block of operating system memory when the application starts and then only work with that for the lifetime of the application by partitioning that into smaller sections and using hierarchical arenas to allocate in a controlled way (“controlled” both in terms of lifetimes and cache friendliness).
  • I have set up the machinery for creating a graphics “engine” and created a D3D12 device
    • The application lets the graphics library create a graphics engine, passing in itself and a block of memory from the operating system that is budgeted for graphics. The graphics engine stores a reference to the application who created it and returns a pointer to itself that the application can refer to. This is my attempt at dependency injection while still allowing easy access to different systems.
    • The graphics engine will have a platform-specific context, and so far I am just creating a D3D12 device to store in it as a proof of concept to show that I am able to access the platform-specific D3D functionality and allocate and store things with the new memory system.

Next Steps

  • Use D3D12 to clear the main window to some arbitrary color
  • Implement the top part of the window as one clear color and the bottom part of the window as a different clear color using ideas discussed at https://blurbusters.com/blur-busters-lagless-raster-follower-algorithm-for-emulator-developers/
    • This isn’t how I intend my engine to run (I will use VSync), but ever since reading about it I have wanted to try and do it to make sure I understood the timing involved, and I think getting this to work would show that I have the fundamentals in place before implementing how I really want my renderer to work.

Type Trait Coding Style

One of my goals while creating a custom game engine is to avoid using the C++ standard library. This post won’t discuss why, but one consequence of that goal that I have run into during this early phase of the project is that I have had to recreate some fundamental machinery that the standard library provides. By “recreate” I don’t mean that I have figured things out myself starting from nothing but first principles, but instead that I have had to look at existing documentation and implementations to understand how something works and then reimplement it in my own style. It has been fun to learn a bit more about how this corner of C++ metaprogramming works; although I have used these features (some of them frequently) I have only vaguely understood how they might have actually been implemented.

Doing this has also had the interesting consequence of presenting me with challenges to my existing coding style and I have had to expand and adapt. This post gives some examples of changes to my coding style that I have been experimenting with in order to accommodate type traits.

Existing Style

My current style uses a lowercase single letter prefix to indicate type names, meaning that if an identifier is spelled e.g. jSomething or pSomething the reader can immediately know that those names identify types, even without knowing what the j or p might mean (and, to be clear, those are just examples, and neither the j nor the p prefix exists (yet)).

Class names start with a c prefix:

class cMyClass;
class cSomeOtherClass;

Struct names start with an s prefix, base class names start with an i prefix (for “interface”, where my goal is that leaf classes are the only ones that are ever instantiated), and enumeration names start with an e prefix:

// When I use a struct instead of a class it means:
//	* The member variables are public instead of private
//	* The member variables use a different naming scheme
struct sMyStruct;
// The reader can tell that the following is a base class
// (and conceptually an abstract base class)
// just from the type name alone:
class iMyBaseClass;
// Using this enum convention makes it
// less annoying to use scope enumerations
// because the prefix makes it instantly identifiable
// and so the identifiers can be chosen accordingly:
enum class eMyState
{
	On,
	Off,
};

Type names start with a t prefix, e.g.:

// Creating a type alias:
using tMySize = int;
// In templates:
template<typename tKey, typename tValue>
class cMyMap;

(I don’t like the common convention of using single letters (e.g. T and U) in templates, and find that it makes code much harder to read for me personally, similarly to how I feel about single letter variable names. This strongly-held conviction has been challenged in some cases by working with type traits, which I discuss below.)

Type Names

Some of the type trait information that I have needed are expressions that are types. The standard uses a _t suffix for this, which is a helper type of a struct, but I have never loved this convention. In my code so far I have used a t prefix for cases like, and hidden the associated struct in a details namespace:

// tNoConst -> Type without a const qualifier
namespace Types::detail
	{
		template<typename T> struct sNoConst { using type = T; };
		template<typename T> struct sNoConst<const T> { using type = T; };
	}
template<typename T>
using tNoConst = typename Types::detail::sNoConst<T>::type;

// An example of this being used:
const int someConstVariable = 0;
tNoConst<decltype(someConstVariable)> someMutableVariable;
someMutableVariable = 0;

This use of the t prefix is not really any different from how I had already named types, and I find that it fits in naturally when used in code. Eagle-eyed readers may notice, however, that I am using a single T as a type, which I had mentioned is something that I strongly dislike and claimed that I don’t do!

In all of my previous template programming I have always been able to come up with some name that made sense in context, even if it was sometimes generic like tValue. While working on these type traits, however, I realized that there are cases (like shown above) where the type really could be any type. I considered tType, but that seemed silly. I considered tAny, and I might still end up changing my mind and refactoring the code to that (or something similar). For now, though, I have capitulated and am using just the T for fully generic fundamental type trait building blocks like the ones discussed in this post (in other code, though, I still intend to strictly adhere to my give-the-type-a-name rule).

Value Names

Some of the type trait information that I have needed are expressions that are values. The standard uses a _v suffix for this, but I have never loved this convention. In fact, I’m not sure that I really understand this convention; unlike with _t where there needs to be some underlying struct for the metaprogramming implementation to work it doesn’t seem like values need this (at least, the ones that I have recreated so far haven’t needed an underlying struct).

I did struggle a bit with how to name these, however. My existing coding convention would prefix global variables names with g_ (that will have to be discussed in a separate post), but these type trait variables feel different from traditional global variables to me. In my mind they conceptually feel more like functions than variables, but functions that I call with <> instead of (). I wanted some alternate convention to make them visually distinct from standard variables.

After some experimentation I eventually settled on keeping the v from the standard but making it a prefix instead of a suffix, and I have been pretty happy so far with the result:

// vIsConst -> Whether a type is const-qualified
template<typename T>
constexpr bool vIsConst = false;
template<typename T>
constexpr bool vIsConst<const T> = true;

// An example of this being used:
if constexpr (vIsConst<decltype(someVariable)>)
{
	// Stuff
}

This convention has added a new member to my pantheon of prefixes, but it has felt natural and like a worthy addition so far. As an additional unexpected bonus it has also given me a new convention for naming template non-type parameters:

// My new convention:
template<typename tSomeType, bool vSomeCondition>
class myClass1;

// My previous convention, which I never loved:
template<typename tSomeType, bool tSomeCondition>
class myClass2;

Having a new way of unambiguously specifying compile-time values has improved the readability of my code for me.

Concept Names

I have encountered one case where I wanted to make a named constraint and I had to think about what to name it. I don’t have enough experience yet to know whether my initial attempt is something that I will end up liking, but this is what I have come up with:

// rIsBaseOf -> Enforces vIsBaseOf
template<typename tDerived, typename tBase>
concept rIsBaseOf = vIsBaseOf<tBase, tDerived>;

// An example of this being used:
template<rIsBaseOf<iBaseClass> tSomeClass>
class cMyConstrainedClass;

I couldn’t use c for “constraint” or “concept” because that was already taken for classes. I finally settled on r for “restraint” (kind of like a mix of “constraint” and “restrict”, with a suggestion of requires) and I don’t hate it so far but I also don’t love it. It feels like it is good enough to do the job for me in my own code, but it also feels like maybe there’s a better convention that I haven’t thought of yet.

Application with an Empty Window

Behold, the main window of an application using my nascent engine:

It does as little as one might expect from the screenshot, but some of the architecture might be of interest.

Build System

This section shows parts of the jpmake project file.

Platforms and Configurations

The different possible platforms and configurations are defined as follows:

-- Platforms
------------

-- Define the platforms by name and type

platform_windows = DefinePlatform("windows", "win64")
local platform_windows = platform_windows

-- Get the specified platform to execute tasks for
platform_target = GetTargetPlatform()
local platform_target = platform_target

-- Configurations
-----------------

-- Define the configurations by name

configuration_unoptimized = DefineConfiguration("unoptimized")
local configuration_unoptimized = configuration_unoptimized

configuration_optimized = DefineConfiguration("optimized")
local configuration_optimized = configuration_optimized

configuration_profile = DefineConfiguration("profile")
local configuration_profile = configuration_profile

configuration_release = DefineConfiguration("release")
local configuration_release = configuration_release

-- Get the specified configuration to execute tasks for
local configuration_current = GetCurrentConfiguration()

I am creating global variables for each platform and configuration so that they are accessible by other Lua files and then immediately assigning them to local variables so that they are cheaper to use in this file. (I currently only have the single Lua project file, but soon I will want to change this and have separate files that can focus on different parts of the project.)

At the moment I am specifying any platform-specific information using a strategy like if platform_target == platform_windows and that works fine (there are several examples later in this post), but I am considering defining something like isWindows = platform_target == platform_windows instead. There won’t be many platforms (only one for the foreseeable future!) and it seems like it would be easier to read and write many platform-specific things with a single boolean rather than with a long comparison. I am doing something similar with the configurations where I define booleans that serve as classification descriptions, and so far it feels nice to me (again, there are examples later in this post).

Directory Structure

The directory structure from the perspective of jpmake is currently defined as follows:

-- Source Files
do
	SetEnvironmentVariable("engineDir", "Engine/")
	SetEnvironmentVariable("appsDir", "Apps/")
end
-- Generated Files
do
	-- Anything in the temp directory should be generated by jpmake executing tasks
	-- and the entire folder should be safely deletable.
	-- Additionally, any files that are not part of the Git repository
	-- should be restricted to this folder.
	SetEnvironmentVariable("tempDir", ConcatenateTable{"temp/", platform_target:GetName(), "/", configuration_current, "/"})
	-- The intermediate directory is for files that must be generated while executing tasks
	-- but which aren't required to run the final applications
	SetEnvironmentVariable("intermediateDir", "$(tempDir)intermediate/")
	SetEnvironmentVariable("intermediateDir_engine", "$(intermediateDir)engine/")
	-- The artifact directory is where jpmake saves files
	-- that it uses to execute tasks
	SetArtifactDirectory("$(intermediateDir)jpmake/")
	-- The staging directory contains the final applications that can be run
	-- independently of the source project and intermediate files
	SetEnvironmentVariable("stagingDir", "$(tempDir)staging/")
end

As a general rule I don’t like abbreviations in variable or function names but I decided to keep the “dir” convention from Visual Studio since these environment variable names will be used so frequently in paths that it seems like a reasonable exception to keep things shorter and more readable. (I did, however, decide to change the first letter to lowercase which fits with my variable naming convention better.)

An issue that I have run into in the past is having trouble deciding how to name directory environment variables to distinguish between source and generated files, and with games where there can be code and assets both for the engine and the application the possible choices are even more complex (and, to make matters worse, with this project I am intending to support multiple applications using the engine and so there is yet a further distinction that must be made). What I have will likely change as time goes on and I write more code, but it feels like a good start. The root repository folder looks like this:

Any files that are generated by the build process are kept quarantined in a single folder (temp/) so that the distinction between source and deletable files is very clear. This is very important to me (as anyone who has worked with me can attest). The temp directory looks like the following, expanded for one platform and configuration:

With such a simple application the only thing in the staging directory is the executable file, but when I develop more complicated applications there will be other files in staging directories (e.g. the assets that the game loads).

One further consideration that is currently missing is what to do with “tools”, those programs that are used during development (either for authoring content or as part of the build process) but that don’t get released to end users. I can imagine that I might want to update some of this directory structure when I start developing tools.

C++ Configuration

The next section in the jpmake project file configures the default settings for how C++ is built for the current target platform and build configuration:

-- C++
------

-- Initialize C++ for the current platform and configuration
cppInfo_common = CreateCppInfo()
local cppInfo_common = cppInfo_common
do
	-- #define VLSH_PLATFORM_SOMENAME for conditional compilation
	do
		local platform_define_suffix
		if (platform_target == platform_windows) then
			platform_define_suffix = "WINDOWS"
		else
			platform_define_suffix = "NONE"
		end
		cppInfo_common:AddPreprocessorDefine(("VLSH_PLATFORM_" .. platform_define_suffix),
			-- There isn't any anticipated reason to check anything other than whether the platform is #defined,
			-- but the name is used as a value because why not?
			platform_target:GetName())
	end
	-- The project directory is used as an $include directory
	-- so that directives like the following can be done to show scope:
	--	#include <Engine/SomeFeature/SomeHeader.hpp>	
	cppInfo_common:AddIncludeDirectory(".")
	local isOptimized = configuration_current ~= configuration_unoptimized
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISOPTIMIZED", isOptimized)
	local isForProfiling = configuration_current == configuration_profile
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISFORPROFILING", isForProfiling)
	local isForRelease = configuration_current == configuration_release
	cppInfo_common:AddPreprocessorDefine("VLSH_CONFIGURATION_ISFORRELEASE", isForRelease)
	do
		local areAssertsEnabled = not isForRelease and not isForProfiling
		cppInfo_common:AddPreprocessorDefine("VLSH_ASSERT_ISENABLED", areAssertsEnabled)
	end
	cppInfo_common.shouldStandardLibrariesBeAvailable = false
	cppInfo_common.shouldPlatformLibrariesBeAvailable = false
	cppInfo_common.shouldExceptionsBeEnabled = false
	cppInfo_common.shouldDebugSymbolsBeAvailable =
		-- Debug symbols would also have to be available for release in order to debug crashes
		not isForRelease
	if platform_target == platform_windows then
		cppInfo_common.VisualStudio.shouldCRunTimeBeDebug = not isOptimized
		cppInfo_common.VisualStudio.shouldIncrementalLinkingBeEnabled =
			-- Incremental linking speeds up incremental builds at the expense of bigger executable size
			not isForRelease
		-- Warnings
		do
			cppInfo_common.VisualStudio.shouldAllCompilerWarningsBeErrors = true
			cppInfo_common.VisualStudio.shouldAllLibrarianWarningsBeErrors = true
			cppInfo_common.VisualStudio.compilerWarningLevel = 4
		end
	end
end

This shows the general approach I am taking towards configuring things (both from the perspective of the game engine and also from the perspective of jpmake and my personal ideal way of configuring software builds). The named configurations (e.g. unoptimized, optimized, profile, release) that I defined earlier are just arbitrary names from the perspective of jpmake and don’t have any semantics associated with them. Instead it is up to the user to specify how each configuration behaves. I can imagine that this would be seen as a negative for most people, but I have a personal issue where I generally prefer to have full control over things.

This section should not be understood as being complete (most notably there actually aren’t any optimization-related settings except for which C run-time to use!) but that is because I haven’t implemented all of the Visual Studio options in jpmake yet.

Engine Static Library

Below is one example of a static library that I have made, which provides base classes for applications (meaning that an actual application can inherit from the provided framework):

do
	local task_application = CreateNamedTask("Application")
	local cppInfo_application = cppInfo_common:CreateCopy()
	do
		if (platform_target == platform_windows) then
			cppInfo_application.shouldPlatformLibrariesBeAvailable = true
		end
	end
	engineLibrary_application = task_application:BuildCpp{
			target = "$(intermediateDir_engine)Application.lib", targetType = "staticLibrary",
			compile = {
				"$(engineDir)Application/iApplication.cpp",
				"$(engineDir)Application/iApplication_windowed.cpp",
				platform_target == platform_windows and "$(engineDir)Application/iApplication_windowed.win64.cpp" or nil,
			},
			link = {
				engineLibrary_assert,
				platform_target == platform_windows and CalculateAbsolutePathOfPlatformCppLibrary("User32.lib", cppInfo_application) or nil,
			},
			info = cppInfo_application,
		}
end
local engineLibrary_application = engineLibrary_application

My current plan is to have the “engine” consist of a collection of static libraries that all get linked into the single application executable.

This named task shows a file specific to Windows that is only compiled for that platform (iApplication_windowed.win64.cpp, where my convention is to try to put as much platform-specific code in separate platform-specific CPP files as possible and then those files have the platform name as a sub-extension), as well as a Windows library that is only needed for linking on that platform (User32.lib) and another static library (engineLibrary_assert, which was defined earlier but that I don’t show in this blog post) that this static library depends on.

As more files get created that are specific to one platform or another I think my style will have to change to make it less annoying to conditionally specify each one.

Applications

Finally, the two proof-of-concept applications that I have created are defined as follows:

-- Hello World
--============

do
	do
		SetEnvironmentVariable("appDir", "$(appsDir)HelloWorld/")
		SetEnvironmentVariable("stagingDir_app", "$(stagingDir)HelloWorld/")
	end
	local cppInfo_helloWorld = cppInfo_common:CreateCopy()
	do
		-- For std::cout
		cppInfo_helloWorld.shouldStandardLibrariesBeAvailable = true
	end
	do
		local helloWorld_task = CreateNamedTask("HelloWorld")
		local application_subTask = helloWorld_task:BuildCpp{
				target = "$(stagingDir_app)HelloWorld.exe", targetType = "consoleApplication",
				compile = {
					"$(appDir)EntryPoint.cpp",
				},
				info = cppInfo_helloWorld,
			}
		helloWorld_task:SetTargetForIde(application_subTask)
	end
end

-- Empty Window
--=============

do
	do
		SetEnvironmentVariable("appDir", "$(appsDir)EmptyWindow/")
		SetEnvironmentVariable("stagingDir_app", "$(stagingDir)EmptyWindow/")
	end
	local cppInfo_emptyWindow = cppInfo_common:CreateCopy()
	do
		cppInfo_emptyWindow:AddIncludeDirectory("$(appDir)")
	end
	do
		local emptyWindow_task = CreateNamedTask("EmptyWindow")
		local application_subTask = emptyWindow_task:BuildCpp{
				target = "$(stagingDir_app)EmptyWindow.exe", targetType = "windowedApplication",
				compile = {
					"$(appDir)EntryPoint.cpp",
				},
				link = {
					engineLibrary_application,
				},
				info = cppInfo_emptyWindow,
			}
		emptyWindow_task:SetTargetForIde(application_subTask)
	end
end

These show the general approach towards making executable applications that I am envisioning, although these both are as simple as possible.

One idiom that I discovered is reusing the same environment variable names but setting them to different values for different applications. This allowed the names to be shorter and thus more readable (before this I had different versions with _helloWorld and _emptyWindow), but I don’t have enough experience to decide if this will work well long term.

The examples also show calls to SetTargetForIde(), which has no effect when executing tasks but is instead used when generating the solution files so that Visual Studio will correctly have its $(TargetPath) set, which makes setting up debugging easier.

Visual Studio Solution

It is now possible for jpmake to generate Visual Studio solution and project files. I did this work to make it easier to write code and debug in Visual Studio. The Solution Explorer currently looks like the following for the jpmake project that I have been showing in this post:

And the properties of the EmptyWindow project have some things filled in:

I had to spend more time on generating these files and additional jpmake features than I had initially anticipated before working on the engine code because I wasn’t able to debug, which felt like a requirement. With the way it works now, however, I was able to write the empty window application and things worked reasonably well.

I did have one discouraging realization, however, which is that Intellisense doesn’t work yet. I was able to complete the empty window application without it but it was more annoying than I would have anticipated. I think I need to take some more time to improve jpmake so that Intellisense will work at least somewhat because not having it has proven to be an annoying impediment.

Devlog 2024-11-11

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 4.

What I have done

  • A jpmake project can now generate Visual Studio IDE files
    • A solution file is created with project files corresponding to every named task
    • The appropriate command line is generated when a project gets built. The output can be generated if the Lua jpmake file specifies it, e.g.:
      • cppNamedTask:SetTargetForIde(myApplication)
    • Additionally, there are two special projects that are generated:
      • Do [JPMAKEPROJECTNAME]
        • This does all tasks, and it is the only Visual Studio project that is configured to build when the entire Visual Studio solution is built
      • Regenerate Solution
        • This can be built when there are changes to the jpmake project file
    • With these improvements it is possible to reasonably work with a jpmake project in Visual Studio. There are still several obvious important missing features (and many more that I could think of that would be nice to have), but I think the status is good enough to start development and add more features as they prove themselves to be issues.
  • It is now possible to specify include directories
    • This seemed important in order to be able to #include engine header files from application code
  • I have added options for generating debug info/symbols
    • After experimenting with debugging in Visual Studio this clearly was important
  • I have added options for handling warnings
    • Specifically, being able to set warning levels and to treat warnings as errors
  • I have created a new git repository to contain the actual engine and application code
    • I have integrated jpmake into this as a submodule
      • This is not what I had initially intended and instead had anticipated just having a jpmake.exe in the repository. It seems, however, that there will probably be frequent jpmake changes in response to me actually working on real projects and realizing that I want or need more features (especially early on) and so a submodule started to make more sense to me than a frequently-changing EXE binary file.
    • I have set up a Windows platform and three different configurations
      • For now I have “unoptimized”, “optimized”, and “release”
    • I have set up some initial directory structure, although I’m sure my mind will change about the best way to do organize things and what environment variables to use as I start authoring actual engine and application code
  • I have created a HelloWorld console application
    • This doesn’t do anything except print “Hello, world!” and so it doesn’t represent any interesting progress over the more complicated examples I had last week
    • It does, however, live within an actual project and so represents the first application in this new game engine repository

Next Steps

  • I need to make an application that displays an empty window
    • This should almost entirely use engine code, where the only thing that the application does is derive an application class from an abstract base application class and then pass on command arguments
  • The next step would be to initialize Direct3D 12 and clear the back buffer to some color
    • This might take more time than it otherwise would because I also want to implement some mechanisms for manual memory management

Devlog 2024-11-04

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 3.

What I have done

  • I have been working on a build system so that I can build code and assets the way that I want to. A dedicated post about it is here.
    • It is possible to define different platforms and configurations (e.g. debug or optimized)
    • It is possible to define three kinds of tasks (C++, copy files, or arbitrary Lua functions)
    • The task dependencies are tracked, and they are only executed if necessary and in the correct order (only single-threaded, although it has been designed for multi-threaded execution)
    • For C++:
      • The installed Visual Studios and Windows SDKs are calculated, and the user can either request the latest by version number or request specific versions if desired
      • Standard library and platform library support can be enabled or disabled (and, when enabled, the appropriate #include paths are set)
      • It is possible to build console applications, windowed applications, shared libraries, or static libraries
      • Exceptions can be enabled or disabled

Next steps

  • It has now been two weeks that I have been working on the build system, and even though I have made good progress it feels like I really need to start working on actual game engine code. There are still obvious missing features in the build system but I think a good strategy now is to add them as they become necessary for the actual code I am working on rather than trying to add more preemptively.
  • One final thing that I think that I want to add is the ability to generate a Visual Studio solution that I can use to debug
    • I think it’s ok initially if I don’t create individual projects for named tasks and instead just have a single project to build the entire solution (it seems ok at least initially to manually find files and open them in the editor)
    • I do, however, want to be able to debug, and having some intellisense would also be nice, and so I might spend some time trying to get that done.
  • Another thing that I know I will want is to be able to set up precompiled header files
    • Being able to do this in a platform-independent way would be nice so that I don’t have to go back and redo it later, but it’s not critical
    • I may or may not add this before moving on to start work on code
  • The initial goal for a program would be to open a window that doesn’t do anything, and that is pretty achievable
  • The next goal would be to clear the color buffer to some hard-coded color
    • This would be quite easy using Direct3D 11, but using Direct3D 12 requires allocating some memory which will require some memory management work with the way I want to do things (rather than just using global new).

Custom Build System (Part 1)

This post is part of a series about creating a custom game engine.

In every job I have had I have become involved with the system of how the software is built, and this is also something I spent time making my graduate students work on in their game engine class. Every time, and in all of my personal projects, I have wished that I had my own system that worked the way that I wanted it to, and while I currently have the time to create a custom game engine I have taken the opportunity to try and make such a build system.

The term “build system” may mean different things to different people, but my usage is to refer to the process of taking any source files (code and assets) and transforming them into the final generated files that can be distributed in order to run the software. In some cases the process can be quite simple conceptually (in order to create an executable application using C++, for example, the source files are compiled into object files and then those object files are linked into an executable), but when creating a game the process becomes much more complex: Not only is there traditional code to deal with but there are also assets (input authored data that is not code) as well as interdependencies between code and assets where the content of one can influence how the other is built.

Wishlist

The following is a non-exhaustive list of what I would want in a build system, which contains elements both of the build system tool and of the implementation of how a piece of software uses a build system:

  • The build system is simple to use
    • There should be exactly one step required to go from only the authored files (the ones contained in source control) to the final product that is ready to be distributed
    • (If there is more than one step that should be the fault of the human and not the build system, and the build system should make it possible to automate any extra steps so that there is only one)
  • Builds are deterministic
    • Building the software should always have the same results
      • If an attempt to build has an error and then a subsequent attempt to build succeeds this is a problem
      • If an attempt to build has an error there is never a question of whether it is a real problem or not
  • Nothing is done if a build is requested and nothing needs to be done
  • It is fast to request a build when nothing needs to be done
    • When I am developing the common case should be to request the entire software to be built (it is exceptional when I want to build some parts but explicitly don’t want other parts to change) and it should be fast for a “no-op” build so that it doesn’t interfere with iteration
  • Dependencies are easy to work with
    • It is easy for a human to specify important dependencies
    • As much as possible, however, the system should figure out the dependencies itself without requiring a human to specify them
    • If an attempt to build has an error there is never a question of whether it is because a dependency was missing
    • When an attempt to build something is made there is never a question of whether the result might not be valid because one of its dependencies was not in a correct state (this is restating the deterministic requirement)
  • Assets (i.e. things that aren’t code) are easy to build
  • Generated code is easy to build
  • It is easy to build abstractions and to document and comment
    • When I am creating the system that builds software I want to be able to use good programming practices

When I am developing software with a build system that behaves as I describe above then I have the confidence to iterate and make changes and know that I am seeing the correct results of the changes I made, without worrying about extra steps or that something might be wrong.

Philosophy

I had to come up with some name and I settled on “jpmake”, a silly variation on the common naming scheme of “make”, “nmake”, “cmake”, etc., for build system software. It is based on some of the principles I have thought about for years, although its development is focused on allowing me to work on the custom engine project and so there are features that I know that I would like that I will intentionally not work on until/unless they are needed.

Despite using the term “build” and even though the motivation behind creating this is in order to build software I am designing things with the mental model that there is a “project” which is a collection of tasks that must be done (and these tasks may or may not have dependencies on each other). This means conceptually that a jpmake project could be used the same way e.g. a BAT file in Windows often is, to automate a series of tasks. Although a requirement is to be able to easily specify and build C++ programs the design goal is to have that be just one kind of task that can be done, with the hope that approaching it this way will make it easy to add other as-yet unanticipated kinds of tasks. The user should be able to focus on defining which tasks must be done and what the inputs and outputs of each task are, and then jpmake should be able to figure out the rest.

Lua

I am a big fan of the Lua scripting language and use it when I want to create a text-based interface. I have in fact used it multiple times in the past for smaller build systems, focusing on being able to build assets as part of a larger code-based project, and so I already have some experience with what I am trying to currently accomplish with jpmake.

My favorite thing about Lua is the ability to create interfaces that are easy to understand and use. It has enough flexibility that I can create files that are understandable even by someone who doesn’t know Lua (often understandable enough to make small changes), but it also has the power of a full programming language behind it so that I don’t feel limited when I want to use what I consider good programming practices.

As an example, observe the following lines from a jpmake project file:

local myTask = CreateNamedTask("myTask")
myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")

Without knowing any details you could probably guess what line 2 does, and even without knowing the correct syntax you could probably copy line 2, modify it, and get the correct result that you wanted and expected.

One issue I have run into with previous large-scale Lua projects that I’ve done, however, is that although I am completely satisfied with using the finished product in terms of the interface it has been very difficult to remember the details of the inner workings and to debug when things go wrong. For this current project I am taking a different approach to try and mitigate this, where I use Lua script files as the project files that specify what tasks to execute and how, but otherwise everything is implemented in C++. In the case of how I anticipate using jpmake this is probably what I would have done anyway because it means that there is a single executable application that can do everything, but it also has the advantage of easier maintainability because of static typing and making it easy to debug. (Additionally, of course, it can be more efficient. I am trying to manage all of the strings that exist in a program like this (i.e. because there are so many file paths) in an efficient way to keep things fast that wouldn’t be possible if the implementation were in pure Lua.)

Example

Details will have to wait for a part 2, but below is an example from the test project file that I have been using while developing:

-- Set Up
--=======

-- Define the platforms that tasks can be executed for for
local platform_windows = DefinePlatform("Windows", "win64")
-- Get the currently-specified (from the command arguments) platform
local platform_current = GetTargetPlatform()
local platformName_current = platform_current:GetName()
-- Define the configurations that can be used when executing tasks
local configuration_debug = "Debug"
local configuration_optimized = "Optimized"
DefineConfigurations{configuration_debug, configuration_optimized}
-- Get the currently-specified (from the command arguments) configuration
local configuration_current = GetConfiguration()

-- Define environment variables that can be used in paths
SetEnvironmentVariable("TempDir", table.concat{"temp/", platformName_current, "/", configuration_current, "/"})
SetEnvironmentVariable("IntermediateDir", "$(TempDir)intermediate/")
SetEnvironmentVariable("OutputDir", "$(TempDir)output/")

-- Define the C++ info that determines how C++ is built
local cppInfo_common = CreateCppInfo()
do
	-- Libraries can be disabled for situations that only use custom code
	do
		cppInfo_common.shouldStandardLibrariesBeAvailable = false
		cppInfo_common.shouldPlatformLibrariesBeAvailable = false
	end
	cppInfo_common.shouldExceptionsBeEnabled = false
	-- Platform-specific configuration
	if (platform_current == platform_windows) then
		-- A #define can be set for platform-specific conditional compilation
		cppInfo_common:AddPreprocessorDefine("PLATFORM_WINDOWS")
		-- Preprocessor symbols can also have values
		cppInfo_common:AddPreprocessorDefine("PLATFORM_NAME", platformName_current)
		-- Use different VC run-times based on the current configuration
		cppInfo_common.VisualStudio.shouldCRunTimeBeDebug = configuration_current == configuration_debug
	end
end
-- Alternate C++ infos can be created from the base configuration and then have targeted changes
local cppInfo_withStandardLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withStandardLibraries.shouldStandardLibrariesBeAvailable = true
end
local cppInfo_withPlatformLibraries = cppInfo_common:CreateCopy()
do
	cppInfo_withPlatformLibraries.shouldPlatformLibrariesBeAvailable = true
end

-- C++ Tasks
--==========

do
	-- A "named task" is just for human organization:
	-- It allows only a subset of sub tasks to be executed (rather than the entire project) by specifying a name
	local cppNamedTask = CreateNamedTask("BuildMyCppProgram")
	local staticLibrary_usingPlatformLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesPlatformLibraries.lib",
			compile = {
				"MyClass.cpp",
			},
			-- Any libraries specified as needing to be linked when defining static library tasks
			-- don't actually influence the creation of the static libraries themselves,
			-- but instead are stored as dependencies
			-- that eventually are linked when an application is created that uses the static libraries
			link = {
				platform_current == platform_windows and CalculateAbsolutePathOfPlatformCppLibrary("Advapi32.lib", cppInfo_withPlatformLibraries) or nil,
			},
			-- This static library looks up a registry value,
			-- and by requesting that platform libraries are available
			-- it allows the #include <windows.h> directive to look in the correct Windows SDK folder
			info = cppInfo_withPlatformLibraries,
		}
	local staticLibrary_usingStandardLibraries = cppNamedTask:BuildCpp{
			targetType = "staticLibrary", target = "$(IntermediateDir)usesStandardLibraries.lib",
			compile = {
				"MySource.cpp",
			},
			-- This static library uses std::cout,
			-- and by requesting that standard libraries are available
			-- it allows the #include <iostream> directive to look in the correct Visual Studio folder
			info = cppInfo_withStandardLibraries,
		}
	local myApplication = cppNamedTask:BuildCpp{
			targetType = "consoleApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "windowedApplication", target = "$(OutputDir)MyApplication.exe",
			--targetType = "sharedLibrary", target = "$(OutputDir)MyLibrary.dll",
			compile = {
				"MyEntryPoint.cpp",
			},
			link = {
				staticLibrary_usingPlatformLibraries,
				staticLibrary_usingStandardLibraries,
			},
			info = cppInfo_common,
		}
end

-- Copy Tasks
--===========

do
	-- Here is an example of a differently-named task that does something other than building C++
	local myTask = CreateNamedTask("myTask")
	myTask:Copy("someAuthoredFile.txt", "$(OutputDir)someDirectory/someIntermediateFile.txt")
	myTask:Copy("$(OutputDir)someDirectory/someIntermediateFile.txt", "$(OutputDir)someDirectory/someOtherDirectory/someCopiedFile.txt")	
end

Creating a Video Game and Engine from Scratch

I am attempting to make a video game without using an available engine. I am documenting the process with two different kinds of posts:

  • Discussions of Features
    • These posts are made when I have something of interest to say rather than on a regular schedule
    • They can be found using the Game Engine Features category
  • Status Updates
    • These posts are intended to be made weekly and are not intended to be of interest to a general audience
    • Instead, these posts are a way for me to hold myself accountable by reporting what I have worked on during the previous week and what I intend to work on during the upcoming week
    • They can be found using the devlogs category

Why am I doing this?

I am currently unemployed by choice. I have been very fortunate professionally that I have been able to work on interesting projects, but there are also some things that I have never had a chance to work on that I am personally interested in and would like to explore. Primary among these are:

  • Graphics
    • Hardware ray tracing
      • I fell in love with all things ray tracing while a student at the University of Utah but I have not had an opportunity to develop anything since hardware support was added to consumer GPUs
    • HDR
      • Although I understand the concepts of tonemapping I have never had the opportunity to implement it, and after all of the work I have done with color and light it is an obvious next step and gap in my experience that I would like to fill
  • Memory Management
    • I have a fascination with manual memory management, and I am interested in attempting to write software that always uses explicit allocators rather than the global new/delete so that all memory is budgeted and uses appropriate allocation strategies
    • I am also interested in trying to go all-in on “data-oriented design” to gain more experience about how to design large scale software with cache-friendliness and good access patterns as a priority
  • Engine
    • I would like a better custom basis that I can use to create applications (not just necessarily games)
    • I have almost entirely worked with graphics in my professional career and I would like to gain experience and learn the technical aspects of other areas
  • Game Design
    • I have ideas of things that I like and things that I don’t like from playing games, but my actual experience of user-facing design is limited to APIs and GUI properties

It can be hard to find the energy or motivation to do personal projects like this with a full-time job, and especially with jobs in the industries that I work in which tend to be high-demand (which, to be fair, is also what makes them fun). With this in mind I have quit my job and am taking a temporary break from employment so that I can focus full time on learning and implementing some of the things listed above.

Goals and Scope

The time frame for working on this project is quite limited before I will have to find another paying job. If the end goal were to actually create a finished game that could be released and sold then the only reasonable strategy would be to use an established game engine (like Unreal or Unity), and even then the only kind of game that a solo developer could create in a short amount of time would have to be extremely limited in scope. The majority of my goals for this project, however, involve creating the technology from scratch. This means that the actual goal is primarily to create a game engine, and any “games” that I make will be more like tech demos.

The meaning of what a game engine is in the popular imagination seems to have changed over the years and my impression is that today many people think of a “game engine” as something like Unreal or Unity where there is an editor and it is meant to be used by external users to make any kind of game imaginable. That definition is not what I mean when I talk about what I want to work on, however: I am interested in making something just for me that serves as a framework to create interactive applications. Although in real game development tools are incredibly important I don’t anticipate (sadly) creating any editors for my current project, and any tools will be programs that build game-ready assets from authored assets.

My ideal outcome would be to actually create some finished experience that I could release in some way, but due to the uncertainty around the time available I don’t believe that that should be my criterion for evaluating whether the project succeeds or fails. Instead, my intent is to try and create several small applications along the way that are unfinished and unpolished (i.e. not releasable to a general audience) but that allow me to have some small scale goal to work towards when implementing features. I will also try to document milestones along the way with posts here. The true measure of success for me personally will be if I have been able to implement some of the features listed in the bullet points above.

With that being said, as a way of setting realistically low expectations my goals for an eventual releasable program would be:

  • Graphics rendered using hardware ray tracing
  • Audio sound effects that play dynamically in response to something
  • A player avatar in some kind of third-person view that can be controlled using an Xbox controller
  • Some kind of action/interaction that the player character can do with the environment

My graduate students in a semester-long class used to accomplish something similar (without the ray tracing) using a starting engine that I provided and so the list above feels achievable. I will have to decide as time passes whether to try and focus on implementing the above points early and then improving things or whether to be content focusing on the individual features that are interesting in the moment even if it means not finishing a final project.