This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 9.

What I have done

• The texture is now uploaded using the copy command queue instead of the graphics command queue
• Made a platform-independent interface for vertex buffers
• Made a platform-independent interface for index buffers and am using it now to draw the simple square
• Made a platform-independent interface for constant buffers and am using it now to draw two squares (where it is being treated as a per-draw constant buffer)
• Added the Tracy profiler
  • Implemented an interface for CPU profiling that doesn’t require adding anything from Tracy in header files
  • Implemented an interface for GPU profiling that doesn’t require adding anything from Tracy in header files
  • Modified the PIX for Windows integration to work more like Tracy
• Build system
  • Added a Lua function to resolve paths
    • This makes it easier and nicer to generate header #include files so that resolving environment variable can be done automatically
  • Fixed a tricky bug when adding multiple levels of filters in Visual Studio
  • Fixed a bug when several different C++ source files specified different ways of building that were independent of the general way specified for the target
  • Found a workaround to help Intellisense give proper x64 suggestions (instead of x86)
    • This didn’t affect the actual compiled or linked files, but it was a relief to finally have pointers and struct sizes reported correctly

Next Steps

I am getting stuck (classic “analysis paralysis”) on trying to figure out how I want frame updates to work. This is kind of where I got stuck previously when I was doing the “beam racing” stuff, and even though I finally gave up and moved on I am running into it again with a different context now while I am trying to figure out how to have the application tell the graphics system what to render and when. It would be easy to just do something simple and move on and maybe I should, but one of the personal goals that I’ve wanted to do with this project is to have fixed simulation timesteps that are independent from the fixed display refresh rate, and what I want to accomplish is pretty straightforward even though I’m getting hung up on details.

• Implementing the Tracy profiler has been a big help, and I am now able to better visualize what my code is actually doing. I need to somehow get something in place, even if it’s just rudimentary, so that I can have simulation updates (that can run both slower and faster than the display refresh rate) and then choose the appropriate time to render an interpolation between two of them.
• I still need to implement a way in the build system to compile shaders offline and then a way to load them at runtime, even if very simple
  • (Recall that I am avoiding the standard library and general-purpose memory management, which is why some of those seemingly-simple tasks involve more work than might be expected)
• I want to add a camera and 3D transforms

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 8.

What I have done

I got very sick this week and so a lot of time was lost.

• Fix precompiled header source linking
  • It turns out that on MSVC it’s not enough to use a precompiled header, but the source (i.e. CPP) file that created it must also be linked. This kind of makes sense in retrospect, but things were somehow working without doing that for a while.
• Created a texture resource in an “upload” heap, mapped and copied texture data to it there, and then copied the resource to the shader-visible default heap
  • This all just followed the Microsoft “hello” sample
• Create constant buffer in different ways and different places to move triangle mesh around
  • I think I am finally understanding how resource management works in D3D12, after an embarrassingly long time and an embarrassing amount of re-reading different documentation and forum posts
  • It’s still unclear to me what the best strategy to manage it all is, but at least that isn’t embarrassing because it seems intentionally flexible and so everyone is a little unclear because there are many different ways to approach it
  • Additionally, since it was originally released there are new techniques for “bindless” rendering and it seems like I will probably want to use those for ray tracing, and so there is still more to learn. One step at a time, though…

Next Steps

The sickness has thrown me off of my schedule and I also find that I am suffering with the mental symptom of wanting-everything-perfect-and-not-knowing-where-to-start with some of the problems that I have to tackle next. I am at the point where I just need to do force myself to do something to make progress, acknowledging that it might not be ideal and will require refactoring later. (I actually have pretty clear ideas of how I would naturally do some of this stuff but I am also trying to approach things in a purely “data-oriented” way for this project and that is contributing to some of my hesitancy because I wonder if my natural tendency is the result of old habits.)

I think there are three main things that I want to do next:

1. Refactor some native D3D12 objects so that I can work with them in a more ergonomic (and potentially platform-independent) way, but more specifically so that I can specify things to render from the application rather than hardcode things in the graphics system
2. Expand the constant buffer scheme so that I can have a camera and multiple objects with transforms and materials (even if these are very simple initially)
3. Add the capability to the build system to run an arbitrary command line so that shaders can be compiled offline, both because I want to have shader errors reported at build time but also so that I can start using shader model 6.6 for the new bindless resources in a more sane way

I have also thought of a variation on breakout that seems like a simple way to incorporate some ray tracing, and so, at least for now, I think that might be my initial small proof-of-concept application to try and make, just so that I have some kind of example application rather than the current empty program that does nothing.

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 7.

What I have done

• Finished the intentional tearing demo, which I wrote a post about
  • This was both satisfying and frustrating. I was happy to get something working that I had wanted to do for a long time but also disappointed that I didn’t get better results. I am also frustrated that I still don’t feel like I have as good of an understanding of swap chains and related flipping and timing as I would like.
  • I finally gave up and moved on and will have to revisit some of this when I have more complicated scenes being rendered, but it felt a bit like accepting defeat
• Improved the code that waits when it happens on the Windows message queue thread
  • Right now the entire application is single threaded, but a rudimentary mechanism is in place to detect the thread (what I am currently calling the “display thread”) and do a different kind of wait so that new messages can be processed if appropriate
• Render a triangle
  • This was mainly just following the Microsoft Hello-Triangle example, and so not particularly impressive
• Improved the array view class
  • This is my implementation of std::span, and the initial implementation revealed some problems when dealing with COM ISomething* pointers, both because of pointer const complexity and because of inheritance
  • I wrote a bunch of tests of situations that I could think of and the class now works better. This presented a fun (thought frustrating) challenge to solve with C++ templates.
• Add WinPixEventRuntime
  • This is the first external code/library that I have included in the project, which is a little unusual for me. Usually I start with {fmt} and Lua in order to have logging and configuration available, but in the current case I am delaying dealing with any strings and so haven’t followed my usual pattern.
  • This integration allows me to annotate captures in PIX for Windows
• Add precompiled header support in the build system
  • Adding the ability to (force) include files wasn’t particularly difficult, and that was the primary motivator for doing this so that I could start writing and structuring code the way that I wanted to. I thought that going one more step with precompiled headers wouldn’t be too difficult but it ended up being more challenging than I had expected.
  • MSVC has some interesting constraints that I hadn’t been aware of where any files using a precompiled header have to use the same PDB file. Most notably, this means that a single PDB file is created and modified in many different steps by many different files getting compiled, which didn’t fit in well to my build system model where every task graph node (i.e. file) was the result of a single sub task. I think that I figured out a satisfactory way around this, though.
• Add ability in the build system to query whether a sub task has a specific input
  • This allows me to decide whether a specific application needs the WinPixEventRuntime DLL
• Add ability in the build system to query for the primary/obvious output target of a sub task
  • This allows me to decide where to stage the WinPixEventRuntime so that it is next to an application executable
• Add ability in the build system to create files
  • This allows me to programmatically create a header file that #includes the WinPixEventRuntime headers using the current version number, sharing code with the LIB and DLL files

Next Steps

• An obvious need is to add something to the build system to deal with shaders
  • Right now I just have a manually-copied shader source file in the application root directory that is compiled at run time, which is obviously not ideal
  • I am a little hesitant to start going down the path of figuring out asset building and loading, however, because I don’t have any string solution (recall that a goal of this project is to not use the standard library and to not use the general purpose new allocator, and so dealing with strings and paths is a big task to tackle)
  • With that being said, I could at least create a new type of build system sub task that allows a command line to be run and inputs and outputs to be specified. This would make it possible to work with the shader close to what I’m doing now but not quite so manually, and seems like an ok first step.
• The other obvious task is to keep adding graphics features beyond the simple triangle and color-changing clears
  • I might at least continue to implement some of the D3D12 “Hello” examples, to increase my familiarity with the changes of D3D12 compared to D3D11 and to give me more time and experience to start thinking about abstractions and how to structure an actual renderer how I would want to

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 6.

What I have done

• Initialized Direct3D 12 and cleared the window
• Calculate timing of display refresh so that vblanks can be predicted
  • I am able to display horizontal color bars on the display without vsync enabled and with only clearing the entire window by properly timing and predicting when to swap the back buffer
  • This is something that I’ve wanted to do for years after reading about it and so it was fun to finally make it happen. I will make a dedicated post about it once I have cleaned up some remaining problems.
• Added the ability to specify C++ compile optimizations and update the project configurations accordingly

Next Steps

• Improve display timing
  • There are still problems that I don’t understand. Rendering without vsync isn’t important because the real engine won’t work that way and so it’s fine if the fun demo of color bars doesn’t work perfectly. What is important, however, is that the underlying timing information is correct for what I want to do with the real engine and I’m concerned with some of the behavior that I occasionally see that I don’t understand because it could lead to hard-to-diagnose problems later.
  • Even though I did the color bars as a fun project they are actually a really good way to visualize whether the timing is correct and so I think it is good to keep working on it before I undo the fun temporary hacky code and make things work the way that they are supposed to.
• Create a separate thread for the display loop and the game logic
  • I was going to delay doing this until later, but as I’ve been working on the swap chain code I think it would be good to get this more formalized early
• Draw a triangle
  • I had thought about adding better input next but for now I can just detect key presses using GetAsyncKeyState() in a hacky way and that means I can quickly experiment without having to go through an application layer. So I think I have changed my mind and that it is a better idea to keep working on the graphics system for the moment.

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 5.

What I have done

• Build system changes:
  • Include directory search paths are used in the dependency hash so that if they change a file is recompiled
  • The use of standard libraries or platform libraries in compiled files that a target depends on now influences its compiled files include directory search paths
    • This is so that if some downstream target #includes a file that #includes <windows.h>, for example, that it will work as expected (meaning that specifying the static library dependency is enough)
  • Added features to help IntelliSense
    • Include directories and preprocessor defines are now added to VCXPROJ files
    • Compile arguments are now added to VCXPROJ files
      • (It turns out that this is important for IntelliSense for things like detecting what version of C++ is used; without it there are lots of false positive red squiggles when using newer features)
    • Files are now added to VCXPROJ files
      • Currently this is done by specifying a file path, along with an optional type (e.g. for C++ compiling or C++ including) and an optional logical display folder (so that files can be organized in a logical file structure in e.g. Solution Explorer).
      • I mostly like this, but one small annoyance is that for C++ files that get compiled it means that each source CPP file has to be listed twice. For now I have decided that the way it is makes sense (rather than e.g. automatically adding these files to be displayed) but it’s something I will have to keep thinking about as I get more experience.
    • Individual files can now be compiled in Visual Studio
      • This involved both adding a pattern for command lines but also adding the ability to jpmake. Previously I had a command line where a file path could be specified to generate just that file (i.e. it was an output file path), but in order to support the Visual Studio functionality the file path must be an input path.
    • Individual C++ files to compile can now have individual C++ build info, rather than always using the C++ build info of the target
      • This was already supported under the hood in jpmake, but I didn’t have a Lua interface for it
• A new EmptyWindow application exists
  • All this does is create and display a main window, so it is the most basic application imaginable, but it was still exciting to be working on real code rather than the build system
  • I made a post about it
• I have added many type traits
  • So far these are just clones of standard library ones that I have needed, but it has been fun to learn more about how some of this template “magic” works
  • I made a post about it
• I added scope guard classes
• I added custom implementations of move and forward
  • MakeMovable() and ForwardReference() are macros based on https://www.foonathan.net/2020/09/move-forward/
    • I discovered accidentally (there was some reddit or stack overflow comment about that link I think) that the macro version enforces actual moves with no copying fallback, which is something that I have wanted but wasn’t sure if or how it was implementable. After figuring out why the macro had this difference (related to const vs. non-const rvalue references) I have also added a MakeMovableIfPossible() in case I ever want it, which just does the same thing that std::move() does.
• I added volatile load and store functions, with relaxed, acquire, or release memory ordering
• I added the ability to allocate and free memory from the operating system
  • (It is also possible to reserve and commit memory in steps if the operating system supports it. This is something I use in jpmake, but don’t currently anticipate using in the game engine itself.)
• I made array view and memory view classes
  • These don’t have all of the functionality that I would like, but they are a good start of a fundamental building block
  • The cMemoryView class can have a view of both const and mutable memory, and allows slices to be made for partitioning memory
• I have added memory arena and allocator classes
  • This is just a starting point, but I do have a monotonic allocator and am able to partition and then logically allocate memory from the operating system
  • I have New()/Delete() functions for working with typed memory and Allocate()/Free() functions for working with untyped memory, all of which require an arena
    • The goal is to allocate a single big block of operating system memory when the application starts and then only work with that for the lifetime of the application by partitioning that into smaller sections and using hierarchical arenas to allocate in a controlled way (“controlled” both in terms of lifetimes and cache friendliness).
• I have set up the machinery for creating a graphics “engine” and created a D3D12 device
  • The application lets the graphics library create a graphics engine, passing in itself and a block of memory from the operating system that is budgeted for graphics. The graphics engine stores a reference to the application who created it and returns a pointer to itself that the application can refer to. This is my attempt at dependency injection while still allowing easy access to different systems.
  • The graphics engine will have a platform-specific context, and so far I am just creating a D3D12 device to store in it as a proof of concept to show that I am able to access the platform-specific D3D functionality and allocate and store things with the new memory system.

Next Steps

• Use D3D12 to clear the main window to some arbitrary color
• Implement the top part of the window as one clear color and the bottom part of the window as a different clear color using ideas discussed at https://blurbusters.com/blur-busters-lagless-raster-follower-algorithm-for-emulator-developers/
  • This isn’t how I intend my engine to run (I will use VSync), but ever since reading about it I have wanted to try and do it to make sure I understood the timing involved, and I think getting this to work would show that I have the fundamentals in place before implementing how I really want my renderer to work.

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 4.

What I have done

• A jpmake project can now generate Visual Studio IDE files
  • A solution file is created with project files corresponding to every named task
  • The appropriate command line is generated when a project gets built. The output can be generated if the Lua jpmake file specifies it, e.g.:
    • cppNamedTask:SetTargetForIde(myApplication)
  • Additionally, there are two special projects that are generated:
    • Do [JPMAKEPROJECTNAME]
      • This does all tasks, and it is the only Visual Studio project that is configured to build when the entire Visual Studio solution is built
    • Regenerate Solution
      • This can be built when there are changes to the jpmake project file
  • With these improvements it is possible to reasonably work with a jpmake project in Visual Studio. There are still several obvious important missing features (and many more that I could think of that would be nice to have), but I think the status is good enough to start development and add more features as they prove themselves to be issues.
• It is now possible to specify include directories
  • This seemed important in order to be able to #include engine header files from application code
• I have added options for generating debug info/symbols
  • After experimenting with debugging in Visual Studio this clearly was important
• I have added options for handling warnings
  • Specifically, being able to set warning levels and to treat warnings as errors
• I have created a new git repository to contain the actual engine and application code
  • I have integrated jpmake into this as a submodule
    • This is not what I had initially intended and instead had anticipated just having a jpmake.exe in the repository. It seems, however, that there will probably be frequent jpmake changes in response to me actually working on real projects and realizing that I want or need more features (especially early on) and so a submodule started to make more sense to me than a frequently-changing EXE binary file.
  • I have set up a Windows platform and three different configurations
    • For now I have “unoptimized”, “optimized”, and “release”
  • I have set up some initial directory structure, although I’m sure my mind will change about the best way to do organize things and what environment variables to use as I start authoring actual engine and application code
• I have created a HelloWorld console application
  • This doesn’t do anything except print “Hello, world!” and so it doesn’t represent any interesting progress over the more complicated examples I had last week
  • It does, however, live within an actual project and so represents the first application in this new game engine repository

Next Steps

• I need to make an application that displays an empty window
  • This should almost entirely use engine code, where the only thing that the application does is derive an application class from an abstract base application class and then pass on command arguments
• The next step would be to initialize Direct3D 12 and clear the back buffer to some color
  • This might take more time than it otherwise would because I also want to implement some mechanisms for manual memory management

This is a regularly-occurring status update. More generally-relevant posts can be found under Features (see Creating a Game and Engine from Scratch for context).

This is the beginning of week 3.

What I have done

• I have been working on a build system so that I can build code and assets the way that I want to. A dedicated post about it is here.
  • It is possible to define different platforms and configurations (e.g. debug or optimized)
  • It is possible to define three kinds of tasks (C++, copy files, or arbitrary Lua functions)
  • The task dependencies are tracked, and they are only executed if necessary and in the correct order (only single-threaded, although it has been designed for multi-threaded execution)
  • For C++:
    • The installed Visual Studios and Windows SDKs are calculated, and the user can either request the latest by version number or request specific versions if desired
    • Standard library and platform library support can be enabled or disabled (and, when enabled, the appropriate #include paths are set)
    • It is possible to build console applications, windowed applications, shared libraries, or static libraries
    • Exceptions can be enabled or disabled

Next steps

• It has now been two weeks that I have been working on the build system, and even though I have made good progress it feels like I really need to start working on actual game engine code. There are still obvious missing features in the build system but I think a good strategy now is to add them as they become necessary for the actual code I am working on rather than trying to add more preemptively.
• One final thing that I think that I want to add is the ability to generate a Visual Studio solution that I can use to debug
  • I think it’s ok initially if I don’t create individual projects for named tasks and instead just have a single project to build the entire solution (it seems ok at least initially to manually find files and open them in the editor)
  • I do, however, want to be able to debug, and having some intellisense would also be nice, and so I might spend some time trying to get that done.
• Another thing that I know I will want is to be able to set up precompiled header files
  • Being able to do this in a platform-independent way would be nice so that I don’t have to go back and redo it later, but it’s not critical
  • I may or may not add this before moving on to start work on code
• The initial goal for a program would be to open a window that doesn’t do anything, and that is pretty achievable
• The next goal would be to clear the color buffer to some hard-coded color
  • This would be quite easy using Direct3D 11, but using Direct3D 12 requires allocating some memory which will require some memory management work with the way I want to do things (rather than just using global new).