A look into the concepts and design phase going into embers, the new rendering engine for luxe.
Not sure if I've said this before but rendering can get pretty complex! At least if you want to do it well, or at any sort of scale, there's a lot to be considered.
What I don't think many people really understand about rendering in particular is just how nuanced and specific it gets on the project level. The considerations you have to make in the renderer layer are very frequently related to the game you want to render (especially when it comes to bigger games). I feel like a lot of developers don't know that (or feel they don't need to). It really helps if you do try to understand that at least a little, though.
In the continued snõwkit spirit of understanding the tools we use to make our games, I'm going to try and elaborate on some of these topics while we work on bringing the new renderer into the engine.
I'd encourage you to spend some time on this site, if you're interested in rendering in general. https://simonschreibt.de/gat/renderhell/
The book parts explains how modern rendering works, described in a way that's easier to digest if your background isn't in rendering, and it isn't super long so it's quite easy to get through in a short amount of time. The rest of the site (the game art tricks) are some fascinating reads from games in production over the years, and how they exploited various techniques to achieve project specific rendering results.
game specific rendering
A really good renderer in my opinion facilitates a lot of project level nuance, without having to write the renderer part again. Getting that right is relatively challenging, for big or small engines alike.
For me personally, my engine needs to allow me to implement project specific rendering, every time (not that I would need to). But my point is that the game specific parts would change - and this is part of the fundamental philosophy that luxe is built on. You do the game part, and the engine facilitates it. A “100% solution” doesn't exist in my opinion, the best case is a tool that gets you very close to your goals with low friction, and doesn't get in your way when you get to the game specifics. The rendering in luxe is being designed in the same way.
With a strong backbone, a variety of solutions each catering to specific needs would have a foundation they can build on, that both beginners and experienced devs can benefit from.
rendering can be simple
Just to note this, that while writing a flexible, portable, longer term renderer is a big task, rendering for a simple 2D game can definitely be pretty simple. Once you pass the initial hurdles of getting a triangle on screen, you draw two triangles together, and presto a sprite is born. A surprising number of games could get away with doing just that, without any further complexity. However the games I want to make are a tad bigger than that, and the engine has to support a wider set of needs.
Over time, you might find that the above doesn't run so well on mobile, and find some of the more common challenges in rendering (mentioned above in the render hell books, the problems chapter). You'll have to look into batching sprites together, which there are numerous approaches each with their own merits. Then you'll want to sort your sprites by their shaders and textures, so you only change them as much as needed. These are all still relatively simple, and manageable for sure.
on the road
The current luxe renderer, phoenix, originated from this sort of path. It was created with a few goals, one of them being automatic batching, and when I first started using it (in c++ back then) it was clean and simple. As my projects grew, we added shaders and over time it evolved. When I ported it to Haxe to use in luxe, I took out some of the 2D assumptions (since geometry is just geometry, and shaders are just shaders!) and had used the renderer for a few 3D projects and had noticed a few issues with the design that obviously wouldn't apply to a renderer that wanted to handle both.
As a simple example of one of the issues (I've covered this in dev log 7 in a bit more detail) is how it batches geometry and sends it off to the GPU. This is the primary bottleneck even for the 2D rendering, and it definitely doesn't scale well when you consider a single mesh in a 3D game could be equivalent to a few hundred sprites. If you wanted to render a small 3D scene the number of vertices grows and well, that's not what the renderer was designed to do!
I believe it's pretty important when considering something as fundamental to the engine as rendering that it be considered with fresh eyes. What I mean is that, given phoenix already exists and could be manipulated (even further than it has been) to solve it's main issues, it will never be able to address it's fundamental design decisions without being rewritten anyway.
Even when starting with a blank project to explore what a new renderer might look like - there are so many things that go into rendering - that it's relatively easy to carry over a number of assumptions with you into the new implementation. Initially, while I was starting out with exploring solutions for embers, I actually was trying to solve specific issues that phoenix presented in different ways. I got a lot of answers, but I hadn't at that time considered whether I was even asking the right questions, or dragging old problems into the new implementation (by way of a solution).
What I figured out and learned is obviously not lost, but it's important to challenge the assumptions that were in my mind, placed there from the years I had been using phoenix, and replace them with questions about the present and the future instead.
If luxe is to facilitate at the ideal level I need it (i.e why I'm making luxe in the first place) it has to be well considered, and this is where we are, now.
Let's consider some things from a renderer that is designed differently, for a second. This is the approach that embers is currently being designed against.
layers for days
The backend layer
One of the key parts of a renderer is abstracting away the graphics API specific details (like OpenGL calls), allowing the rendering part of an engine to be ported easier. What you want is typically the backend, which does all the talking and follows all the rules of the graphics API all self contained so that it can do it's thing without jumping through a lot of hoops.
The primary goal of the backend then, is to abstract specifically the graphics API (OpenGL, Metal, Vulkan and so on), and to port your engine to a new renderer, you just port the backend.
The low level layer
This layer talks to the backend for you, and describes slightly higher level concepts that make up rendering. Take for instance, sending vertices to the GPU. The backend handles the actual details of that, but the low level layer is the one that describes the vertices, their organization with other geometry properties (like the vertex colors, vertex uv's), as well as other low level pieces. Things like a render pass, the ability to define where rendering ends up (a texture, the window backbuffer), what a shader is and how that is defined, and so on.
The mid level layer
This is the layer that typically would describe a geometry as a whole, rather than a collection of parts. In other words, it would be the one where a unique mesh or sprite could defined, and rendered from. It would also probably define the overarching structure of the frame, like is there just one big list of geometry? Are there groups? What does a material mean? Where are the textures stored?
The high level layer
Now that we can define a frame in renderer terms, and we can define how things look, we might also want to provide facilities like batching. Batching is essentially putting many other geometry instances into a single one, so it can talk to the mid level layer. Other higher level constructs could include level of detail systems, culling and so on, but those can also go on the mid level, depending on the goals. Keeping in mind not every game needs that stuff in their default render path, it makes more sense to be in the higher level, provided as facilitators and not something to work around.
The game specifics layer
And finally, you have all the rendering specifics for the game. In 2D terms this might be a custom 2D shadows implementation, or it could be a ascii renderer or some customized tilemap implementation for rendering large scale worlds that stream efficiently, things that the renderer shouldn't know about, but can easily facilitate.
What you can hopefully deduce from the above described layers is that there's a lot of stuff to fit in ones head. Often the lower levels are defined by needs of the higher level, and vice versa. It takes a good amount of time to validate choices and assumptions, because there's a lot of things to build up before you should start exploring other layers. Like, how do you test a renderer? Ideally with a game-sized test, but how do you test the low level, when drawing a single sprite takes 60 lines of code? If you wrap that into a class, aren't you defining a higher level in the low level test? Will that cloud your view or mask issues with the workflow on the low level?
Then there's choices. Choices made in any level below another definitely impact the ones above it... At least on the lower levels there is a lot of well defined areas so it's relatively easy to progress (there are only so many ways to feed geometry to the GPU for instance). On the mid and higher levels you usually have to make choices in order to progress, and those choices stack into the final outcome. Whether those choices fit every type of game is debatable, and why I think the better option is described above, which puts the choices on the higher levels and focuses the implementation details on facilitating that choice instead.
Yea, it's a lot of work!
My point here is that in the design phase, there are a lot of questions. To answer those questions, it feels like you have to be in multiple stages all at once. It would be easier to test the lower levels by having the higher levels around, but you need the low level in place to do so. These are the sorts of things that add to the complexity of writing a decent renderer, because you may have to back pedal and chuck out some choices to allow the higher level ones to make sense.
It's not so much that any particular part of this is overly difficult, but there are numerous challenges on numerous levels, and there's a good amount of thinking and planning required to keep complexity and coupling in check, all while balancing flexibility. Fitting it all into a neat, maintainable, future focused, portable, light weight renderer that's also efficient isn't exactly an evening of relaxed coding. It takes deliberate consideration, and inordinate amounts of reading.
Before I ramble much more, I just wanted to note that due to the nature of the challenges and due to the goals of luxe, tied with the amount of stuff that's hinged on the renderer, it's not that there's slow or quiet progress on the engine, just that I don't really want to rewrite the renderer again any time soon so I don't want to limit the potential by rushing choices.
The above is all describing relatively straight forward stuff, it doesn't even get into scalability (like multithreaded rendering) or complex 3D render pipelines with multiple stages and passes, compute shaders or any of that.
The good news is that there's a lot of people that have forged good paths, and while my particular goals are somewhat specific to my ideals and standards, there's a lot of good knowledge and information to lean on. It's not like running around in the dark, and I'm definitely leaning on those that went before and soaking in tons of experience from others.
Also, I know it's not super ideal for this type of design exploring not to be done on the repo itself (in branches), but it's the best way I know how to work efficiently and design carefully without being easily distracted. Rendering scope is frequently overwhelming, and I also have art work to do on Drifter of course. When I have to consider so much and fit a lot in my head, it's infinitely harder to try and account for what others are thinking as well. I just don't have the bandwidth for that.
My exploring process in this phase is more like a bunch of napkins in a fish bowl. Having to explain why there is a slice of cheese taped to the side of a fish and what the napkins are for, is time spent not answering questions in the code that actually matter. It's not meant to make sense yet!
I will, as always, go into details about all the choices and the paths to them, rest assured. I definitely plan to write a series of posts on my personal dev blog as well as here about rendering (like I did with shaders), and like I'm doing right now.
I appreciate that the community understands this about my workflow, and I know that we're going to land somewhere good with regards to the renderer.
Rendering is the biggest major system left in the engine, and informs a lot about the rest of it. As I progress, the pieces start falling into place, suddenly everything becomes clearer, and we'll be continuing forward as usual in a much better place.
In other good news, I'm making great progress on the design exploration! Things are progressing really well in fact, and before long I'll have a complete structure ready to upstream and work on in the repo.
I'll leave what's going on here to your imagination for now.
There is (as usual) lots of fun bugs when doing rendering.
In the next post, I'll elaborate on the plans with luxe alpha-4.0.
I'll also try to cover some of the choices I've already made with regards to embers, and give a general overview of where it's at, since I think that a lot more will be solidified by then.
I also dug up a bunch of the old images from when I initially implemented phoenix in luxe, it will be fun to compare the approaches, since phoenix was a place holder during alpha and embers is a more considered and designed renderer for the engine.
Till next time!