Of Delays and Silence - Part I

Dear All,

Before we jump in; first of all I’d like to apologize for this update coming so late and for us offering little detail of what has been going on. The reasons for that are explained in more detail below, but mainly, this is because we've been working ourselves to the bone to try and steer the ship back on course, as well as not having full clarity over factors that we are simply not in control of. With just two weeks left until the end of our release window, this update is very late, but it is important to note that it is the result of an all-out effort rather than the opposite.

To dive right in, this is a difficult update to write, as failing to deliver and to disappoint you all is generally not a fun experience- especially not at the tail end of a multiple-year marathon; and even more so when it’s the result of both difficult technical processes and some externalities that we do not fully control. This is further multiplied by being something that has a far lasting impact on our team and reputation. The runup to the release of another one of your babies is supposed to be a fun, if stressful event - full of joy, laughter and problem-solving before sitting back and sharing our passion and work with everyone. In the case of the Phantom, it has been like a relentless battle against the elements and the most complicated set of problems we’ve ever had to solve, and despite burning every ounce of energy we could muster in the pursuit of success, we’ve fallen just a little short.

As we have just over two weeks until the end of march and the end of our stated release window, we have made the exceptionally tough decision to postpone the release of DCS: F-4E until early Spring, with an absolute worst case scenario launch in May. We are aiming for an April release, and are working with our partners to get the Phantom to you all as soon as possible. While we have a lot of confidence in our product and we believe very strongly that we would be able to still ship inside of March, we would be risking that any minor miss-step or development element going awry will cause the timeline to slip. This would be unfair to you all; and quite frankly the amount of stress such a condensed release process would incur upon our minds and bodies, and that on our partners, is outside of what is reasonable after a tough few months. The decision hasn’t been easy, and of note is that the F-4 Phantom itself is, for most intents and purposes very close to being ready for launch - minus a few warts here and there.

That said, something that would make us feel even worse is working on something for years, through toils, turmoil and difficulty, only to stumble completely - both in terms of timeline and ultimate quality - right at the finish line. No amount of loss of revenue, ire, reputational loss or even our own fears of the above is worth risking a lack of pride and fulfilment from sharing our passion with everyone. This is a small niche, and we have no complaints as to the commercial viability of our business, but the satisfaction and enjoyment of doing the best possible job we can, with the most rich and enjoyable features we can muster is without a doubt the only driving force behind Heatblur. So it has been for a decade, and so it will continue to be. 

We feel very strongly about needing to go into detail on why we are delaying, in order to give you an idea of exactly why this message comes quite late. Some of the new features and advances that we’ve undertaken have come with tougher technical challenges than expected, with some key elements proving incredibly elusive and adding strong elements of uncertainty. Let’s dive in depth on what we actually mean:

Why are we Delaying?

We’re delaying for two primary reasons: 

• COVID. It seems that every company on the planet has gone through their COVID product and COVID phase, and I suppose that we were only going to evade this for so long. Since December 1st, an estimated 90% of the team has been, in some way, significantly ill, almost exclusively with COVID, and a few with significant forms of long COVID. The team has been at full-strength for, at most, a few weeks since last fall. There isn’t much more to say about this. In isolation, this would not have been an issue, the rest of us rolled up our sleeves and covered for our colleagues, but in combination with the dragon that is..
  
  
• HBUI. We’ll dive into more detail on what this feature is below; but in summary; HBUI is one of the new and entirely novel additions we’ve developed for DCS World as part of the F-4. It is a rendering backend that allows for many new UI elements in DCS World, spanning from browsers, interactive manuals, to a more fully-featured Jester wheel. It is a cornerstone of our plans for today and the future, and also what we think will eventually be a welcome addition to DCS World as a whole.
  
  When we began mass testing of the F-4 in January, we found that an estimated 5-10% of users had complete, game breaking, unusable lag with any and all UI elements due to a GPU bottleneck, and despite all-out efforts, truly day and night, to resolve these as soon as possible, we found ourselves at the mercy of a complex onion of PC hardware and OS interactions. Every time we thought we had a breakthrough, we’d run into another set of hardware and software combinations and be right back at step one. Every time this happened, we thought we were back on track, only to find ourselves behind schedule once again. Every time, this forced us back to the drawing board to reassess and try new solutions, and further shifted our timeline, yet disguised itself as something that appeared to be solvable in time.
  
  While concerning, we initially approached the issue with confidence. We’ve solved worse in the runup to the release of the F-14 and Viggen. Or so we thought. The core issue causing this turned out to be an incredibly complex, multi-layered onion involving both the Windows GPU scheduler as well as GPU process scheduling in DCS itself, mixed in with some deep, hardware level connections across various technologies, such as HAGS, resizable bar, and even specific GPU drivers.
  
  It took us hundreds of hours of debugging and bombarding the affected hardware and individuals with experimental builds, and in the end, despite also reaching out to industry contacts, we found no solution but to rewrite significant parts of our UI layer to solve the issue. Not only did we spend an inordinate amount of time trying every combination of solution we could think of, seemingly at the brink of a breakthrough, but in the end found no reasonable solution other than to go back and rewrite this key piece of the F-4.
  
  In summary, this issue has diverted a significant amount of resources to fix. Elusive issues with many elements out of our control can be a nightmare, especially when it comes to features which underpin the entirety of our product, and for the very first time in our entire decade long history… we just couldn’t find a solution except ripping it apart and building it back together. 

Ultimately, being a small indie developer can be a risky affair. Being a small indie flight simulation developer can be another step up the risk ladder. Adding massively to this risk are sometimes the types of endeavors we electively (and I mean this truly with no arrogance) undertake: adding new, novel, mostly untested technologies and features, developed exclusively by ourselves, at own cost and fully at our own risk - mixed together with what is ultimately the most complex and high fidelity simulation we’ve ever created (the F-4E) and mixed into a fantastic and complex flight simulator (DCS). Not only is DCS: F-4E, especially for our scale, a complex simulation and game-dev project, but also very complex and difficult from a general software engineering perspective. The triple headed trifecta of upfront cost, risk associated with untested solutions, and the reputational hit of not upholding our promises is a bitter pill, but we really hope that you will all eventually agree that the trials and tribulations will be worth it, especially after refinement during the course of EA.

To focus on that substance more than anything; let's dive in-depth on what these UI technologies are, what they bring to the table, and why exactly they’re so fundamental and a valid cause for delay;

So, What is Heatblur UI Exactly?

Simply put, Heatblur UI is a Chromium Embedded Framework backend, with a custom interaction and integration layer, integrated into DCS World. It allows for the use of unrestricted web technologies to build UIs or simply render content just like any other Chromium based browser, directly in game. We envision the use of HBUI for checklists, planning tools, fully fledged EFBs where needed, customization tools, and of course already announced key F-4 features such as the Interactive manual and Bombing calculator. If you’re familiar with some of these types of features in Microsoft Flight Simulator or other simulators, you know exactly what we’re talking about.

The Interactive Manual and Bombing Calculator are just two features that HBUI has enabled.

Moreover, it allows for quicker development of UIs and for far more advanced features in existing UIs such as the Jester wheel. Previously, doing keyboard input, or mouse related input would have been far more difficult. Text handling. Translation, subtitling, and a lot of other features are more effectively achieved, especially for a smaller team. Why? Well, primarily because these things can be developed in a tried and tested, proven, web development based workflow. There are drawbacks; such as increased performance requirements to render simpler UIs versus using native technologies available in DCS, but using the kneeboard as some sort of pseudo aircraft computer was getting.. a little tiring. And you’ve just seen the tip of the iceberg.

Why Delay over this, of all things? You’re making planes.

While you might be thinking to yourself; is it really worth it to delay and to cause issues over something like an interactive manual and bombing calculator? While these two features are important to us and we love that we’re now able to ship these kinds of things into the game; the answer is no.

The reason why this particular issue has been such a critical blocking item for release, is because it also drives the JESTER wheel and Crew Chief menus. Meaning; for those drastically affected (where UI elements would freeze completely) - the aircraft would become completely unflyable (you wouldn’t be able to cold start!) - or you would simply be unable to operate jester using the UI. While even these two issues could in theory be overcome- as we now offer key binds for every single jester wheel entry, we’re a little too stubborn to give up.

Unfortunately, this time around, it simply took a few hundred hours longer to figure out a solution than we expected. In combination with so many illnesses, it has simply put too much pressure on the team. We’ve been pushing ourselves incredibly hard to wrap things up regardless, and I couldn’t be prouder of the team at getting things practically across the finish line at this point, but we also recognize the fact that it’s best to cool things for a couple more weeks, and ensure that the i’s are dotted, and the t’s crossed.

HBUI underpins the most important UI element in the F-4E: The JESTER wheel

You really should have told us sooner

We agree, and we apologize for stretching it so thin. However, we have simply been working down to the wire. The inconsistency of illnesses and the issue which we believed to have solved several times over exacerbated the issue greatly - as we were constantly moving in and out of confidence about our window. To be perfectly clear, even with all the illnesses, we were somewhat ahead of schedule versus the state of the F-14 at the same moment during all of this winter. Unfortunately the tail end of our development process saw a steeper decline in readiness state versus that launch.

To offer some more defense based on past experiences: generally, the way we approach a release window is by setting a broad target window that accounts for things going wrong. Even if you cumulatively would pile up bad news; people getting sick, bad bugs happening, some fundamental technology having issues to resolve- all of those should be resolvable and you should be able to ship- even at the tail end of your window.

Thus, even when things go sideways, you’ll just have to pile on the coal and probably make some adjustments to the EA feature list; make some minor compromises in general, but you’ll make it. This is the approach we have had, and took even with this major issue. You work until the last second to resolve them and ship on time. We’re not here to plod along when we have to deliver; you straighten your back and get to work; but in this particular case, the issues sucked up so much time, effort and energy that we’ve been left with an uncomfortable amount of time left.

We’ve in this instance finally decided to extend things just slightly to give ourselves a little bit more breathing room to ensure we don’t fundamentally undermine our vision.

We were caught off guard by the severity and difficulty of resolving this issue; and even though the team was somewhat split on actually being for or against extending our timeline, we think that even making this late decision, frustrating as it is, is for the best.

The UI system is also used for the Dialogue Wheel

What are you doing to make this right?

We take our promises to you all very seriously. While we may try to share best guess estimates throughout development, generally once we open a pre-order window, we consider that to be an explicit promise rather than an estimate.

We’ve always had an unspoken, no questions asked policy and this has not changed, and if you so wish we will continue to honour this, regardless of where you purchased the F-4 pre-order. Further, if you decide to take a refund for your purchase, we will extend the pre-order discount to you for two weeks past release day as a thank you for your support, should you find initial reviews favourable and wish to re-purchase.

In the future, we will ensure that any new and fundamental technologies which may be impacted by any factors that we are not fully in control of (e.g. hardware!) will be more broadly tested before we open pre-orders and set a launch window. In this case we were unlucky that not a single dev or early tester machine exhibited the issue (though that is essentially a 90%~ chance) - but that was a false sense of security. It worked, until it didn’t, and threw plans into chaos at the most inopportune moment.

In Summary

In summary, we had a massive, fundamental, game breaking issue sneaking around in one of the most fundamental features of our product, and we had absolutely no idea until we started mass-testing. We prematurely declared victory every time it looked like we had broken through the issue - only to find ourselves at square one - further delaying and torpedoing our timeline, and our disclosure timelines.

There rarely is a perfect storm, but we certainly feel like we’ve found ourselves in one with the F-4. We’ve had our share of difficulties in the past; in the runup to the F-14 suddenly all multicrew synchronisation stopped working prompting an extremely urgent total rewrite. We had a game breaking bug that would kill all audio in the game for months. We rolled up our sleeves and got it done, and we did not expect any different from the F-4E. Game development is inherently quite chaotic, but we will try to strengthen our processes to avoid similar issues in the future.

If there is a silver lining; we’re now through the worst of creating the entirety of this new simulation framework, and over time, the stability and experience of leveraging it for future products will reduce the risk of these sorts of issues. As mentioned previously the F-4E represents a step up in terms of realism, fidelity and complexity - and the very first product always represents the greatest risk.

We hope that the above is helpful in understanding the why and how. Both in terms of the technical aspects, as well as the reasons underpinning our decision making and limited comms. 

With that out of the way to some extent; let's follow up with Part 2 below - which is a more standard development update.

Thank you as always for your support and sincerity, and we truly hope you will enjoy the Phantom on launch day.

Sincerely,
Nicholas Dackard

Continue to Part II - Development Update