Somehow, despite the eternal procrastination that re-writing one of my larger internal projects has become, I’ve finally managed to make some progress and replace my extremely inefficient method for achieving ‘stream-proof’ visuals with something a lot easier to implement.
Wait.. what are ‘stream-proof’ visuals? Put simply, it refers to rendering that is visible to the end user, but invisible when livestreamed or recorded to a video. For example, someone broadcasting with a “stream-proof” wallhack would be able to see players from behind walls while those watching on a livestream would be none the wiser.
A common implementation involves the creation of a transparent window that is placed on top of the game window. You can then create a rendering device and draw over the game to your liking. This is what I’ve been using for a few years now, but the inevitable performance loss and inability to play the game in fullscreen while using this method is a dealbreaker for most.
The method we’ll be using today makes use of behavior exhibited by most recording software, that is, to exclude in-game overlays from the final video. The most common overlay I can think of is Steam – so that’s what we’ll be using today. This post will show you how to find the functions needed, implement the hooks, and of course, we’ll finish off with an example implementation.
First things first, be aware that the following method specifically targets the Steam in-game overlay and games using DirectX 9. It’s possible that some parts will be applicable to other graphics APIs but that’s beyond the scope of this post. Consider this as more of a brief introduction to utilitising the Steam overlay for game hacking purposes.
With that out of the way, let’s get started. Our journey begins with GameOverlayRenderer.dll – a generic library packaged with Steam that, as the name suggests, handles rendering of the in-game overlay. Let’s assume we found this by comparing the game module list when the overlay was disabled to another list from when the overlay was enabled.
You’ll find the library in the folder you installed Steam to. Either right-click your Steam shortcut and click ‘Open file location’ or just browse to the folder manually. Among the files, you should see the aforementioned library (along with a 64-bit variant) and a log file with the same name. If you don’t see the log file I’ve included one here. Not essential, but it’ll come in handy later.
Open up GameOverlayRenderer.dll in IDA and take a look at DLLMain. This is a rather unwieldy function when decompiled but taking a quick look at it now will provide some useful hints for the next part.
The first part of this function is invoked when the library is unloading, as evidenced by the fdwReason check and the ‘GameOverlayRenderer.dll detaching’ string. We’re not concerned about unhooking, though, so scroll down nearer the bottom until you see a few “failed hooking ...” strings.
All these strings are making this a lot easier for us.
With all the error strings littered around, it’s pretty obvious that sub_10071260 is a hooking subroutine. That’s very relevant to our interests so let’s follow it. Upon arrival, we’re greeted with more strings confirming that this is indeed, a hooking function. Since this is referred to as ‘HookFunc’ by the logging strings, take this time to rename the function accordingly.
A cursory glance at the function and the usage we just saw in DLLMain indicates the first argument is the ‘original function’ to be hooked, while the second is the ‘replacement function’. The return value is likely to contain the address of the original function.
There’s not much else to see in DLLMain right now so let’s refer to that log file we saw earlier.
You’ll immediately notice a bunch of strings we saw previously in DLLMain appear here. This log file gives us a good idea of the order in which events occur. Everything after the list of modules is uncharted territory at the moment so let’s go from there.
Sun Aug 20 10:50:47 2017 UTC - IWrapIDirect3D9Ex::CreateDevice hook called (type: 1, behavior flags: 56!) Sun Aug 20 10:50:47 2017 UTC - Creating D3D9Ex renderer
We’ll need a device pointer to use later and that looks like a good place to get one. Perform a quick search for that “IWrapIDirect3D9Ex::CreateDevice hook called” string in IDA and jump to the referencing function.
First thing to look at is the function call around v11. There’s seven arguments there, and the string below suggests it’s some kind of call to CreateDevice. Looking at the documentation for IDirect3D9::CreateDevice confirms this, which would indicate the pointer we’re looking for is in contained in the variable a7.
This is further evidenced by the int** cast in the funtion call to sub_1006C750. Let’s jump there now.
Can you see the virtual functions?
Now this is interesting! We’re seeing some hooks initialised on the device here. There’s probably something useful in here but it’s a little hard to follow without proper type information. Let’s set the type of a1 to IDirect3DDevice9** and look again.
Getting there..
That’s a little better, but it’s still a bit.. off. Try setting the type of v10 to IDirect3DDevice9Vtbl* instead.
..and there it is.
Perfect. We’ve now found exactly what we’re looking for. If you remember from earlier, the first argument passed to HookFunc is the original function, while the second argument is the replacement function. Rename sub_10069F20 to HookedPresent and dword_100EDBFC to OriginalPresent, then jump to the HookedPresent function.
As expected, near the bottom of the function is a call to OriginalPresent. We’ll store the address OriginalPresent points to, then overwrite it to point to our own Present function. Finally, we’ll invoke the original and call it a day.
We’re almost done here now. The only thing left to do in IDA is to come up with patterns for the original Reset and Present functions. If we check the cross-references to OriginalPresent, we can see it is initialised in the function we were previously looking at, and called once in the HookedPresent. Creating a pattern from the latter is going to be a lot easier.
Here's one for Present.
FF 15 ? ? ? ? 8B F8 85 DB 74 1F— Regular or 'IDA-style' signature.\xFF\x15\x00\x00\x00\x00\x8B\xF8\x85\xDB\x74\x1F— Code-style signature.xx????xxxxxx— Code-style mask.
And here's one for Reset.
FF 15 ? ? ? ? 8B F8 85 FF 78 18— Regular or 'IDA-style' signature.\xFF\x15\x00\x00\x00\x00\x8B\xF8\x85\xFF\x78\x18— Code-style signature.xx????xxxxxx— Code-style mask.
The first two bytes aren’t relevant, so make sure you add +2 to the result of your pattern scan.
With that, our work in IDA is finished and we’re finally ready to write some code. As far as pre-requisites go, you’ll only need a pattern scanner. I won’t go into too much detail here, just getting the hooks in-place is more than enough to get you started. After all, everything after this point is generic DirectX stuff that you can learn from anywhere.
We’ll start by defining the original function prototypes and the placeholder replacement functions.
#include <d3d9.h>
HRESULT (STDMETHODCALLTYPE *original_present) (IDirect3DDevice9*, const RECT*, const RECT*, HWND, const RGNDATA*);
HRESULT (STDMETHODCALLTYPE *original_reset) (IDirect3DDevice9*, D3DPRESENT_PARAMETERS*);
HRESULT STDMETHODCALLTYPE present(IDirect3DDevice9* thisptr, const RECT* src, const RECT* dest, HWND wnd_override, const RGNDATA* dirty_region) {
return original_present(thisptr, src, dest, wnd_override, dirty_region);
}
HRESULT STDMETHODCALLTYPE reset(IDirect3DDevice9* thisptr, D3DPRESENT_PARAMETERS* params) {
return original_reset(thisptr, params);
}
Now in our initialisation function, we’ll perform the scanning and put the hooks in place. Starting off with the pattern scan in gameoverlayrenderer.dll, which we can assume will be present in the game process at the time our library is loaded.
std::uintptr_t present_addr = FindPattern("gameoverlayrenderer.dll", "FF 15 ? ? ? ? 8B F8 85 DB 74 1F") + 2;
std::uintptr_t reset_addr = FindPattern("gameoverlayrenderer.dll", "FF 15 ? ? ? ? 8B F8 85 FF 78 18") + 2;
original_present = **reinterpret_cast<decltype(&original_present)*>(present_addr);
original_reset = **reinterpret_cast<decltype(&original_reset)*>(reset_addr);
We make use of the decltype keyword here to save us from re-defining the function prototype. Now with the original functions stored in the original_present and original_reset variables, all that remains is to perform the replacement.
**reinterpret_cast<void***>(present_addr) = reinterpret_cast<void*>(&present);
**reinterpret_cast<void***>(reset_addr) = reinterpret_cast<void*>(&reset);
You could compile and load your library right now but you might be a bit disappointed with the result.
Of course, this is because the replacement Present function is still empty.
HRESULT STDMETHODCALLTYPE present(IDirect3DDevice9* thisptr, const RECT* src, const RECT* dest, HWND wnd_override, const RGNDATA* dirty_region) {
D3DCOLOR color = D3DCOLOR_XRGB(255, 0, 0);
D3DRECT position = {100, 100, 200, 200};
thisptr->Clear(1, &position, D3DCLEAR_TARGET | D3DCLEAR_TARGET, color, 0, 0);
return original_present(thisptr, src, dest, wnd_override, dirty_region);
}
Nothing too fancy, just a red square. Now you’re free to compile & test this out.
Did you see it? Or more importantly, did you not see it? Switch over to OBS, add a Game Capture source and point it to your game of choice. You shouldn’t see the red square in the preview window. If you do, make sure the ‘capture third-party overlays’ option is unchecked in the options window of the game capture source.
With that, we’re finished! Yeah, I know, not the most exciting example but don’t worry, I’ve got you covered. Head on over to this GitHub repository if you want something a bit more practical to play around with.
On an unrelated note, this post marks the 1 year anniversary of this blog. It’s still a bit strange to think this all started when someone in Twitch chat asked about missing Counter-Strike: Source textures in Momentum Mod. Anyway, I hope to continue writing here for the foreseeable future – even if it takes me longer than usual to get this stuff published.