Helix Mod: Collection of bookmarked posts related to shader hacking

In light of the fact that I'm no longer able to dedicate any significant amount of time to working on fixes (and, unfortunately, I'm not the only recent 'retiree'), and that we seem to have several new eager and capable individuals looking to learn the tricks of the trade of shaderhacking, I want to be able to provide whatever support I could for the transition to this generation.

There's no denying the fact that in order to learn the fundamentals of shader hacking that there's a LOT to read, learn, and understand. I, nor anyone else can change that, but what I can do is help provide a listing of many useful posts that I had uncovered while I had lots of time on my hands. I spent hours scouring the two largest threads on the Geforce 3DVision forum for this information, and now I'll share it all so that you don't have to!

Mind you, this is all supplementary reading for AFTER you've completed and understood the basics that are laid out and taught through bo3b's School for Shaderhacking. Even though it teaches the use of Helixmod on DX9 titles, and most newly release games are DX11 and require the use of 3DMigoto, which uses an entirely different programming language (HLSL, which is essentially the same as C++ in many regards), those lessons will provide the foundation on which all of this other information builds upon.

I've grouped and categorized the bookmarks to help with organization and navigation. But first I'll start with the holy grail; the one single post that probably taught me more than all the rest combined! (well, again, this was after learning and understanding the basics. Won't make any sense until then):

Fixing shadows, various transformation stages, and fixes for each

... and now on to the rest.

Fix examples

And lastly, I provided a lengthy explanation in response to a PM I received from an inquiring soul regarding fixing HUDs, that goes over some basics in some easy to understand language. Since I can't provide a bookmark to the PMs, I'll copy and past the discussion here:

PM Received:

If you have time could you do me a favour and take a quick look at the shader below, and if you can point me in the right direction for some example code to stereoize it. It is just the HUD shader which id like to push into depth a little bit. I always learn best by comparing examples, so it would be a great help if you could spend a couple of minutes, I'd really appreciate it.


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
cbuffer cb0 : register(b0)
{
  float4 cb0[4];
}








// 3Dmigoto declarations
#define cmp -
Texture1D IniParams : register(t120);
Texture2D StereoParams : register(t125);




void main( 
  float2 v0 : POSITION0,
  float2 v1 : TEXCOORD0,
  float4 v2 : COLOR0,
  float2 v3 : TEXCOORD1,
  out float4 o0 : SV_POSITION0,
  out float2 o1 : TEXCOORD0,
  out float2 p1 : TEXCOORD1,
  out float4 o2 : COLOR0)
{
  float4 r0;
  uint4 bitmask, uiDest;
  float4 fDest;


  r0.xyzw = cb0[1].xyzw * v0.yyyy;
  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  o0.xyzw = cb0[3].xyzw + r0.xyzw;
  o1.xy = v1.xy;
  p1.xy = v3.xy;
  o2.xyzw = v2.xyzw;
  return;
}


/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//
// Generated by Microsoft (R) D3D Shader Disassembler
//
//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
//
//
// Input signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// POSITION                 0   xy          0     NONE   float   xy  
// TEXCOORD                 0   xy          1     NONE   float   xy  
// COLOR                    0   xyzw        2     NONE   float   xyzw
// TEXCOORD                 1   xy          3     NONE   float   xy  
//
//
// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION              0   xyzw        0      POS   float   xyzw
// TEXCOORD                 0   xy          1     NONE   float   xy  
// TEXCOORD                 1     zw        1     NONE   float     zw
// COLOR                    0   xyzw        2     NONE   float   xyzw
//
vs_5_0
dcl_globalFlags refactoringAllowed
dcl_constantbuffer cb0[4], immediateIndexed
dcl_input v0.xy
dcl_input v1.xy
dcl_input v2.xyzw
dcl_input v3.xy
dcl_output_siv o0.xyzw, position
dcl_output o1.xy
dcl_output o1.zw
dcl_output o2.xyzw
dcl_temps 1
mul r0.xyzw, v0.yyyy, cb0[1].xyzw
mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
add o0.xyzw, r0.xyzw, cb0[3].xyzw
mov o1.xy, v1.xyxx
mov o1.zw, v3.xxxy
mov o2.xyzw, v2.xyzw
ret 
// Approximately 0 instruction slots used


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

... and my response:

So, first off, let's cover one of the basic things, which is the input/output declarations that comes right after the "void main(" line

float2 v0 : POSITION0,
float2 v1 : TEXCOORD0,
float4 v2 : COLOR0,
float2 v3 : TEXCOORD1,
out float4 o0 : SV_POSITION0,
out float2 o1 : TEXCOORD0,
out float2 p1 : TEXCOORD1,
out float4 o2 : COLOR0

In pretty much all PS and VS shaders, inputs begin with the letter 'v' and outputs begin with the letter 'o' (or, as in this case, sometimes 'p' as well, but I've really never come across a case where I've had to modify those ones). Inputs are usually coordinates or data that has been passed into this shader for it's use to calculate something, and outputs are what that shader has calculated and are passing out to the next stage in the rendering pipeline.

Often, there will be more than one type of input and/or output, and they have different properties of what kind of data they hold. This shader uses most of the common types: SV_POSITION0 contains the position of the vertex on the screen, TEXCOORD# will contain data on the texture(s) that is being mapped/layered into the vertex, and COLOR#, as you've probably guessed, contains some colour data (although that we usually don't need to bother with, since we're really only ever concerned with correcting the positioning of things for stereo purposes).

In this case, it's a vertex shader, so this shader is responsible for calculating where all of vertexes related to this particular shader will appear on the screen. If this was for an actual 3D model of some sort, the inputs would be carrying data related to where the object exists in the game world, and the vertex shader would be converting that into which pixels on the screen they will appear on. In this case, though, it's a HUD shader so it's a bit more simplified, so the inputs are probably just static values, which is why there is not a lot of calculations going on, and you see a lot of "output = input" stuff happening.

Generally speaking, when we want to stereoize something that is not in stereo at all, we will want to add some amount of separation value to the position of the vertex (which, if we've been paying attention, in this shader is o0 because it was declared as SV_POSITION).

If the HUD items are completely 2D, then they have 0 depth (or 0 separation) and are considered at the front of the screen. Think of "1 separation" representing the entire range of depth all the way to "infinity", or otherwise the back of the screen. So if we added the full value of separation to the output, it would shift from being at the very front of the screen to the very back of the screen now. That usually would not be ideal, so instead we might add only 0.5 separation (half), which would effectively put the HUD items at 50% depth, or halfway into the screen, or maybe just 0.25 at 25%, or 0.10 for 10%, etc. (or even better, we might set up a button that will change the values on the fly, but that's a little more advanced and we'll get to that later).

So now that I've covered all the theory, here's how you actually DO it. Thankfully, since this is a DX11 game, it's a lot easier since HLSL code is a lot easier to work with than ASM code.

On line 32, we see the output position gets it's value's calculated: o0.xyzw = cb0[3].xyzw + r0.xyzw;
Since we simply want to add some separation to it (we're not actually "fixing" something that's broken), that's all perfectly fine. At any point after line 32, but before line 36 (the "return;" function, which is essentially the equivalent of "the end" in code) we need to add our separation.

Now, another quick thing to note is that o0 has 4 different values:
x - the coordinate along the horizontal left/right axis
y - the coordinate along the vertical up/down axis
z - the coordinate along the depth in/out of the screen axis
w - kinda the same thing as z, but during certain calculations

Now, we might think that we would want to be adjusting the z or w axis in order to push things further back, but that's not correct. We are almost always adjusting the x coordinate instead (because separation is a left/right thing). So, in this case, we want to add some separation to the o0.x coordinate.

Now, before we can add separation, we first need to create a variable (aka a container) that will contain the value of separation. Fortunately, 3DMigoto automatically provides us with a value for separation (and even convergence too, but that's not necessary for this operation), we simply need to 'declare' that we are going to use it. To declare a variable, at any point inside the main operation (between lines 25 and 36), we need to add the following line of code:

float4 stereo = StereoParams.Load(0);

"stereo" now holds 4 different pieces of data, although you will probably only ever be concerned with 2 of them:

stereo.x - contains separation
stereo.y - contains convergence

Some people, like bo3b, encourage taking an additional step for the sake of clarity, which would be adding the next 2 lines of code:

float separation = stereo.x;
float convergence = stereo.y;

So now you actually have 2 variables named "separation" and "convergence" instead of having to use "stereo.x" and "stereo.y" which might be confusing for you. I, personally, don't bother with this extra step, so if you ever review the code I use in my fixes, you'll see I simply use "stereo.x" for separation. I'll provide you with both examples in just a moment.

Ok, so now we've declared our variable for separation, we are now able to add that to the output. So, again, any time after line 32, but before line 36, we will do this. For the sake of simplicity we will be adding 0.5 separation to push it 50% into the screen, which should be a noticeable enough shift to visibly see, without being too far in that it will probably be too far. We do this with the following line of code:

o0.x = o0.x + separation * 0.5;

or

o0.x = o0.x + stereo.x * 0.5;

In actuality, though, this can be simplified a little bit to be:

o0.x += separation * 0.5;

or

o0.x += stereo.x * 0.5;

Because the += operator means to add whatever this value is to the existing value. There are equivalent operators -= *= and even /=.

So, finally, lets round this all up. Here are 2 perfectly equivalent ways of doing this:

Easy to understand way:


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
cbuffer cb0 : register(b0)
{
  float4 cb0[4];
}








// 3Dmigoto declarations
#define cmp -
Texture1D IniParams : register(t120);
Texture2D StereoParams : register(t125);




void main( 
  float2 v0 : POSITION0,
  float2 v1 : TEXCOORD0,
  float4 v2 : COLOR0,
  float2 v3 : TEXCOORD1,
  out float4 o0 : SV_POSITION0,
  out float2 o1 : TEXCOORD0,
  out float2 p1 : TEXCOORD1,
  out float4 o2 : COLOR0)
{
  float4 r0;
  uint4 bitmask, uiDest;
  float4 fDest;


  r0.xyzw = cb0[1].xyzw * v0.yyyy;
  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  o0.xyzw = cb0[3].xyzw + r0.xyzw;
  o1.xy = v1.xy;
  p1.xy = v3.xy;
  o2.xyzw = v2.xyzw;


float4 stereo = StereoParams.Load(0);
float separation = stereo.x;
float convergence = stereo.y;


o0.x = o0.x + separation * 0.5;


  return;
}


/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//
// Generated by Microsoft (R) D3D Shader Disassembler
//
//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
//
//
// Input signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// POSITION                 0   xy          0     NONE   float   xy  
// TEXCOORD                 0   xy          1     NONE   float   xy  
// COLOR                    0   xyzw        2     NONE   float   xyzw
// TEXCOORD                 1   xy          3     NONE   float   xy  
//
//
// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION              0   xyzw        0      POS   float   xyzw
// TEXCOORD                 0   xy          1     NONE   float   xy  
// TEXCOORD                 1     zw        1     NONE   float     zw
// COLOR                    0   xyzw        2     NONE   float   xyzw
//
vs_5_0
dcl_globalFlags refactoringAllowed
dcl_constantbuffer cb0[4], immediateIndexed
dcl_input v0.xy
dcl_input v1.xy
dcl_input v2.xyzw
dcl_input v3.xy
dcl_output_siv o0.xyzw, position
dcl_output o1.xy
dcl_output o1.zw
dcl_output o2.xyzw
dcl_temps 1
mul r0.xyzw, v0.yyyy, cb0[1].xyzw
mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
add o0.xyzw, r0.xyzw, cb0[3].xyzw
mov o1.xy, v1.xyxx
mov o1.zw, v3.xxxy
mov o2.xyzw, v2.xyzw
ret 
// Approximately 0 instruction slots used


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

and here's how my code might look like:


// ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
cbuffer cb0 : register(b0)
{
  float4 cb0[4];
}








// 3Dmigoto declarations
#define cmp -
Texture1D IniParams : register(t120);
Texture2D StereoParams : register(t125);




void main( 
  float2 v0 : POSITION0,
  float2 v1 : TEXCOORD0,
  float4 v2 : COLOR0,
  float2 v3 : TEXCOORD1,
  out float4 o0 : SV_POSITION0,
  out float2 o1 : TEXCOORD0,
  out float2 p1 : TEXCOORD1,
  out float4 o2 : COLOR0)
{
  float4 r0;
  uint4 bitmask, uiDest;
  float4 fDest;


  r0.xyzw = cb0[1].xyzw * v0.yyyy;
  r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  o0.xyzw = cb0[3].xyzw + r0.xyzw;


float4 stereo = StereoParams.Load(0);
o0.x += stereo.x * 0.5;


  o1.xy = v1.xy;
  p1.xy = v3.xy;
  o2.xyzw = v2.xyzw;
  return;
}


/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//
// Generated by Microsoft (R) D3D Shader Disassembler
//
//   using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
//
//
// Input signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// POSITION                 0   xy          0     NONE   float   xy  
// TEXCOORD                 0   xy          1     NONE   float   xy  
// COLOR                    0   xyzw        2     NONE   float   xyzw
// TEXCOORD                 1   xy          3     NONE   float   xy  
//
//
// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION              0   xyzw        0      POS   float   xyzw
// TEXCOORD                 0   xy          1     NONE   float   xy  
// TEXCOORD                 1     zw        1     NONE   float     zw
// COLOR                    0   xyzw        2     NONE   float   xyzw
//
vs_5_0
dcl_globalFlags refactoringAllowed
dcl_constantbuffer cb0[4], immediateIndexed
dcl_input v0.xy
dcl_input v1.xy
dcl_input v2.xyzw
dcl_input v3.xy
dcl_output_siv o0.xyzw, position
dcl_output o1.xy
dcl_output o1.zw
dcl_output o2.xyzw
dcl_temps 1
mul r0.xyzw, v0.yyyy, cb0[1].xyzw
mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
add o0.xyzw, r0.xyzw, cb0[3].xyzw
mov o1.xy, v1.xyxx
mov o1.zw, v3.xxxy
mov o2.xyzw, v2.xyzw
ret 
// Approximately 0 instruction slots used


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/

And that should effectively teach you everything you need to know to add a static value to the HUD. Let me know if that works (and maybe play around with the amount of separation to find what is a good amount), and if you understand all that then we can discuss adding a button to cycle values, or other types of fixes.

... and some Q&A related to frame analysis:

Q: I might need to pick your brain a little bit over that frame analysis stuff. I got it going and did a dump, but I am not sure how to identify which shader it is

A: Each file that gets dumped will have have the shaders that are used in the filename, for example:

000122-o0=!MU!=957b6ea2-vs=b229b3faa24a4aac-ps=ef7eaa7c9cab3eb3.jps

can be broken down as

000122 - the draw call number

o0 - which output from the vertex shader that this draw call is pertaining to. Sometimes it will be o1, o2, etc

!MU! - flags that may indicate an error in the texture hash to follow. Not important until you start looking at doing advanced things like texture filtering.

957b6ea2 - texture hash

vs=b229b3faa24a4aac - vertex shader b229b3faa24a4aac is being used

ps=ef7eaa7c9cab3eb3 - pixel shader ef7eaa7c9cab3eb3 is being used

Q: Is there a option to dump all shaders?

A: Yes, although I'm not sure if you are referring to dump EVERY shader a game uses, or simply all the shaders currently being used in a scene. There are ways for each.

To dump every shader a game uses, change the following options in the d3dx.ini file to:

export_fixed=1
export_shaders=1
export_hlsl=2
dump_usage=1

Then all the shaders will be dumped into a folder called ShaderCache when you first launch it (which might take a minute or two). Some games, however, will only dump shaders as they are generated, rather than everything all at once, which is a pain.

To dump all the shaders in a scene, well, there is not a built in method to do that, but you can manually just cycle through every shader and dump each one. Of course that is tedious, so what I suggest is to create a macro of some sort (I use Razer Synapse since I have a Razer Naga mouse, but I could also have used MadCatz software for my keyboard, or Logitech software, or even AutoHotkey) that simply alternates between numpad 2 and 3 to dump all pixel shaders, and numpad 5 and 6 to dump all vertex shaders. Just be sure to make note of the timestamp on the files generated to know which ones you've dumped, or simply remove all other files out of the ShaderFixes folder temporarily before you start dumping.

Q: I tried to do so manually but it seems I can never successfully hunt down the shadow in particular it's like it doesn't exists.

A: Yeah, dumping manually for certain types of effects is next to impossible. Back in DX9, shadows used to not be so bad, but in DX11 I'm finding more and more that they are not obvious to dump (hence why I now rely on frame analysis instead). One thing that might help is to try changing the option 'marking_mode' to 'pink' which causes the selected shader to appear as pink, or sometimes disappear (and in some cases will make things that didn't seem to be affected by 'skip' to react).

16 comments:

neovadApril 28, 2017 at 9:26 AM
Thanks a lot. Unfortunately searching in Nvidia forum is awfull. It's hard to find out how to fix issues especially if English is not your native language. It seems to me if somebody would made simple instructions with screenshot with issue and how to fix it. For example: in first case shown in this screenshot we should fix shadow shader from vertex shader via that method but in second case geometry of shadow is right but we see holes from objects situated between camera and shadow so we should fix pixel shader by those method.
HelifaxApril 30, 2017 at 7:52 AM
Awesome info right there!
Big Big thank you for collecting all of this here!
neovadApril 30, 2017 at 8:09 AM
Today I've fixed "Mordeheim City of the Damned" via DarkStarSword scripts without any knowledge of linux, python, bash, cygwin (and almost completelly broke my brain). Is somebody need instructions how to do Unity 5.3 DX11 fixes like mine because I've spent almost day on reverse engineering of bash and python scripts to find out which *.exe, *.py, *.sh and where to put them?
NecropantsMay 1, 2017 at 2:46 AM
Wow, this looks like a lot of work to compile, I'll be definitely trying to work through it. One of the reasons I tend to get frustrated and give up is how difficult it is too typically find the answer in the forums you are looking for. Very sad that both you and 4everawake are departing. Tekken 7, my favourite game franchise is comming out on pc (unreal4) in a month and I'll be heart broken if I cannot play it in 3d, which is likely as it seems you were the only one really fixing fighters. Best of luck and thanks for contributions and stuff.
cicicletaMay 1, 2017 at 1:40 PM
Wow, thanks a lot!!!!!
Nephilim JonesMay 3, 2017 at 11:49 PM
I am trying to find out how to fix Shadows in Black Desert.. the posts here help but I still need to do more reading.. Can you shed any light DJ_RK on How you fixed shadows in Elite Dangerous ..?
Nephilim JonesMay 4, 2017 at 2:46 PM
https://forums.geforce.com/default/topic/766890/3d-vision/bo3bs-school-for-shaderhackers/post/4715553/#4715553

that thread there seems to be the key but I will need to go over it again and again.. the other links for for HelixMod which I dont understand too much but this code DSS said made more sence I will try and sort out my issues but I am unsure If I can get it too work or not..
Nephilim JonesMay 4, 2017 at 4:37 PM
Well Like Bo3b said it never hurts to go back over the lessons and the whitepapers.. I have a good understanding on what I able to fix so far and have learned tons..
I would love to be able to fix the shadows in BlackDesert as that adds so much to that already beautiful game..

I will go over the whitepapers and do some more research thnx so far.. If you can try and visit my Black Desert thread at GeForce Forums ;)

link bar

Friday, April 28, 2017

Collection of bookmarked posts related to shader hacking

3DMigoto Specific

HelixMod Specific

General / Theory / Formulas

Fix examples

... and my response:

16 comments: