Friday, April 28, 2017

Collection of bookmarked posts related to shader hacking

In light of the fact that I'm no longer able to dedicate any significant amount of time to working on fixes (and, unfortunately, I'm not the only recent 'retiree'), and that we seem to have several new eager and capable individuals looking to learn the tricks of the trade of shaderhacking, I want to be able to provide whatever support I could for the transition to this generation.

There's no denying the fact that in order to learn the fundamentals of shader hacking that there's a LOT to read, learn, and understand. I, nor anyone else can change that, but what I can do is help provide a listing of many useful posts that I had uncovered while I had lots of time on my hands. I spent hours scouring the two largest threads on the Geforce 3DVision forum  for this information, and now I'll share it all so that you don't have to!

Mind you, this is all supplementary reading for AFTER you've completed and understood the basics that are laid out and taught through bo3b's School for Shaderhacking. Even though it teaches the use of Helixmod on DX9 titles, and most newly release games are DX11 and require the use of 3DMigoto, which uses an entirely different programming language (HLSL, which is essentially the same as C++ in many regards), those lessons will provide the foundation on which all of this other information builds upon.

I've grouped and categorized the bookmarks to help with organization and navigation. But first I'll start with the holy grail; the one single post that probably taught me more than all the rest combined! (well, again, this was after learning and understanding the basics. Won't make any sense until then):

Fixing shadows, various transformation stages, and fixes for each

... and now on to the rest.

3DMigoto Specific


3DMigoto Releases & New Features Explained
3DM Release v1.1.34 - Texture and RenderTarget size filtering
3DM Release v1.2.4: Arbitrary resource copying features
3DM Release v1.2.22: Custom shader injecting
3DM Release v1.2.24: Hooking functionality
3DM Release v1.2.28: Custom resource creation/import, copy_description
3DM Release: v1.2.30: cmd_Decompiler and OM Blend override
3DM Release: v1.2.31: Better custom shader functionality with examples

Frame Analysis
Using frame analysis explained

Texture Filtering
Texture filtering preamble
Texture filtering Q&A's (see posts to follow as well)
More Texture filtering Q&A
Finding texture hashes
Texture tracking and explaining some functionality
Texture replacement example
My own explanation on finding texture hashes

Partner Shaders
Partner shader filtering and some Q&A's about texture filtering
More partner shader matching and Iteration function

HUD Fixes
More on FarCry 4 HUD using texture replacement/filtering
HUD fixing using render target size filtering or partner shaders
HUD fix Q&As
Advanced fix: DSS FC4 filtering HUD icons by texture color for custom separation. Dumping index/vertex buffers

Arbitrary Resource Copying
Resource copying - very indepth
Q&A on resource copying, lots of specifics

3D Crosshair / Auto-depth Custom Shader
Auto-depth/3D crosshair instructions
Auto-crosshair and texture filtering Q's
Auto-crosshair and texture filtering A's

Miscellaneous
Active depth target filtering for UI elements
Indepth using render target copying, texture injection, using Present to clear resources
Using Vertex buffer copying explained
Unity fullscreen alt-tab functionality
Some vertex fixing Q&A
No matching PS for VS in shaderusage.txt
Nvidia profile editing
ASM if conditions
Display constant buffer values via custom shader
Inverse-cs



HelixMod Specific


HelixMod Feature List
Texture filtering
LUA script for UE3   
Partner shader overrides
Modify render targets/surfaces
Creating mono/stereo surfaces. Using grep to getin matches from log.txt







General / Theory / Formulas



Z-buffering - Wikipedia
Fixing halos explained-bo3b
Theory:Objects to infinity, far clipping, W coordinate by DSS
PostProcessing effect adjustments
Fixing fixed effects being clipped via the VS
Theory: DSS fixes Batman CS shader lights
Tips for finding right shadow shaders
ASM: Calculating byte offsets of buffers
Explaining Homogeneous Coordinates; Projective Geometry — Tom Dalling
Introduction to Shaders - RB Whitaker's Wiki
VIDEO Tutorial 3D Vision Shadow Fix Tips & Tricks - Demonicon - YouTube
ASM inverse matrix formula
3d-fix/inverseMatrix.asm at master · mx-2/3d-fix · GitHub
3D HBAO+ Normal Map Artefact Fix - GeForce Forums
More shadows fixing theory
Projection Transform (Direct3D 9) (Windows)
Near and far clipping values from projection matrix
World space fix via frustum corners method
Stereo correction without projection matrix answers
UI Depth adjustment w/ texture isolating
ASM-cmp operator
Nvidia profiles codes
Canonical Stereo Code - Bo3b's School for Shaderhackers
Fixing shadows by Mike_ar69
DirectXTutorial.com
Unity fixes (Halos, specular, lighting) explained by DSS
Fixing GodRays using screen postprocessing-DSS
Moon depth adjustment
ASM-Ways to disable a shader
UE3 shadows fixing pattern
DSS Shadertool.py instructions
More shadertool.py instructions
Depth buffer filtering for driver heuristics issue
Diff Checker - Online diff tool to compare text to find the difference between two text files
Lots of background info to study


Fix examples



Reflections fix
FCPrimal: Fix tile lighting clipping · DarkStarSword/3d-fixes@135aeed · GitHub
FCPrimal: Fix Ambient Occlusion · DarkStarSword/3d-fixes@00be9c1 · GitHub
Far Cry Primal: Fix reflections via stereo2mono technique · DarkStarSword/3d-fixes@594d528 · GitHub
Far Cry Primal: Fix Lens Flares · DarkStarSword/3d-fixes@0f09870 · GitHub
FCPrimal: Fix shadow volumes / light shafts (includes debugging shader) · DarkStarSword/3d-fixes@f72ca4c · GitHub
FCPrimal: Move bloom to infinity · DarkStarSword/3d-fixes@6b66f3a · GitHub
FCPrimal: Apply scripted volumetric fog fix · DarkStarSword/3d-fixes@53458ac · GitHub
Lichdom: Fix light shafts · DarkStarSword/3d-fixes@76af7b9 · GitHub
Lichdom: Accurate fix for CryEngine directional shadows!!! · DarkStarSword/3d-fixes@0680548 · GitHub
Lichdom: Fix specular highlights from moonlight on ground in tutorial · DarkStarSword/3d-fixes@fd10e5a · GitHub
Lichdom: Use render target filtering to avoid adjusting UI elements i… · DarkStarSword/3d-fixes@6ebe6e7 · GitHub
Lichdom: Fix reflections on 'very high' quality water · DarkStarSword/3d-fixes@4d3881b · GitHub
Mad Max: Fix environment reflections + specular highlights · DarkStarSword/3d-fixes@5e29434 · GitHub
Mad Max: Fix bloom/lens flare opacity being calculated from wrong spo… · DarkStarSword/3d-fixes@a0f71fc · GitHub
The Park: Fix UE4 ReflectionEnvironmentComputeShaders.ucf shader · DarkStarSword/3d-fixes@9d526e0 · GitHub
The Park: Move sun reflection to infinity · DarkStarSword/3d-fixes@5d74514 · GitHub
The Park: Fix light shafts · DarkStarSword/3d-fixes@62c0ec7 · GitHub
The Park: Fix UE4 tile lighting · DarkStarSword/3d-fixes@8ecb3ce · GitHub
The Park: Move screen space reflections to surface depth · DarkStarSword/3d-fixes@8b1bed8 · GitHub
The Witness: Adjust UI depth for puzzles · DarkStarSword/3d-fixes@0a29067 · GitHub
The Witness: Auto adjust the mouse cursor depth · DarkStarSword/3d-fixes@2b07c19 · GitHub
The Witness: Disable UI adjustment when no depth buffer is in use · DarkStarSword/3d-fixes@13867a8 · GitHub
The Witness: Substantial improvements to auto cursor for some puzzles · DarkStarSword/3d-fixes@e5f07f4 · GitHub
Eleusis: Fix river water refraction at shoreline · DarkStarSword/3d-fixes@7e65c89 · GitHub
ABZU: Move specular highlights to correct depth · Also world space adjustment example
ABZU: Fix deferred lighting - Easy world space adj example
           

And lastly, I provided a lengthy explanation in response to a PM I received from an inquiring soul regarding fixing HUDs, that goes over some basics in some easy to understand language. Since I can't provide a bookmark to the PMs, I'll copy and past the discussion here:

PM Received:
If you have time could you do me a favour and take a quick look at the shader below, and if you can point me in the right direction for some example code to stereoize it. It is just the HUD shader which id like to push into depth a little bit. I always learn best by comparing examples, so it would be a great help if you could spend a couple of minutes, I'd really appreciate it.                                                                                                   

  1. // ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  2. cbuffer cb0 : register(b0)
  3. {
  4. float4 cb0[4];
  5. }
  6. // 3Dmigoto declarations
  7. #define cmp -
  8. Texture1D IniParams : register(t120);
  9. Texture2D StereoParams : register(t125);
  10. void main(
  11. float2 v0 : POSITION0,
  12. float2 v1 : TEXCOORD0,
  13. float4 v2 : COLOR0,
  14. float2 v3 : TEXCOORD1,
  15. out float4 o0 : SV_POSITION0,
  16. out float2 o1 : TEXCOORD0,
  17. out float2 p1 : TEXCOORD1,
  18. out float4 o2 : COLOR0)
  19. {
  20. float4 r0;
  21. uint4 bitmask, uiDest;
  22. float4 fDest;
  23. r0.xyzw = cb0[1].xyzw * v0.yyyy;
  24. r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  25. o0.xyzw = cb0[3].xyzw + r0.xyzw;
  26. o1.xy = v1.xy;
  27. p1.xy = v3.xy;
  28. o2.xyzw = v2.xyzw;
  29. return;
  30. }
  31. /*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  32. //
  33. // Generated by Microsoft (R) D3D Shader Disassembler
  34. //
  35. // using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  36. //
  37. //
  38. // Input signature:
  39. //
  40. // Name Index Mask Register SysValue Format Used
  41. // -------------------- ----- ------ -------- -------- ------- ------
  42. // POSITION 0 xy 0 NONE float xy
  43. // TEXCOORD 0 xy 1 NONE float xy
  44. // COLOR 0 xyzw 2 NONE float xyzw
  45. // TEXCOORD 1 xy 3 NONE float xy
  46. //
  47. //
  48. // Output signature:
  49. //
  50. // Name Index Mask Register SysValue Format Used
  51. // -------------------- ----- ------ -------- -------- ------- ------
  52. // SV_POSITION 0 xyzw 0 POS float xyzw
  53. // TEXCOORD 0 xy 1 NONE float xy
  54. // TEXCOORD 1 zw 1 NONE float zw
  55. // COLOR 0 xyzw 2 NONE float xyzw
  56. //
  57. vs_5_0
  58. dcl_globalFlags refactoringAllowed
  59. dcl_constantbuffer cb0[4], immediateIndexed
  60. dcl_input v0.xy
  61. dcl_input v1.xy
  62. dcl_input v2.xyzw
  63. dcl_input v3.xy
  64. dcl_output_siv o0.xyzw, position
  65. dcl_output o1.xy
  66. dcl_output o1.zw
  67. dcl_output o2.xyzw
  68. dcl_temps 1
  69. mul r0.xyzw, v0.yyyy, cb0[1].xyzw
  70. mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
  71. add o0.xyzw, r0.xyzw, cb0[3].xyzw
  72. mov o1.xy, v1.xyxx
  73. mov o1.zw, v3.xxxy
  74. mov o2.xyzw, v2.xyzw
  75. ret
  76. // Approximately 0 instruction slots used
  77. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/








... and my response:


So, first off, let's cover one of the basic things, which is the input/output declarations that comes right after the "void main(" line

float2 v0 : POSITION0,
float2 v1 : TEXCOORD0,
float4 v2 : COLOR0,
float2 v3 : TEXCOORD1,
out float4 o0 : SV_POSITION0,
out float2 o1 : TEXCOORD0,
out float2 p1 : TEXCOORD1,
out float4 o2 : COLOR0

In pretty much all PS and VS shaders, inputs begin with the letter 'v' and outputs begin with the letter 'o' (or, as in this case, sometimes 'p' as well, but I've really never come across a case where I've had to modify those ones). Inputs are usually coordinates or data that has been passed into this shader for it's use to calculate something, and outputs are what that shader has calculated and are passing out to the next stage in the rendering pipeline.

Often, there will be more than one type of input and/or output, and they have different properties of what kind of data they hold. This shader uses most of the common types: SV_POSITION0 contains the position of the vertex on the screen, TEXCOORD# will contain data on the texture(s) that is being mapped/layered into the vertex, and COLOR#, as you've probably guessed, contains some colour data (although that we usually don't need to bother with, since we're really only ever concerned with correcting the positioning of things for stereo purposes).

In this case, it's a vertex shader, so this shader is responsible for calculating where all of vertexes related to this particular shader will appear on the screen. If this was for an actual 3D model of some sort, the inputs would be carrying data related to where the object exists in the game world, and the vertex shader would be converting that into which pixels on the screen they will appear on. In this case, though, it's a HUD shader so it's a bit more simplified, so the inputs are probably just static values, which is why there is not a lot of calculations going on, and you see a lot of "output = input" stuff happening.

Generally speaking, when we want to stereoize something that is not in stereo at all, we will want to add some amount of separation value to the position of the vertex (which, if we've been paying attention, in this shader is o0 because it was declared as SV_POSITION). 

If the HUD items are completely 2D, then they have 0 depth (or 0 separation) and are considered at the front of the screen. Think of "1 separation" representing the entire range of depth all the way to "infinity", or otherwise the back of the screen. So if we added the full value of separation to the output, it would shift from being at the very front of the screen to the very back of the screen now. That usually would not be ideal, so instead we might add only 0.5 separation (half), which would effectively put the HUD items at 50% depth, or halfway into the screen, or maybe just 0.25 at 25%, or 0.10 for 10%, etc. (or even better, we might set up a button that will change the values on the fly, but that's a little more advanced and we'll get to that later).

So now that I've covered all the theory, here's how you actually DO it. Thankfully, since this is a DX11 game, it's a lot easier since HLSL code is a lot easier to work with than ASM code.

On line 32, we see the output position gets it's value's calculated: o0.xyzw = cb0[3].xyzw + r0.xyzw;
Since we simply want to add some separation to it (we're not actually "fixing" something that's broken), that's all perfectly fine. At any point after line 32, but before line 36 (the "return;" function, which is essentially the equivalent of "the end" in code) we need to add our separation.

Now, another quick thing to note is that o0 has 4 different values:
x - the coordinate along the horizontal left/right axis
y - the coordinate along the vertical up/down axis
z - the coordinate along the depth in/out of the screen axis
w - kinda the same thing as z, but during certain calculations

Now, we might think that we would want to be adjusting the z or w axis in order to push things further back, but that's not correct. We are almost always adjusting the x coordinate instead (because separation is a left/right thing). So, in this case, we want to add some separation to the o0.x coordinate.

Now, before we can add separation, we first need to create a variable (aka a container) that will contain the value of separation. Fortunately, 3DMigoto automatically provides us with a value for separation (and even convergence too, but that's not necessary for this operation), we simply need to 'declare' that we are going to use it. To declare a variable, at any point inside the main operation (between lines 25 and 36), we need to add the following line of code:

float4 stereo = StereoParams.Load(0);

"stereo" now holds 4 different pieces of data, although you will probably only ever be concerned with 2 of them:

stereo.x - contains separation
stereo.y - contains convergence

Some people, like bo3b, encourage taking an additional step for the sake of clarity, which would be adding the next 2 lines of code:

float separation = stereo.x;
float convergence = stereo.y;

So now you actually have 2 variables named "separation" and "convergence" instead of having to use "stereo.x" and "stereo.y" which might be confusing for you. I, personally, don't bother with this extra step, so if you ever review the code I use in my fixes, you'll see I simply use "stereo.x" for separation. I'll provide you with both examples in just a moment.

Ok, so now we've declared our variable for separation, we are now able to add that to the output. So, again, any time after line 32, but before line 36, we will do this. For the sake of simplicity we will be adding 0.5 separation to push it 50% into the screen, which should be a noticeable enough shift to visibly see, without being too far in that it will probably be too far. We do this with the following line of code:

o0.x = o0.x + separation * 0.5;

or 

o0.x = o0.x + stereo.x * 0.5;


In actuality, though, this can be simplified a little bit to be:

o0.x += separation * 0.5;

or 

o0.x += stereo.x * 0.5;

Because the += operator means to add whatever this value is to the existing value. There are equivalent operators -= *= and even /=.


So, finally, lets round this all up. Here are 2 perfectly equivalent ways of doing this:

Easy to understand way:

  1. // ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  2. cbuffer cb0 : register(b0)
  3. {
  4. float4 cb0[4];
  5. }
  6. // 3Dmigoto declarations
  7. #define cmp -
  8. Texture1D IniParams : register(t120);
  9. Texture2D StereoParams : register(t125);
  10. void main(
  11. float2 v0 : POSITION0,
  12. float2 v1 : TEXCOORD0,
  13. float4 v2 : COLOR0,
  14. float2 v3 : TEXCOORD1,
  15. out float4 o0 : SV_POSITION0,
  16. out float2 o1 : TEXCOORD0,
  17. out float2 p1 : TEXCOORD1,
  18. out float4 o2 : COLOR0)
  19. {
  20. float4 r0;
  21. uint4 bitmask, uiDest;
  22. float4 fDest;
  23. r0.xyzw = cb0[1].xyzw * v0.yyyy;
  24. r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  25. o0.xyzw = cb0[3].xyzw + r0.xyzw;
  26. o1.xy = v1.xy;
  27. p1.xy = v3.xy;
  28. o2.xyzw = v2.xyzw;
  29. float4 stereo = StereoParams.Load(0);
  30. float separation = stereo.x;
  31. float convergence = stereo.y;
  32. o0.x = o0.x + separation * 0.5;
  33. return;
  34. }
  35. /*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  36. //
  37. // Generated by Microsoft (R) D3D Shader Disassembler
  38. //
  39. // using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  40. //
  41. //
  42. // Input signature:
  43. //
  44. // Name Index Mask Register SysValue Format Used
  45. // -------------------- ----- ------ -------- -------- ------- ------
  46. // POSITION 0 xy 0 NONE float xy
  47. // TEXCOORD 0 xy 1 NONE float xy
  48. // COLOR 0 xyzw 2 NONE float xyzw
  49. // TEXCOORD 1 xy 3 NONE float xy
  50. //
  51. //
  52. // Output signature:
  53. //
  54. // Name Index Mask Register SysValue Format Used
  55. // -------------------- ----- ------ -------- -------- ------- ------
  56. // SV_POSITION 0 xyzw 0 POS float xyzw
  57. // TEXCOORD 0 xy 1 NONE float xy
  58. // TEXCOORD 1 zw 1 NONE float zw
  59. // COLOR 0 xyzw 2 NONE float xyzw
  60. //
  61. vs_5_0
  62. dcl_globalFlags refactoringAllowed
  63. dcl_constantbuffer cb0[4], immediateIndexed
  64. dcl_input v0.xy
  65. dcl_input v1.xy
  66. dcl_input v2.xyzw
  67. dcl_input v3.xy
  68. dcl_output_siv o0.xyzw, position
  69. dcl_output o1.xy
  70. dcl_output o1.zw
  71. dcl_output o2.xyzw
  72. dcl_temps 1
  73. mul r0.xyzw, v0.yyyy, cb0[1].xyzw
  74. mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
  75. add o0.xyzw, r0.xyzw, cb0[3].xyzw
  76. mov o1.xy, v1.xyxx
  77. mov o1.zw, v3.xxxy
  78. mov o2.xyzw, v2.xyzw
  79. ret
  80. // Approximately 0 instruction slots used
  81. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/


and here's how my code might look like:
  1. // ---- Created with 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  2. cbuffer cb0 : register(b0)
  3. {
  4. float4 cb0[4];
  5. }
  6. // 3Dmigoto declarations
  7. #define cmp -
  8. Texture1D IniParams : register(t120);
  9. Texture2D StereoParams : register(t125);
  10. void main(
  11. float2 v0 : POSITION0,
  12. float2 v1 : TEXCOORD0,
  13. float4 v2 : COLOR0,
  14. float2 v3 : TEXCOORD1,
  15. out float4 o0 : SV_POSITION0,
  16. out float2 o1 : TEXCOORD0,
  17. out float2 p1 : TEXCOORD1,
  18. out float4 o2 : COLOR0)
  19. {
  20. float4 r0;
  21. uint4 bitmask, uiDest;
  22. float4 fDest;
  23. r0.xyzw = cb0[1].xyzw * v0.yyyy;
  24. r0.xyzw = v0.xxxx * cb0[0].xyzw + r0.xyzw;
  25. o0.xyzw = cb0[3].xyzw + r0.xyzw;
  26. float4 stereo = StereoParams.Load(0);
  27. o0.x += stereo.x * 0.5;
  28. o1.xy = v1.xy;
  29. p1.xy = v3.xy;
  30. o2.xyzw = v2.xyzw;
  31. return;
  32. }
  33. /*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  34. //
  35. // Generated by Microsoft (R) D3D Shader Disassembler
  36. //
  37. // using 3Dmigoto v1.2.49 on Sat Dec 03 21:48:30 2016
  38. //
  39. //
  40. // Input signature:
  41. //
  42. // Name Index Mask Register SysValue Format Used
  43. // -------------------- ----- ------ -------- -------- ------- ------
  44. // POSITION 0 xy 0 NONE float xy
  45. // TEXCOORD 0 xy 1 NONE float xy
  46. // COLOR 0 xyzw 2 NONE float xyzw
  47. // TEXCOORD 1 xy 3 NONE float xy
  48. //
  49. //
  50. // Output signature:
  51. //
  52. // Name Index Mask Register SysValue Format Used
  53. // -------------------- ----- ------ -------- -------- ------- ------
  54. // SV_POSITION 0 xyzw 0 POS float xyzw
  55. // TEXCOORD 0 xy 1 NONE float xy
  56. // TEXCOORD 1 zw 1 NONE float zw
  57. // COLOR 0 xyzw 2 NONE float xyzw
  58. //
  59. vs_5_0
  60. dcl_globalFlags refactoringAllowed
  61. dcl_constantbuffer cb0[4], immediateIndexed
  62. dcl_input v0.xy
  63. dcl_input v1.xy
  64. dcl_input v2.xyzw
  65. dcl_input v3.xy
  66. dcl_output_siv o0.xyzw, position
  67. dcl_output o1.xy
  68. dcl_output o1.zw
  69. dcl_output o2.xyzw
  70. dcl_temps 1
  71. mul r0.xyzw, v0.yyyy, cb0[1].xyzw
  72. mad r0.xyzw, v0.xxxx, cb0[0].xyzw, r0.xyzw
  73. add o0.xyzw, r0.xyzw, cb0[3].xyzw
  74. mov o1.xy, v1.xyxx
  75. mov o1.zw, v3.xxxy
  76. mov o2.xyzw, v2.xyzw
  77. ret
  78. // Approximately 0 instruction slots used
  79. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/


And that should effectively teach you everything you need to know to add a static value to the HUD. Let me know if that works (and maybe play around with the amount of separation to find what is a good amount), and if you understand all that then we can discuss adding a button to cycle values, or other types of fixes.




... and some Q&A related to frame analysis:


Q: I might need to pick your brain a little bit over that frame analysis stuff. I got it going and did a dump, but I am not sure how to identify which shader it is

A: Each file that gets dumped will have have the shaders that are used in the filename, for example: 

000122-o0=!MU!=957b6ea2-vs=b229b3faa24a4aac-ps=ef7eaa7c9cab3eb3.jps

can be broken down as

000122 - the draw call number

o0 - which output from the vertex shader that this draw call is pertaining to. Sometimes it will be o1, o2, etc

!MU! - flags that may indicate an error in the texture hash to follow. Not important until you start looking at doing advanced things like texture filtering.

957b6ea2 - texture hash

vs=b229b3faa24a4aac - vertex shader b229b3faa24a4aac is being used

ps=ef7eaa7c9cab3eb3 - pixel shader ef7eaa7c9cab3eb3 is being used




Q: Is there a option to dump all shaders?

A: Yes, although I'm not sure if you are referring to dump EVERY shader a game uses, or simply all the shaders currently being used in a scene. There are ways for each.

To dump every shader a game uses, change the following options in the d3dx.ini file to:

export_fixed=1
export_shaders=1
export_hlsl=2
dump_usage=1

Then all the shaders will be dumped into a folder called ShaderCache when you first launch it (which might take a minute or two). Some games, however, will only dump shaders as they are generated, rather than everything all at once, which is a pain.

To dump all the shaders in a scene, well, there is not a built in method to do that, but you can manually just cycle through every shader and dump each one. Of course that is tedious, so what I suggest is to create a macro of some sort (I use Razer Synapse since I have a Razer Naga mouse, but I could also have used MadCatz software for my keyboard, or Logitech software, or even AutoHotkey) that simply alternates between numpad 2 and 3 to dump all pixel shaders, and numpad 5 and 6 to dump all vertex shaders. Just be sure to make note of the timestamp on the files generated to know which ones you've dumped, or simply remove all other files out of the ShaderFixes folder temporarily before you start dumping.



Q: I tried to do so manually but it seems I can never successfully hunt down the shadow in particular it's like it doesn't exists.

A: Yeah, dumping manually for certain types of effects is next to impossible. Back in DX9, shadows used to not be so bad, but in DX11 I'm finding more and more that they are not obvious to dump (hence why I now rely on frame analysis instead). One thing that might help is to try changing the option 'marking_mode' to 'pink' which causes the selected shader to appear as pink, or sometimes disappear (and in some cases will make things that didn't seem to be affected by 'skip' to react).

15 comments:

  1. Thanks a lot. Unfortunately searching in Nvidia forum is awfull. It's hard to find out how to fix issues especially if English is not your native language. It seems to me if somebody would made simple instructions with screenshot with issue and how to fix it. For example: in first case shown in this screenshot we should fix shadow shader from vertex shader via that method but in second case geometry of shadow is right but we see holes from objects situated between camera and shadow so we should fix pixel shader by those method.

    ReplyDelete
    Replies
    1. Agreed, searching the forums is awful, which is why I actually started right from page 1 of both "bo3b's School for Shaderhackers" and "3DMigoto is now open source" threads, and read through them to their entirety to make this collection of the most useful posts.

      As for what you're suggesting, I've considered picking up where bo3b left off, and continuing providing tutorials and lessons, but time is short and it's not quite as easy as you'd think to prepare such material. Maybe at some point in the future I'll reconsider it, but for now I think the above links should be plenty to put people on the right path.

      Delete
  2. Awesome info right there!
    Big Big thank you for collecting all of this here!

    ReplyDelete
  3. Today I've fixed "Mordeheim City of the Damned" via DarkStarSword scripts without any knowledge of linux, python, bash, cygwin (and almost completelly broke my brain). Is somebody need instructions how to do Unity 5.3 DX11 fixes like mine because I've spent almost day on reverse engineering of bash and python scripts to find out which *.exe, *.py, *.sh and where to put them?

    ReplyDelete
    Replies
    1. Yeah, I remember being in the exact same boat, figuring all that out on my own... Googling, sorting through all sorts of unrelated stuff until I found what I was looking for, moving past that obstacle just to run smack dab right into another... and so on, and on.

      So if you want, feel free to do a post on the Geforce forum and I'll go ahead and add the link for that in the General section. Thanks for wanting to help out and contribute, that's definitely a not well covered area anywhere else.

      Delete
    2. Yes, that would be great if you'd put together a guide for how to use the DSS scripts. Our resident master of the Unity auto-fix scripts was 4everawake, and with him leaving, it would be terrific to have other people able to use the scripts effectively.

      It might make sense to go ahead and make a HelixModBlog post for that detail, because it will have long term value, and be easier to link against. I'm going to add DJ-RK's page here to the Guides section, and it would make sense to add your blog post too.

      Delete
    3. OK, I'll try to fix one more DX9 game in Unity 5.3 and make instructions

      Delete
  4. Wow, this looks like a lot of work to compile, I'll be definitely trying to work through it. One of the reasons I tend to get frustrated and give up is how difficult it is too typically find the answer in the forums you are looking for. Very sad that both you and 4everawake are departing. Tekken 7, my favourite game franchise is comming out on pc (unreal4) in a month and I'll be heart broken if I cannot play it in 3d, which is likely as it seems you were the only one really fixing fighters. Best of luck and thanks for contributions and stuff.

    ReplyDelete
    Replies
    1. It's not so much that I'm departing, I just don't have time to work on fixes like I used to, however I'll still do the occasional fix as time permits, and I'll do my best to find time for Tekken 7 (and possibly Injustice 2)! ;)

      Delete
  5. I am trying to find out how to fix Shadows in Black Desert.. the posts here help but I still need to do more reading.. Can you shed any light DJ_RK on How you fixed shadows in Elite Dangerous ..?

    ReplyDelete
  6. https://forums.geforce.com/default/topic/766890/3d-vision/bo3bs-school-for-shaderhackers/post/4715553/#4715553

    that thread there seems to be the key but I will need to go over it again and again.. the other links for for HelixMod which I dont understand too much but this code DSS said made more sence I will try and sort out my issues but I am unsure If I can get it too work or not..

    ReplyDelete
    Replies
    1. Yep, that was exactly the post I was going to suggest you commit to studying and understanding in order to learn to fix shadows. The shadows fix in Elite Dangerous is not a good example, because it's really a glorified "halo" fix. I actually spent a tremendous amount of time trying to fix shadows using the standard approach, only have a hilarious LOL when I decided to take a simpler approach and found it worked (which is hardly ever the case with shadows). Another resource that is extremely helpful (also from our resident genius DSS) is the video link provided above: https://www.youtube.com/watch?v=GssPDoVtQfA

      Delete
    2. ... but I also have to stress the fact that the post by DSS really only helps bring all the pieces together, by itself I don't think it will make total sense. In one of bo3b's lessons, he provides links to certain Nvidia whitepapers which talk about the different stages in the rendering pipeline, and the transformations they go through. At first when I read them they seemed totally abstract, but then DSS's response there helped me learn how that theory related to working with shaders, so again, I emphasize the importance of doing the prior reading and learning done in bo3b's lessons. You need to learn to walk before you can run! (not meaning YOU specifically, I'm not sure where you're basic understanding lies at this point)

      Delete
  7. Well Like Bo3b said it never hurts to go back over the lessons and the whitepapers.. I have a good understanding on what I able to fix so far and have learned tons..
    I would love to be able to fix the shadows in BlackDesert as that adds so much to that already beautiful game..

    I will go over the whitepapers and do some more research thnx so far.. If you can try and visit my Black Desert thread at GeForce Forums ;)

    ReplyDelete