d3d11: samplerstate hashing fix

[Velro]

there's a case where you can request two different sampler states and they end up resolving down to the same sampler state in d3d11 internally, causing DX_CHECK_REFCOUNT(sampler, 1); to trigger.

Namely if you do
BGFX_SAMPLER_MIN_ANISOTROPIC | BGFX_SAMPLER_MAG_ANISOTROPIC
and
BGFX_SAMPLER_MIN_ANISOTROPIC | BGFX_SAMPLER_MAG_ANISOTROPIC | BGFX_SAMPLER_MIP_POINT
it'll trigger that assert.

you can repro the assert with this patch to bump.cpp example

diff --git a/examples/06-bump/bump.cpp b/examples/06-bump/bump.cpp
index 31afa7200..d99022568 100644
--- a/examples/06-bump/bump.cpp
+++ b/examples/06-bump/bump.cpp
@@ -282,8 +282,11 @@ public:
                                                bgfx::setIndexBuffer(m_ibh);

                                                // Bind textures.
-                                               bgfx::setTexture(0, s_texColor,  m_textureColor);
-                                               bgfx::setTexture(1, s_texNormal, m_textureNormal);
+                                               const uint32_t samplerFlagsA = BGFX_SAMPLER_MIN_ANISOTROPIC | BGFX_SAMPLER_MAG_ANISOTROPIC;
+                                               const uint32_t samplerFlagsB = BGFX_SAMPLER_MIN_ANISOTROPIC | BGFX_SAMPLER_MAG_ANISOTROPIC | BGFX_SAMPLER_MIP_POINT;
+
+                                               bgfx::setTexture(0, s_texColor,  m_textureColor, samplerFlagsA);
+                                               bgfx::setTexture(1, s_texNormal, m_textureNormal, samplerFlagsB);

this PR fixes things so these two requests do end up hashing to the same value. I'm hashing more memory than before, there's probably a way to compact the D3D11_TEXTURE_ADDRESS_MODE's better I didn't explore it much.

[Velro]

sorry the git history is all weird I am super weak git-foo. If you have a quick tip on how to fix it I'll do it otherwise feel free to just yoink this and paste it

[bkaradzic]

Ok, I looked into your PR, but doesn't make sense why passing just addressing modes would fix this. I'll need to investigate.

[bkaradzic]

sorry the git history is all weird I am super weak git-foo.

Btw, if you syncing branches with latest, use rebase instead of merge.

[bkaradzic]

I'm trying to repro this but it's not triggering the error you have.

[Velro]

I just repro'd the assert on another machine with that patch against latest. Here's all the debug output up to hitting the assert if that gives any hints

bump_example_debug_output.txt

[Velro]

genie command
..\bx\tools\bin\windows\genie --with-windows="10.0" --with-examples --with-tools vs2019
no command-line args to bump.exe