MSVC & SDK dirs suddenly hidden from compiler.include_dirs

[kxrob]

#153 broke the pywin32 builds by incompatibly hiding the MSVC & SDK dirs from compiler_instance.include_dirs - and decent public access at all. Same for libs.

The problem happened at
https://github.com/mhammond/pywin32/blob/fc69a1ecc150a13433a7a71d61bac52a5f3df42b/setup.py#L695
/ mhammond/pywin32@fc69a1e) ( @zooba )
(And further problems would also occur later in _why_cant_build_extension() at least.)

The VC & SDK dirs are needed for inspection in complex projects.
At best in a compatible way again.

#153 uses (hidden) class vars compiler_instance.__class__.include_dirs which though are set only after instance.initialize() .

PR #172 (tested with pywin32) suggests a potential way of solution and tests - see further comments and commit comments there.

Without solution, strange application code / workarounds like this would be required:
https://github.com/mhammond/pywin32/blob/b52884ae5417fd2f857007ae1c74d4f753591d36/setup.py#L688
mhammond/pywin32@b52884a
mhammond/pywin32#1936

[jaraco]

Thanks kxrob for the report, and specifically for what's needed. I think I understand now.

I'm unsure why the initialization is deferred, but the include/lib dirs are dependent on the plat_name indicated by .initialize(plat_name), so it's necessary to initialize before accessing those dirs. That's no different than before.

I think I agree that a compiler should expose the include/lib dirs as they would resolve when compiling, rather than hiding them in _fix_compiler_args. And maybe they shouldn't be stored in class attributes, but that's an implementation detail if we can expose the full set through an API.

[zooba]

I think I agree that a compiler should expose the include/lib dirs as they would resolve when compiling, rather than hiding them in _fix_compiler_args.

Are you going to do the same for gcc? Because I bet you're not ;)

Hiding the default arguments is just trying to give both gcc and MSVC an equivalent API. If that API is "distutils knows what the default compiler paths will be", you'll likely need to do more work to extract those paths from gcc to provide them. Otherwise, you're just entrenching different APIs for different platforms.

The pywin32 need for this part of the API was to work around the underlying issue, which is now fixed. With the fix in place, they don't need it anymore. (They were also previously using it to guess where certain other headers and libraries might be, but that logic is wrong and I already replaced it with a more robust option that doesn't rely on being able to read out the runtime include/lib dirs.)

[kxrob]

If that API is "distutils knows what the default compiler paths will be", you'll likely need to do more work to extract those paths from gcc to provide them.

I don't think its about knowing all that behind the curtain for abstract reasons - but simple knowing what the MSVC / gcc ... is fed with by distutils.
So that e.g. we can use the same paths compatibly for _build_scintilla() - which uses a makefile of a sub-project outside of distutils.

pywin32 need for this part of the API was to work around the underlying issue, which is now fixed. With the fix in place, they don't need it anymore.

Well, the _fixup_sdk_dirs() and its digging mechanism find_platform_sdk_dir() (competing with distutils and actually finding a different kit version on github CI) for the VC/SDK dirs can now be removed - but only when so far the above PR also peeks the hidden VC&SDK dirs out of distutils - which are still needed, see below.
These dirs should be better exposed "public".
And why not simply in the .include_dirs itself - complete ?

They were also previously using it to guess where certain other headers and libraries might be, but that logic is wrong and I already replaced it with a more robust option that doesn't rely on being able to read out the runtime include/lib dirs.

Well, actually your own (!) last changes there
https://github.com/mhammond/pywin32/blob/fc69a1ecc150a13433a7a71d61bac52a5f3df42b/setup.py#L695
/ mhammond/pywin32@fc69a1e) ( @zooba )

introduced, not removed the inspection of compiler.include_dirs .
And that is even exactly where the crash occurred first after the setuptools version stepped up on github CI.
(Besides that there is another use case in _why_cant_build_extension() at least, used for every extension - which wants to know beforehand via that inspection which of the extension builds need to be skipped when certain special libs are not present. )

And there may be many other reasons for other users to simply know "how the cow is fed" . Why hide important things?
And also to replace dirs in special cases - e.g. the SDK dirs for choosing a preferred version. (The effect is achieved also by searching the alternative user provided dirs first (which is the case), but produces absurd and sometimes conflicting command lines - which we have currently on CI logs until the above PR)

Besides: You also introduced another maybe competing mechanism (vswhere) in find_visual_studio_file(). Don't know if that is meant to find ATLMFC dirs even outside of the VC dirs which distutils uses, or not? In the latter case (otherwise there could be library conflicts?) that would compete with the different mechanism in distutils._msvccompiler and might potentially find a different VC installation. That finder would probably also better replaced by using distutils defaults as a starting point for search in the same tree as "...VC\INCLUDE" or so? (Not so far in the above PR.)

different APIs for different platforms

There are many platform specific functions in standard Python - and needed. But generally trying to make it as similar as practically possible.
Here so far there is no extra API needed when e.g. simply all dirs are exposed.
Or as a second-best compiler.default_cmdline_include_dirs or so are exposed for all platforms, sometimes empty (but why?).
Or third-best compiler.default_msvc_cmdline_include_dirs (double why??).
Just like for example getcwd() returns a path with drive letter in front on Win, but we don't have os.ms_getcwd() ...

pywin32 of course needs to do Windows specific things - but still produce distutils / Python compatible extensions, wheels, ...

[zooba]

Well, actually your own (!) last changes there https://github.com/mhammond/pywin32/blob/fc69a1ecc150a13433a7a71d61bac52a5f3df42b/setup.py#L695 / mhammond/pywin32@fc69a1e) ( @zooba )

introduced, not removed the inspection of compiler.include_dirs . And that is even exactly where the crash occurred first after the setuptools version stepped up on github CI.

Yep. How do you think I knew where setuptools needed to be fixed, and also that the bit I added would stop working after it was fixed upstream? I'm not magic, I'm just trying to be both the cause and the solution! 😆

Besides: You also introduced another maybe competing mechanism (vswhere) in find_visual_studio_file(). Don't know if that is meant to find ATLMFC dirs even outside of the VC dirs which distutils uses, or not?

It's not competing - it's correct. The ATLMFC redistributable libraries are separate from UCRT and WinSDK, which is what vcvarsall will configure by default. You can't find them by looking in a fixed relative directory from the headers, though that did work for a number of versions of Visual Studio. It doesn't work any more.

vswhere is also the official way to find VS installs these days. The registry keys haven't been reliable since VS 2017. And yeah, it'll find different versions at times - a fundamental problem with distutils is that it oversimplifies the build process (a big reason why I wrote pymsbuild for my own Windows-only projects 😉). But most users need a simple build process that works the same across platforms - only a few of us really want to specialise for one target.

Here so far there is no extra API needed when e.g. simply all dirs are exposed.

Except UnixCCompiler won't expose the directories. It never even tries. So the subclass of CCompiler you get might or might not expose the directories you want, and you won't know without having special knowledge about it and about how it works.

Under the current change, it's far simpler. CCompiler does not expose the directories, and if you need them for some reason, you have to find them yourself (and if it matters, pass them to CCompiler to override its defaults, which is an already existing public API that behaves identically for all supported compilers).

[kxrob]

Yep. How do you think I knew where setuptools needed to be fixed, and also that the bit I added would stop working after it was fixed upstream? I'm not magic, I'm just trying to be both the cause and the solution! 😆

There doesn't seem to be a warning comment at that code position and/or hint for resolution, your mhammond/pywin32#1897 says: I haven't tried, but I believe the code in your setup.py will still work, and none of the asserts will trigger. So you may not notice anything at all ...
How should that be solved there best without getting the dirs out of distutils - using one of the distutil's spawn()'s to call nmake or so? (kind of public API?)
And in _why_cant_build_extension() ?

The ATLMFC redistributable libraries are separate from UCRT and WinSDK, which is what vcvarsall will configure by default.

But in case its compatible, are those dirs usually near the corresponding VC? Did not see anything else so far. And had no problem so far as long as the ATL stuff was clicked for installation.

vswhere is also the official way to find VS installs these days. The registry keys haven't been reliable since VS 2017. And yeah, it'll find different versions at times - a fundamental problem with distutils is ...

In the end distutils should do the "probably best" / offical base search, whatever it is, and pywin32 probably just follow without (1st priority) risking a completely incompatible dir for ATL. So I guess it should remain a goal to get rid of a completely independent search mechanism "from bottom up" - just do some search from a distutils provided start?

Except UnixCCompiler won't expose the directories.

It just inherits and exports the empty list - trivially. And in a similar use case as above: the extra makefile sub-project for gcc e.g. could be fed the same way.

you have to find them yourself

With an independent often incompatible search mechanism again? That would be back to square one.
Think it would still be better to hack peek into the "top secret" distutils private internals and fixup every few years ...

[jaraco]

I think I agree that a compiler should expose the include/lib dirs as they would resolve when compiling, rather than hiding them in _fix_compiler_args.

Are you going to do the same for gcc? Because I bet you're not ;)

You're correct. I was not explicitly planning to do the same, in part because it's not obvious to me what that would entail. It seems to me the difference here is that gcc infers the default libs for the platform but Windows requires them to be supplied explicitly. If so, then the path to convergence is for distutils to supply those defaults. I was not intending to extract those and present them for a downstream consumer.

What I do believe is valuable to convey is "what behavior does distutils add" or "how does distutils configure a compile operation", so a consumer could reasonably infer how an operation (compile, link) is going to behave on any platform.

[zooba]

What you probably want is to add a new setuptools feature to allow running an arbitrary command as if it's the compiler (same environment, basically). It's already inherited a number of helper methods for certain types of commands (static lib, preprocess, etc.), but adding a new one for any command would satisfy the need for building Scintilla, and could also be implemented consistently across all platforms.

Jason owns distutils now as far as I'm concerned, but I'd expect it to be deprecated across the board now with the intent to merge all the functionality into setuptools APIs over the next couple of years (and likely tidy some up along the way). That's why I say it's probably a setuptools feature.

[zooba]

But in case its compatible, are those dirs usually near the corresponding VC? Did not see anything else so far. And had no problem so far as long as the ATL stuff was clicked for installation.

One day soon you will have a problem, because I encountered the problem while cross-compiling for a platform that you don't support yet 😉 With the fixes I contributed (and the later fixes for the distutils change), it'll be trivial for you to support Windows ARM64 devices once you decide it's worthwhile (hopefully during CPython 3.11, which has an ARM64 installer). But I acknowledge that you likely haven't seen that issue before, even though I think that's due to luck rather than any intentional design on the part of the MSVC installer.

In the end distutils should do the "probably best" / offical base search, whatever it is, and pywin32 probably just follow without (1st priority) risking a completely incompatible dir for ATL. So I guess it should remain a goal to get rid of a completely independent search mechanism "from bottom up" - just do some search from a distutils provided start?

As I mentioned above, setuptools may choose to expose more information or access to the compilation environment. But distutils never did it on purpose (and never provided enough to be accurate, unless you got lucky, as you apparently have been), and it's unlikely to get new features to provide it now.

In this particular case, you'd only get a mismatch if the build was going to fail anyway, so nothing breaks that was already working. But the point is true that in more complex cases, there may be components installed in a newer VS instance that "shadow" the one that included MSVC and cause problems. It won't be ATLMFC though (which has a dependency on MSVC in VS).