pythoncom and pywintypes module import issue

[marunguy]

• test environment

  • windows 10 64bit
  • python 3.8.10 64bit
  • py2exe 0.13.0.0
  • pywin32 305

• When import pywintypes, pythoncom, I have same problem with Inconsistent builds: ExtensionDLL copying order #149
  AttributeError: module 'pywintypes' has no attribute '__import_pywin32_system_module__'

• My Workaround

  • change bundle_files to 2
  • change data_files as below

import pythoncom
import pywintypes
...
    py2exe.freeze(
        ...
        data_files=[
            ("", [pythoncom.__file__]),
            ("", [pywintypes.__file__]),
        ],
        ...
    }

• But, I want to use with bundle_files=0.

• Is there any way to fix the hook_pythoncom and hook_pywintypes?

• Below is the result of my analyze.

• execution result of frozen main.py

D:\test\dist_py2exe>test.exe
main: start
main: frozen=True
main: __import_pywin32_system_module__=False

• Because pythoncom38.dll and pywintypes38.dll cannot be found in sys.path, ImportError is raised at line 44 at 'import_pywin32_system_module' in pywintypes.py
• But, it is swallowed by py2exe.hooks.hook_pywintypes and the remaining code in __import_pywin32_system_module__ is not executed.

            raise ImportError(
                "Module '%s' isn't in frozen sys.path %s" % (modname, sys.path)
            )

• uncomment del py2exe.hooks.hook_pywintypes in main.py
• ImportError is raised correctly.

D:\test\dist_py2exe>test.exe
Traceback (most recent call last):
  File "zipextimporter.pyc", line 96, in load_module
  File "<frozen zipimport>", line 259, in load_module
  File "pywintypes.pyc", line 123, in <module>
  File "pywintypes.pyc", line 44, in __import_pywin32_system_module__
ImportError: Module 'pywintypes' isn't in frozen sys.path ['D:\\test\\dist_py2exe\\test.exe']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 5, in <module>
  File "zipextimporter.pyc", line 131, in load_module
zipimport.ZipImportError: can't find module pywintypes

• change data_files in dist_py2exe.py

import pythoncom
import pywintypes
...
    py2exe.freeze(
        ...
        data_files=[
            ("", [pythoncom.__file__]),
            ("", [pywintypes.__file__]),
        ],
        ...
    }

D:\test\dist_py2exe>dir
2022-12-20  15:31           717,824 pythoncom38.dll
2022-12-20  15:31           140,800 pywintypes38.dll
2022-12-20  16:33         9,409,558 test.exe

D:\test\dist_py2exe>test.exe
Traceback (most recent call last):
  File "zipextimporter.pyc", line 96, in load_module
  File "<frozen zipimport>", line 259, in load_module
  File "pywintypes.pyc", line 123, in <module>
  File "pywintypes.pyc", line 113, in __import_pywin32_system_module__
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing pywintypes: 지정된 모듈을 찾을 수 없습니다.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
  File "zipextimporter.pyc", line 131, in load_module
zipimport.ZipImportError: can't find module pywintypes

• pythoncom38.dll and pywintypes38.dll depend on python38.dll
• change bundle_files from 0 to 2 in dist_py2exe.py

D:\test\dist_py2exe>dir
2021-05-03  11:54         3,406,016 libcrypto-1_1.dll
2021-05-03  11:54            32,792 libffi-7.dll
2021-05-03  11:54           690,368 libssl-1_1.dll
2021-05-03  11:54         4,211,376 python38.dll
2022-12-20  15:31           717,824 pythoncom38.dll
2022-12-20  15:31           140,800 pywintypes38.dll
2022-12-20  16:41         3,630,883 test.exe

• Good: __import_pywin32_system_module__=True

D:\test\dist_py2exe>test.exe
main: start
main: frozen=True
main: __import_pywin32_system_module__=True

• dist_py2exe.py source

from typing import List

import py2exe
import py2exe.hooks

excludes: List[str] = [
    '_gtkagg', '_tkagg', 'bsddb', 'curses', 'pywin.debugger', 'pywin.debugger.dbgcon',
    'pywin.dialogs', 'tcl', 'tkinter'
]

target_1 = {
    "script": "main.py",
    "icon_resources": [],
    "bitmap_resources": [],
    "other_resources": [],
    "dest_base": "test",
}

def main() -> None:
    # del py2exe.hooks.hook_pywintypes
    # del py2exe.hooks.hook_pythoncom

    py2exe.freeze(
        console=[target_1],
        data_files=None,
        zipfile=None,
        options={
            "includes": [],
            "excludes": excludes,
            "packages": [],
            "dll_excludes": [],
            "dist_dir": "dist_py2exe",
            "compressed": 1,
            "optimize": 1,
            "bundle_files": 0,
            "verbose": 4,
        },
    )

if __name__ == "__main__":
    main()

• main.py source

import sys

#import pythoncom
import pywintypes

def main() -> None:
    print("main: start")
    print(f"main: frozen={hasattr(sys, 'frozen')}")
    print(f"main: __import_pywin32_system_module__={hasattr(pywintypes, '__import_pywin32_system_module__')}")

if __name__ == "__main__":
    main()

• pywintypes.py source

# Magic utility that "redirects" to pywintypesxx.dll
import importlib.util, importlib.machinery, sys, os


def __import_pywin32_system_module__(modname, globs):
    # This has been through a number of iterations.  The problem: how to
    # locate pywintypesXX.dll when it may be in a number of places, and how
    # to avoid ever loading it twice.  This problem is compounded by the
    # fact that the "right" way to do this requires win32api, but this
    # itself requires pywintypesXX.
    # And the killer problem is that someone may have done 'import win32api'
    # before this code is called.  In that case Windows will have already
    # loaded pywintypesXX as part of loading win32api - but by the time
    # we get here, we may locate a different one.  This appears to work, but
    # then starts raising bizarre TypeErrors complaining that something
    # is not a pywintypes type when it clearly is!

    # So in what we hope is the last major iteration of this, we now
    # rely on a _win32sysloader module, implemented in C but not relying
    # on pywintypesXX.dll.  It then can check if the DLL we are looking for
    # lib is already loaded.
    # See if this is a debug build.
    suffix = "_d" if "_d.pyd" in importlib.machinery.EXTENSION_SUFFIXES else ""
    filename = "%s%d%d%s.dll" % (
        modname,
        sys.version_info[0],
        sys.version_info[1],
        suffix,
    )
    if hasattr(sys, "frozen"):
        # If we are running from a frozen program (py2exe, McMillan, freeze)
        # then we try and load the DLL from our sys.path
        # XXX - This path may also benefit from _win32sysloader?  However,
        # MarkH has never seen the DLL load problem with py2exe programs...
        for look in sys.path:
            # If the sys.path entry is a (presumably) .zip file, use the
            # directory
            if os.path.isfile(look):
                look = os.path.dirname(look)
            found = os.path.join(look, filename)
            if os.path.isfile(found):
                break
        else:
            raise ImportError(
                "Module '%s' isn't in frozen sys.path %s" % (modname, sys.path)
            )
    else:
        # First see if it already in our process - if so, we must use that.
        import _win32sysloader

        found = _win32sysloader.GetModuleFilename(filename)
        if found is None:
            # We ask Windows to load it next.  This is in an attempt to
            # get the exact same module loaded should pywintypes be imported
            # first (which is how we are here) or if, eg, win32api was imported
            # first thereby implicitly loading the DLL.

            # Sadly though, it doesn't quite work - if pywintypesxx.dll
            # is in system32 *and* the executable's directory, on XP SP2, an
            # import of win32api will cause Windows to load pywintypes
            # from system32, where LoadLibrary for that name will
            # load the one in the exe's dir.
            # That shouldn't really matter though, so long as we only ever
            # get one loaded.
            found = _win32sysloader.LoadModule(filename)
        if found is None:
            # Windows can't find it - which although isn't relevent here,
            # means that we *must* be the first win32 import, as an attempt
            # to import win32api etc would fail when Windows attempts to
            # locate the DLL.
            # This is most likely to happen for "non-admin" installs, where
            # we can't put the files anywhere else on the global path.

            # If there is a version in our Python directory, use that
            if os.path.isfile(os.path.join(sys.prefix, filename)):
                found = os.path.join(sys.prefix, filename)
        if found is None:
            # Not in the Python directory?  Maybe we were installed via
            # easy_install...
            if os.path.isfile(os.path.join(os.path.dirname(__file__), filename)):
                found = os.path.join(os.path.dirname(__file__), filename)

        # There are 2 site-packages directories - one "global" and one "user".
        # We could be in either, or both (but with different versions!). Factors include
        # virtualenvs, post-install script being run or not, `setup.py install` flags, etc.

        # In a worst-case, it means, say 'python -c "import win32api"'
        # will not work but 'python -c "import pywintypes, win32api"' will,
        # but it's better than nothing.

        # We use the same logic as pywin32_bootstrap to find potential location for the dll
        # Simply import pywin32_system32 and look in the paths in pywin32_system32.__path__

        if found is None:
            import pywin32_system32

            for path in pywin32_system32.__path__:
                maybe = os.path.join(path, filename)
                if os.path.isfile(maybe):
                    found = maybe
                    break

        if found is None:
            # give up in disgust.
            raise ImportError("No system module '%s' (%s)" % (modname, filename))
    # After importing the module, sys.modules is updated to the DLL we just
    # loaded - which isn't what we want. So we update sys.modules to refer to
    # this module, and update our globals from it.
    old_mod = sys.modules[modname]
    # Load the DLL.
    loader = importlib.machinery.ExtensionFileLoader(modname, found)
    spec = importlib.machinery.ModuleSpec(name=modname, loader=loader, origin=found)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)

    # Check the sys.modules[] behaviour we describe above is true...
    assert sys.modules[modname] is mod
    # as above - re-reset to the *old* module object then update globs.
    sys.modules[modname] = old_mod
    globs.update(mod.__dict__)


__import_pywin32_system_module__("pywintypes", globals())

[albertosottile]

Thanks for this detailed investigation. In general, support for bundle_files <=2 is limited to packages in the Python standard library, see:

py2exe/docs/py2exe.freeze.md

Line 69 in d34da51

- <b>``bundle_files <=2``</b>: these values are supported only for packages in the Python standard library. Issues occurring with external packages and lower values of `bundle_files` will not be investigated.

)

because I simply do not have the capacity to provide support for the whole Python ecosystem.

That being said, you are more than welcome to submit a PR with patches for py2exe.hooks that are suitable for your needs.

[marunguy]

@albertosottile Thank you for your reply.

The frozen exe with bundle_files=0,1,2 (test.exe in this case) include both pythoncom38.dll and pywintypes38.dll.
(When the frozen exe is extracted using 7zip, there are pythoncom38.dll and pywintypes38.dll.)

Although the extension of pythoncom38.dll and pywintypes38.dll is .dll, these are Python extension modules.

How to import pythoncom38.dll and pywintypes38.dll as module dynamically from the frozen exe(test.exe) in runtime?

[albertosottile]

DLL loading should be done automatically when you import win32com from Python, assuming that everything works. If you are looking for a test for pywin32, you can borrow the one I use in CI: https://github.com/py2exe/py2exe/blob/master/tests/functional/pywin32_test/pywin32_test.py

[marunguy]

I'm looking for a way to import pythoncom38.dll and pywintypes38.dll directly from frozen exe in runtime.

pywintypes_dll_mod = import_from_frozen("pywintypes38.dll", ...)

[albertosottile]

I do not follow you, why would you want to load the DLL manually? The pywin32 package has to load the DLLs when needed. If you really want to do so, I'd inspect the code of pywin32 to check how the DLLs are loaded (I suspect ctypes is used for that).

EDIT: by looking at the code you pasted above, you should inspect _win32sysloader in the pywin32 code.

[marunguy]

Following is the reason to import pythoncom38.dll and pywintypes38.dll directly from frozen exe in runtime.

• pywintypes38.dll is imported using __import_pywin32_system_module__ function pywintypes.py
• pywintypes38.dll - pywintypes.py line 123

__import_pywin32_system_module__("pywintypes", globals())

• search pywintypes38.dll in sys.path: pywintypes.py line 35-42

        for look in sys.path:
            # If the sys.path entry is a (presumably) .zip file, use the
            # directory
            if os.path.isfile(look):
                look = os.path.dirname(look)
            found = os.path.join(look, filename)
            if os.path.isfile(found):
                break

• Because pywintypes38.dll not in sys.path, ImportError raised. : pywintypes.py line 44-45

raise ImportError(
                "Module '%s' isn't in frozen sys.path %s" % (modname, sys.path)

• But, ImportError exception is swallowd by py2exe.hooks.hook_pywintypes
• As a result, pywintypes module don't have __import_pywin32_system_module__ attribute.
• pythoncom38.dll - pythoncom.py line 4

pywintypes.__import_pywin32_system_module__("pythoncom", globals())

• Because pywintypes module don't have __import_pywin32_system_module__ attribute, AttributeError is raised.

AttributeError: module 'pywintypes' has no attribute '__import_pywin32_system_module__'

• If continue by copying pywintypes38.dll to sys.path, pywintypes38.dll is directly imported using absolute path - pywintypes.py line 110-114

    # Load the DLL.
    loader = importlib.machinery.ExtensionFileLoader(modname, found)
    spec = importlib.machinery.ModuleSpec(name=modname, loader=loader, origin=found)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)

• So, I decide to try to replace __import_pywin32_system_module__ function.
• And, It need a way to import pythoncom38.dll and pywintypes38.dll directly from frozen exe in runtime.

[albertosottile]

But, ImportError exception is swallowd by py2exe.hooks.hook_pywintypes

Why do you think the hook is swallowing this exception? I cannot see an except clause in the hook nor in its chain. Unfortunately, I am on holiday and so I cannot run this myself directly. Can you show me a stack trace that points exactly where this exception is caught?

And, It need a way to import pythoncom38.dll and pywintypes38.dll directly from frozen exe in runtime.

In general, the code you pasted here:

    # Load the DLL.
    loader = importlib.machinery.ExtensionFileLoader(modname, found)
    spec = importlib.machinery.ModuleSpec(name=modname, loader=loader, origin=found)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)

should also work from "frozen exe in runtime". This works OOTB when bundle_files==3 because the DLL is in the file system, and it automatically makes use of zipextimporter when bundle_files<=2.

To facilitate debugging, I would advise you to start with bundle_files=2 (so that you can manually inspect the zip file) and adjust found to point to the path of e.g. pythoncom38.dll inside the zip file (e.g. C:\Users\yourname\Documents\test\dist\library.zip\pythoncom38.dll).

[SeaHOH]

#174 2eae517 may work with bundle_files<=2.