os.stat() on Windows succeeds for nonexistent paths with trailing spaces

[laloch]

BPO	40238
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @gaborbernat, @laloch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-09.14:25:30.374>
labels = ['3.8', 'type-bug', 'library', '3.9', 'OS-windows']
title = 'os.stat() on Windows succeeds for nonexistent paths with trailing spaces'
updated_at = <Date 2020-04-10.13:57:20.426>
user = 'https://github.com/laloch'

bugs.python.org fields:

activity = <Date 2020-04-10.13:57:20.426>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2020-04-09.14:25:30.374>
creator = 'laloch'
dependencies = []
files = []
hgrepos = []
issue_num = 40238
keywords = []
message_count = 6.0
messages = ['366054', '366063', '366111', '366120', '366123', '366125']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'gaborjbernat', 'laloch']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40238'
versions = ['Python 3.8', 'Python 3.9']

Linked PRs

• gh-84419: Fix the execute permissions in os.stat() on Windows #155392
• [3.15] gh-84419: Fix the execute permissions in os.stat() on Windows (GH-155392) #156629
• [3.14] gh-84419: Fix the execute permissions in os.stat() on Windows (GH-155392) #156630
• [3.13] gh-84419: Fix the execute permissions in os.stat() on Windows (GH-155392) #156631

[gaborbernat]

While I agree that Windows is safe to transform paths as he wishes to, the bug reported here is that os.stat/os.path.isdir behaves differently than os.scandir. Can we make them behave the same?

[eryksun]

While I agree that Windows is safe to transform paths as he wishes to,
the bug reported here is that os.stat/os.path.isdir behaves
differently than os.scandir. Can we make them behave the same?

os.listdir and os.scandir can be modified to behave like os.stat, but not the other way around. They differ because a "*.*" wildcard component is appended to the path that's passed to FindFirstFileW, and trailing spaces and dots only get stripped in the final path component.

To implement this without hard-coding Windows filename rules, the path needs to be normalized via WINAPI GetFullPathNameW before appending the "*.*" component (or just "*"; the ".*" is superfluous) -- but only for normal paths, i.e. paths that do not begin with exactly "\\\\?\\". The functions that would need to be updated are _listdir_windows_no_opendir and os_scandir_impl in Modules/posixmodule.c.

Another option would be to rewrite listdir and scandir to use CreateFileW and GetFileInformationByHandleEx: FileIdBothDirectoryInfo 1. This query provides two additional fields in comparison to the classic find data: ChangeTime (Unix st_ctime) and FileId (Unix st_ino). If the file is flagged as a reparse point in its attributes, then the reparse tag is set in the EaSize field of the directory info, since extended attributes can't be set on a reparse point; see [MS-FSCC] 2.4.17 2.

[laloch]

Hi Eryk, thanks for your time and for the explanation.

The Windows file API normalizes paths to replace forward slashes with backslashes; resolve relative paths and "." and ".." components; strip trailing spaces and dots from the final component; and map reserved DOS device names in the final component of non-UNC paths to device paths (e.g. "C:/Temp/con " -> r"\\.\con").

OK, I understand. I know that Win32 documentation suggests to avoid using paths with trailing spaces and that the paths are subject to normalization. Then I'd say os.path.normpath() should perform the same (GetFullPathNameW?) normalization as os.stat() and friends do.

Currently:

>>> import os
>>> path = r"c:\Program Files "
>>> os.path.normpath(path)
'c:\\Program Files '
>>> os.path.realpath(path)
'C:\\Program Files'

[eryksun]

I'd say os.path.normpath() should perform the same (GetFullPathNameW?)
normalization as os.stat() and friends do.

ntpath.abspath calls GetFullPathNameW (i.e. nt._getfullpathname) in Windows, but ntpath.normpath is pure Python. I agree that normpath should trim trailing spaces and dots from the last component. It should also normalize device paths and extended paths that start with "\\\\.\\" and "\\\\?\\". An extended path only skips normalization in an open or create context.

[serhiy-storchaka]

Interesting, os.stat() returns different results for path with and without a trailing space. st_mode is different.

>>> os.stat('python.bat')
os.stat_result(st_mode=33279, st_ino=562949953457810, st_dev=4917565571596533356, st_nlink=1, st_uid=0, st_gid=0, st_size=489, st_atime=1747648131, st_mtime=1725455997, st_ctime=1725455997)
>>> os.stat('python.bat ')
os.stat_result(st_mode=33206, st_ino=562949953457810, st_dev=4917565571596533356, st_nlink=1, st_uid=0, st_gid=0, st_size=489, st_atime=1747648131, st_mtime=1725455997, st_ctime=1725455997)
>>> oct(os.stat('python.bat').st_mode)
'0o100777'
>>> oct(os.stat('python.bat ').st_mode)
'0o100666'

I guess this is because it tests the extension against the sets of literals:

            if (_wcsicmp(fileExtension, L".exe") == 0 ||
                _wcsicmp(fileExtension, L".bat") == 0 ||
                _wcsicmp(fileExtension, L".cmd") == 0 ||
                _wcsicmp(fileExtension, L".com") == 0) {
                result->st_mode |= 0111;
            }

So what is the plan?

• Make normpath() trimming trailing spaces and dots.
• Trim them also from the path in os.stat(). At least before checking extension.
• Trim them also in os.listdir().

What about splitext()?

Note also that '...' and '.. ' are equivalent to '.', but not to '..'.

[zooba]

Ignoring trailing spaces in the hack to check extensions in stat is the only viable way (and strictly speaking they should only be ignored when the path doesn't start with a \\?\ prefix). (I considered getting the final path from the open file handle, but I'm pretty sure CreateProcess validates the path given to it rather than the resolved file, and so e.g. a symlink without .exe isn't executable even if the target is one.)

We should avoid modifying paths that come from the OS, to ensure that users can round-trip paths perfectly. normpath is our exception, but it's also not required, and so our own functions that try to interpret paths need to handle some of the normalisation that the OS would do when actually resolving the path.

[serhiy-storchaka]

The difference between os.stat() and os.scandir()/os.listdir() was fixed in gh-150880: the path is now normalized before appending the wildcard, so all of them ignore trailing dots and spaces.

What is left is the execute permissions in st_mode, which were guessed from the not normalized extension. #155392 fixes this.