tests: test suite break when path contains #

Building from https://aur.archlinux.org/packages/python-cyclonedx-lib, I am getting ~400 errors if the path to the project contains a # character.
Log attached
error.log

Could you provide a reproducible setup/example?

dug into https://github.com/CycloneDX/cyclonedx-python-lib/files/14281372/error.log

line of interest:

======================================================================
ERROR: test_cases_render_valid_05_XML_1_5 (tests.test_enums.TestEnumComponentScope.test_cases_render_valid_05_XML_1_5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ddt.py", line 221, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/mock.py", line 1375, in patched
    return func(*newargs, **newkeywargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp#Le1C31tGnA/python-cyclonedx-lib/src/cyclonedx-python-lib-6.4.1/tests/test_enums.py", line 256, in test_cases_render_valid
    super()._test_cases_render(bom, of, sv)
  File "/tmp/tmp#Le1C31tGnA/python-cyclonedx-lib/src/cyclonedx-python-lib-6.4.1/tests/test_enums.py", line 145, in _test_cases_render
    validation_errors = make_schemabased_validator(of, sv).validate_str(output)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp#Le1C31tGnA/python-cyclonedx-lib/src/cyclonedx-python-lib-6.4.1/cyclonedx/validation/xml.py", line 61, in validate_str
    return self._validata_data(
           ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp#Le1C31tGnA/python-cyclonedx-lib/src/cyclonedx-python-lib-6.4.1/cyclonedx/validation/xml.py", line 67, in _validata_data
    validator = self._validator  # may throw on error that MUST NOT be caught
                ^^^^^^^^^^^^^^^
  File "/tmp/tmp#Le1C31tGnA/python-cyclonedx-lib/src/cyclonedx-python-lib-6.4.1/cyclonedx/validation/xml.py", line 91, in _validator
    self.__validator = XMLSchema(file=schema_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/lxml/xmlschema.pxi", line 89, in lxml.etree.XMLSchema.__init__
lxml.etree.XMLSchemaParseError: element decl. '{http://cyclonedx.org/schema/bom/1.5}id', attribute 'type': The QName value '{http://cyclonedx.org/schema/spdx}licenseId' does not resolve to a(n) type definition., line 644

It appears, that XMLSchema()'s file argument must not include a #, as it would be stripped, since the path is treated as a url.

I need to check with the leveraged https://pypi.org/project/lxml/ project, maybe there is a fix, or workaround, or an existing report/solution/advice.

@hseg

Thank you for your report, @hseg

Since this project is community-driven, and contributions are welcome, I would ask you: are you interested in providing a solution to the problem? If so, please let me know. Please see our contribution guideline.

On Wed, Feb 14, 2024 at 07:16:32AM -0800, Jan Kowalleck wrote: Could you give a reproducible setup/example?

Reproducing the steps run by pacman: ```sh cd "$(mktemp -td tmp#XXXXX)" wget https://github.com/CycloneDX/cyclonedx-python-lib/archive/refs/tags/v6.4.1.tar.gz tar xzf v6.4.1.tar.gz cd cyclonedx-python-lib-6.4.1/ find tests -name 'invalid-metadata-timestamp-*.json' -exec rm -v '{}' ';' find tests -name 'valid-signatures-*.json' -exec rm -v '{}' ';' python -m build --wheel --no-isolation python -m venv --clear --system-site-packages .venv source .venv/bin/activate pip install --force-reinstall --no-deps dist/*.whl python -m unittest discover -v deactivate ``` By contrast, if the `#` in the invocation to `mktemp` there is replaced with `.`, everything passes.

On Wed, Feb 14, 2024 at 07:29:47AM -0800, Jan Kowalleck wrote: Thank you for your report, @hseg Since this project is community-driven, and contributions are welcome, I would ask you: are you interested in providing a solution to the problem? If so, please let me know. Please see our [contribution guideline](https://github.com/CycloneDX/cyclonedx-python-lib/blob/main/CONTRIBUTING.md).

I don't have much free time at the moment, if no one has fixed this by 2024-03-17 I'll take it on.

re: #551 (comment)
Thanks for providing the setup snippet. this should make it easy for others to work on this issue.

re: #551 (comment)
I flagged this issue as "help wanted" as in "this is free for contributors".
Whoever wants to tackle it, just drop a note here to let the others know.

Tried tackling this, one complication to look out for is that `lxml` makes heavy use of cython, which I'm having trouble debugging. I'm giving up on this one. One thing I _have_ noted playing around with this is that what is significant is the real path to the test files - in particular, a usable workaround is to have a clone somewhere with a saner path, and to have a symlink to it in the desired location. El 14 de febrero de 2024 17:42:12 GMT+02:00, Jan Kowalleck ***@***.***> escribió:

…

re: #551 (comment) Thanks for providing the setup snippet. this should make it easy for others to work on this issue. re: #551 (comment) I flagged this issue as "help wanted" as in "this is free for contributors". Whoever wants to tackle it, just drop a note here to let the others know. -- Reply to this email directly or view it on GitHub: #551 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

I've run a couple of exhaustive tests trying to find the forbidden patterns.
Data is attached below. From it, I conjencture that besides #, it is also forbidden to have paths containing %xx where x are hexadecimal characters (though I've only checked lowercase...), excepting the following accepted hexadecimal sequences:

%01..%19
%20,%22,%23,%25,%60
%80..%99
These can't be referring to control characters, otherwise eg %0a..%0f, %1a..1f would've been accepted. However, I do note that the second sequence of accepted characters corresponds to
"
#
%
`
respectively. Perhaps someone can make better use of this data than me.

failures-sorted.log
successes-sorted.log

looks like these chars are all XML special chars.
makes sense, since the XML tooling crashes when it tries to resolve paths ...

Right. ig the weird part about this is that the XML tooling is embedding raw paths into the files it constructs. I would expect any such serialization/deserialization to include a quote/unquote step as well. El 26 de mayo de 2024 16:44:33 GMT+03:00, Jan Kowalleck ***@***.***> escribió:

…

looks like these chars are all XML special chars. makes sense, since the XML tooling crashes when it tries to resolve paths ... -- Reply to this email directly or view it on GitHub: #551 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

Confirmed on current main (8edee5c) with Python 3.12, lxml 5.3.2, and libxml2 2.12.10 under WSL2: loading the bundled schemas from a checkout path containing # fails because the relative spdx.SNAPSHOT.xsd import is not resolved. The same regression test passes with lxml 6.1.1.

I am preparing a narrow fix that passes an encoded local file: URI to XMLSchema, plus a regression test. I will link the PR to this issue when it is ready.

fixed via https://github.com/CycloneDX/cyclonedx-python-lib/releases/tag/v11.11.2

Navigation Menu

Repository navigation

tests: test suite break when path contains `#` #551

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tests: test suite break when path contains # #551

Description

Activity

jkowalleck commented on Feb 14, 2024

jkowalleck commented on Feb 14, 2024

jkowalleck commented on Feb 14, 2024

hseg commented on Feb 14, 2024

hseg commented on Feb 14, 2024

jkowalleck commented on Feb 14, 2024

hseg commented on Apr 30, 2024

hseg commented on May 26, 2024

jkowalleck commented on May 26, 2024

hseg commented on May 26, 2024

CAOShurong commented on Aug 12, 2026

jkowalleck commented on Aug 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

tests: test suite break when path contains `#` #551