Skip to content

Unvendor libmpdec sources #115119

Closed
1 of 1 issue completed
Closed
Unvendor libmpdec sources#115119
1 of 1 issue completed
Assignees
zware
Labels
buildThe build process and cross-buildextension-modulesC modules in the Modules dir

Description

@zware
zware
opened on Feb 7, 2024

To facilitate cleaner updates of the externally-maintained libmpdec required by the _decimal module, we should migrate away from the bundled copy in Modules/_decimal/libmpdec and towards an "external" in cpython-source-deps for Windows and --with-system-libmpdec by default elsewhere.

My tentative plan is as follows:

Linked PRs

Activity

  1. zware
    added
    buildThe build process and cross-build
    on Feb 7, 2024
  2. zware
    self-assigned this
    on Feb 7, 2024
  3. erlend-aasland
    added
    extension-modulesC modules in the Modules dir
    on Feb 7, 2024

  4. erlend-aasland commented on Feb 7, 2024

    @erlend-aasland
    erlend-aasland
    on Feb 7, 2024
    Contributor

    [...] with a fallback to the bundled copy (maybe with a warning?)

    +1 for an autoconf warning.

  5. ned-deily commented on Feb 8, 2024

    @ned-deily
    ned-deily
    on Feb 8, 2024
    Member

    Adding an external libmpdec to the macOS installer build shouldn't be a big issue. Assign it to me if you proceed with this.

  6. erlend-aasland commented on Feb 8, 2024

    @erlend-aasland
    erlend-aasland
    on Feb 8, 2024
    Contributor

    FTR: the source deps repo is now updated.

  7. zware
    added 2 commits that reference this issue on Feb 8, 2024

    pythongh-115119: Switch Windows build to mpdecimal external

    3cfe39b

    pythongh-115119: Switch Windows build to mpdecimal external

    7fc2f57

  8. erlend-aasland commented on Feb 8, 2024

    @erlend-aasland
    erlend-aasland
    on Feb 8, 2024
    Contributor

    @ned-deily: for build-installer.py, is there more to do than adding a configuration for mpdecimal? I've used the following configure options: ['--disable-cxx', 'MACHINE=universal']

  9. ned-deily commented on Feb 8, 2024

    @ned-deily
    ned-deily
    on Feb 8, 2024
    Member

    Likely yes, I'll take care of it. Thanks.

  10. ned-deily commented on Feb 13, 2024

    @ned-deily
    ned-deily
    on Feb 13, 2024
    Member

    I took a quick look at this and be aware that this change proposal will likely impact everyone building Python 3.13 on macOS. macOS does not ship a copy of libmpdec so, if the bundled version is removed, anyone building Python on macOS will now need to provide a local copy, either building from source or using a version from one of the widely-used third-party package managers, like Homebrew or MacPorts. Both currently have mpdecimal packages at the 4.0.0 level. (Recall that, unlike Windows builds, we do not currently provide non-vendored third-party library binaries for macOS users, other than those built for and linked into the Pythons provided by python.org installers for macOS.) In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected. And, in any case, the devguide instructions for installing dependencies will need to be updated to cover libmpdec. We should add all this and get this working before moving on to changing the installer.

  11. erlend-aasland commented on Feb 13, 2024

    @erlend-aasland
    erlend-aasland
    on Feb 13, 2024
    Contributor

    In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

    Last time I checked, libmpdec did not provide pkgconfig files:

    $ ls $(brew --prefix libmpdec)/lib/pkgconfig
ls: /opt/homebrew/opt/mpdecimal/lib/pkgconfig: No such file or directory
$ cd mpdecimal-2.5.1; find . | grep pkg
$

  12. ned-deily commented on Feb 13, 2024

    @ned-deily
    ned-deily
    on Feb 13, 2024
    Member

    With the current 4.0.0 version, it appears both Homebrew and MacPorts ship them:

    $ pkg-config libmpdec --cflags
-I/opt/homebrew/Cellar/mpdecimal/4.0.0/include
$ pkg-config libmpdec --libs
-L/opt/homebrew/Cellar/mpdecimal/4.0.0/lib -lmpdec -lm

...

$ pkg-config libmpdec --libs
-L/opt/macports/lib -lmpdec -lm

  13. erlend-aasland commented on Feb 13, 2024

    @erlend-aasland
    erlend-aasland
    on Feb 13, 2024
    Contributor

    Yes, the 4.0.0 version does indeed provide pkg-config configuration files. I'll work on an Autoconf patch.

  14. erlend-aasland
    added a commit that references this issue on Feb 13, 2024

    pythongh-115119: Detect _decimal dependencies using pkg-config

    cff99f3

  15. 133 remaining items

  16. zware commented on May 13, 2026

    @zware
    zware
    on May 13, 2026
    MemberAuthor

    I consider the disappearance of libmpdec{,-dev} in Ubuntu 24.04 to be a bug in that version of Ubuntu; lobby them to fix it.

  17. asottile commented on May 13, 2026

    @asottile
    asottile
    on May 13, 2026
    Contributor

    it's an LTS, they don't add new packages. I've tried many times

  18. asottile commented on May 13, 2026

    @asottile
    asottile
    on May 13, 2026
    Contributor

    it's also not present in debian:bookworm

  19. vstinner commented on May 13, 2026

    @vstinner
    vstinner
    on May 13, 2026
    Member

    Can't you add libmpdev in deadsnakes PPA?

  20. eli-schwartz commented on May 13, 2026

    @eli-schwartz
    eli-schwartz
    on May 13, 2026
    Contributor

    @eli-schwartz I don't control the ubuntu repositories just the deadsnakes ones. unless you're suggesting "backporting" via revendoring in a patch?

    I quite literally mean backport the *.deb packaging format package for a dpkg package named "libmpdec". If I'd meant reverting a patch to revendor the source directory I'd say "forward port the libmpdec sources from older python", which is the opposite of a backport.

    The repository that you control is deadsnakes, hence backport from Ubuntu 26.04 to deadsnakes.

  21. skirpichev commented on May 14, 2026

    @skirpichev
    skirpichev
    on May 14, 2026
    Member

    (noticing this because of nightly builds for deadsnakes) -- removing the bundled copy makes python unbuildable on ubuntu 24.04 because there is no system libmpdec there

    @asottile, could you please point to build logs or elaborate a bit exact reason of failure? Missing libmpdec shouldn't prevent CPython 3.16 from build:

    $ ./configure -q
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
$ make -s
Written build/lib.linux-x86_64-3.16/_sysconfigdata__linux_x86_64-linux-gnu.py
Written build/lib.linux-x86_64-3.16/_sysconfig_vars__linux_x86_64-linux-gnu.json
The necessary bits to build these optional modules were not found:
_decimal
To find the necessary bits, look in configure.ac and config.log.

Checked 116 modules (37 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 1 missing, 0 failed on import)
$ ./python -m test test_decimal
Using random seed: 1990606034
0:00:00 load avg: 2.12 Run 1 test sequentially in a single process
0:00:00 load avg: 2.12 [1/1] test_decimal
C tests skipped: no module named _decimal.
0:00:15 load avg: 2.24 [1/1] test_decimal passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 15.6 sec
Total tests: run=579 skipped=198
Total test files: run=1/1
Result: SUCCESS

    It might be that differences of pure-Python decimal module wrt the C extension break some your CI tests. In this case, please open issues.

    I consider the disappearance of libmpdec{,-dev} in Ubuntu 24.04 to be a bug in that version of Ubuntu; lobby them to fix it.

    They blindly follow unprofessional actions of Debian maintainers: https://bugs.debian.org/1056785.

    BTW, this is not a problem for ubuntu 22.04 or 26.04. Tentative schedule for new GA images: actions/runner-images#13964 (comment)

    Edit:

    I would prefer waiting to remove the bundled copy until 24.04 is end-of-lifed

    That means April 2034, i.e ~10 years. Any good reasons for this huge delay, given that the libmpdec is available in PPA and it's trivial to build it from sources?

  22. stbergmann
    added a commit that references this issue on May 18, 2026

    python3: upgrade to 3.14.5rc1

    622d1c6
  23. zware
    added a commit that references this issue on Jun 1, 2026

    gh-115119: Remove superfluous TEST_COVERAGE private macro from _decim…

    59abdf8
  24. vstinner
    added a commit that references this issue on Jun 2, 2026

    gh-115119: Fix libmpdec detection when pkg-config files are missing (#…

    025cc26
  25. ned-deily
    added 2 commits that reference this issue on Jun 9, 2026

    [3.13] gh-115119: Update macOS installer to use libmpdecimal 4.0.1. (G…

    7372a14

    [3.14] gh-115119: Update macOS installer to use libmpdecimal 4.0.1. (G…

    3ece6b5

  26. edmorley commented on Sep 3, 2026

    @edmorley
    edmorley
    on Sep 3, 2026

    I consider the disappearance of libmpdec{,-dev} in Ubuntu 24.04 to be a bug in that version of Ubuntu; lobby them to fix it.

    Opened https://support-portal.canonical.com/tickets/00448718

  27. hugovk
    added a commit that references this issue on Sep 18, 2026

    gh-115119: Fix licence page after removing bundled copy of libmpdec (#…

    e66424a

  28. edmorley commented on Sep 22, 2026

    @edmorley
    edmorley
    on Sep 22, 2026

    Opened https://support-portal.canonical.com/tickets/00448718

    Which resulted in a (currently in-progress) SRU request to add libmpdec-dev to noble, here:
    https://bugs.launchpad.net/ubuntu/+source/mpdecimal/+bug/2167730

  29. freakboy3742
    added a commit that references this issue on Oct 12, 2026

    gh-115119: Remove deprecated --with-system-mpdec option from iOS buil…

    87d4d44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

buildThe build process and cross-buildextension-modulesC modules in the Modules dir

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions