Performance degradation in google-api-core

[Crowiant]

Determine this is the right repository

• I determined this is the correct repository in which to report this bug.

Summary of the issue

Context
Upgrading google-api-core from version 2.27.0 to 2.28.1 (and later) has caused a significant spike in CPU usage. While version 2.27.0 maintains a steady 20–40% utilization, versions 2.28.1 and above consistently reach 100% CPU consumption.

Expected Behavior:
Same consumption of CPU as in 2.27.0

Actual Behavior:
CPU consumption increased up to 5 times.

API client name and version

google-api-core v. 2.28.1 - 2.30.0

Reproduction steps: code

file: main.py

import time
import importlib
import google.api_core

time.sleep(5)

def test_api_core():
    importlib.reload(google.api_core)


for _ in range(2000):
    test_api_core()

print("Successfully imported 2000 times.")

Reproduction steps: supporting files

file: Dockerfile

FROM python:3

WORKDIR /app

RUN pip install --no-cache-dir google-api-core==2.28.1 grpcio

COPY main.py .

CMD ["python", "main.py"]

Reproduction steps: actual results

docker stats:
2.30.0

2.27.0

Reproduction steps: expected results

OS & version + platform

No response

Python environment

No response

Python dependencies

No response

Additional context

No response

[daniel-sanche]

Do you see a performance degredation during usage, or just at import time?

This seems likely related to the new version support check. But that should be a one-time cost

[Crowiant]

Hello @daniel-sanche I see performance degradation during usage. It is really a one-time cost for 2.27.0 . CPU usage spikes once and then drops to 0. But for version higher than 2.27.0 spike lasts during all imports. So each import causes CPU consumption up to 100%.

[bonauer-pf]

This is likely the same regression that also leads to #15015, which is only reproducible in large venvs.

We are also affected by this, and every google cloud library takes 3-4s to import since the version bump, inflating our total startup times by >10s.

[bonauer-pf]

Isolated the slow call to be importlib.metadata.package_distributions(), which has to scann all package metadata in the venv and takes >1s. Code location:

google-cloud-python/packages/google-api-core/google/api_core/_python_version_support.py

Line 159 in 281eaae

module_to_distributions = metadata.packages_distributions()

This gets called at import time for every Google Cloud module, sometimes multiple times.

[vchudnov-g]

@bonauer-pf 's #16579 seems like a promising fix. Thanks!

[vchudnov-g]

To clarify: I believe the issue arises here: recent versions of api_core now include these symbols, and so we were calling the function doing these checks every time we imported a new library. The memoization in #16579 should alleviate that.

[daniel-sanche]

This should be fixed in the latest version, but let me know if this is still an issue