pip install is not working on Apple M1 silicone chip

[Yarden92]

I'll just raise it here as well since nobody initiated any speak for that.

same as scipy, scikit-learn etc, this one as well is not installable on Apple's silicone M1 Chips.

with scipy I managed to install it via homebrew and then manually copy the folder to my venv, but this one is not listed on the brew packages.

here's what happen when I run the command:
pip install scikit-commpy

ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/1k/xrnsys5s7b39s_ylmkgng4fw0000gn/T/pip-install-m_y1ehyi/scipy_f4cd9f203f3b496f9359d8942a67d319/setup.py'"'"'; __file__='"'"'/private/var/folders/1k/xrnsys5s7b39s_ylmkgng4fw0000gn/T/pip-install-m_y1ehyi/scipy_f4cd9f203f3b496f9359d8942a67d319/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/1k/xrnsys5s7b39s_ylmkgng4fw0000gn/T/pip-record-csv2ix36/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/scipy

(full error i in the attached log file)

log.txt

any solution / workaround?

[YairMZ]

By looking at your log it seems the error is raised when trying to build numpy.
It seems, that when building scipy and other dependencies, the wheel building of these dependencies ignores the numpy numpy version you already have.

You can see this in the log:

Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from scikit-commpy) (1.21.2)
Collecting scipy
  Using cached scipy-1.7.1.tar.gz (36.1 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /private/var/folders/1k/xrnsys5s7b39s_ylmkgng4fw0000gn/T/pip-standalone-pip-xmf7j3wp/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/1k/xrnsys5s7b39s_ylmkgng4fw0000gn/T/pip-build-env-xi9mridd/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'wheel<0.37.0' 'setuptools<58.0.0' 'Cython>=0.29.18,<3.0' 'pybind11>=2.4.3,<2.7.0' pythran==0.9.11 'numpy==1.19.2; 

As you can see pip install specifies pip install --ignore-installed . This is of course specifed in hte build process of scipy, and as such, the problem doesn't originate from this repo, and cannot be fixed here.

You may open an issue at scipy, but a short google search seems to show they are aware of the problems.

That being said, this SO thread seems to have solutions that enable people to build numpy locally on Apple M1, which is where the process failed for you.

If you can build numpy your problem may be resolved.

[Yarden92]

@YairMZ thank you so much for the detailed comment!

I am not sure I completely understood you.
numpy was easily installable via pip, it is installed both globally as well as in my project's venv.

scipy I have managed to installed using homebrew and then manually copying the folder from the homebrew's packages directory (which was something like /opt/homebrew/lib/python3.9/site-packages/scipy) into my venv's packages directory (which is <project_dir>/venv/lib/python**/site_packages/___HERE__)

can you please elaborate how exactly do I use the above in order to get CommPy (or any other scikit module that does not exist on homebrew)?

[YairMZ]

@Yarden92 You're welcome.
Always happy to help. I'll try to explain, and you're welcome to ask more questions.

When you install packages via pip, basically there are two options for installing. The distributor of the package distributes the source code for the package, and sometimes he distributes also compiled binaries called wheels.

These wheels are helpful as they allow faster install (no need to build it yourself), and they're especially useful for complicated packages such as numpy / scipy for which the build process is complex, as you encountered.

The problem with wheels is that since they ship compiled code and not source, they are architecture dependent. Apparently, for M1 macs, wheel support is still lacking.

Now, I see that you already have scipy and numpy, but for some reason the installation process tries to build other versions of these packages. I'm not sure if it is a result of the CommPy build process or of scipy, but from the log it seems the issue is with scipy.

None the less, it may be required for instance for compatibility reasons. It seems your version of numpy is newer and there may be problems with backward compatability.

To make a long story short, the installation failed on trying to build scipy. When you installed it via homebrew, you used compiled code and didn't build it on your machine. To build it, you may be lacking libraries, compilers or both.

The link I shared earlier, discusses how people were able to build the libraries and not just install them. Again' I'm not sure why this is even needed and why wheels aren't used. If you want to check, I would start with a clean venv, and try to pip install the earlier version of numpy which is being built while installing CommPy (the version appears in the log you sent). Then after this is succsseful, try intalling CommPy in this venv.

[BastienTr]

CommPy does not specify any version for Numpy/Scipy. It accepts newest version of both libraries (Cf. numpy>=1.9.2 and scipy>=0.15.0). I think that pip is not aware that Numpy/Scipy already exist when you manually copied it from homebrew to your venv. Thus, pip try to reinstall it as these library are required for Commpy and this lead to the bug. Are you sure that pip detects properly Numpy/Scipy in your venv after copying it?