Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPython Console not working when connecting standalone app to external interpreter on Mac #21575

Closed
shubhamg13 opened this issue Dec 2, 2023 · 26 comments
Closed

IPython Console not working when connecting standalone app to external interpreter on Mac #21575

shubhamg13 opened this issue Dec 2, 2023 · 26 comments
Assignees
@mrclary
Labels
component:IPython Console OS:MacOS type:Bug

Comments

@shubhamg13
Copy link

Description

What steps will reproduce the problem?

  1. Change Python Interpretor to System's Python path
  2. Install spyder-kernal

Traceback

Traceback (most recent call last):
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 91, in wrapper
    self._ready.set_result(None)
asyncio.exceptions.InvalidStateError: invalid state

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "qtconsole/manager.pyc", line 27, in poll
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/restarter.py", line 143, in poll
    self.kernel_manager.restart_kernel(now=True, newports=newports)
  File "jupyter_core/utils/__init__.pyc", line 173, in wrapped
  File "asyncio/base_events.pyc", line 647, in run_until_complete
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 596, in _async_restart_kernel
    await self._async_start_kernel(**self._launch_args)
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 94, in wrapper
    self._ready.set_exception(e)
asyncio.exceptions.InvalidStateError: invalid state

Versions

  • Spyder version: 5.5.0 b4c4e1a (standalone)
  • Python version: 3.9.14 64-bit
  • Qt version: 5.15.11
  • PyQt5 version: 5.15.10
  • Operating System: macOS-14.1.1-x86_64-i386-64bit

Dependencies

# Mandatory:
atomicwrites >=1.2.0             :  1.4.1 (OK)
chardet >=2.0.0                  :  5.2.0 (OK)
cloudpickle >=0.5.0              :  3.0.0 (OK)
cookiecutter >=1.6.0             :  2.5.0 (OK)
diff_match_patch >=20181111      :  20230430 (OK)
intervaltree >=3.0.2             :  3.1.0 (OK)
IPython >=8.13.0,<9.0.0,!=8.17.1 :  8.17.2 (OK)
jedi >=0.17.2,<0.20.0            :  0.19.1 (OK)
jellyfish >=0.7                  :  1.0.3 (OK)
jsonschema >=3.2.0               :  4.20.0 (OK)
keyring >=17.0.0                 :  24.3.0 (OK)
nbconvert >=4.0                  :  7.11.0 (OK)
numpydoc >=0.6.0                 :  1.6.0 (OK)
parso >=0.7.0,<0.9.0             :  0.8.3 (OK)
pexpect >=4.4.0                  :  4.8.0 (OK)
pickleshare >=0.4                :  0.7.5 (OK)
psutil >=5.3                     :  5.9.6 (OK)
pygments >=2.0                   :  2.17.1 (OK)
pylint >=2.5.0,<3.1              :  3.0.2 (OK)
pylint_venv >=3.0.2              :  None (OK)
pyls_spyder >=0.4.0              :  0.4.0 (OK)
pylsp >=1.9.0,<1.10.0            :  1.9.0 (OK)
pylsp_black >=1.2.0,<3.0.0       :  1.3.0 (OK)
qdarkstyle >=3.2.0,<3.3.0        :  3.2.1 (OK)
qstylizer >=0.2.2                :  0.2.2 (OK)
qtawesome >=1.2.1                :  1.2.3 (OK)
qtconsole >=5.5.0,<5.6.0         :  5.5.1 (OK)
qtpy >=2.1.0                     :  2.4.1 (OK)
rtree >=0.9.7                    :  1.1.0 (OK)
setuptools >=49.6.0              :  69.0.2 (OK)
sphinx >=0.6.6                   :  5.1.1 (OK)
spyder_kernels >=2.5.0,<2.6.0    :  2.5.0 (OK)
textdistance >=4.2.0             :  4.6.0 (OK)
three_merge >=0.1.1              :  0.1.1 (OK)
watchdog >=0.10.3                :  3.0.0 (OK)
zmq >=22.1.0                     :  25.1.1 (OK)

# Optional:
cython >=0.21                    :  3.0.5 (OK)
matplotlib >=3.0.0               :  3.8.2 (OK)
numpy >=1.7                      :  1.26.2 (OK)
pandas >=1.1.1                   :  2.1.3 (OK)
scipy >=0.17.0                   :  1.11.4 (OK)
sympy >=0.7.3                    :  1.12 (OK)
@dalthviz
Copy link
Member

dalthviz commented Dec 6, 2023

Hi @shubhamg13 thank you for hte feedback! Seems like this is a duplicate of #21338 Could you check the version of jupyter_core and see if it's at least 5.3.2 (which seems like is the version that fixes things - #21338 (comment)).

Let us know!

@dalthviz dalthviz self-assigned this Dec 6, 2023
@ccordoba12 ccordoba12 mentioned this issue Jan 4, 2024
Closed
@ccordoba12 ccordoba12 changed the title Python Console not working IPython Console not working when connecting stand alone app to external interpreter on Mac Jan 4, 2024
@ccordoba12 ccordoba12 added this to the v5.5.1 milestone Jan 4, 2024
@ccordoba12 ccordoba12 changed the title IPython Console not working when connecting stand alone app to external interpreter on Mac IPython Console not working when connecting standalone app to external interpreter on Mac Jan 4, 2024
@ccordoba12 ccordoba12 mentioned this issue Jan 4, 2024
Closed
@ccordoba12
Copy link
Member

ccordoba12 commented Jan 4, 2024 •
edited
Loading

@mrclary, all users who have reported this issue during the last couple of months are using our standalone app.

Besides what @dalthviz mentioned above (using an old Jupyter-core version), I think this could be caused by using an external interpreter which is compiled for arm64 while our app is based on Intel. Could you check that?

If those two things are not the cause of the problem, could you try to replicate the bug in other ways? Thanks!

@ccordoba12 ccordoba12 assigned mrclary and unassigned dalthviz Jan 4, 2024
@mrclary
Copy link
Contributor

mrclary commented Jan 4, 2024

Our standalone application for Spyder 5.5.0 ships with jupyter_core=5.5.0 (for both macOS and Windows). From the METADATA file in the distribution:

Metadata-Version: 2.1
Name: jupyter_core
Version: 5.5.0
Summary: Jupyter core package. A base package on which Jupyter projects rely.

I'll investigate possible arm64 complications.

@shubhamg13, could elaborate on how to reproduce the issue? In particular:

  • When you changed the interpreter to the System's Python path, do you mean /usr/bin/python3 that ships with macOS, or /usr/local/bin/python3 installed by Python.org or Homebrew, or some other path?
  • How did you install spyder-kernels? /usr/bin/python3 -m pip install spyder-kernels?

@tomb1949
Copy link

tomb1949 commented Jan 5, 2024 •
edited by mrclary
Loading

@mrclary: I'm replying here because I opened a similar issue. I downloaded the standalone for Mac and attempted to bind it to Python 3.12.0 for Mac, the latest release. The errors in the interactive console suggest that the standalone is expecting Intel 64 binaries. But there is only one Mac download for this and other recent versions of Python. @ccordoba said this is a known issue and no fix is planned, so is the standalone version just not going to support recent Python on the Mac? (My issue was #21672, you can see the stack trace there.). Thanks, TB

@mrclary
Copy link
Contributor

mrclary commented Jan 5, 2024

So far here is what I've found:

I have not been able to reproduce the OP on my M1 machine. I have successfully used an external arm64 (not universal) conda environment. I'm waiting for a response from @shubhamg13 to test a different environment. Although I will likely not test the system /usr/bin/python3 environment as it is strongly discouraged to alter this in any way, and I'm certain that Spyder deliberately excludes this from the list of possible Python interpreters to use for the IPython console.

On my M1 machine I see the following:

  1. From Terminal: 
    (spy-test) >> arch
arm64
(spy-test) >> file $(which python)
/Users/ryan/mambaforge/envs/spy-test/bin/python: Mach-O 64-bit executable arm64
(spy-test) >> file /Applications/Spyder.app/Contents/MacOS/python 
/Applications/Spyder.app/Contents/MacOS/python: Mach-O 64-bit executable x86_64
(spy-test) >> file /Applications/Spyder.app/Contents/MacOS/spyder 
/Applications/Spyder.app/Contents/MacOS/spyder: Mach-O 64-bit executable x86_64
    This shows that I'm running on arm64, that the conda environment Python executable is arm64 only, and that Spyder standalone is x86_64 only.
  2. In conda test environment Python interpreter: 
    >>> import platform; platform.platform()
'macOS-14.2.1-arm64-arm-64bit'
    This shows that the test environment Python indicates the correct machine and processor.
  3. In Spyder's internal console: 
    >>> import platform; platform.platform()
'macOS-14.2.1-x86_64-i386-64bit'
>>> platform.version()
'Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103'
    This shows that Spyder's internal Python indicates the Rosetta translation, i.e. it thinks it's running on x86_64 machine and i386 processor. However, there is a clue in the version that indicates that the true machine is arm64.
  4. In Spyder's IPython console using the same test conda environment: 
    In [1]: import platform; platform.platform()
Out[1]: 'macOS-14.2.1-arm64-i386-64bit'
    This shows a discrepency from item 2 above, in which the machine is correctly reported but the processor is not. This must be a result of the iPython kernel (arm64) being started by a subprocess of Spyder (x86_64), yielding a mixed report.

Note: our Spyder standalone application (including our alpha conda-based installer) is x86_64 only (not universal) and when opened macOS will prompt the user to install Rosetta.

It is true that we will not build the current standalone application in an arm64 variant. However, our planned release of a conda-based application for version 6 is expected to have an arm64 variant, pending availability of arm64 runners on GitHub early this year.

@tomb1949
Copy link

tomb1949 commented Jan 5, 2024

Ryan: Thank you for looking at this.
Some possible things of interest.

  1. I don't have a usr/bin/python instance.
  2. Anaconda used to be installed but I removed it because I kept having failed update issues. So I've been building a standalone environment (successfully, so far) except for this problem.
  3. The python interpreter that is failing to load in Spyder iConsole is:
    /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
    There is also a python3.12-intel64 executable in that 'bin' but it doesn't help if I point to it explicity in Spyder Preferences. It would want to be started by Rosettta 2, I think.
  4. I was NOT prompted to install Rosetta 2 when i started Spyder standalone. I know it's on my machine (and that it works) because I needed to install it for MATLAB.
  5. Happy to try something if you have ideas.
    Thanks, TB

@mrclary
Copy link
Contributor

mrclary commented Jan 5, 2024

TL;DR

What I've learned is that for Spyder (x86_64) to work with external environments on arm64:

  • If the environment Python executable is universal, then all binaries in the Python environment must be x86_64. See below on how to accomplish this.
  • If the environment Python executable is a single architecture, then all the binaries in the Python environment must be the same architecture.

Details

Ryan: Thank you for looking at this. Some possible things of interest.

  1. I don't have a usr/bin/python instance.

Actually, you should have /usr/bin/python3, which comes with macOS, but it is best to pretend that you don't have anything there 😉.

  1. Anaconda used to be installed but I removed it because I kept having failed update issues. So I've been building a standalone environment (successfully, so far) except for this problem.

Sorry to hear that. I prefer to use miniforge since it is much smaller.

  1. The python interpreter that is failing to load in Spyder iConsole is:
    /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
    There is also a python3.12-intel64 executable in that 'bin' but it doesn't help if I point to it explicity in Spyder Preferences. It would want to be started by Rosettta 2, I think.

Okay, this is the Python.org installation and it is a universal2 binary (has both x86_64 and arm64 built in). As you observe, there are also python3.12-intel64 and python3.12-arm64 executables; these are also universal binaries but only have 1 architecture. You are correct that these executables may not be useful in solving any of the problems that you observe. There are two problems I think may be occurring for you here.

  1. You should avoid using the base Python executable environment. This is true for any Python install, even for conda (Anaconda or miniforge). One should always use a virtual environment instead.
    For a conda install you would do the following. 
    $ conda create -n my-venv python=3.12 spyder-kernels=2.5
    For any other Python install, use venv. 
    $ /usr/local/bin/python3 -m venv /path/to/desired/my-venv
$ source /path/to/desired/my-venv/bin/activate
(my-venv) $ python -m pip install spyder-kernels==2.5
    You should point Spyder to the python executable in the virtual environment.
    If you were using the Anaconda base environment to "work" in, then I suspect that is the reason you had update issues. I've also run into all sorts of problems whenever I've used base environments (even accidentally).
  2. When I create a virtual environment as above with /usr/local/bin/python3 (which just points to the universal binary /Library/Frameworks/Python.framework/Versions/3.12/bin/python3) and install spyder-kernels I get the following error in Spyder's IPython console: 
    ImportError: dlopen(/Users/ryan/my-venv/lib/python3.12/site‑packages/psutil/_psutil_osx.abi3.so, 0x0002): tried: '/Users/ryan/my-venv/lib/python3.12/site‑packages/psutil/_psutil_osx.abi3.so' (mach‑o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
    I think the problem here is that _psutil_osx.abi3.so is not the same architecture as the Python executable (x86_64 version selected by Rosetta). As I've found with conda environments, when the Python executable is arm64 and _psutil_osx.abi3.so is arm64, then there doesn't seem to be a problem.
    The workaround is to ensure x86_64 binaries are installed in the virtual environment. After creating the virtual environment as shown above, when installing spyder-kernels (or any package in the environment) use the arch -x86_64 pre-command: 
    $ /usr/local/bin/python3 -m venv /path/to/desired/my-venv
$ source /path/to/desired/my-venv/bin/activate
(my-venv) $ arch -x86_64 python -m pip install spyder-kernels==2.5
    This ensures that x86_64 packages are installed instead of arm64.
  1. I was NOT prompted to install Rosetta 2 when i started Spyder standalone. I know it's on my machine (and that it works) because I needed to install it for MATLAB.

It will only prompt if Rosetta is not already installed.

  1. Happy to try something if you have ideas.
    Thanks, TB

See above.

@mrclary
Copy link
Contributor

mrclary commented Jan 5, 2024

To be clear, my previous comment resolves cross-architecture issues, but I still have not been able to reproduce the OP.

@mrclary
Copy link
Contributor

mrclary commented Jan 5, 2024 •
edited
Loading

I don't see anything in the OP traceback (or in #21672 traceback) that indicates an arm64/x86_64 conflict.

@shubhamg13, since platform.platform misrepresents the actual machine and architecture when running under Rosetta, can you also confirm that you are running on an M1 or M2 Apple processor?

@tomb1949
Copy link

tomb1949 commented Jan 5, 2024 via email

@tomb1949
Copy link

tomb1949 commented Jan 5, 2024 via email

@mrclary
Copy link
Contributor

mrclary commented Jan 5, 2024

Thanks @tomb1949. That is exactly the same error that I was seeing, but that was not shown in the OP to this issue or in #21672. So that appears to be a completely different issue from the OP and we may have highjacked this issue 😦. But, hey, @ccordoba12 did initiate the idea of arm64/x86_64 conflict 😉.

@tomb1949
Copy link

tomb1949 commented Jan 5, 2024 via email

@ccordoba12
Copy link
Member

ccordoba12 commented Jan 8, 2024 •
edited
Loading

@tomb1949, according to what you posted above, please try these commands to check if they fix your problem:

pip uninstall psutil
pip install --no-binary :all: psutil

@mrclary
Copy link
Contributor

mrclary commented Jan 8, 2024

@tomb1949, according to what you posted above, please try these commands to check if they fix your problem:

pip uninstall psutil
pip install --no-binary :all: psutil

I would note that from the above traceback, all we know is that psutil is the first problematic binary encountered. It is possible that other packages will also have arm64 only architecture installed.

@tomb1949
Copy link

tomb1949 commented Jan 9, 2024 via email

@mrclary
Copy link
Contributor

mrclary commented Jan 9, 2024

I'm mulling Ryan's recommendation to get a more complex Python environment going versus going back to Anaconda, which is what I think is what both of you would see as a preferred solution.

I'm glad to see we are making progress. I would not worry about going back to Anaconda right now. With your existing Python.org installation, you must be adding packages using pip. Let me reiterate my recommendation to use a virtual environment and the architecture pre-command to install your packages.

$ /usr/local/bin/python3 -m venv $HOME/my-venv
$ source $HOME/my-venv/bin/activate
(my-venv) $ arch -x86_64 python -m pip install spyder-kernels==2.5 openai

This creates a virtual environment my-env in your home directory (but you can specify anywhere you want) with spyder-kernels and openai (and their dependencies) installed.
Just point the Spyder Python interpreter to $HOME/my-venv/bin/python and things should work fine.

If you have several packages you need to install, I would recommend using a simple requirements.txt file that lists the packages you want installed, e.g.

# requirements.txt
spyder-kernels==2.5
openai

and

(my-venv) $ arch -x86_64 python -m pip install -r requirements.txt

@mrclary
Copy link
Contributor

mrclary commented Jan 9, 2024

I'm fine with closing the issue

We cannot close the issue until I get a response from @shubhamg13 regarding comments #21575 (comment) and #21575 (comment)

@ccordoba12
Copy link
Member

@mrclary, is it possible to detect if an external environment is universal or not on Mac? That way I think we could post your recommendations as a message in the console:

What I've learned is that for Spyder (x86_64) to work with external environments on arm64:

  • If the environment Python executable is universal, then all binaries in the Python environment must be x86_64. See below on how to accomplish this.
  • If the environment Python executable is a single architecture, then all the binaries in the Python environment must be the same architecture.

@mrclary
Copy link
Contributor

mrclary commented Jan 10, 2024

@mrclary, is it possible to detect if an external environment is universal or not on Mac? That way I think we could post your recommendations as a message in the console:

What I've learned is that for Spyder (x86_64) to work with external environments on arm64:

  • If the environment Python executable is universal, then all binaries in the Python environment must be x86_64. See below on how to accomplish this.
  • If the environment Python executable is a single architecture, then all the binaries in the Python environment must be the same architecture.

It is possible to detect whether the environment Python executable is x86_64, arm64, or Universal. It is possible to detect whether Spyder is running under Rosetta since platform.version will contain ARM64 (the true architecture), while platform.platform will indicate x86_64.

>>> platform.version()                                            
'Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103'

>>> platform.platform()
'macOS-14.2.1-x86_64-i386-64bit'

I'll reinvestigate building the standalone installer on arm64. As I recall, there was an issue because PyQt5 was not available in arm64. If I can build a native arm64 standalone, I think other issues may be resolved as well (#20201, #20629).

We can discuss possible strategies for moving forward at our next developer meeting.

@ccordoba12 ccordoba12 modified the milestones: v5.5.1, v6.1.0 Jan 21, 2024
@mrclary
Copy link
Contributor

mrclary commented Jan 21, 2024

@tomb1949, I've uploaded an arm64 build of Spyder 5.5.0 to our release page. This is a full standalone version, but built locally on my M1 iMac, using a universal/arm64 Python environment only. You may want to give that a try; this should not require any of the workarounds that I posted earlier.
https://github.com/spyder-ide/spyder/releases/download/v5.5.0/Spyder-arm64.dmg

@tomb1949
Copy link

tomb1949 commented Jan 23, 2024 via email

@tomb1949
Copy link

tomb1949 commented Mar 13, 2024 via email

@mrclary
Copy link
Contributor

mrclary commented Mar 13, 2024

Thanks @tomb1949! Your appreciation is much appreciated 😁

@ccordoba12 ccordoba12 mentioned this issue Mar 14, 2024
Closed
@mrclary
Copy link
Contributor

mrclary commented May 13, 2024

I'm closing this issue due to inactivity. @shubhamg13, if you think this issue is not resolved, please reopen and respond to my comments #21575 (comment) and #21575 (comment).

@mrclary mrclary closed this as completed May 13, 2024
@tomb1949
Copy link

tomb1949 commented May 13, 2024 via email

@ccordoba12 ccordoba12 removed this from the v6.1.0 milestone May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrclary mrclary

Labels
component:IPython Console OS:MacOS type:Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ccordoba12 @mrclary @dalthviz @shubhamg13 @tomb1949