Skip to content

pyenv-virtualenv is very slow on macOS Sequoia #490

Closed
Closed
pyenv-virtualenv is very slow on macOS Sequoia#490

Description

@kozlek
kozlek
opened on Sep 20, 2024

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Prerequisite

  • Make sure your problem is not listed in the common build problems.
  • Make sure no duplicated issue has already been reported in the pyenv-virtualenv issues. You should look in closed issues, too.
  • Make sure you are not asking us to help solving your specific issue.
    • GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
  • Make sure your problem is not derived from packaging (e.g. Homebrew).
    • Please refer to the package documentation for the installation issues, etc.
  • Make sure your problem is not derived from other plugins.
    • This repository is maintaining the pyenv-virtualenv plugin only. Please refrain from reporting issues of other plugins here.

Description

  • Platform information (e.g. Ubuntu Linux 20.04): macOS 15.0 Sequoia
  • OS architecture (e.g. amd64): amd64 (MacbookPro 14,3)
  • pyenv version: v2.4.13
  • pyenv-virtualenv version: v1.2.4
  • Python version: 3.12.6
  • virtualenv version (if installed): none

I was using pyenv & pyenv-virtualenv on macOS 12.x Monterey for 2 years with great success.
Yesterday, I updated to macOS 15.0 Sequoia (clean install).
The shell was very slow with pyenv-virtualenv autoloading enabled.
It takes around 1.2 seconds when hitting a blank line in the shell.

I downgraded to macOS 13.x Ventura (clean install again) and the shell was instant again.

I don't expect you to fix macOS, but I wanted to report my experience with Sequoia to see if others users have the same problem. Apart from pyenv-virtualenv, pyenv + jenv + rbenv were also a bit slower. The rest of the apps were working well. I suspect Apple to have changed something with the terminal / shell...

I'll stick to Ventura for sometimes, and I'll give a try to Sonoma to see.

Activity

  1. kozlek commented on Sep 21, 2024

    @kozlek
    kozlek
    on Sep 21, 2024
    Author

    I did a macOS 14.7 Sonoma clean install on an external SSD and pyenv-virtualenv is behaving normally, without any delays.

    At least on my platform, macOS Sequoia has a huge issue with some shell programs.

    I recommend to postpone the Sequoia upgrade until this is fixed on macOS side.

  2. aphedges commented on Sep 26, 2024

    @aphedges
    aphedges
    on Sep 26, 2024

    I encountered this same problem yesterday on my Intel Mac running macOS 14.7 (Sonoma) when I updated from Xcode 15.3 to Xcode 16. I presume that your fresh install of macOS 15 (Sequoia) had Xcode 16, and I'd be curious to see whether downgrading only Xcode without downgrading the OS would solve the problem. I'm also curious about whether this is a problem only on Intel Macs or if it's a problem on Apple Silicon as well.

    I noticed a variety of pyenv and pyenv-virtualenv commands being much slower, and I was able to figure out that the command pyenv sh-activate --quiet command run in the shell hook was taking most of the time when I hit "Enter" on a blank line. I ran the following command to help test out the problem:

    time env -i bash -cx 'export PYENV_DEBUG=1; command pyenv sh-activate --quiet'

    With Xcode 16, the command takes over 10 s to execute, while with Xcode 15.3, the command takes ~0.3 s to execute.

    I can help with debugging the problem because I can't delay upgrading Xcode indefinitely. However, I have no clue how to profile shell scripts. If anyone has tooling recommendations, I can hopefully figure out exactly where the regression is happening.

  3. native-api commented on Sep 26, 2024

    @native-api
    native-api
    on Sep 26, 2024
    Member

    However, I have no clue how to profile shell scripts.

    #259 (comment):

    Try localizing the issue by timing the code that runs at prompt with:

    export PS4='+($(date "+%s.%N")) (${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -x
<reproduce the problem>
set +x

    Also add the same date invocation to PS4 assignments throughout Pyenv and Pyenv-Virtualenv and set PYENV_DEBUG=1

  4. aphedges commented on Oct 23, 2024

    @aphedges
    aphedges
    on Oct 23, 2024

    I forgot to mention this, but I switched to an Apple silicon laptop a week or two after encountering this issue, and I had no problems using Xcode 16 on it. I think this regression is purely for Intel Mac users.

    Someone else is obviously free to debug this, but I'm not going to spend the time debugging it myself.

  5. andrewaylett commented on Dec 27, 2024

    @andrewaylett
    andrewaylett
    on Dec 27, 2024

    Here's the timed output from my machine:

    $ PS4='+($(date "+%s.%N")) (${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }' time bash -x /usr/local/bin/pyenv sh-activate --quiet
+(1735257413.707286000) (/usr/local/bin/pyenv:2): set -e
+(1735257413.717032000) (/usr/local/bin/pyenv:4): '[' sh-activate = --debug ']'
+(1735257413.729358000) (/usr/local/bin/pyenv:9): '[' -n '' ']'
+(1735257413.741455000) (/usr/local/bin/pyenv:23): enable -f /usr/local/bin/../libexec/pyenv-realpath.dylib realpath
+(1735257413.754514000) (/usr/local/bin/pyenv:30): '[' -z '' ']'
++(1735257413.767650000) (/usr/local/bin/pyenv:32): type -P readlink
+(1735257413.778545000) (/usr/local/bin/pyenv:32): READLINK=/usr/local/opt/coreutils/libexec/gnubin/readlink
+(1735257413.788991000) (/usr/local/bin/pyenv:33): '[' -n /usr/local/opt/coreutils/libexec/gnubin/readlink ']'
+(1735257413.802350000) (/usr/local/bin/pyenv:58): '[' -z /Users/axa/.pyenv ']'
+(1735257413.815300000) (/usr/local/bin/pyenv:61): PYENV_ROOT=/Users/axa/.pyenv
+(1735257413.828289000) (/usr/local/bin/pyenv:63): export PYENV_ROOT
+(1735257413.839098000) (/usr/local/bin/pyenv:65): '[' -z '' ']'
+(1735257413.852871000) (/usr/local/bin/pyenv:66): PYENV_DIR=/Users/axa
+(1735257413.868479000) (/usr/local/bin/pyenv:69): '[' '!' -d /Users/axa ']'
+(1735257413.882083000) (/usr/local/bin/pyenv:69): '[' '!' -e /Users/axa ']'
++(1735257413.897309000) (/usr/local/bin/pyenv:73): cd /Users/axa
++(1735257413.910416000) (/usr/local/bin/pyenv:73): echo /Users/axa
+(1735257413.921637000) (/usr/local/bin/pyenv:73): PYENV_DIR=/Users/axa
+(1735257413.932492000) (/usr/local/bin/pyenv:74): export PYENV_DIR
+(1735257413.943222000) (/usr/local/bin/pyenv:77): shopt -s nullglob
++(1735257413.953742000) (/usr/local/bin/pyenv:79): abs_dirname /usr/local/bin/pyenv
++(1735257413.963952000) (/usr/local/bin/pyenv:40): abs_dirname(): local path=/usr/local/bin/pyenv
++(1735257413.974300000) (/usr/local/bin/pyenv:44): abs_dirname(): '[' -n /usr/local/bin/pyenv ']'
++(1735257413.983151000) (/usr/local/bin/pyenv:45): abs_dirname(): cd_path=/usr/local/bin
++(1735257413.992398000) (/usr/local/bin/pyenv:46): abs_dirname(): [[ /usr/local/bin != \/\u\s\r\/\l\o\c\a\l\/\b\i\n\/\p\y\e\n\v ]]
++(1735257414.001208000) (/usr/local/bin/pyenv:47): abs_dirname(): cd /usr/local/bin
++(1735257414.009919000) (/usr/local/bin/pyenv:49): abs_dirname(): name=pyenv
+++(1735257414.019198000) (/usr/local/bin/pyenv:50): abs_dirname(): resolve_link pyenv
+++(1735257414.027892000) (/usr/local/bin/pyenv:36): resolve_link(): /usr/local/opt/coreutils/libexec/gnubin/readlink pyenv
++(1735257414.044531000) (/usr/local/bin/pyenv:50): abs_dirname(): path=../Cellar/pyenv/2.5.0/bin/pyenv
++(1735257414.053510000) (/usr/local/bin/pyenv:44): abs_dirname(): '[' -n ../Cellar/pyenv/2.5.0/bin/pyenv ']'
++(1735257414.062451000) (/usr/local/bin/pyenv:45): abs_dirname(): cd_path=../Cellar/pyenv/2.5.0/bin
++(1735257414.071110000) (/usr/local/bin/pyenv:46): abs_dirname(): [[ ../Cellar/pyenv/2.5.0/bin != \.\.\/\C\e\l\l\a\r\/\p\y\e\n\v\/\2\.\5\.\0\/\b\i\n\/\p\y\e\n\v ]]
++(1735257414.080631000) (/usr/local/bin/pyenv:47): abs_dirname(): cd ../Cellar/pyenv/2.5.0/bin
++(1735257414.089569000) (/usr/local/bin/pyenv:49): abs_dirname(): name=pyenv
+++(1735257414.101663000) (/usr/local/bin/pyenv:50): abs_dirname(): resolve_link pyenv
+++(1735257414.111082000) (/usr/local/bin/pyenv:36): resolve_link(): /usr/local/opt/coreutils/libexec/gnubin/readlink pyenv
++(1735257414.129154000) (/usr/local/bin/pyenv:50): abs_dirname(): path=../libexec/pyenv
++(1735257414.137597000) (/usr/local/bin/pyenv:44): abs_dirname(): '[' -n ../libexec/pyenv ']'
++(1735257414.146025000) (/usr/local/bin/pyenv:45): abs_dirname(): cd_path=../libexec
++(1735257414.155138000) (/usr/local/bin/pyenv:46): abs_dirname(): [[ ../libexec != \.\.\/\l\i\b\e\x\e\c\/\p\y\e\n\v ]]
++(1735257414.164070000) (/usr/local/bin/pyenv:47): abs_dirname(): cd ../libexec
++(1735257414.172947000) (/usr/local/bin/pyenv:49): abs_dirname(): name=pyenv
+++(1735257414.182324000) (/usr/local/bin/pyenv:50): abs_dirname(): resolve_link pyenv
+++(1735257414.191762000) (/usr/local/bin/pyenv:36): resolve_link(): /usr/local/opt/coreutils/libexec/gnubin/readlink pyenv
+++(1735257414.207854000) (/usr/local/bin/pyenv:50): abs_dirname(): true
++(1735257414.217041000) (/usr/local/bin/pyenv:50): abs_dirname(): path=
++(1735257414.226502000) (/usr/local/bin/pyenv:44): abs_dirname(): '[' -n '' ']'
++(1735257414.235250000) (/usr/local/bin/pyenv:53): abs_dirname(): echo /usr/local/Cellar/pyenv/2.5.0/libexec
+(1735257414.245037000) (/usr/local/bin/pyenv:79): bin_path=/usr/local/Cellar/pyenv/2.5.0/libexec
+(1735257414.255293000) (/usr/local/bin/pyenv:80): for plugin_bin in "${bin_path%/*}"/plugins/*/bin
+(1735257414.263904000) (/usr/local/bin/pyenv:81): PATH=/usr/local/Cellar/pyenv/2.5.0/plugins/python-build/bin:/opt/homebrew/opt/curl/bin:/Users/axa/.asdf/shims:/usr/local/opt/asdf/libexec/bin:/usr/local/Cellar/pyenv-virtualenv/1.2.4/shims:/Users/axa/.pyenv/shims:/Users/axa/.pyenv/bin:/Users/axa/.cargo/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/sbin:/usr/local/bin:/Users/axa/.local/bin:/usr/local/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/axa/Library/Android/sdk/platform-tools
+(1735257414.272779000) (/usr/local/bin/pyenv:85): '[' /usr/local/Cellar/pyenv/2.5.0 '!=' /Users/axa/.pyenv ']'
+(1735257414.282625000) (/usr/local/bin/pyenv:90): export PATH=/usr/local/Cellar/pyenv/2.5.0/libexec:/usr/local/Cellar/pyenv/2.5.0/plugins/python-build/bin:/opt/homebrew/opt/curl/bin:/Users/axa/.asdf/shims:/usr/local/opt/asdf/libexec/bin:/usr/local/Cellar/pyenv-virtualenv/1.2.4/shims:/Users/axa/.pyenv/shims:/Users/axa/.pyenv/bin:/Users/axa/.cargo/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/sbin:/usr/local/bin:/Users/axa/.local/bin:/usr/local/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/axa/Library/Android/sdk/platform-tools
+(1735257414.291685000) (/usr/local/bin/pyenv:90): PATH=/usr/local/Cellar/pyenv/2.5.0/libexec:/usr/local/Cellar/pyenv/2.5.0/plugins/python-build/bin:/opt/homebrew/opt/curl/bin:/Users/axa/.asdf/shims:/usr/local/opt/asdf/libexec/bin:/usr/local/Cellar/pyenv-virtualenv/1.2.4/shims:/Users/axa/.pyenv/shims:/Users/axa/.pyenv/bin:/Users/axa/.cargo/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/sbin:/usr/local/bin:/Users/axa/.local/bin:/usr/local/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/axa/Library/Android/sdk/platform-tools
+(1735257414.300668000) (/usr/local/bin/pyenv:92): PYENV_HOOK_PATH=:/Users/axa/.pyenv/pyenv.d
+(1735257414.309510000) (/usr/local/bin/pyenv:93): '[' /usr/local/Cellar/pyenv/2.5.0 '!=' /Users/axa/.pyenv ']'
+(1735257414.319152000) (/usr/local/bin/pyenv:95): PYENV_HOOK_PATH=:/Users/axa/.pyenv/pyenv.d:/usr/local/Cellar/pyenv/2.5.0/pyenv.d
+(1735257414.328590000) (/usr/local/bin/pyenv:97): PYENV_HOOK_PATH=:/Users/axa/.pyenv/pyenv.d:/usr/local/Cellar/pyenv/2.5.0/pyenv.d:/usr/etc/pyenv.d:/usr/local/etc/pyenv.d:/etc/pyenv.d:/usr/lib/pyenv/hooks
+(1735257414.337478000) (/usr/local/bin/pyenv:101): PYENV_HOOK_PATH=/Users/axa/.pyenv/pyenv.d:/usr/local/Cellar/pyenv/2.5.0/pyenv.d:/usr/etc/pyenv.d:/usr/local/etc/pyenv.d:/etc/pyenv.d:/usr/lib/pyenv/hooks
+(1735257414.347152000) (/usr/local/bin/pyenv:102): export PYENV_HOOK_PATH
+(1735257414.356199000) (/usr/local/bin/pyenv:104): shopt -u nullglob
+(1735257414.365002000) (/usr/local/bin/pyenv:107): command=sh-activate
+(1735257414.374405000) (/usr/local/bin/pyenv:108): case "$command" in
++(1735257414.385512000) (/usr/local/bin/pyenv:121): command -v pyenv-sh-activate
+(1735257414.394983000) (/usr/local/bin/pyenv:121): command_path=/usr/local/bin/pyenv-sh-activate
+(1735257414.403592000) (/usr/local/bin/pyenv:122): '[' -z /usr/local/bin/pyenv-sh-activate ']'
+(1735257414.412640000) (/usr/local/bin/pyenv:130): shift 1
+(1735257414.421487000) (/usr/local/bin/pyenv:131): '[' --quiet = --help ']'
+(1735257414.430203000) (/usr/local/bin/pyenv:138): exec /usr/local/bin/pyenv-sh-activate --quiet
false
        1.05 real         0.12 user         0.20 sys

    Then the call to pyenv-sh-activate:

    $ PATH="/usr/local/Cellar/pyenv/2.5.0/libexec:$PATH" PS4='+($(date "+%s.%N")) (${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }' PYENV_HOOK_PATH=/Users/axa/.pyenv/pyenv.d:/usr/local/Cellar/pyenv/2.5.0/pyenv.d:/usr/etc/pyenv.d:/usr/local/etc/pyenv.d:/etc/pyenv.d:/usr/lib/pyenv/hooks time bash -x /usr/local/bin/pyenv-sh-activate --quiet
+(1735257993.285294000) (/usr/local/bin/pyenv-sh-activate:14): set -e
+(1735257993.295018000) (/usr/local/bin/pyenv-sh-activate:15): '[' -n '' ']'
+(1735257993.305093000) (/usr/local/bin/pyenv-sh-activate:17): '[' -z /Users/axa/.pyenv ']'
+(1735257993.314298000) (/usr/local/bin/pyenv-sh-activate:25): unset FORCE
+(1735257993.323911000) (/usr/local/bin/pyenv-sh-activate:26): unset QUIET
+(1735257993.333091000) (/usr/local/bin/pyenv-sh-activate:31): declare -a before_hooks after_hooks
+(1735257993.342439000) (/usr/local/bin/pyenv-sh-activate:44): OLDIFS='
'
+(1735257993.351376000) (/usr/local/bin/pyenv-sh-activate:45): IFS='
'
+(1735257993.360294000) (/usr/local/bin/pyenv-sh-activate:45): scripts=(`pyenv-hooks activate`)
++(1735257993.369588000) (/usr/local/bin/pyenv-sh-activate:45): pyenv-hooks activate
+(1735257993.398014000) (/usr/local/bin/pyenv-sh-activate:46): IFS='
'
+(1735257993.407585000) (/usr/local/bin/pyenv-sh-activate:49): '[' 1 -gt 0 ']'
+(1735257993.416895000) (/usr/local/bin/pyenv-sh-activate:50): case "$1" in
+(1735257993.425927000) (/usr/local/bin/pyenv-sh-activate:60): QUIET=1
+(1735257993.435898000) (/usr/local/bin/pyenv-sh-activate:73): shift 1
+(1735257993.444574000) (/usr/local/bin/pyenv-sh-activate:49): '[' 0 -gt 0 ']'
+(1735257993.454808000) (/usr/local/bin/pyenv-sh-activate:81): no_shell=
+(1735257993.463386000) (/usr/local/bin/pyenv-sh-activate:82): versions=("$@")
+(1735257993.473687000) (/usr/local/bin/pyenv-sh-activate:83): current_versions=()
+(1735257993.485190000) (/usr/local/bin/pyenv-sh-activate:84): '[' -z '' ']'
+(1735257993.495583000) (/usr/local/bin/pyenv-sh-activate:85): no_shell=1
+(1735257993.506443000) (/usr/local/bin/pyenv-sh-activate:86): get_current_versions
+(1735257993.515385000) (/usr/local/bin/pyenv-sh-activate:77): get_current_versions(): local IFS=:
+(1735257993.524527000) (/usr/local/bin/pyenv-sh-activate:78): get_current_versions(): current_versions=($(pyenv-version-name 2> /dev/null))
++(1735257993.534925000) (/usr/local/bin/pyenv-sh-activate:78): get_current_versions(): pyenv-version-name
+(1735257993.617218000) (/usr/local/bin/pyenv-sh-activate:87): versions=("${current_versions[@]}")
+(1735257993.626618000) (/usr/local/bin/pyenv-sh-activate:90): '[' -z 1 ']'
+(1735257993.635276000) (/usr/local/bin/pyenv-sh-activate:96): venv=system
+(1735257993.644949000) (/usr/local/bin/pyenv-sh-activate:98): '[' -n '' ']'
+(1735257993.654518000) (/usr/local/bin/pyenv-sh-activate:111): pyenv-virtualenv-prefix system
+(1735257993.704050000) (/usr/local/bin/pyenv-sh-activate:113): '[' -n system ']'
+(1735257993.712654000) (/usr/local/bin/pyenv-sh-activate:114): new_venv=system/envs/system
+(1735257993.722326000) (/usr/local/bin/pyenv-sh-activate:115): pyenv-virtualenv-prefix system/envs/system
+(1735257993.916236000) (/usr/local/bin/pyenv-sh-activate:119): '[' -z 1 ']'
+(1735257993.925403000) (/usr/local/bin/pyenv-sh-activate:122): echo false
false
+(1735257993.934596000) (/usr/local/bin/pyenv-sh-activate:123): exit 1
        0.67 real         0.21 user         0.35 sys

    It's a 2019 13-inch MacBook Pro, 2.8GHz i7, Sequoia 15.1.1.

    If I'm not outputting debug information it's faster, but not particularly fast:

    $ time _pyenv_virtualenv_hook

real	0m0.410s
user	0m0.131s
sys	0m0.200s

  6. samdoran commented on Feb 13, 2025

    @samdoran
    samdoran
    on Feb 13, 2025
    Contributor

    I'm not sure if this is related to the new macOS version or not, but I have long noticed that pyenv-virtualenvs is incredibly slow, especially if you have many virtual environments (I currently have 128). Since that is called during pyenv-sh-activate, I imagine that may be part of the problem.

    I've been digging into this and found the call from pyenv-virtualenvs --> pyenv-virtualenv-prefix --> pyenv-prefix to be the source of the slowness.

    I'll put up a PR soon with a fix.

  7. samdoran commented on Feb 13, 2025

    @samdoran
    samdoran
    on Feb 13, 2025
    Contributor

    @kozlek Try out the changes in #502 and see if that helps things any.

  8. native-api commented on Mar 4, 2026

    @native-api
    native-api
    on Mar 4, 2026
    Member

    pyenv/pyenv#3411 has an example of a profiling technique to find hot spots in Bash code

  9. native-api commented on Mar 25, 2026

    @native-api
    native-api
    on Mar 25, 2026
    Member

    #523

  10. native-api
    closed this as completedon Mar 25, 2026

  11. jakelodwick commented on Mar 26, 2026

    @jakelodwick
    jakelodwick
    on Mar 26, 2026
    Contributor

    Fixed in v1.3.0 via #523. The per-prompt activate calls that caused the slowness have been replaced with a stat-based cache. The slowness was the same underlying fork overhead documented in #259.

    @samdoran's pointer to #502 for the pyenv virtualenvs listing cost remains relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions