python环境管理conda和virtualenv

[TOC]

1. conda形态

  • anaconda powershell prompt 命令行
  • anaconda 客户端
  • pycharm 客户端
    路径:interpreter管理-local-conda envs
  • 物理文件夹
    路径:...\anaconda3\envs

2. conda提供服务

  • python环境 - 管理多版本环境
  • python环境 - 管理包
  • python环境 - 管理本地notebook kernel

3. python环境管理

  • 查看环境
    conda env list
  • 创建python环境
    方式一:终端运行 conda create --name test3810; conda activate test3810;conda install python=3.8.10;
    方式二:终端运行 conda create -y --name test3810 python=3.8.10
    方式三:anaconda客户端,点击创建env
    方式四:pycharm客户端,点击添加本地conda环境
  • 问题:激活环境后,有冗余包的解决。*
    排查:执行python -m site,可以看到conda虚拟环境会从C:\Users\cat\AppData\Roaming\Python\Python38\site-packages带入很多包,污染了。这是因为安装时,用了 --user, 这个命令会安装到用户级别的python包环境下,并被conda创建的环境共享。
    解决:激活环境下,把这些包都卸载了。再次安装即可
    验证:在激活环境里再次安装包,只会出现在虚拟环境的包目录下,不会到用户级别环境的包目录。
  • 另一种虚拟环境管理:pycharm+virtualenv管理虚拟环境,也可以提供多套虚拟环境。*
  • pip镜像源
    官方源 https://pypi.org/simple/
    腾讯源 https://mirrors.cloud.tencent.com/pypi/simple
    阿里源 https://mirrors.aliyun.com/pypi/simple/
    豆瓣 https://pypi.doubanio.com/simple/
    网易源 https://mirrors.163.com/pypi/simple/
    科大源 https://pypi.mirrors.ustc.edu.cn/simple/
    清华源 https://pypi.tuna.tsinghua.edu.cn/simple/
  • 包安装卸载
    查看包源构成:python -m site
    查看包信息:pip show setuptools
    升级pip包:python -m pip install pip==23.0
    升级setup包:python -m pip install setuptools==41.6.0
    pip安装包(安装到python环境):pip install ffn
    pip安装包(安装到用户共享python环境):pip install ffn --user
    pip安装包(从指定的镜像源安装): pip install ffn -i https://pypi.tuna.tsinghua.edu.cn/simple/
    pip安装wheel包:pip install resources/whl/TA_Lib-0.4.19-cp37-cp37m-win_amd64.whl
    pip安装tar包:pip install resources/tar/TA_Lib-0.4.19-cp37-cp37m-win_amd64.tar
    pip安装git源码:pip install git+https://github.com/bollwyvl/nb-mermaid.git@103502e6
    conda安装包:conda install -c conda-forge ffn
    卸载失败的解决事例
    (quant3810) PS C:\Users\cat> pip uninstall -y pyzmq
    WARNING: No metadata found in d:\codepython\anaconda3\envs\quant3810\lib\site-packages
    Found existing installation: pyzmq 25.1.0
    ERROR: Cannot uninstall pyzmq 25.1.0, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps pyzmq==25.1.0'.
    (quant3810) PS C:\Users\cat> pip install --force-reinstall --no-deps --ignore-installed pyzmq==25.0.2
    >>
    Looking in indexes: https://mirrors.ustc.edu.cn/pypi/web/simple
    Collecting pyzmq==25.0.2
      Using cached https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/04/f1/feaf5d091173036ddbebbe0657453b122de167fd055e8d94a76aff862e99/pyzmq-25.0.2-cp38-cp38-win_amd64.whl (1.1 MB)
    Installing collected packages: pyzmq
    Successfully installed pyzmq-25.0.2
    (quant3810) PS C:\Users\cat> pip uninstall -y pyzmq
    Found existing installation: pyzmq 25.0.2
    Uninstalling pyzmq-25.0.2:
      Successfully uninstalled pyzmq-25.0.2
  • 清理缓存
    pip清除缓存: pip cache purge
    conda清除全部缓存: conda clean -a
    conda清除包缓存: conda clean -p
    conda清除tar包缓存: conda clean -t
  • 环境控制
    查询变更记录:conda list --revisions
    回滚:conda install --revision REV_NUM
    激活环境:conda activate test3810
    退出环境:conda deactivate
    删除环境:conda remove --all -y -n test3810
    会删除环境物理文件,并从conda环境列表自动移除;而pycharm环境列表需要手动移除

4. notebook kernel 管理

  • 查看kernel
    方式一:notebook选择kernel下拉列表
    方式二:jupyter kernelspec list
  • 创建kernel
    conda install ipykernel
    python -m ipykernel install --user --name test3810 --display-name Python (test3810)"
  • 启动notebook服务
    pycharm终端:jupyter notebook --port 8889
    开始菜单:Jupyter Notebook (quant3810).lnk
  • 删除kernel
    方式一:jupyter kernelspec uninstall test3810
    方式二:python -m ipykernel uninstall --name test3810

5. virtualenv管理多环境

  • 存在形态
    pycharm客户端:interpreter-local-virtualenv
    物理文件夹
  • 查看环境列表
    在硬盘目录下,无激活概念,pycharm选择使用后,可进激活环境的终端
  • 创建venv
    直接用pycharm客户端,创建local virtualenv
  • 删除env
    删除文件夹和pycharm的环境项

6. 任何环境的重置脚本

#!/usr/bin/env python3
# coding=utf-8
"""
@File : utils_reset_python_env.py
@Author  : AT
@Create  : 2023/6/16 15:52
@Desc    : 切换到需要重置清空的python环境,运行main方法或本文件即可
"""
import sys
import platform
import pkg_resources
import subprocess

def reset_packages_init(_init_list=['pip', 'setuptools', 'wheel']):
    _print_python_interpreter()
    _exist_list = _get_packages()
    print(f"当前环境:初始化为: ({' '.join(_init_list)})")
    _remove_list = list(set(_exist_list) - set(_init_list))
    if _remove_list:
        print(f"当前环境:pip uninstall -y {' '.join(_remove_list)}")
    user_input = input("当前环境:pip uninstall continue: 0-No(默认), 1-Yes")
    print(f'当前环境:pip uninstall continue:: 你的选择 = {user_input}')
    if user_input == '1':
        for _package in _remove_list:
            _uninstall_package(_package)
        _get_packages()
    sys.exit()


def _print_python_interpreter():
    print(f"当前环境:python版本={platform.python_version()}, 解释器路径={sys.executable})")

def _get_packages():
    # 获取当前 Python 系统中所有已安装包的列表
    installed_packages = pkg_resources.WorkingSet()
    installed_packages_name = [dist.project_name for dist in installed_packages]
    print(f"当前环境:installed_packages=({' '.join(installed_packages_name)})")
    return installed_packages_name

def _uninstall_package(package):
    try:
        subprocess.run(['pip', 'uninstall', '-y', package])
    except Exception as e:
        print(f"卸载:{package} 失败!An error occurred: {e}")

if __name__ == '__main__':
    reset_packages_init()

运行上面代码的演示日志如下

当前环境:python版本=3.6.9, 解释器路径=D:\codePython\python\venv\Y7.2.8_terminal_369_0\Scripts\python.exe)
当前环境:installed_packages=(pytz setuptools pip python-dateutil numpy six pandas wheel)
当前环境:初始化为: (pip setuptools wheel)
当前环境:pip uninstall -y python-dateutil pytz six pandas numpy
Found existing installation: python-dateutil 2.8.2
Uninstalling python-dateutil-2.8.2:
  Successfully uninstalled python-dateutil-2.8.2
Found existing installation: pytz 2023.3
Uninstalling pytz-2023.3:
  Successfully uninstalled pytz-2023.3
Found existing installation: six 1.16.0
Uninstalling six-1.16.0:
  Successfully uninstalled six-1.16.0
Found existing installation: pandas 1.1.5
Uninstalling pandas-1.1.5:
  Successfully uninstalled pandas-1.1.5
Found existing installation: numpy 1.19.5
Uninstalling numpy-1.19.5:
  Successfully uninstalled numpy-1.19.5
当前环境:installed_packages=(setuptools pip wheel)