Sep 9

Vim is not an IDE, episode three thousand

Update (2020-01-12): The sane approach to all of this is not to have a Vim-specific Python which contains the virtualenv at all. Bram Moolenar's recent experimental work on "Vim 9" eliminates (I think) the Python interpreter in Vim in favour of running Python out-of-process, which seems sensible -- though I wish he would make the interface a proper API instead, rather than a way to invoke Vimscript -- then he could make the next obvious improvement and eliminate Vimscript!

Writing Python in Vim has become more fun, or at least more interesting, in the last few years with the preponderance of "opinionated" (which means not configurable) Python checkers and linters, and packages like ale for Vim. However, Python and Vim are certainly not opinionated and there are hundreds of ways to structure your projects and workflow. This freedom is wonderful and amazing but every so often it means an hour or two of digging to find out why something's not working properly.

Let's consider the case of running a single linter, Pylint, inside Vim. Typically I'm working on more than one project at once -- often a library and a user of that library, or perhaps a client and a server. This means that there may be a couple of virtualenvs involved for different projects, or possibly a combination of one project with a virtualenv and one project without.

Under ale, pylint runs as a separate process, and ale's python integration decides which one to run like this:

  • First, it looks for a virtualenv, which is a directory called "virtualenv", "venv" or a couple of other options in the directory of the file being edited.
  • If it doesn't find one, it moves up one directory and checks again, repeating this process on failure until it gets to the root.
  • If it still hasn't found a virtualenv, it looks at the VIRTUAL_ENV environment variable and uses that.
  • If that environment variable isn't set, it uses whatever first matches "pylint" (by default) in the PATH.

If all the files you're editing use virtualenvs, and you're not doing something like editing files on an exported fileshare from a virtual machine running a different operating system, which unfortunately I do quite often, this works fine.

If, however, you happened to be inside a virtualenv when you started vim, the VIRTUAL_ENV environment variable will be set (and the PATH will be modified to put virtualenv/bin first). This means, following ale's rules above, that any file which doesn't have its own virtualenv will end up using the virtualenv of whatever was active when you started.

This becomes problematic when you're using project-specific Pylint extensions, such as pylint_django, because pylint will fail to start if it can't load the extension, and to decide whether to use the extension or not basically involves replicating ale's virtualenv-searching process and looking for both pylint and pylint_django.

Here's how I did that, in ~/.vim/after/ftplugin/python.vim:

python3 <<EOF
import vim
import imp
import os
import glob

def find_virtualenv(virtualenv_names):
    cwd = vim.eval('getcwd()')
    while cwd != '/':
        for virtualenv_name in virtualenv_names:
            venv_path = os.path.join(cwd, virtualenv_name)
            if os.path.exists(venv_path):
                return venv_path

        cwd, _ignored = os.path.split(cwd)

    return os.environ.get('VIRTUAL_ENV')

# If we have a virtualenv, check to see whether it contains pylint and the
# django module. If we don't, just try to import both.  We can't even use
# ale's "ale_virtualenv_dir_names" here, because it's not set yet.

virtualenv_path = find_virtualenv(['virtualenv', 'venv'])  #vim.eval('ale_virtualenv_dir_names')

if virtualenv_path:
    has_pylint_django = glob.glob(os.path.join(virtualenv_path, 'lib/*/site-packages/pylint_django'))
        has_pylint_django = True
    except ImportError:
        has_pylint_django = False

if has_pylint_django:
    vim.command("let b:ale_python_pylint_options = '--load-plugins pylint_django'")

(Note that this will be using a possibly-completely-different Python, i.e. the one that Vim was compiled with.)

Using VIRTUAL_ENV makes sense, though using it as a last-resort fallback doesn't: it should probably be the first option picked. One sane way of using Vim would be to have a single off-path virtualenv for all the files you're editing. But that's not the way I work, so after some pain I eventually decided that the best approach is to ignore any virtualenvs in the environment, and just look for them on the path. Active virtualenvs set VIRTUAL_ENV and modify the path, so I added the following to my ~/.vimrc:

" Remove any virtualenv-specific directories from the PATH, and remove the
" VIRTUAL_ENV environment variable if it's set.
" The problem is that running vim from the command line pulls in the
" VIRTUAL_ENV environment variable if it's set, which is not usually helpful
" -- it means that Python modules without a virtualenv will end up using the
" one that happened to be active when I started vim. 
python3 <<EOF
import os
import vim
virtualenv_dir = os.environ.get('VIRTUAL_ENV')
if virtualenv_dir:
    newpath = ':'.join(elem for elem in os.environ.get('PATH').split(':') if not elem.startswith(virtualenv_dir))

    vim.command("let $PATH='%s'" % (newpath,))
    vim.command("let $VIRTUAL_ENV=''")

After all of this, which works and produces a somewhat-sane-feeling editing environment, I'm still not convinced I'm actually doing the right thing. IDEs, being project-based, have a much easier time. Basically it feels like I'm at the stage where I need to decide whether I'm actually creating a nice environment for editing, or whether I should give it up and move to VS Code.