A Morlock's Guide to Lua, Part 2: Imports, Modules, and Code Organization
Modules are tables of functions, PLUS: Everything you didn't know you wanted to know about how Lua finds files when you run "require"
Greetings, Morlocks, you weird little people who keep the machines running — with Neovim! I’m Nat Bennett, your morlock host, and you’re reading Mastering Neovim, a newsletter about getting the absolute best performance out the text editor that makes sense, dang it.
This is the second entry in a three-part series on using Lua in Neovim. If you’re just joining us, check out last week’s issue on running Neovim for instructions on how to set up an environment where you can test out the things we’ll be talking about today — modules, imports, and code organization.
Part 2: Imports, modules, and code organization
Part 3: Using Lua to configure NeoVim
Maybe I’m weird but when I’m learning a new language the part I’m most interested in isn’t syntax — that’s usually pretty easy. It’s getting programs to actually run — building and configuring them. And that usually also requires getting a solid handle on how the language handles imports and dependencies.
Lua imports code with the “require” keyword
Lua has a few ways to bring in code, but the one you’ll encounter most in Neovim is require. You might remember this friendly little function from the lspconfig plugin.
-- Setup language servers.
local lspconfig = require('lspconfig')
lspconfig.pyright.setup {}
lspconfig.tsserver.setup {}This tells us two important things about require in Neovim.
One is that you have to be explicit about importing Lua functions. Just installing a plugin won’t make an API available.
The second is that require returns something that we can bind to a local value. That “something” is typically a table of functions. When you then write code like
lspconfig.pyright.setup {}you’re calling functions from that table.
“Require” does three things
Require is basically just “execute this Lua script” with a couple of extra steps.
First, it finds the file. You don’t need to know the absolute path to the file on the system it’s running in.
Second, it checks that the file hasn’t already been loaded. You can write “require” as many times as you want in a Lua script, and the script it refers to will only run once.
Finally, it runs the code, in what Lua calls a “chunk.”
Modules are tables of functions
As far as I can tell “module” is the term of art in Lua for “a function loaded from a file that returns a table of functions.” There’s no “def module” or anything similar in the language itself.
The Lua community and Lua tools also sometimes refer to this mechanism of code organization as a package, especially when distributing code. From the docs on Lua.org:
Lua does not provide any explicit mechanism for packages. However, we can implement them easily with the basic mechanisms that the language provides. The main idea is to represent each package by a table, as the basic libraries do.
(Technically you can also register functions into the table on the top of the stack, so it’s possible for require(“bees”) to create a table named bees without explicitly assigning the return value of the require, but I think this is something that only C packages can do. Not 100% sure about that though — if you know, write in.)
Put Lua modules under ~/.config/nvim/lua
The full description of how “require” finds files is complicated and involves the phrase “cartesian product,” but the default behavior is — okay I’m not going to say simple because that always gets us into trouble, but, look—
If you write:
require("bees")Nvim will first try to load ~/.config/nvim/lua/bees.lua
And if that file doesn’t exist, it tries to load ~/.config/nvim/lua/bees/init.lua
So those are the two places you should probably put any custom Lua code you write yourself, at least if you’re not bundling it up as a proper Nvim plugin. If it’s a single file just put it straight in the /lua directory, and if it’s more than file, put them all in a named directory and create an entry point named init.lua.
How does “require” find files?
Especially if you use a plugin manager you may end up needing to figure out where code other places is coming from. I got majorly nerd-sniped by figuring this out in detail and I think I’m pretty close but there are still a couple of things I don’t understand.
The main thing that’s complicated here is that the search process takes several passes, and the list of things to search are built up in a couple of similar-but-slighty-different ways.
First, require first checks the directories in runtimepath, with lua appended to the directory. It checks for lua/modulename.lua first, and then for lua/modulename/init.lua. If it doesn’t find anything that matches, it then falls back to normal Lua resolution.
Then, it checks the same locations for files with suffixes from Lua’s package.cpath. This is usually .so and .dll. (This probably won’t come up very often, but apparently Lua is pretty good at running C code. For a simple example of what a Lua C module looks like, take a look at norman/hello-lua.)
If it doesn’t find anything there, either, it falls back to Lua’s normal package finding mechanism, which is based on the values of package.path.
An example of ordering
The Neovim Lua docs have an ordering example but it uses foo and bar in a way that I personally found really confusing. So here’s another slightly more realistic one in case that also helps you grok this stuff.
Say your runtimepath looks like this:
~/.config/nvim~/.local/share/nvim/lazy/nvim-lspconfig
And your package.path looks like this:
./?.lua/usr/local/share/lua/5.1/?.lua/usr/local/share/lua/5.1/?/init.lua
And finally your package.cpath looks like this
./?.so
Then when you run require(“honeycomb”) require will search for and load modules in this order:
~/.config/nvim/lua/honeycomb.lua
~/.config/nvim/honeycomb/init.lua
~/.local/share/nvim/lazy/nvim-lspconfig/lua/honeycomb.lua
~/.local/share/nvim/lazy/nvim-lspconfig/lua/honeycomb/init.lua
~/.config/nvim/lua/honeycomb.so
~/.local/share/nvim/lazy/nvim-lspconfig/lua/honeycomb.so
./honeycomb.lua
/usr/local/share/lua/5.1/honeycomb.lua
/usr/local/share/lua/5.1/honeycomb/init.luaLua’s path values are patterns, not directories
It’s worth taking a look at your Lua path right now with
:lua =package.pathIf you haven’t done anything tricky to your configuration it should look something like this.
./?.lua;
/usr/local/share/luajit-2.1.0-beta3/?.lua;
/usr/local/share/lua/5.1/?.lua;
/usr/local/share/lua/5.1/?/init.lua(I added spaces after the semicolons just to make it a little easier to read.)
The main thing I want to draw your attention to is that these are patterns, not directories, and that ? is a Lua-specific wildcard character. This is fundamentally how require is doing that “check for a file named ?.lua, then for a init.lua in a directory named ?” thing.
The other thing that I got caught up on here was that first entry, which checks the current directory. I got very confused about how all this worked for an hour or so because I had a Lua file in a directory that the runtimepath based mechanism couldn’t find, but I thought it was finding it, because I happened to be starting Nvim from the directory that the file was in. So it was falling back to this value and tricking me.
This is otherwise a pretty convenient behavior though, because it means that if you’re just doing a quick bit of Lua hacking, you can just start Nvim in the directory your file is in and require it up.
The part that still confuses me
There’s one thing that I still don’t understand and I’m hoping one of ya’ll will know: How does Neovim intercept Lua’s “normal” package finding behavior and get it to look for things on runtimepath first?
The documentation here says
Nvim automatically adjusts package.path and package.cpath according to the effective 'runtimepath' value.
and at first I assumed that this meant that it was changing package.path. But as you can see above the value of this on my current setup doesn’t have any values from runtimepath in it!
This does kind of make sense, because building the “real” package.path based on the value of both it and runtimepath at the moment that require actually runs would be a lot cleaner than trying to change package.path every time runtimepath changes. But… it does bug me that I don’t have a way to get the actual list of the places that Neovim’s Lua will actually search when I run require. And it makes me think I don’t actually understand this as well as I think I do. So, I may do more experiments in the future.
In summary
You bring in modules to a Lua program with require and then assign the return value to a local variable
Modules are functions that return tables of functions
How require finds functions is a little bit complicated but you can probably assume code comes from
~/.config/nvim/lua/
Write in!
Thoughts? Questions? Corrections? Leave a comment or e-mail me.
Are you interested in reading about “motions” — Vim key commands for moving the cursor around and editing text? Right now the outline for this newsletter is very heavy on configuration and code navigation and doesn’t have much about editing within a file. So if that’s something you’re interested in let me know.