Elixir Metaprogramming Basics

Metaprogramming is a powerful, yet pretty complex technique, that means a program can analyze or even modify itself during runtime. Many modern languages support this feature, and Elixir is no exception.

With metaprogramming, you may create new complex macros, dynamically define and defer code execution, which allows you to write more concise and powerful code. This is indeed an advanced topic, but hopefully after reading this article you will get a basic understanding of how to get started with metaprogramming in Elixir.

In this article you will learn:

What the abstract syntax tree is and how Elixir code is represented under the hood.
What the quote and unquote functions are.
What macros are and how to work with them.
How to inject values with binding.
Why macros are hygienic.

Before starting, however, let me give you a small piece of advice. Remember Spider Man's uncle said "With great power comes great responsibility"? This can be applied to metaprogramming as well because this is a very powerful feature that allows you to twist and bend code to your will.

Still, you must not abuse it, and you should stick to simpler solutions when it is sane and possible. Too much metaprogramming may make your code much harder to understand and maintain, so be careful about it.

Abstract Syntax Tree and Quote

The first thing we need to understand is how our Elixir code is actually represented. These representations are often called Abstract Syntax Trees (AST), but the official Elixir guide recommends calling them simply quoted expressions.

It appears that expressions come in the form of tuples with three elements. But how can we prove that? Well, there is a function called quote that returns a representation for some given code. Basically, it makes the code turn into an unevaluated form. For example:

1	quote do
2	1 + 2
3	end # => {:+, [context: Elixir, import: Kernel], [1, 2]}

So what's going on here? The tuple returned by the quote function always has the following three elements:

Atom or another tuple with the same representation. In this case, it is an atom :+, meaning we are performing addition. By the way, this form of writing operations should be familiar if you have come from the Ruby world.
Keyword list with metadata. In this example we see that the Kernel module was imported for us automatically.
List of arguments or an atom. In this case, this is a list with the arguments 1 and 2.

The representation may be much more complex, of course:

1	quote do
2	Enum.each([1,2,3], &(IO.puts(&1)))
3	end # => {{:., [], [{:__aliases__, [alias: false], [:Enum]}, :each]}, [],
4	# [[1, 2, 3],
5	# {:&, [],
6	# [{{:., [], [{:__aliases__, [alias: false], [:IO]}, :puts]}, [],
7	# [{:&, [], [1]}]}]}]}

On the other hand, some literals return themselves when quoted, specifically:

atoms
integers
floats
lists
strings
tuples (but only with two elements!)

In the next example, we can see that quoting an atom returns this atom back:

1	quote do
2	:hi
3	end # => :hi

Now that we know how the code is represented under the hood, let's proceed to the next section and see what macros are and why quoted expressions are important.

Macros

Macros are special forms like functions, but the ones that return quoted code. This code is then placed into the application, and its execution is deferred. What's interesting is that macros also do not evaluate the parameters passed to them—they are represented as quoted expressions as well. Macros can be used to create custom, complex functions used throughout your project.

Bear in mind, however, that macros are more complex than regular functions, and the official guide states that they should be used only as the last resort. In other words, if you can employ a function, do not create a macro because this way your code becomes needlessly complex and, effectively, harder to maintain. Still, macros do have their use cases, so let's see how to create one.

It all starts with the defmacro call (which is actually a macro itself):

defmodule MyLib do
  defmacro test(arg) do
    arg |> IO.inspect
  end
end

This macro simply accepts an argument and prints it out.

Also, it's worth mentioning that macros can be private, just like functions. Private macros can be called only from the module where they were defined. To define such a macro, use defmacrop.

Now let's create a separate module that will be used as our playground:

defmodule Main do
  require MyLib

  def start! do
    MyLib.test({1,2,3})
  end
end

Main.start!

When you run this code, {:{}, [line: 11], [1, 2, 3]} will be printed out, which indeed means that the argument has a quoted (unevaluated) form. Before proceeding, however, let me make a small note.

Require

Why in the world did we create two separate modules: one to define a macro and another one to run the sample code? It appears that we have to do it this way, because macros are processed before the program is executed. We also must ensure that the defined macro is available in the module, and this is done with the help of require. This function, basically, makes sure that the given module is compiled before the current one.

You might ask, why can't we get rid of the Main module? Let's try doing this:

defmodule MyLib do
  defmacro test(arg) do
    arg |> IO.inspect
  end
end


MyLib.test({1,2,3})

# => ** (UndefinedFunctionError) function MyLib.test/1 is undefined or private. However there is a macro with the same name and arity. Be sure to require MyLib if you intend to invoke this macro
#    MyLib.test({1, 2, 3})
#    (elixir) lib/code.ex:376: Code.require_file/2

Unfortunately, we get an error saying that the function test cannot be found, though there is a macro with the same name. This happens because the MyLib module is defined in the same scope (and the same file) where we are trying to use it. It may seem a bit strange, but for now just remember that a separate module should be created to avoid such situations.

Also note that macros cannot be used globally: first you must import or require the corresponding module.

Macros and Quoted Expressions

So we know how Elixir expressions are represented internally and what macros are... Now what? Well, now we can utilize this knowledge and see how the quoted code can be evaluated.

Let's return to our macros. It is important to know that the last expression of any macro is expected to be a quoted code which will be executed and returned automatically when the macro is called. We can rewrite the example from the previous section by moving IO.inspect to the Main module:

defmodule MyLib do
  defmacro test(arg) do
    arg
  end
end

defmodule Main do
  require MyLib

  def start! do
    MyLib.test({1,2,3}) |> IO.inspect
  end
end

Main.start! # => {1, 2, 3}

See what happens? The tuple returned by the macro is not quoted but evaluated! You may try adding two integers:

1	MyLib.test(1 + 2) \|> IO.inspect # => 3

Once again, the code was executed, and 3 was returned. We can even try to use the quote function directly, and the last line will still be evaluated:

defmodule MyLib do
  defmacro test(arg) do
    arg |> IO.inspect
    quote do
      {1,2,3}
    end
  end
end

# ...

def start! do
    MyLib.test(1 + 2) |> IO.inspect
    # => {:+, [line: 14], [1, 2]}
    # {1, 2, 3}
end

The arg was quoted (note, by the way, that we can even see the line number where the macro was called), but the quoted expression with the tuple {1,2,3} was evaluated for us as this is the last line of the macro.

We may be tempted to try using the arg in a mathematical expression:

  defmacro test(arg) do
    quote do
      arg + 1
    end
  end

But this will raise an error saying that arg does not exist. Why so? This is because arg is literally inserted into the string that we quote. But what we'd like to do instead is evaluate the arg, insert the result into the string, and then perform the quoting. To do this, we will need yet another function called unquote.

Unquoting the Code

unquote is a function that injects the result of code evaluation inside the code that will be then quoted. This may sound a bit bizarre, but in reality things are quite simple. Let's tweak the previous code example:

  defmacro test(arg) do
    quote do
      unquote(arg) + 1
    end
  end

Now our program is going to return 4, which is exactly what we wanted! What happens is that the code passed to the unquote function is run only when the quoted code is executed, not when it is initially parsed.

Let's see a slightly more complex example. Suppose we'd like to create a function which runs some expression if the given string is a palindrome. We could write something like this:

1	def if_palindrome_f?(str, expr) do
2	if str == String.reverse(str), do: expr
3	end

The _f suffix here means that this is a function as later we will create a similar macro. However, if we try to run this function now, the text will be printed out even though the string is not a palindrome:

1	def start! do
2	MyLib.if_palindrome_f?("745", IO.puts("yes")) # => "yes"
3	end

The arguments passed to the function are evaluated before the function is actually called, so we see the "yes" string printed out to the screen. This is indeed not what we want to achieve, so let's try using a macro instead:

  defmacro if_palindrome?(str, expr) do
    quote do
      if(unquote(str) == String.reverse( unquote(str) )) do
        unquote(expr)
      end
    end
  end
  
  # ...
  
  MyLib.if_palindrome?("745", IO.puts("yes"))

Here we are quoting the code containing the if condition and use unquote inside to evaluate the values of the arguments when the macro is actually called. In this example, nothing will be printed out to the screen, which is correct!

Injecting Values With Bindings

Using unquote is not the only way to inject code into a quoted block. We can also utilize a feature called binding. Actually, this is simply an option passed to the quote function that accepts a keyword list with all the variables that should be unquoted only once.

To perform binding, pass bind_quoted to the quote function like this:

1	quote bind_quoted: [expr: expr] do
2	end

This can come in handy when you want the expression used in multiple places to be evaluated only once. As demonstrated by this example, we can create a simple macro that outputs a string twice with a delay of two seconds:

defmodule MyLib do
  defmacro test(arg) do
    quote bind_quoted: [arg: arg] do
      arg |> IO.inspect
      Process.sleep 2000
      arg |> IO.inspect
    end
  end
end

Now, if you call it by passing system time, the two lines will have the same result:

1	:os.system_time \|> MyLib.test
2	# => 1547457831862272
3	# => 1547457831862272

This is not the case with unquote, because the argument will be evaluated twice with a small delay, so the results are not the same:

  defmacro test(arg) do
    quote do
      unquote(arg) |> IO.inspect
      Process.sleep(2000)
      unquote(arg) |> IO.inspect
    end
  end
  
  # ...
  def start! do
    :os.system_time |> MyLib.test
    # => 1547457934011392
    # => 1547457936059392
  end

Converting Quoted Code

Sometimes, you may want to understand what your quoted code actually looks like in order to debug it, for example. This can be done by using the to_string function:

  defmacro if_palindrome?(str, expr) do
    quoted = quote do
      if(unquote(str) == String.reverse( unquote(str) )) do
        unquote(expr)
      end
    end

    quoted |> Macro.to_string |> IO.inspect
    quoted
  end

The printed string will be:

1	"if(\"745\" == String.reverse(\"745\")) do\n IO.puts(\"yes\")\nend"

We can see that the given str argument was evaluated, and the result was inserted right into the code. \n here means "new line".

Also, we can expand the quoted code using expand_once and expand:

  def start! do
    quoted = quote do
      MyLib.if_palindrome?("745", IO.puts("yes"))
    end
    quoted |> Macro.expand_once(__ENV__) |> IO.inspect
  end

Which produces:

1	{:if, [context: MyLib, import: Kernel],
2	[{:==, [context: MyLib, import: Kernel],
3	["745",
4	{{:., [],
5	[{:__aliases__, [alias: false, counter: -576460752303423103], [:String]},
6	:reverse]}, [], ["745"]}]},
7	[do: {{:., [],
8	[{:__aliases__, [alias: false, counter: -576460752303423103], [:IO]},
9	:puts]}, [], ["yes"]}]]}

Of course, this quoted representation can be turned back to a string:

1	quoted \|> Macro.expand_once(__ENV__) \|> Macro.to_string \|> IO.inspect

We will get the same result as before:

1	"if(\"745\" == String.reverse(\"745\")) do\n IO.puts(\"yes\")\nend"

The expand function is more complex as it tries to expand every macro in a given code:

1	quoted \|> Macro.expand(__ENV__) \|> Macro.to_string \|> IO.inspect

The result will be:

1	"case(\"745\" == String.reverse(\"745\")) do\n x when x in [false, nil] ->\n nil\n _ ->\n
2	IO.puts(\"yes\")\nend"

We see this output because if is actually a macro itself that relies on the case statement, so it gets expanded too.

In these examples, __ENV__ is a special form that returns environment information like the current module, file, line, variable in the current scope, and imports.

Macros Are Hygienic

You may have heard that macros are actually hygienic. What this means is they do not overwrite any variables outside of their scope. To prove it, let's add a sample variable, try changing its value in various places, and then output it:

  defmacro if_palindrome?(str, expr) do
    other_var = "if_palindrome?"
    quoted = quote do
      other_var = "quoted"
      if(unquote(str) == String.reverse( unquote(str) )) do
        unquote(expr)
      end
      other_var |> IO.inspect
    end
    other_var |> IO.inspect

    quoted
  end
  
  # ...
  
  def start! do
    other_var = "start!"
    MyLib.if_palindrome?("745", IO.puts("yes"))
    other_var |> IO.inspect
  end

So other_var was given a value inside the start! function, inside the macro, and inside the quote. You will see the following output:

1	"if_palindrome?"
2	"quoted"
3	"start!"

This means that our variables are independent, and we're not introducing any conflicts by using the same name everywhere (though, of course, it would be better to stay away from such an approach).

If you really need to change the outside variable from within a macro, you may utilize var! like this:

  defmacro if_palindrome?(str, expr) do
    quoted = quote do
      var!(other_var) = "quoted"
      if(unquote(str) == String.reverse( unquote(str) )) do
        unquote(expr)
      end
    end

    quoted
  end
  
  # ...
  
  def start! do
    other_var = "start!"
    MyLib.if_palindrome?("745", IO.puts("yes"))
    other_var |> IO.inspect # => "quoted"
  end

By using var!, we are effectively saying that the given variable should not be hygienized. Be very careful about using this approach, however, as you may lose track of what is being overwritten where.

Conclusion

In this article, we have discussed metaprogramming basics in the Elixir language. We have covered the usage of quote, unquote, macros and bindings while seeing some examples and use cases. At this point, you are ready to apply this knowledge in practice and create more concise and powerful programs. Remember, however, that it is usually better to have understandable code than concise code, so do not overuse metaprogramming in your projects.

If you'd like to learn more about the features I've described, feel free to read the official Getting Started guide about macros, quote and unquote. I really hope this article gave you a nice introduction to metaprogramming in Elixir, which can indeed seem quite complex at first. At any rate, don't be afraid to experiment with these new tools!

I thank you for staying with me, and see you soon.