`callPackage`

CAVEAT LECTOR
I’m not a Nix(OS|pkgs)? expert, and none of this has been reviewed by anyone who is. Please consider creating an issue or pull request if you have any suggestions or corrections. Thank you!

Officially, callPackage is only indirectly documented in the Nixpkgs manual, but there are efforts to get this rectified.⁰ The closest to an official documentation is the Package parameters and overrides with callPackage article on nix.dev.

^{[0]: See nix.dev issue #651, Nixpkgs issue #36354, and Nixpkgs PR #270696.}

Below are just my notes:

0. Executive summary

callPackage f attr_set will

call function f by automatically populating its input attribute set
^{from attribute set attr_set and Nixpkgs ( which contains all the available Nix packages, convenience functions, etc.)}
return an overridable attribute set

1. Context

1.1 General usage

callPackage is used overwhelmingly on functions that produce derivations (a build recipe for a Nix package). Creating a Nix package can require a lot of dependencies that won’t have to be supplied manually to the function this way.

“DEFINITION”
Nix package can mean anything digital that requires assembly from one or more sources. A Nix package can be a software, a dataset, a document, a piece of configuration, etc. (The term “package” is quite overloaded, so it may mean something else entirely in other contexts.)

“DEFINITION”
A derivation is a “build recipe” for a Nix package. The term “derivation” can either refer to a “store derivation” or a “derivation attribute set”, depending on the context. They can be used interchangeably most of the time, but there is a fundamental difference between the two: a “store derivation” is produced from a “derivation attribute set” during “instantiation”, and the former is a Nix expression while the latter is “stored on-disk in ATerm format” (from Nix manual).

“DEFINITION”
store derivation TODO finish this. Meanwhile, here’s the store derivation (and its prettified counterpart) of hello.nix, produced with callPackage ./hello.nix {}.

“DEFINITION”
derivation attribute set TODO finish this. Meanwhile, here’s the derivation attribute set (~95k lines, so here’s the short version) of hello.nix, produced with callPackage ./hello.nix {}.

1.2 A common example use case from Nixpkgs

An important part of Nixpkgs is pkgs/top-level/all-packages.nix, where all the available packages are listed. When the Nixpkgs repo (which is “just” a huge Nix expression) is imported, all these packages (and some extra attributes) will be composed into one huge attribute set¹.

^{[1]: … in here in pkgs/top-level/stage.nix. See comment at the top.}

The majority of its ~40k lines either are or boil down to a callPackage call, such as this:

godot_4 = callPackage ../development/tools/godot/4 { };

which is equivalent to

godot_4_nix = import ../development/tools/godot/4/default.nix;

godot_4 = callPackage ┌godot_4_nix┐ ┌{         }┐ ;
#                     │           │ │           │
#                     └─ FUNCTION ┘ └ ATTRIBUTE ┘
#                         ▲    ▲         SET
#                         │    │          │
#      "NIXPKGS"² ────────┘ //³└──────────┘
#

where "NIXPKGS"² stands for the resulting attribute set when the huge Nix expression that is the Nixpkgs repo is evaluated (e.g., with import <nixpkgs> {}).

^{[2]: I used quotes because some attributes are actually omitted.}

^{[3]: See the // operator.}

“DEFINITION”
A “Nix expression” is any piece of code written using the Nix language.

The godot_4_nix function has 30+ input parameters; most are dependecies (the ones with no default values) and some are options (the ones with default values, provided with the ? operator). This means that to build the Godot 4 application, one would have to write out all the input dependencies manually - unless it is called with callPackage.

Thus, extensive use of callPackage when “pre-evaluating” package definitions (i.e., “derivation-returning functions”) in pkgs/top-level/all-packages.nix cuts down on a lot of space and reduces manual labour.

NOTE
TODO this section needs to be clearer… “Pre-evaluation” here does not mean that all attributes in pkgs/top-level/all-packages.nix will be evaluated on the spot when evaluating Nixpkgs! The Nix language uses lazy evaluation, therefore these attributes will only get evaluated when they are actually used (in e.g., the nix repl, a Nix shell, system configuration in NixOS, etc.).

TODO: example

What I mean by “pre-evaluation” is to call (or, because of Nix’s laziness, schedule a call of) a “derivation-returning function” with the arguments it needs. This was already the practice before callPackage was introduced, it just took up way more space and time.

TODO add links to the functions in the Nixpkgs manual

Because of pre-evaluation, invoking callPackage on a Nixpkgs package produces a “derivation attribute set” right away - but this is where overridability comes in the picture: callPackage will also add two extra attributes⁴ to the produced attribute set, override and overrideDerivation⁵, making it possible to tweak the future derivation(e.g., by specifying a different source, change the options, etc.) instead of accepting the defaults.

^{[4]: … by eventually calling makeOverridable.}

^{[5]: It is officially recommended to use overrideAttrs⁶ over overrideDerivation; see <pkg>.overrideDerivation in the Nixpkgs manual.}

^{[6]: Here’s a beautifully simple explanation how override and overrideAttrs/overrideDerivation differ.}

TODO: override hell

see “override vs overlay nix” tab

see “override vs overrideAttrs” tab

general intro to overriding (There is Nix pill to consider though)

^footnum

Here’s an example that proves all the points above:

TODO: quite a bad example here and it should be in the Examples section as a counter-example for the “rule” that callPackage’s 2nd input can only have parameters that are either declared in Nixpkgs or in the head of attr_set_function.

# No semicolons because of `nix repl`.

f = let
      id = x: x;
    in
      pkgs.callPackage id { lofa = "balabab"; }

#=> { override = { ... }; overrideDerivation = ...; }

f.override { lofa = 27; miez = 9; }

#=> { lofa = 27
#=> ; miez = 9
#=> ; override = { ... }; overrideDerivation = ... ; }

This means that when the Nix expression comprising Nixpkgs is evaluated (e.g,. with import <nixpkgs> {}), then every(?) package definition (i.e., “derivation-returning function”) in there will be called with callPackage. Users then either accept the defaults and build the packages as is, or create their own Nix expression and override the inputs to their heart’s desire.

TODO: derivation attribute sets vs store derivations!

2. An informal specification

One way to write callPackage’s type signature:

callPackage ::    INPUT_1  path_to_file_with_attr_set_function
               -> INPUT_2  ...        │
               -> OUTPUT   ...        │ ( paths are `import`ed )
                                      │ (     automatically    )
                                      │
                                      ▼
callPackage ::    INPUT_1  attr_set_function            :~> package dependencies and options
               -> INPUT_2  overriding_attr_set          :~>  custom dependencies and options
               -> OUTPUT   maybe_overridable_attr_set   :~> derivation attribute set

where

attr_set_function :: initial_attr_set -> maybe_attr_set

NOTE The Nix language is dynamically typed, so this is (mostly) wishful thinking and it only serves demonstration purposes; see Examples section below.

The callPackage function accepts two arguments:

attr_set_function: a Nix function (or a file containing one) whose input can be
- either an attribute set with restrictions (see “2.1.1 initial_attr_set”)
- or a variable that will be only be used in contexts that is befitting an attribute set
```
pkgs = import <nixpkgs> {}

pkgs.callPackage (i: i) {}
#=> OK

pkgs.callPackage (i: i // { lofa = 27; } ) {}
#=> OK

pkgs.callPackage (i: i + 2) {}
#=> ERROR
```
an attribute set that must be a subset of the input function’s input attribute set

2.1 What is `attr_set_function`?

Most often, attr_set_function tends to be a “derivation-returning function” to build a package where

the function parameters (in this case, initial_attr_set) contain the dependencies and options for building and configuring it, and
the function body will (most often) call a Nix expression that will ultimately invoke the derivation primitive.^footnum The skeleton of attr_set_function’s body usually looks along the lines of
```
{ stdenv
, ...
}:

stdenv.mkDerivation { ... }
```

^{[footnum]: Then again, callPackage’s first argument doesn’t have to be a function that produces a derivations at all (see Examples section). callPackage’s convenience properties (auto-call & overridability) can be taken advantage of in other situations as well. I don’t have a list of such edge use cases, but would love to see some.}

2.1.1 `initial_attr_set`

The input for attr_set_function is

initial_attr_set, a regular attribute set whose attributes cannot be completely arbitrary, depending on the answer to the question:

Is the attribute name present in Nixpkgs?⁵

(That is, in the evaluated Nixpkgs expression. For example, if pkgs = import <nixpkgs> {} and initial_attr_set = { stdenv, cowsay }, then pkgs.stdenv and pkgs.cowsay need to be valid expressions.)

TODO: Does callPackage only look at top-level attributes?

If the answer is
- yes, attribute name is IN Nixpkgs⁵
  
  then these can stand freely, without having a default values defined for them.
```
(import <nixpkgs> {}).callPackage ( { lib, cowsay }@s: s ) {}
#=> OK

(import <nixpkgs> {}).callPackage ( { lofa }@s: s ) {}
#=> ERROR
```
- no, attribute name is NOT IN Nixpkgs⁵
  
  then these either must either have a default values or have to be declared in overriding_attr_set.
```
pkgs = import <nixpkgs> {}

pkgs.callPackage ( { lofa ? 27 }@s: s ) {}
#=> OK

pkgs.callPackage ( { lofa }@s: s ) { lofa = 27; }
#=> OK
```
^{[5]: It is actually a limited subset of Nixpkgs; see here.}

2.2 How does `callPackage` call `attr_set_function`?

Here’s the visual representation of what is happening at a very high level:

    ==================================================================
    === DECLARATIONS IN NIX REPL =====================================
    ==================================================================

    ATTR_SET_FUNCTION =            OVERRIDING_ATTR_SET =  ┌─┐
     ┌── { stdenv                    { lofa = 27; ────────┤ │
     ├── , lib                         cowsay = "miez"; ──┤2│
     ├── , cowsay                    }                    └┬┘
     │   , lofa ? 12                                       │
     │   }:                                                │
     │   { inherit lofa cowsay; }                          │
     ▼                                                     │
    PKGS = import <nixpkgs> {}                             │
     │ ┌─┐                                     ┌───────────┘
     └─┤1├───┐                                 │
       └─┘   │                                 │
             │                                 │
  ┌──────────────────────────────────────────────────────────────────┐
  ├          │                                 │                     │
  │          ▼                                 │                     │
  │┌────────────────┐                          ▼                     │
∙∙││PKGS.callPackage│   ATTR_SET_FUNCTION   OVERRIDING_ATTR_SET      │
: │└────────────────┘                          │           ┌─┐       │
: │               │                            └───────────┤2├────┐  │
: │               │                                        └─┘    │  │
: └──────────────────────────────────────────────────────────────────┘
b                 │                                               │
e   ==============│===============================================│===
c   === EVALUATION│STEPS =========================================│===
o   ==============│===============================================│===
m                ┌┴┐                                              │
e                │1│                                              │
s                └┬┘                                              │
:                 │                                               │
:       ATTR_SET_F│UNCTION           \  ATTR_SET_FUNCTION         │
:         {       ▼                   \   {                       │
∙∙∙∙>       stdenv = PKGS.stdenv;     /     stdenv = PKGS.stdenv; ▼
            lib    = PKGS.lib;     ┌─┐      lib    = PKGS.lib;    │
       ┌─►  cowsay = PKGS.cowsay; ─┤ ├────► cowsay = "miez";      │
       ├─►  lofa   = 12; ──────────┤2├────► lofa   = 27;          │
       │  }                        └─┘    }                      ┌┴┐
      ┌┴┐                                          │             │2│
      │2│                                          │             └┬┘
      └┬┘                                          │              │
       └───────────────────────────◄──────────────────────────────┘
                                                   │
                                                  ┌┴┐
                                                  │3│
                                                  └┬┘
                                                   ▼

                                      { cowsay = "miez";
                                        lofa   = 27;
                                        override           = {...};
                                        overrideDerivation = ...;
                                      }

PKGS = import <nixpkgs> {}    ATTR_SET_FUNCTION =  OVERRIDING_ATTR_SET =
                                {                    {
                                  stdenv               lofa   = 27;
                                , lib                  cowsay = "miez";
                                , cowsay             }
                                , lofa ? 12
                                }:
                                { inherit
                                    lofa cowsay;
                                }

PKGS.callPackage              ATTR_SET_FUNCTION    OVERRIDING_ATTR_SET

PKGS.callPackageWith PKGS     ATTR_SET_FUNCTION    OVERRIDING_ATTR_SET

See Nix Pill chapters 14 and 17. (The latter introduces the concept of “fixed point” which is used - but not mentioned - in the former.)

2.3 Why is the output `maybe_overridable_attr_set`?

maybe, because
overridable, because
attr_set, because

Level 2. More precise but also more abstruse

callPackage ::    ( attr_set_function | path_to_file_with_attr_set_function )
               -> constrained_attr_set
               -> ( overridable_attr_set | arbitrary_nix_value )

attr_set_function :: input_attr_set -> attr_set

^{The Nix language is dynamically typed, so this is just a personal notation.}

where

input_attr_set is a regular attribute set with caveats depending on whether its attributes are Nixpkgs or not:
- IN Nixpkgs If the attribute names match attributes in the limited subset of Nixpkgs package set¹, then these won’t have to be specified in constrained_attr_set as callPackage will auto-populate them.
- NOT in Nixpkgs These either must have a default value or have to be specified in constrained_attr_set.
constrained_attr_set is an attribute set that can only be a subset of attr_set_function’s input attribute set.
overridable_att_set - See this section of the nix.dev tutorial.

Level 3. Source

https://github.com/NixOS/nixpkgs/blame/e7f2456df49b39e816e9ed71622c09a42446a581/pkgs/top-level/splice.nix#L125-L144

  splicedPackagesWithXorg = splicedPackages // builtins.removeAttrs splicedPackages.xorg [
    "callPackage"
    "newScope"
    "overrideScope"
    "packages"
  ];

  # ...

  callPackage = pkgs.newScope { };
  callPackages = ...;
  newScope = extra: lib.callPackageWith (splicedPackagesWithXorg // extra);

^{callPackages (note the extra s!) is also defined here, but omitted it, lest I muddy the waters even more.}

callPackageWith is quite long, so only posting the link, but the resources in the question explain the nitty-gritty - and the core of it is this line:

callPackage = f: args:
  f ((builtins.intersectAttrs (builtins.functionArgs f) pkgs) // args);

^{… and this is an email from 2009 that introduces it.}

Examples

The term pkgs below will refer to the Nixpkgs package set. Such as the result of any of the following calls in nix repl:

pkgs = import <nixpkgs> {}
pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/06278c77b5d162e62df170fec307e83f1812d94b.tar.gz") {}
… and so on

(The semicolon is missing from the end of the Nix expressions to make it easier to try them out in nix repl.)

TODO: put the texts BEFORE the examples

✅
```
pkgs.callPackage ({ lib, stdenv }: { ... }) {}
```
lib and stdenv are valid attributes in Nixpkgs, so these will be auto-populated from there. Roughly equivalent to: pkgs.callPackage ({ lib, stdenv }: ... }) { lib = pkgs.lib; stdenv = pkgs.stdenv; }
✅
```
pkgs.callPackage ({ lib, stdenv, lofa ? 27 }: { ... }) {}
```
Same things apply as in the previous item, except that Nixpkgs has no lofa attribute, but there is a default value provided for it, so it will work.

✅

( pkgs.callPackage
    ({ lib, stdenv, lofa }: { inherit lofa lib; })
    { lofa = 27; lib = "whatever"; }
)

Same as in previous item, but attributes lofa and lib are explicitly provided.

✅

 f = let
       id = x: x;
     in
       pkgs.callPackage id { lofa = "balabab"; }

 #=> { override = { ... }; overrideDerivation = ...; }

 f.override { lofa = 27; miez = 9; }

 #=> { lofa = 27
 #=> ; miez = 9
 #=> ; override = { ... }; overrideDerivation = ... ; }

callPackage’s input function (1st parameter) is called with an attribute set, so this still works - and it becomes overridable too.

❌
```
pkgs.callPackage (i: i + 2) {}
```
callPackage will choke because i will be an attribute set and the + operator will try to coerce it into a string.

  pkgs.callPackage (i: 2) {} #=> 2

callPackage does not care if an attribute set is returned or not, but only attribute sets will be made overridable (for obvious reasons).

❌
```
pkgs.callPackage ({ lib, stdenv, lofa }: { inherit lofa; }) {}
```
will raise error: Function called without required argument "lofa".
✅
```
$ echo "{lib, stdenv, lofa ? 27}: {inherit lofa;}" > f.nix

$ nix repl

nix-repl> pkgs = import <nixpkgs> {}
nix-repl> pkgs.callPackage ./f.nix {}
```
Providing a valid file system path containing a Nix expression (that conforms to what callPackage is accepting) works too as the file will be automatically imported first.

nix-repl> pkgs.callPackage ({ stdenv, lib, lofa ? 27 }: lofa) {} 27

nix-repl> pkgs.callPackage ({ stdenv, lib, lofa ? 27 }: lofa) {lofa = 9;} 9

nix-repl> pkgs.callPackage ({ stdenv, lib, lofa ? 27 }: lofa) {lofa = 9; miez = 8; } error: anonymous function at «string»:1:19 called with unexpected argument ‘miez’

   at /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:80:16:

       79|     let
       80|       result = f origArgs;
         |                ^
       81|

nix-repl> pkgs.callPackage (x: x) { lofa = 27; } { lofa = 27; override = { … }; overrideDerivation = «lambda @ /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:95:32»; }

nix-repl> pkgs.callPackage ({miez ? 7}: x) { lofa = 27; } error: undefined variable ‘x’

   at «string»:1:31:

        1| pkgs.callPackage ({miez ? 7}: x) { lofa = 27; }
         |                               ^

nix-repl> pkgs.callPackage ({miez ? 7}@x: x) { lofa = 27; } error: anonymous function at «string»:1:19 called with unexpected argument ‘lofa’

   at /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:80:16:

       79|     let
       80|       result = f origArgs;
         |                ^
       81|

nix-repl> pkgs.callPackage ({}: x) { lofa = 27; } error: undefined variable ‘x’

   at «string»:1:23:

        1| pkgs.callPackage ({}: x) { lofa = 27; }
         |                       ^

nix-repl> pkgs.callPackage ({}: {}) { lofa = 27; } error: anonymous function at «string»:1:19 called with unexpected argument ‘lofa’

   at /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:80:16:

       79|     let
       80|       result = f origArgs;
         |                ^
       81|

nix-repl> pkgs.callPackage ({}: {miez = 9; }) { } { miez = 9; override = { … }; overrideDerivation = «lambda @ /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:95:32»; }

nix-repl> pkgs.callPackage (x: {miez = 9; }) { } { miez = 9; override = { … }; overrideDerivation = «lambda @ /nix/store/h8ankbqczm1wm84libmi3harjsddrcqx-nixpkgs/nixpkgs/lib/customisation.nix:95:32»; }

The result of attr_set_function is usually a Nix derivation expression (which is an attribute set denoting a derivation - or something like it; look this up).

[1]: https://github.com/NixOS/nixpkgs/blame/e7f2456df49b39e816e9ed71622c09a42446a581/pkgs/top-level/splice.nix#L140-L144

callPackage