The Purely Functional Software

Deployment Model

Het puur functionele softwaredeploymentmodel

(met een samenvatting in het Nederlands)

Proefschrift ter verkrijging van de graad van doctor aan de Universiteit Utrecht op gezag

van de Rector Magniﬁcus, Prof. dr. W. H. Gispen, ingevolge het besluit van het College

voor Promoties in het openbaar te verdedigen op woensdag 18 januari 2006 des middags

te 12.45 uur

door

Eelco Dolstra

geboren op 18 augustus 1978, te Wageningen

Promotor: Prof. dr. S. Doaitse Swierstra, Universiteit Utrecht

Copromotor: Dr. Eelco Visser, Universiteit Utrecht

The work in this thesis has been carried out under the auspices of the research school IPA

(Institute for Programming research and Algorithmics). Dit proefschrift werd mogelijk

gemaakt met ﬁnanciële steun van CIBIT|SERC ICT Adviseurs.

Cover illustration: Arthur Rackham (1867–1939), The Rhinegold & The Valkyrie (1910),

illustrations for Richard Wagner’s Der Ring des Nibelungen: the gods enter Walhalla as

the Rhinemaidens lament the loss of their gold.

ISBN 90-393-4130-3

Acknowledgements

This thesis could not have come about without the help of many individuals. Foremost I

wish to thank my supervisor and copromotor Eelco Visser. He was also my master’s thesis

supervisor, and I was quite happy that we were able to continue to work together on the

Variability/TraCE project. He shared my eventual interest in conﬁguration management-

related issues, and package management in particular. His insights have improved this

research and this thesis at every point.

The Software Engineering Research Center (SERC), now known as CIBIT|SERC ICT

Adviseurs, funded my PhD position. Gert Florijn initiated the variability project and the

discussions with him were a valuable source of insights. The results of the present thesis

are probably not what any of us had expected at the start; but then again, the nice thing

about a term like “variability” is that it can take you in so many directions.

My promotor Doaitse Swierstra gave lots of good advice—but I ended up not imple-

menting his advice to keep this thesis short. I am grateful to the members of the reading

committee—Jörgen van den Berg, Sjaak Brinkkemper, Paul Klint, Alexander Wolf and

Andreas Zeller—for taking the time and effort to go through this book. Karl Trygve Kalle-

berg, Armijn Hemel, Arthur van Dam and Martin Bravenboer gave helpful comments.

Several people have made important contributions to Nix. Notorious workaholic Martin

Bravenboer in particular was an early adopter who contributed to almost every aspect of

the system. Armijn Hemel (“I am the itch programmers like to scratch”) contributed to

Nixpkgs and initiated the NixOS, which will surely conquer the world. René de Groot

coined the marketing slogan for NixOS—“Nix moet, alles kan!” Roy van den Broek has

replaced my hacky build farm supervisor with something much nicer (and Web 2.0 compli-

ant!). Eelco Visser, Rob Vermaas and Merijn de Jonge also contributed to Nixpkgs and the

build farm. In addition, it is hard to overestimate the importance of the work done by the

free and open source software community in providing a large body of non-trivial modular

components suitable for validation. Thanks!

The Software Technology group is a very pleasant work environment, and I thank

all my current and former colleagues for that. I especially thank my roommates Dave

Clarke (“The cause of the software crisis is software”), Frank Atanassow (who demanded

“pointable research”), Merijn de Jonge, Martin Bravenboer, Rob Vermaas and Stefan Hold-

ermans. Andres Löh, Bastiaan Heeren, Arthur Baars, Karina Olmos and Alexey Rodriguez

were more than just good colleagues. Daan Leijen insisted on doing things the right way.

The #klaplopers provided a nice work environment in virtual space—though whether IRC

is a medium that increases productivity remains an unanswered empirical question.

Arthur van Dam and Piet van Oostrum provided valuable assistance in wrestling with

X. Atze Dijkstra, Bastiaan Heeren and Andres Löh had helpful advice (and code!) for

the preparation of this thesis.

And ﬁnally I would like to thank my family—Mom, Dad, Jitske and Menno—for their

support and love through all these years.

iii

Contents

I. Introduction 1

1. Introduction 3

1.1. Software deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2. The state of the art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.4. The Nix deployment system . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.5. Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.6. Outline of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.7. Notational conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2. An Over view of Nix 19

2.1. The Nix store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.2. Nix expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.3. Package management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2.4. Store derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.5. Deployment models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.6. Transparent source/binary deployment . . . . . . . . . . . . . . . . . . . . . 44

II. Foundations 47

3. Deployment as Memory Management 49

3.1. What is a component? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.2. The ﬁle system as memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.3. Closures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.4. A pointer discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.5. Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4. The Nix Expression Language 61

4.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.2. Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.2.1. Lexical syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.2.2. Context-free syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.3. Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.3.1. Basic values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.3.2. Compound values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Contents

4.3.3. Substitutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.3.4. Evaluation rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.4. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5. The Extensional Model 87

5.1. Cryptographic hashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.2. The Nix store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.2.1. File system objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.2.2. Store paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.2.3. Path validity and the closure invariant . . . . . . . . . . . . . . . . . . . . . 95

5.3. Atoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

5.4. Translating Nix expressions to store derivations . . . . . . . . . . . . . . . . 100

5.4.1. Fixed-output derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.5. Building store derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

5.5.1. The simple build algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5.5.2. Distributed builds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

5.5.3. Substitutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.5.4. The parallel build algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.6. Garbage collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

5.6.1. Garbage collection roots . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

5.6.2. Live paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

5.6.3. Stop-the-world garbage collection . . . . . . . . . . . . . . . . . . . . . . . 128

5.6.4. Concurrent garbage collection . . . . . . . . . . . . . . . . . . . . . . . . . 131

5.7. Extensionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

6. The Intensional Model 135

6.1. Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

6.2. Local sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6.3. A content-addressable store . . . . . . . . . . . . . . . . . . . . . . . . . . 140

6.3.1. The hash invariant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

6.3.2. Hash rewriting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

6.4. Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

6.4.1. Equivalence class collisions . . . . . . . . . . . . . . . . . . . . . . . . . . 147

6.4.2. Adding store objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

6.4.3. Building store derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

6.4.4. Substitutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

6.5. Trust relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

6.6. Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

6.7. Multiple outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

6.8. Assumptions about components . . . . . . . . . . . . . . . . . . . . . . . . 162

III. Applications 165

Contents

7. Software Deployment 167

7.1. The Nix Packages collection . . . . . . . . . . . . . . . . . . . . . . . . . . 167

7.1.1. Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.1.2. The standard environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

7.1.3. Ensuring purity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

7.1.4. Supporting third-party binary components . . . . . . . . . . . . . . . . . . . 180

7.1.5. Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

7.2. User environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

7.3. Binary deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

7.4. Deployment policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

7.5. Patch deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

7.5.1. Binary patch creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

7.5.2. Patch chaining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

7.5.3. Base selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

7.5.4. Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

7.6. Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

8. Continuous Integration and Release Management 209

8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

8.2. The Nix build farm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

8.3. Distributed builds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

8.4. Discussion and related work . . . . . . . . . . . . . . . . . . . . . . . . . . 218

9. Service Deployment 221

9.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

9.1.1. Service components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

9.1.2. Service conﬁguration and deployment . . . . . . . . . . . . . . . . . . . . . 223

9.1.3. Capturing component compositions with Nix expressions . . . . . . . . . . . 224

9.1.4. Maintaining the service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

9.2. Composition of services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

9.3. Variability and crosscutting conﬁguration choices . . . . . . . . . . . . . . . 226

9.4. Distributed deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

9.5. Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

9.6. Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

10. Build Management 233

10.1. Low-level build management using Nix . . . . . . . . . . . . . . . . . . . . 233

10.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238

10.3. Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

IV. Conclusion 243

11. Conclusion 245

11.1. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

vii

Contents

viii

Part I.

Introduction

1. Introduction

This thesis is about getting computer programs from one machine to another—and having

them still work when they get there. This is the problem of software deployment. Though

it is a part of the ﬁeld of Software Conﬁguration Management (SCM), it has not been a

subject of academic study until quite recently [169]. The development of principles and

tools to support the deployment process has largely been relegated to industry, system

administrators, and Unix hackers. This has resulted in a large number of often ad hoc

tools that typically automate manual practices but do not address fundamental issues in a

systematic and disciplined way.

This is evidenced by the huge number of mailing list and forum postings about de-

ployment failures, ranging from applications not working due to missing dependencies, to

subtle malfunctions caused by incompatible components. Deployment problems also seem

curiously resistant to automation: the same concrete problems appear time and again. De-

ployment is especially difﬁcult in heavily component-based systems—such as Unix-based

open source software—because the effort of dealing with the dependencies can increase

super-linearly with each additional dependency.

This thesis describes a system for software deployment called Nix that addresses many

of the problems that plague existing deployment systems. In this introductory chapter I

describe the problem of software deployment, give an overview of existing systems and

the limitations that motivated this research, summarise its main contributions, and outline

the structure of this thesis.

1.1. Software deployment

Software deployment is the problem of managing the distribution of software to end-user

machines. That is, a developer has created some piece of software, and this ultimately has

to end up on the machines of end-users. After the initial installation of the software, it

might need to be upgraded or uninstalled.

Presumably, the developer has tested the software and found it to work sufﬁciently well,

so the challenge is to make sure that the software works just as well, i.e., the same, on the

end-user machines. I will informally refer to this as correct deployment: given identical

inputs, the software should behave the same on an end-user machine as on the developer

machine

This should be a simple problem. For instance, if the software consists of a set of ﬁles,

then deployment should be a simple matter of copying those to the target machines. In

I’m making several gross simpliﬁcations, of course. First, in general there is no single “developer”. Second,

there are usually several intermediaries between the developer and the end-user, such as a system administra-

tor. However, for a discussion of the main issues this will sufﬁce.

1. Introduction

practice, deployment turns out to be much harder. This has a number of causes. These fall

into two broad categories: environment issues and manageability issues.

Environment issues The ﬁrst category is essentially about correctness. The software

might make all sorts of demands about the environment in which it executes: that certain

other software components are present in the system, that certain conﬁguration ﬁles exist,

that certain modiﬁcations were made to the Windows registry, and so on. If any of those

environmental characteristics does not hold, then there is a possibility that the software

does not work the same as it did on the developer machine. Some concrete issues are the

following:

• A software component is almost never self-contained; rather, it depends on other

components to do some work on its behalf. These are its dependencies. For correct

deployment, it is necessary that all dependencies are identiﬁed. This identiﬁcation

is quite hard, however, as it is often difﬁcult to test whether the dependency speciﬁ-

cation is complete. After all, if we forget to specify a dependency, we don’t discover

that fact if the machine on which we are testing already happens to have the depen-

dency installed.

• Dependencies are not just a runtime issue. To build a component in the ﬁrst place we

need certain dependencies (such as compilers), and these need not be the same as the

runtime dependencies, although there may be some overlap. In general, deployment

of the build-time dependencies is not an end-user issue, but it might be in source-

based deployment scenarios; that is, when a component is deployed in source form.

This is common in the open source world.

• Dependencies also need to be compatible with what is expected by the referring

component. In general, not all versions of a component will work. This is the case

even in the presence of type-checked interfaces, since interfaces never give a full

speciﬁcation of the observable behaviour of a component. Also, components often

exhibit build-time variability, meaning that they can be built with or without certain

optional features, or with other parameters selected at build time. Even worse, the

component might be dependent on a speciﬁc compiler, or on speciﬁc compilation

options being used for its dependencies (e.g., for Application Binary Interface (ABI)

compatibility).

• Even if all required dependencies are present, our component still has to ﬁnd them,

in order to actually establish a concrete composition of components. This is often

a rather labour-intensive part of the deployment process. Examples include setting

up the dynamic linker search path on Unix systems [160], or the CLASSPATH in the

Java environment.

• Components can depend on non-software artifacts, such as conﬁguration ﬁles, user

accounts, and so on. For instance, a component might keep state in a database that

has to be initialised prior to its ﬁrst use.

1.1. Software deployment

• Components can require certain hardware characteristics, such as a speciﬁc proces-

sor type or a video card. These are somewhat outside the scope of software deploy-

ment, since we can at most check for such properties, not realise them if they are

missing.

• Finally, deployment can be a distributed problem. A component can depend on

other components running on remote machines or as separate processes on the same

machine. For instance, a typical multi-tier web service consists of an HTTP server,

a server implementing the business logic, and a database server, possibly all running

on different machines.

So we have two problems in deployment: we must identify what our component’s re-

quirements on the environment are, and we must somehow realise those requirements in

the target environment. Realisation might consist of installing dependencies, creating or

modifying conﬁguration ﬁles, starting remote processes, and so on.

Manageability issues The second category is about our ability to properly manage the

deployment process. There are all kinds of operations that we need to be able to perform,

such as packaging, transferring, installing, upgrading, uninstalling, and answering various

queries; i.e., we have to be able to support the evolution of a software system. All these

operations require various bits of information, can be time-consuming, and if not done

properly can lead to incorrect deployment. For example:

• When we uninstall a component, we have to know what steps to take to safely undo

the installation, e.g., by deleting ﬁles and modifying conﬁguration ﬁles. At the same

time we must also take care never to remove any component still in use by some

other part of the system.

• Likewise, when we perform a component upgrade, we should be careful not to over-

write any part of any component that might induce a failure in another part of the

system. This is the well-known DLL hell, where upgrading or installing one applica-

tion can cause a failure in another application due to shared dynamic libraries. It has

been observed that software systems often suffer from the seemingly inexplicable

phenomenon of “bit rot,” i.e., that applications that worked initially stop working

over time due to changes in the environment [26].

• Administrators often want to perform queries such as “to what component does this

ﬁle belong?”, “how much disk space will it take to install this component?”, “from

what sources was this component built?”, and so on.

• Maintenance of a system means keeping the software up to date. There are many

different policy choices that can be made. For instance, in a network, system ad-

ministrators may want to push updates (such as security ﬁxes) to all client machines

periodically. On the other hand, if users are allowed to administer their own ma-

chines, it should be possible for them to select components individually.

• When we upgrade components, it is important to be able to undo, or roll back the

effects of the upgrade, if the upgrade turns out to break important functionality. This

1. Introduction

requires both that we remember what the old conﬁguration was, and that we have

some way to reproduce the old conﬁguration.

• In heterogeneous networks (i.e., consisting of many different types of machines),

or in small environments (e.g., a home computer), it is not easy to stay up to date

with software updates. In particular in the case of security ﬁxes this is an important

problem. So we need to know what software is in use, whether updates are available,

and whether such updates should be performed.

• Components can often be deployed in both source and binary form. Binary pack-

ages have to be built for each supported platform, and sometimes in several variants

as well. For instance, the Linux kernel has thousands of build-time conﬁguration

options. This greatly increases the deployment effort, particularly if packaging and

transfer of packages is a manual or semi-automatic process.

• Since components often have a huge amount of variability, we sometimes want to

expose that variability to certain users. For instance, Linux distributors or system

administrators typically want to make speciﬁc feature selections. A deployment

system should support this.

1.2. The state of the art

Having seen some of the main issues in the ﬁeld of software deployment, we now look at

some representative deployment tools. This section does not aim to provide a full survey

of the ﬁeld; rather, the goal is to show the main approaches to deployment. Section 7.6 has

a more complete discussion of related work.

Package management is a perennial problem in the Unix community [2]. In fact, entire

operating system distributions rise and fall on the basis of their deployment qualities. It can

be argued that Gentoo Linux’s [77] quick adoption in the Linux community was entirely

due to the perceived strengths of its package management system over those used by other

distributions. This interest in deployment can be traced to Unix’s early adoption in large,

advanced and often academic installations (in contrast to the single PC, single user focus

in the PC industry in a bygone era).

Also, for better or for worse, Unix systems have traditionally insisted on storing com-

ponents in global namespaces in the ﬁle system such as the /usr/bin directory. This makes

management tools indispensable. But more importantly, modern Unix components have

ﬁne-grained reuse, often having dozens of dependencies on other components. Since it

is not desirable to use monolithic distribution (as is generally done in Windows and Mac

OS X, as discussed below), a package management tool is absolutely required to support

the resulting deployment complexity. Therefore Unix (and speciﬁcally, Linux) package

management is what we will look at ﬁrst.

RPM The most widely used Linux package management system is the Red Hat Package

Manager (RPM) [62], and as it is a good, mature tool it is instructive to look at it in some

detail. RPM is a low-level deployment tool: it installs and manages components, keeping

meta-information about them to prevent unsafe upgrades or uninstalls.

1.2. The state of the art

Summary: Hello World program

Name: hello

Version: 1.0

Source0: %{name}-%{version}.tar.gz

%description

This program prints out "Hello

World".

%build

./configure --prefix=%{_prefix}

make

%install

make install

%files

/usr/bin/hello

/usr/share/man/man1/hello.1.gz

/etc/hello.conf

Figure 1.1.: Spec ﬁle for hello

Summary: Hello World GUI

Name: xhello

Version: 1.4

Source0: %name-%version.tar.gz

Requires: hello >= 1.0

Requires: XFree86

%description

This program provides a graphical

user interface around Hello.

%build

./configure --prefix=%_prefix

make

%install

make install

%files

/usr/bin/xhello

Figure 1.2.: Spec ﬁle for xhello

A software component is deployed by packaging it into an RPM package (or simply

RPM), which is an archive ﬁle that consists of the ﬁles that constitute the component, and

some meta-information about the package. This includes a speciﬁcation of the dependen-

cies of the package, scripts to be run at various points in the install and uninstall process,

and administrative information such as the originator of the package. RPMs are built from

spec ﬁles. Figure 1.1 shows a trivial spec ﬁle for an imaginary component called Hello that

provides a command /usr/bin/hello.

An RPM package is typically produced from source as follows

$ rpmbuild -ba hello.tar.gz

where hello.tar.gz contains the sources of the component and its spec ﬁle. The resulting

RPM (say, hello-1.0.i686.rpm) must then be transferred to the client machines in some way

that the RPM tool itself does not specify. Once it is on a client machine, it can be installed:

$ rpm -i hello-1.0.i686.rpm

Similarly, we can upgrade the package when a new RPM with the same name comes along

(rpm -u hello-2.0.i686.r pm), and uninstall it (rpm -e hello). To perform upgrades and unin-

stalls cleanly, it is necessary to track which ﬁles belong to what packages. This information

can be queried by users:

$ rpm -ql hello

In this thesis, the preﬁx $ in code samples indicates a Unix shell command.

1. Introduction

/usr/bin/hello

/etc/hello.conf

$ rpm -qf /usr/bin/hello

hello-1.0

RPM also maintains a cryptographic hash [145] of the contents of each ﬁle as it existed at

installation time. Thus, users can verify that ﬁles have not been tampered with.

RPM maintains the integrity of installed packages with respect to the dependency re-

quirements. Figure 1.2 shows another RPM package, xhello, that provides a GUI front-end

for hello and thus depends on the hello package (expressed at point 1 ). When we have the

xhello package installed, we cannot uninstall hello unless xhello is uninstalled also:

$ rpm -e hello

hello is needed by (installed) xhello-1.4

Similarly, RPM does not allow two packages that contain ﬁles with equal path names

to exist simultaneously in the system. For instance, when we have our hello-1.0 package

installed, it is not possible to install another package that also installs a ﬁle with path name

/usr/bin/hello. In general, this means that we cannot have multiple versions of the same

component installed simultaneously. It also means that it is hard for multiple users to

independently select, install and manage software.

A fundamental problem of RPM and essentially all general package management sys-

tems is that they cannot reliably determine the correctness of dependency speciﬁcations.

In our example, xhello depends on hello, and so its spec ﬁle will contain a dependency

speciﬁcation to that effect, e.g.,

Requires: hello

But what happens when we forget to specify this? When the developer builds and tests the

RPM package, the component will probably work because in all likelihood the developer

has hello installed. If we then deploy xhello to clients, the component will work if hello

happens to have been installed previously. If it was not, xhello may fail mysteriously (Fig-

ure 1.3; black ovals denote broken components). Thus, it is intrinsically hard to validate

dependency speciﬁcations. (It is also hard to prevent unnecessary dependencies, but that

does not harm correctness, just efﬁciency.) An analysis of the actual number of depen-

dency errors in a large RPM-based Linux distribution is described in [87]. The number of

dependency errors turned out to be quite low, but this is likely to be at least in part due to

the substantial effort invested in specifying complete dependencies. Missing dependencies

lead to incomplete deployment; correct deployment on the other hand requires complete

deployment.

Related to the inability to validate dependency speciﬁcations is the fact that dependen-

cies tend to be inexact. Above, xhello required that a component named hello is present—

but it makes no requirements on the actual properties or interfaces of that component. That

is, the dependency speciﬁcation is nominal (determined by name only), not by contract

(requiring that the dependency has certain properties). So any component named hello

satisﬁes the dependency. Actually, we can require speciﬁc versions of the dependency:

Requires: hello >= 1.0

1.2. The state of the art

Producer Site

Application

App

Libraries

LibA

version 0.5

LibB

version 1.3

Consumer Site

Application

App

Libraries

LibA

version 0.3

Figure 1.3.: Incomplete deploy-

ment

Applications

App1 App2 App3

Libraries

Lib1 Lib2

Applications

App1 App2 App3

Libraries

Applications

App1 App2' App3

Libraries

Upgrade of App2

Applications

App1 App2

Libraries

Lib2

App1 App2

Removal of App3

LibA LibB

LibA LibB' LibB

Figure 1.4.: Interference

which excludes version 1.0. However, such version speciﬁcations involve a high degree of

wishful thinking, since we can never in general rely on the fact that any version in an open

range works. For instance, there is no way to know whether future release 1.3.1 of hello

will be backwards compatible. Even “exact” dependencies such as

Require: hello = 1.0

are unsafe, because this is still a nominal dependency: we can conceive of any number

of component instances with name hello and version number 1.0 that behave completely

differently. In fact, this is a real problem: Linux distributions from different vendors can

easily have components with equal names (e.g., glibc-2.3.5) that actually have vendor-

speciﬁc patches applied, have been built with speciﬁc options, compilers, or ABI options,

and so on.

Incomplete dependencies and inexact notions of component compatibility give rise to

interference between components, which is the phenomenon that an operation on one com-

ponent can “break” an essentially unrelated component. Figure 1.4 shows two examples of

interference. In the left scenario, the upgrading of application App2 breaks App1 because

the new version App2’ requires a newer version of LibB’, which happens not to be sufﬁ-

ciently backwards compatible. In the right scenario, the uninstallation of App3 also breaks

App1, because the package manager has removed LibA under the mistaken belief that App3

was the sole user of LibA.

A more subtle problem in RPM (and most other deployment tools) is that it has the prop-

erty of destructive upgrading, which means that components are upgraded by overwriting

the ﬁles of the old version with those of the new one. This is the case because the new ver-

1. Introduction

python-2.3.4

bittorrent-3.4.2

wxPython-2.4.2.4

zapping-0.7

coreutils-5.2.1

perl-5.8.5

glibc-2.3.3

gcc-3.4.2 zlib-1.2.1

wxGTK-2.4.2

libjpeg-6b

libpng-1.2.7 atk-1.2.4

gtk+-2.2.4

glib-2.2.3

libtiff-3.6.1 pango-1.2.5

libX11-6.2.1

libXft-2.1.6

fontconfig-2.2.3libXrender-0.8.4

expat-1.95.8freetype-2.1.5

libXext-6.4.3

libXau-0.1.1

libXt-0.1.4-cvs

libICE-6.3.3

libSM-6.0.3

libgnomecanvas-2.4.0

glib-2.4.7

libgnomeui-2.4.0.1

popt-1.7

libbonoboui-2.4.1

libXv-2.2.2

libxml2-2.6.13gtk+-2.4.13

libart_lgpl-2.3.16

ORBit2-2.8.3

libbonobo-2.4.2

gnome-vfs-2.4.2

esound-0.2.32

libgnome-2.0.6

GConf-2.4.0.1

pango-1.4.1

zvbi-0.2.8

rte-0.5.2

atk-1.6.1

libglade-2.0.1

libIDL-0.8.2audiofile-0.2.3

xlib-1.0

libXtrans-0.1 xproto-6.6.1xextensions-1.0.1

renderext-0.8

Figure 1.5.: The dependency hell: the runtime dependency graph of Mozilla Firefox

sion typically lives in the same paths in the ﬁle system, e.g., hello-2.0 will still install into

/usr/bin/hello and /etc/hello.conf. Apart from the resulting inability to have multiple ver-

sions installed at the same time, this gives rise to a temporary inconsistency in the system:

there is a time window in which we have some of the ﬁles of the old version, and some of

the new version. Thus, upgrading is not atomic. If a user were to start the Hello program

during this time window, the program might not work at all or misbehave in certain ways.

The main criticism leveled at RPM by some of its users is the difﬁculty in obtaining RPM

packages. If we want to install a complex, multi-component system such as X.org, KDE,

or GNOME, we have to manually download a possibly large number of RPM packages

until all missing dependencies are resolved. That is, RPM veriﬁes that an installation of a

package proceeds safely according to its spec ﬁle, but has no way to resolve problems by

itself if they do occur. This gives rise to the dependency hell, where users ﬁnd themselves

chasing down RPMs that may not be easy to obtain. Figure 1.5 shows the dependency

graph of Mozilla Firefox, a popular web browser

. Each node is a component that must be

present. This gives an intuition as to the magnitude of the problem.

However, it is not actually a problem that RPM does not obtain packages automatically:

it is in fact a good separation of mechanism and policy. Higher-level deployment tools

can be built on top of RPM that do provide automatic fetching of packages. Installation

This picture was produced from the Firefox component in the Nix Packages collection using the nix-env --query

--graph command and Graphviz [54].

1.2. The state of the art

Client machine

/etc/yum.conf

Server 1

refers to

Server 2

refers to

Server 3

refers to

yum

reads

rpm

calls

user

invokes

Figure 1.6.: Software deployment using yum

tools such as SuSE Linux’s YaST have a global view of all available packages in a Linux

distribution, and will automatically download them from a network site or prompt for the

appropriate installation CD or DVD. The tool yum (short for Yellow Dog Updater, Modi-

ﬁed) [174], used among others by the Fedora Core Linux distribution, is a wrapper around

RPM to add support for network repositories from which the tool can automatically ac-

quire packages. Figure 1.6 illustrates a typical yum deployment scenario. Client machines

have a ﬁle /etc/yum.conf that contains a list of RPM repositories. We can then install a

package with all its dependencies by saying, e.g.,

$ yum install firefox

Yum will consult the repositories deﬁned in /etc/yum.conf to ﬁnd the appropriate RPM

packages, download them, and install them by calling the RPM tool.

Source deployment models In the open source community there are several operating

system distributions that deploy packages not (primarily) in binary form but in source form.

The main advantage of source deployment is that it allows greater ﬂexibility by allowing

users to make speciﬁc build time conﬁguration choices. For instance, we can optimise

components for our speciﬁc processor architecture, or remove all unneeded optional func-

tionality in components. Whether this is actually a desirable feature for end-users is up for

debate, but the ability to easily construct a customised set of component instances is quite

valuable to some users, such as developers, system administrators, deployers of complex

installations, and distributors.

The archetypical source deployment system is the FreeBSD Ports Collection [73]. Pack-

ages in source form (called ports) are essentially Makeﬁles [56] that describe how to recur-

sively build the dependencies of the package, download the sources of the package, apply

possible FreeBSD-speciﬁc patches, build the package, and ﬁnally install it in the ﬁle sys-

tem. By passing arguments to Make (or editing the Makeﬁle, associated ﬁles, and options

in global conﬁguration ﬁles), the component can be adapted to local requirements.

An obvious downside to source deployment is its rather considerable resource consump-

tion. While binary deployment only requires disk space for the binaries, and possibly net-

work resources for obtaining the binary package, source deployment requires resources to

obtain and store the sources, CPU time and memory to build the package, and disk space

for temporary ﬁles and the ﬁnal results. To make matters worse, binary deployment only

1. Introduction

involves runtime dependencies, while source deployment may involve additional depen-

dencies that are only used at build time, such as compilers. Unsurprisingly, FreeBSD

therefore also offers a binary deployment option, called packages, which are pre-built

ports. However, ports and packages are not quite integrated: for instance, when installing

a port, dependencies that might be installed using packages are still built from source.

A further problem with source deployment is that it increases the risk of deployment

failure, as now not just runtime dependencies but also build-time dependencies can affect

the result of the deployment. For instance, if the user’s installed C compiler is not quite

what a component’s developer tested against, there is a slight probability that the com-

ponent fails to build or run correctly—and the product of sufﬁciently many slight failure

probabilities is a large probability.

Windows and Mac OS Windows and Mac OS X tend to use monolithic deployment

for applications: except for some large-grained dependencies on the operating system en-

vironment (e.g., on the kernel, the core GUI libraries, or Direct X), dependencies tend

to be distributed as part of the application itself, with no sharing of dependencies be-

tween applications. This can be accomplished through static linking or by having dy-

namic libraries be part of the private namespace (directory tree) of the application (e.g.,

C:\Program Files\MyApp\Foo.DLL). While this reduces deployment complexity at the end-

user machine, it has several downsides. First, it removes sharing: if two applications use

the “same” component, they will nevertheless end up using private copies. The result is

increased resource consumption in terms of disk space, RAM, cache efﬁciency, download

time, and so on. Clearly, it is bad if all or many of Firefox’s dependencies in Figure 1.5

were unshared. In the worst case, we get a quadratic blowup in the disk space require-

ments: if we have N applications that share the same M dependencies, then we need disk

space Θ(NM) instead of Θ(N + M). Second, it still requires the developer to obtain and

compose the components, typically through a semi-manual process.

Especially elegant from an end-user perspective are Mac OS’s application bundles,

which are directory trees containing all ﬁles belonging to an application. Generally, such

bundles are self-contained, except for operating system component dependencies. Con-

trary to typical Windows applications, they do not have other environment dependencies

such as registry settings. This means that bundles can be copied or moved around in the

ﬁle system arbitrarily. For instance, the whole of Microsoft Ofﬁce X on Mac OS X can be

copied between machines by dragging it from one disk to another. Again, the limitation of

this approach is that it falls apart when components have dependencies on each other. That

is, the bundle approach works only for “top level” components, i.e., end-user applications

.NET Historically, Windows has suffered from the DLL hell, a result of an unmanaged

global namespace being used to store shared dependencies, e.g., the directory C:\Windows-

\System. An installation or uninstallation of one application frequently caused other ap-

plications to break because a shared library (DLL) would be replaced with an incompat-

ible version, or deleted altogether. This is a classic example of component interference.

For instance, the management of Mac OS X’s BSD-based Unix foundations is a complete mess, which has

prompted the development of tools such as Fink [138], which is based on Debian’s APT system (discussed in

Section 7.6).

1.3. Motivation

Microsoft’s .NET platform [17] improves on this situation through its assemblies, which

are ﬁles that store code and other resources. Assemblies can have strong names, which

are globally unique identiﬁers signed by the assembly producer. Assemblies can depend

on each other through these strong names. The Global Assembly Cache (GAC) holds a

system-wide repository of assemblies, indexed by strong name. If assemblies in the GAC

differ in their strong names, they will not overwrite each other. Thus, interference does not

happen.

However, the GAC is only intended for shared components. Application code and mis-

cellaneous resources are typically not stored in the GAC, but in application-speciﬁc di-

rectories, e.g., C:\Program Files\MyApp. That is, management of those components must

be handled through other mechanisms. Also, the GAC only holds components that use a

speciﬁc component technology—.NET assemblies.

Other systems The Zero Install system [112] installs components on demand from net-

work repositories. On-demand installation is enabled by using a virtual ﬁle system that

intercepts ﬁle system requests for component ﬁles. For instance, when a process accesses

/uri/0install/abiword.org/abiword, and this ﬁle is not already present in the local component

cache, it is fetched from a repository, at which point the requesting process resumes. Users

can independently install software in this manner, and are unaffected by the actions of other

users. Unfortunately, systems depending on non-portable extensions to the ﬁle system face

difﬁculty in gaining wide adoption

. This is especially the case for deployment systems as

they should support a wide variety of platforms.

1.3. Motivation

From the previous discussion of existing deployment systems it should be clear that they

lack important features to support safe and efﬁcient deployment. In particular, they have

some or all of the following problems:

• Dependency speciﬁcations are not validated, leading to incomplete deployment.

• Dependency speciﬁcations are inexact (e.g., nominal).

• It is not possible to deploy multiple versions or variants of a component side-by-side.

• Components can interfere with each other.

• It is not possible to roll back to previous conﬁgurations.

• Upgrade actions are not atomic.

• Applications must be monolithic, i.e., they must statically contain all their depen-

dencies.

For instance, Apple is moving away from resource forks to bundles in recent releases of its Mac OS. The

former approach stores streams of metadata (such as icons) in ﬁles in addition to its regular data contents,

which is not portable. The latter approach on the other hand uses directory trees of regular ﬁles to keep such

data elements together, which is portable.

1. Introduction

• Deployment actions can only be performed by administrators, not by unprivileged

users.

• There is no link between binaries and the sources and build processes that built them.

• The system supports either source deployment or binary deployment, but not both;

or it supports both but in a non-uniﬁed way.

• It is difﬁcult to adapt components.

• Component composition is manual.

• The component framework is narrowly restricted to components written in a speciﬁc

programming language or framework.

• The system depends on non-portable techniques.

The objective of the research described in this thesis is to develop a deployment system

that does not have these problems.

1.4. The Nix deployment system

This thesis describes the Nix deployment system, which overcomes the limitations of con-

temporary deployment tools described above. I describe the concepts and implementation

(how it works), the underlying principles (why it works), our experiences and empirical

validation (that it works), and the application areas to which it can be applied (where it

works).

The main idea of the Nix approach is to store software components in isolation from each

other in a central component store, under path names that contain cryptographic hashes of

all inputs involved in building the component, such as /nix/store/rwmfbhb2znwp...-ﬁrefox-

1.0.4. As I show in this thesis, this prevents undeclared dependencies and enables support

for side-by-side existence of component versions and variants.

Availability At the time of writing, the Nix system is available as free software at the

homepage of the TraCE project [161].

1.5. Contributions

The main contribution of this thesis is the development of a purely functional deployment

model, which we implemented in the Nix system. In this model a binary component is

uniquely deﬁned by the declared inputs used to build the component. This enables arbitrary

component instances to exist side by side. I show how such a model can be “retroﬁtted”

onto existing software that was designed without such a model in mind.

Concretely, the contributions of this thesis are the following.

• The cryptographic hashing scheme used by the Nix component store prevents un-

declared dependencies, giving us complete deployment. Furthermore it provides

support for side-by-side existence of component versions and variants.

1.5. Contributions

• Isolation between components prevents interference.

• Users can install software independently from each other, without requiring mutual

trust relations. Components that are common between users are shared, i.e., stored

only once.

• Upgrades are atomic; there is no time window in which the system is in an inconsis-

tent state.

• Nix supports O(1)-time rollbacks to previous conﬁgurations.

• Nix supports automatic garbage collection of unused components.

• The Nix component language describes not just how to build individual components,

but also compositions. The language is a lazy, purely functional language. This is

a good basis for a component composition language, as it allows dependencies and

variability to be expressed in an elegant and ﬂexible way.

• Nix has a transparent source/binary deployment model that marries the best parts

of source deployment models such as the FreeBSD Ports Collection, and binary

deployment models such as RPM. In essence, binary deployment is an automatic

optimisation of source deployment.

• Nix is policy-free; it provides mechanisms to implement various deployment poli-

cies, but does not enforce a speciﬁc one. Some policies described in this thesis are

channels (in push and pull variants), one-click installations, and pure source deploy-

ments.

• I show that a purely functional model supports efﬁcient component upgrades despite

the fact that a change to a fundamental component can propagate massively through

the dependency graph.

• Nix supports distributed multi-platform builds in a transparent manner, i.e., a remote

build is indistinguishable from a local build from the perspective of the user.

• Nix provides a good basis for the implementation of a build farm, which supports

continuous integration and portability testing. I describe a Nix build farm that has

greatly improved manageability over other build farms and is integrated with release

management, that is, it builds concrete installable components that can be deployed

directly through Nix.

• The use of Nix for software deployment extends naturally to the ﬁeld of service de-

ployment. Services are running processes that provide some useful facility to users,

such as web servers. They consist of software components, static conﬁguration and

data, and mutable state. The ﬁrst two aspects can be managed using Nix, and its

advantages—such as rollbacks and side-by-side deployment—apply here as well.

• Though Nix is typically used to build large-grained components (i.e., traditional

packages), it can also be used to build small-grained components such as individual

source ﬁles. When used in this way it is a superior alternative to build managers

1. Introduction

such as Make [56], ensuring complete dependency speciﬁcations and enabling more

sharing between builds.

The above may sound like a product feature list, which is not necessarily a good thing

as it is always possible to add any feature to a system given enough ad-hockery. However,

these contributions all follow from a small set of principles, foremost among them the

purely functional model.

1.6. Outline of this thesis

This thesis is divided in four parts. Part I (which includes this chapter) introduces the

problem domain and motivates the Nix system. Chapter 2, “An Overview of Nix”, intro-

duces the Nix system from a high-level perspective (i.e., that of users or authors of Nix

components), and gives some intuitions about the underlying principles.

Part II is about the principles and formal semantics of Nix. The basic philosophy behind

Nix—to solve deployment problems by treating them analogously to memory management

issues in programming languages—is introduced in Chapter 3, “Deployment as Memory

Management”.

The syntax and semantics of the Nix expression language—a purely functional language

used to describe components and compositions—is given in Chapter 4, “The Nix Expres-

sion Language”. This chapter also discusses some interesting aspects of the implementa-

tion of the Nix expression evaluator.

The next two chapters form the technical core of this thesis, as they describe the Nix

component store and the fundamental operations on it. In fact, there are two models for

the Nix store that are subtly different with important consequences. Chapter 5, “The Ex-

tensional Model”, formalises the extensional model, which was the ﬁrst model and the one

with which we have gained the most experience. This chapter describes the objects that

live in the Nix store, the building of Nix expressions, binary deployment as an optimisation

of source deployment, distributed and multi-platform builds, and garbage collection.

The extensional model however has several downsides, in particular the inability to se-

curely share a Nix store between multiple users. These problems are ﬁxed in the more

recent intensional model, described in Chapter 6, “The Intensional Model”.

Part III discusses in detail the various applications of Nix. Chapter 7, “Software Deploy-

ment”, is about Nix’s raison d’être, software deployment. It describes principles for Nix

expression writers that help ensure correct deployment, and various policies that deploy

components to clients. The main vehicle for this discussion is the Nix Packages collection

(Nixpkgs), a large set of existing Unix components that we deploy through Nix. Most of

our experience with the Nix system is in the context of the Nix Packages collection, so this

chapter provides a certain level of empirical validation for the Nix approach.

Chapter 8, “Continuous Integration and Release Management”, shows that Nix pro-

vides a solid basis for the implementation of a build farm, which is a facility that automati-

cally builds software components from a version management repository. This is useful to

support continuous integration testing, which provides developers with almost immediate

feedback as to whether their changes integrate cleanly with the work of other developers.

1.7. Notational conventions

Also, a Nix build farm can produce concrete releases that can be automatically deployed

to user machines, for instance through the channel mechanism described in Chapter 7.

Chapter 9, “Service Deployment”, takes Nix beyond deployment of software compo-

nents and into the realm of service deployment. It shows that Nix extends gracefully into

this domain using several examples of production services deployed using Nix.

Chapter 10, “Build Management”, extends Nix in another direction, towards support for

low-level build management. As stated above, Nix provides exactly the right technology

to implement a build manager that ﬁxes the problems of most contemporary build tools.

Roadmap Different readers will ﬁnd different parts of this thesis of interest. Chapter 2,

“An Overview of Nix” should be read by anyone. The semantics of the Nix system in Part II

can be skipped—initially, at least—by readers who are interested in Nix as a tool, but is

required reading for those who wish to modify Nix itself. Nix users will be most interested

in the various applications discussed in Part III. Chapter 7, “Software Deployment” on

deployment should not be skipped as it provides an in-depth discussion on Nix expressions,

builders, and deployment mechanisms and policies that are also relevant to the subsequent

application chapters.

Nix users and developers will also ﬁnd the online manual useful [161].

Origins Parts of this thesis are adapted from earlier publications. Chapter 2, “An Over-

view of Nix” is very loosely based on the LISA ’04 paper “Nix: A Safe and Policy-

Free System for Software Deployment”, co-authored with Merijn de Jonge and Eelco

Visser [50]. Chapter 3, “Deployment as Memory Management” is based on Sections 3–6 of

the ICSE 2004 paper “Imposing a Memory Management Discipline on Software Deploy-

ment”, written with Eelco Visser and Merijn de Jonge [52]. Chapter 6, “The Intensional

Model” is based on the ASE 2005 paper “Secure Sharing Between Untrusted Users in a

Transparent Source/Binary Deployment Model” [48]. Section 7.5 on patch deployment

appeared as the CBSE 2005 paper “Efﬁcient Upgrading in a Purely Functional Compo-

nent Deployment Model” [47]. Chapter 9, “Service Deployment” is based on the SCM-12

paper “Service Conﬁguration Management”, written with Martin Bravenboer and Eelco

Visser [49]. Snippets of Section 10.2 were taken from the SCM-11 paper “Integrating

Software Construction and Software Deployment” [46].

1.7. Notational conventions

Part II of this thesis contains a number of algorithms in a pseudo-code notation in an im-

perative style with some functional elements. Keywords in algorithms are in boldface,

variables are in italic, and function names are in sans ser if. Callouts like this 1

are fre-

quently used to refer to points of interest in algorithms and listings from the text. Variable

assignment is written x ← value.

Sets are written in curly braces, e.g., {1,2,3}. Since many algorithms frequently extend

variables of set type with new elements, there is a special notation x

∪

← set which is syn-

tactic sugar for x ← x ∪ set. For instance, x

∪

← {2,3} extends the set stored in x with the

elements 2 and 3.

1. Introduction

Set comprehensions are a concise notation for set construction: {expr | conditions} pro-

duces a set by applying the expression expr to all values from a (usually implicit) universe

that meet the predicate conditions. For example, {x

| x ∈ {2, 3,4} ∧ x 6= 3} produces the

set {4, 16}.

Ordered lists are written between square brackets, e.g., [1, 2, 1, 3]. Lists can contain ele-

ments multiple times. Lists are sometimes manipulated in a functional (Haskell-like [135])

style using the function map and λ -abstraction: x ← map(λ v . f (v),y) constructs a list x by

applying a function f (v) to each element in the list x. For instance, map(λ x . x

, [4,2,6])

produces the list [16, 4, 36]. Tuples or pairs are enclosed in parentheses, e.g., (a,b,c).

Throughout this thesis, storage sizes are given in IEC units [33]: 1 KiB = 1024 bytes,

1 MiB = 1024

bytes, and 1 GiB = 1024

bytes.

Data types are deﬁned in a Haskell-like notation. For instance, the deﬁnition

data Foo = Foo {

x : String,

ys : [String],

zs : {String}

}

deﬁnes a data type Foo with three ﬁelds: a string x, a list of strings ys, and a set of strings

zs. An example of the construction of a value of this type is as follows:

Foo {x = "test", ys = ["hello", "world"],zs = /0}

In addition, types can have unnamed ﬁelds and multiple constructors. For example, the

deﬁnition

data Tree = Leaf String | Branch Tree Tree

deﬁnes a binary tree type with strings stored in the leaves. An example value is

Branch (Leaf "a") (Branch (Leaf "b") (Leaf "c")).

2. An Overview of Nix

The previous chapter discussed the features that we require from a software deployment

system, as well as the shortcomings of current technology. The subject of this thesis is

the Nix deployment system, which addresses many of these problems. This chapter gives

a gentle introduction to Nix from a user perspective (where “user” refers to component

deployers as well as end-users). The succeeding parts of this thesis will present a more

formal exposition of Nix’s semantics (Part II) and applications (Part III).

2.1. The Nix store

Nix is a system for software deployment. The term component will be used to denote the

basic units of deployment. These are simply sets of ﬁles that implement some arbitrary

functionality through an interface. In fact, Nix does not really care what a component ac-

tually is. As far as Nix is concerned a component is just a set of ﬁles in a ﬁle system. That

is, we have a very weak notion of component, weaker even than the commonly used deﬁ-

nition in [155]. (What we call a component typically corresponds to the ambiguous notion

of a package in package management systems. Nix’s notion of components is discussed

further in Section 3.1.)

Nix stores components in a component store, also called the Nix store. The store is

simply a designated directory in the ﬁle system, usually /nix/store. The entries in that

directory are components (and some other auxiliary ﬁles discussed later). Each component

is stored in isolation; no two components have the same ﬁle name in the store.

A subset of a typical Nix store is shown in Figure 2.1. It contains a number of applica-

tions—GNU Hello 2.1.1 (a toy program that prints “Hello World”, Subversion 1.1.4 (a ver-

sion management system), and Subversion 1.2.0—along with some of their dependencies.

These components are not single ﬁles, but directory trees. For instance, Subversion con-

sists of a directory called bin containing a program called svn, and a directory lib containing

many more ﬁles belonging to the component. (For simplicity, many ﬁles and dependencies

have been omitted from the example.) The arrows denote runtime dependencies between

components, which will be described shortly.

The notable feature in Figure 2.1 is the names of the components in the Nix store,

such as /nix/store/bwacc7a5c5n3...-hello-2.1.1. The initial part of the ﬁle names, e.g.,

bwacc7a5c5n3..., is what gives Nix the ability to prevent undeclared dependencies and

component interference. It is a representation of the cryptographic hash of all inputs in-

volved in building the component.

Cryptographic hash functions are hash functions that map an arbitrary-length input onto

a ﬁxed-length bit string. They have the property that they are collision-resistant: it is

computationally infeasible to ﬁnd two inputs that hash to the same value. Cryptographic

hashes are discussed in more detail in Section 5.1. Nix uses 160-bit hashes represented in

2. An Overview of Nix

/nix/store

h7yw7a257m1i...-db4-4.3.28

lib

libdb-4.so

v2cc475f6nv1...-subversion-1.1.4

bin

svn

lib

libsvn...so

72by2iw5wd8i...-glibc-2.3.5

lib

libc.so.6

bwacc7a5c5n3...-hello-2.1.1

bin

hello

man

man1

hello1

5jq6jgkamxjj...-openssl-0.9.7d

lib

libssl.so.0.9.7

dg7c6zr5vqrj...-subversion-1.2.0

bin

svn

lib

libsvn...so

8ggmblhvzciv...-openssl-0.9.7f

lib

libssl.so.0.9.7

Figure 2.1.: The Nix store

/nix/store

h7yw7a257m1i...-db4-4.3.28

lib

libdb-4.so

v2cc475f6nv1...-subversion-1.1.4

bin

svn

lib

libsvn...so

72by2iw5wd8i...-glibc-2.3.5

lib

libc.so.6

bwacc7a5c5n3...-hello-2.1.1

bin

hello

man

man1

hello1

5jq6jgkamxjj...-openssl-0.9.7d

lib

libssl.so.0.9.7

dg7c6zr5vqrj...-subversion-1.2.0

bin

svn

lib

libsvn...so

8ggmblhvzciv...-openssl-0.9.7f

lib

libssl.so.0.9.7

Figure 2.2.: Closure in the store

a base-32 notation, so each hash is 32 characters long. In this thesis, hashes are abridged

to ellipses (...) most of the time for reasons of legibility. The full path of a directory entry

in the Nix store is referred to as its store path. An example of a full store path is:

/nix/store/bwacc7a5c5n3qx37nz5drwcgd2lv89w6-hello-2.1.1

So the ﬁle bin/hello in that component has the full path

/nix/store/bwacc7a5c5n3qx37nz5drwcgd2lv89w6-hello-2.1.1/bin/hello

The hash is computed over all inputs, including the following:

• The sources of the components.

• The script that performed the build.

• Any arguments or environment variables [152] passed to the build script.

• All build time dependencies, typically including the compiler, linker, any libraries

used at build time, standard Unix tools such as cp and tar, the shell, and so on.

Cryptographic hashes in store paths serve two main goals. They prevent interference

between components, and they allow complete identiﬁcation of dependencies. The lack

2.1. The Nix store

of these two properties is the cause for the vast majority of correctness and ﬂexibility

problems faced by conventional deployment systems, as we saw in Section 1.2.

Preventing interference Since the hash is computed over all inputs to the build process

of the component, any change, no matter how slight, will be reﬂected in the hash. This

includes changes to the dependencies; the hash is computed recursively. Thus, the hash

essentially provides a unique identiﬁer for a conﬁguration. If two component compositions

differ in any way, they will occupy different paths in the store (except for dependencies that

they have in common). Installation or uninstallation of a conﬁguration therefore will not

interfere with any other conﬁguration.

For instance, in the Nix store in Figure 2.1 there are two versions of Subversion. They

were built from different sources, and so their hashes differ. Additionally, Subversion

1.2.0 uses a newer version of the OpenSSL cryptographic library. This newer version of

OpenSSL likewise exists in a path different from the old OpenSSL. Thus, even though

installing a new Subversion entails installing a new OpenSSL, the old Subversion instance

is not affected, since it continues to use the old OpenSSL.

Recursive hash computation causes changes to a component to “propagate” through the

dependency graph. This is illustrated in Figure 2.3, which shows the hash components of

the store paths computed for the Mozilla Firefox component (a web browser) and some of

its build time dependencies, both before and after a change is made to one of those depen-

dencies. (An edge from a node A to a node B denotes that the build result of A is an input to

the build process of B.) Speciﬁcally, the GTK GUI library dependency is updated from ver-

sion 2.2.4 to 2.4.13. That is, its source bundle (gtk+-2.2.4.tar.bz2 and gtk+-2.4.13.tar.bz2,

respectively) changes. This change propagates through the dependency graph, causing dif-

ferent store paths to be computed for the GTK component and the Firefox component.

However, components that do not depend on GTK, such as Glibc (the GNU C Library),

are unaffected.

An important point here is that upgrading only happens by rebuilding the component in

question and all components that depend on it. We never perform a destructive upgrade.

Components never change after they have been built—they are marked as read-only in the

ﬁle system. Assuming that the build process for a component is deterministic, this means

that the hash identiﬁes the contents of the components at all times, not only just after it has

been built. Conversely, the build-time inputs determine the contents of the component.

Therefore we call this a purely functional model. In purely functional programming

languages such as Haskell [135], as in mathematics, the result of a function call depends

exclusively on the deﬁnition of the function and on the arguments. In Nix, the contents of

a component depend exclusively on the build inputs. The advantage of a purely functional

model is that we obtain strong guarantees about components, such as non-interference.

Identifying dependencies The use of cryptographic hashes in component ﬁle names

also prevents the use of undeclared dependencies, which (as we saw in Section 1.2) is the

major cause of incomplete deployment. The reason that incomplete dependency informa-

tion occurs is that there is generally nothing that prevents a component from accessing

another component, either at build time or at runtime. For instance, a line in a build script

or Makeﬁle such as

2. An Overview of Nix

acyn3qg8z5y4...-

gtk+-2.2.4.tar.bz2

zmhd227j57rx...-

gtk+-2.2.4

mkmpxqr8d7f7...-

firefox-1.0

ja7kjr1njh6b...-

glibc-2.3.5.tar.bz2

72by2iw5wd8i...-

glibc-2.3.5

x3cgx0xj1p4i...-

gcc-3.4.3

pmdmxsff5kpm...-

gcc-3.4.3.tar.bz2

lrclv7w3fb13...-

firefox-1.0.tar.bz2

Before

cak0k28vxcsh...-

gtk+-2.4.13.tar.bz2

j4v4n2an42jw...-

gtk+-2.4.13

r0ndfgl4w95m...-

firefox-1.0

ja7kjr1njh6b...-

glibc-2.3.5.tar.bz2

72by2iw5wd8i...-

glibc-2.3.5

x3cgx0xj1p4i...-

gcc-3.4.3

pmdmxsff5kpm...-

gcc-3.4.3.tar.bz2

lrclv7w3fb13...-

firefox-1.0.tar.bz2

After

Figure 2.3.: Propagation of dependency changes through the store paths of the build-time

component dependency graph

gcc -o program main.c ui.c -lssl

which links a program consisting of two C modules against a library named ssl, causes

the linker to search in a set of standard locations for a library called ssl

. These standard

locations typically include “global” system directories such as /usr/lib on Unix systems. If

the library happens to exist in one of those directories, we have incurred a dependency.

However, there is nothing that forces us to include this dependency in the dependency

speciﬁcations of the deployment system (e.g., in the RPM spec ﬁle of Figure 1.2).

At runtime we have the same problem. Since components can perform arbitrary I/O

actions, they can load and use other components. For instance, if the library ssl used

above is a dynamic library, then program will contain an entry in its dynamic linkage

meta-information that causes the dynamic linker to search for the library when program is

started. The dynamic linker similarly searches in global locations such as /lib and /usr/lib.

Of course, there are countless mechanisms other than static or dynamic linking that es-

tablish a component composition. Some examples are including C header ﬁles, importing

Java classes at compile time, calling external programs found through the PATH environ-

ment variable, and loading help ﬁles at runtime.

The hashing scheme neatly prevents these problems by not storing components in global

locations, but in isolation from each other. For instance, assuming that all components in

the system are stored in the Nix store, the linker line

gcc -o program main.c ui.c -lssl

will simply fail to ﬁnd libssl. Unless the path to an OpenSSL instance (e.g., /nix/store/-

5jq6jgkamxjj...-openssl-0.9.7d) was explicitly fed into the build process and added to the

linker’s search path, no such instance will be found by the linker.

To be precise, libssl.a or libssl.so on Unix.

2.1. The Nix store

00000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 |.ELF............|

00000010 02 00 03 00 01 00 00 00 40 bb 04 08 34 00 00 00 |........@...4...|

...

00000130 04 00 00 00 2f 6e 69 78 2f 73 74 6f 72 65 2f 37

37 |..../nix/store/7

00000140 32 62 79 32 69 77 35 77 64 38 69 68 72 35 79 37

32 62 79 32 69 77 35 77 64 38 69 68 72 35 79 37

32 62 79 32 69 77 35 77 64 38 69 68 72 35 79 37 |2by2iw5wd8ihr5y7

2by2iw5wd8ihr5y7

2by2iw5wd8ihr5y7|

00000150 6e 31 31 31 6a 66 77 6c 37 33 71 32 68 36 37

6e 31 31 31 6a 66 77 6c 37 33 71 32 68 36 37

6e 31 31 31 6a 66 77 6c 37 33 71 32 68 36 37 2d |n111jfwl73q2h67

n111jfwl73q2h67

n111jfwl73q2h67-|

00000160 67 6c 69 62 63 2d 32 2e 33 2e 35 2f 6c 69 62 2f |glibc-2.3.5/lib/|

00000170 6c 64 2d 6c 69 6e 75 78 2e 73 6f 2e 32 00 00 00 |ld-linux.so.2...|

...

00002670 73 74 6f 72 65 2f 35 6a 71 36 6a 67 6b 61 6d 78

35 6a 71 36 6a 67 6b 61 6d 78

35 6a 71 36 6a 67 6b 61 6d 78 |store/5jq6jgkamx

5jq6jgkamx

5jq6jgkamx|

00002680 6a 6a 64 64 6c 61 76 67 76 63 39 6e 76 30 6c 72

6a 6a 64 64 6c 61 76 67 76 63 39 6e 76 30 6c 72

6a 6a 64 64 6c 61 76 67 76 63 39 6e 76 30 6c 72 |jjddlavgvc9nv0lr

jjddlavgvc9nv0lr

jjddlavgvc9nv0lr|

00002690 6d 38 66 79 73 37

6d 38 66 79 73 37

6d 38 66 79 73 37 2d 6f 70 65 6e 73 73 6c 2d 30 |m8fys7

m8fys7

m8fys7-openssl-0|

000026a0 2e 39 2e 37 64 2f 6c 69 62 3a 2f 6e 69 78 2f 73 |.9.7d/lib:/nix/s|

...

Figure 2.4.: Retained dependencies in the svn executable

Also, we go to some lengths to ensure that component builders are pure, that is, not

inﬂuenced by outside factors. For example, the builder is called with an empty set of

environment variables (such as the PATH environment variable, which is used by Unix

shells to locate programs) to prevent user settings such as search paths from reaching tools

invoked by the builder. Similarly, at runtime on Linux systems, we use a patched dynamic

linker that does not search in any default locations—so if a dynamic library is not explicitly

declared with its full path in an executable, the dynamic linker will not ﬁnd it.

Thus, when the developer or deployer fails to specify a dependency explicitly (in the Nix

expression formalism, discussed below), the component will fail deterministically. That

is, it will not succeed if the dependency already happens to be available in the Nix store,

without having been speciﬁed as an input. By contrast, the deployment systems discussed

in Section 1.2 allow components to build or run successfully even if some dependencies

are not declared.

Retained dependencies A rather tricky aspect to dependency identiﬁcation is the oc-

currence of retained dependencies. This is what happens when the build process of a

component stores a path to a dependency in the component. For instance, the linker invo-

cation above will store the path of the OpenSSL library in program, i.e., program will have

a “hard-coded” reference to /nix/store/5jq6jgkamxjj...-openssl-0.9.7d/lib/libssl.so.

Figure 2.4 shows a dump of parts of the Subversion executable stored in the ﬁle /nix/-

store/v2cc475f6nv1...-subversion-1.1.4/bin/svn. Offsets are on the left, a hexadecimal rep-

resentation in the middle, and an ASCII representation on the right. The build process for

Subversion has caused a reference to the aforementioned OpenSSL instance to be stored in

the program’s executable image. The path of OpenSSL has been stored in the RPATH ﬁeld

of the header of the ELF executable, which speciﬁes a list of directories to be searched

by the dynamic linker at runtime [160]. Even though our patched dynamic linker does

not search in /nix/store/5jq6jgkamxjj...-openssl-0.9.7d/lib by default, it will ﬁnd the library

anyway through the executable’s RPATH.

This might appear to be bad news for our attempts to prevent undeclared dependencies.

Of course, we happen to know the internal structure of Unix executables, so for this speciﬁc

ﬁle format we can discover retained dependencies. But we do not know the format of every

2. An Overview of Nix

ﬁle type, and we do not wish to commit ourselves to a single component composition

mechanism. E.g., Java and .NET can ﬁnd retained dependencies by looking at class ﬁles

and assemblies, respectively, but only for their speciﬁc dynamic linkage mechanisms (and

not for dependencies loaded at runtime).

The hashing scheme comes to the rescue once more. The hash part of component paths

is highly distinctive, e.g., 5jq6jgkamxjj.... Therefore we can discover retained dependen-

cies generically, independent of speciﬁc ﬁle formats, by scanning for occurrences of hash

parts. For instance, the executable image in Figure 2.4 contains the highlighted string

5jq6jgkamxjj..., which is evidence that an execution of the svn program might need that

particular OpenSSL instance. Likewise, we can see that it has a retained dependency on

some Glibc instance (/nix/store/72by2iw5wd8i.... Thus, we automatically add these as run-

time dependencies of the Subversion component. Using this scanning approach, we ﬁnd

the runtime dependencies indicated in Figure 2.1.

This approach might seem a bit dodgy. After all, what happens when a ﬁle name is rep-

resented in a non-standard way, e.g., in UTF-16 [34, Section 2.5] or when the executable

is compressed? In Chapter 3 the dependency problem is cast in terms of memory man-

agement in programming languages, which justiﬁes the scanning approach as an analogue

of conservative garbage collection. Whether this is a legitimate approach is an empirical

question, which is addressed in Section 7.1.5.

Note that strictly speaking it is not the use of cryptographic hashes per se, but globally

unique identiﬁers in ﬁle names that make this work. A sufﬁciently long pseudo-random

number also does the trick. However, the hashes are needed to prevent interference, while

at the same time preventing unnecessary duplication of identical components (which would

happen with random paths).

Closures Section 1.2 ﬁrst stated the goal of complete deployment: safe deployment re-

quires that there are no missing dependencies. This means that we need to deploy closures

of components under the “depends-on” relation. That is, when we deploy (i.e., copy) a

component X to a client machine, and X depends on Y , then we also need to deploy Y to

the client machine.

The hash scanning approach gives us all runtime dependencies of a component, while

hashes themselves prevent undeclared build-time dependencies. Furthermore, these de-

pendencies are exact, not nominal (see page 8). Thus, Nix knows the entire dependency

graph, both at build time and runtime. With full knowledge of the dependency graph, Nix

can compute closures of components. Figure 2.2 shows the closure of the Subversion 1.1.4

instance in the Nix store, found by transitively following all dependency arrows.

In summary, the purely functional model, and its concrete implementation in the form

of the hashing approach used by the Nix store, prevents interference and enables complete

deployment. It makes deployment much more likely to be correct, and is therefore one

of the major results of this thesis. However, the purely functional model does provoke a

number of questions, such as:

• Hashes do not appear to be very user-friendly. Can we hide them from users in

everyday interaction?

• Can we deploy upgrades efﬁciently? That is, suppose that we want to upgrade Glibc

2.2. Nix expressions

(a dependency of all other components in Figure 2.1). Can we prevent a costly

redownload of all dependent components?

As we will see in the remainder of this thesis, the answer to these questions is “yes”.

2.2. Nix expressions

Nix components are built from Nix expressions. The language of Nix expressions is a sim-

ple purely functional language used to describe components and the compositions thereof.

This section introduces the Nix expression language using a simple example. In com-

puter science’s ﬁnest tradition we will start with “Hello World”; to be precise, a Nix ex-

pression that builds the GNU Hello package. GNU Hello is a program that prints out the

text Hello World and so “allows nonprogrammers to use a classic computer science tool

which would otherwise be unavailable to them” [67]. It is representative of what one must

do to deploy a component using Nix. Generally, to deploy a component one performs the

following three steps:

• Write a Nix expression for the component (Figure 2.6), which describes all the in-

puts involved in building the component, such as dependencies (other components

required by the component), sources, and so on.

• Write a builder (Figure 2.7)—typically a shell script—that actually builds the com-

ponent from the inputs.

• Create a composition (Figure 2.8). The Nix expression written in the ﬁrst step is a

function: in order to build a concrete component, it requires that the dependencies are

ﬁlled in. To compose the component with its dependencies, we must write another

Nix expression that calls the function with appropriate arguments.

The Nix Packages collection The GNU Hello example is taken from the Nix Packages

collection (Nixpkgs), a large set of Nix expressions for common and not-so-common soft-

ware components. Since Nixpkgs contains many components and also describes their com-

positions, many ﬁles are involved in building a component. It is useful to have a mental pic-

ture of how the different parts ﬁt together. Figure 2.5 shows the directory structure of parts

of the Nix Packages collection, including some of the Nix expressions, builders, and mis-

cellaneous inputs contained therein. The function that builds Hello components is deﬁned

in pkgs/applications/misc/hello/default.nix. Likewise, many other components are deﬁned,

hierarchically organised by component purpose or topic. Builders are placed in the same

directory as the Nix expression that uses them, e.g., pkgs/applications/misc/hello/builder.sh

for the Hello build script.

Compositions are deﬁned in pkgs/system/all-packages-generic.nix, which imports the

various component-speciﬁc Nix expressions and puts them all together. The ﬁle has

generic in its name because it is itself a function that returns compositions for vari-

ous machine architecture and operating system platforms. The function pkgs/system/all-

packages.nix does actually evaluate to a set of concrete components that can be built and

installed on the current platform.

2. An Overview of Nix

nixpkgs

pkgs

applications

misc

hello

default.nix

builder.sh

networking

browsers

ﬁrefox

default.nix

builder.sh

writable-copies.patch

development

libraries

glibc

default.nix

builder.sh

compilers

gcc

...

interpreters

perl

...

system

all-packages-generic.nix

all-packages.nix

stdenvs.nix

Figure 2.5.: Organisation of the Nix Packages collection

There is nothing in Nix that requires this organisation; it is merely a useful convention.

It is perfectly possible to place component expressions and builders in the same directory

by naming them appropriately, e.g., hello.nix, hello-builder.sh, ﬁrefox.nix, and so on. It is

also possible to put all Nix expressions in a single ﬁle, e.g., put everything in all-packages-

generic.nix, which of course would make this ﬁle quickly spiral out of control. In fact,

it is not even necessary to deﬁne components as functions and compose them separately,

as done in the 3-step procedure above; we could remove all variability and write Nix

expressions that describe exactly one build action, rather than a family of build actions.

Clearly, that is not a very general approach.

We now look at the parts that constitute the Hello component in detail.

Nix expression for the Hello component Figure 2.6 shows a Nix expression for GNU

Hello. It has a number of salient features:

2 This states that the expression is a function that expects to be called with three ar-

guments: stdenv, fetchurl, and perl. They are needed to build Hello, but we don’t

know how to build them here; that’s why they are function arguments. stdenv is a

component that is used by almost all Nix Packages components; it provides a “stan-

dard” environment consisting of the things one expects in a basic Unix environment:

a C/C++ compiler (GCC, to be precise), the Bash shell, fundamental Unix tools

2.2. Nix expressions

{stdenv, fetchurl, perl}:

stdenv.mkDerivation {

name = "hello-2.1.1";

builder = ./builder.sh;

src = fetchurl {

url = http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz;

md5 = "70c9ccf9fac07f762c24f2df2290784d";

};

inherit perl;

}

Figure 2.6.: pkgs/applications/misc/hello/default.nix: Nix expression for GNU Hello

such as cp, grep, tar, etc. fetchurl is a function that downloads ﬁles. perl is the Perl

interpreter.

Nix functions generally have the form {x, y, ..., z}: e where x, y, etc. are the names

of the expected arguments, and where e is the body (result) of the function. So here,

the entire remainder of the ﬁle is the body of the function; when given the required

arguments, the body describes how to build an instance of the Hello component.

3 The result of the function in this case is a derivation. This is Nix-speak for a com-

ponent build action, which derives the component from its inputs. We perform a

derivation by calling stdenv.mkDerivation. mkDerivation is a function provided by

stdenv that builds a component from a set of attributes

. An attribute set is just a

list of key/value pairs where each value is an arbitrary Nix expression. They take the

general form {name

= expr

; ... name

= expr

;}.

The attributes given to stdenv.mkDerivation are the concrete inputs to the build ac-

tion.

4 A few of the attributes to derivations are special. The attribute name speciﬁes the

symbolic name and version of the component. It is appended to the cryptographic

hash in store paths (see, e.g., Figure 2.1). Nix doesn’t care very much about it most

of the time.

5 The attribute builder speciﬁes the script that actually builds the component. This

attribute can sometimes be omitted, in which case stdenv.mkDerivation will ﬁll in

a default builder (which essentially performs a “standard” Unix component build

sequence consisting of the commands conﬁgure; make; make install). Hello is suf-

ﬁciently simple that the default builder sufﬁces; but in this case, for educational

purposes an actual builder is shown in Figure 2.7, discussed below.

6 The builder has to know what the sources of the component are. Here, the attribute

src is bound to the result of a call to the fetchurl function. Given a URL and an MD5

hash of the expected contents of the ﬁle at that URL, this function builds a derivation

mkDerivation is actually a wrapper around the primitive operation derivation, shown on page 80.

2. An Overview of Nix

source $stdenv/setup

PATH=$perl/bin:$PATH

tar xvfz $src

cd hello-*

./configure --prefix=$out

make

make install

Figure 2.7.: pkgs/applications/misc/hello/builder.sh: Builder for GNU Hello

that downloads the ﬁle and checks its hash. So the sources are dependencies that like

all other dependencies are built before Hello itself is built.

Instead of src any other name could have been used, and in fact there can be any

number of sources (bound to different attributes). However, src is customary, and it

is also expected by the default builder (which we don’t use in this example).

7 Since the derivation requires Perl, we have to pass the value of the perl function

argument to the builder. All attributes in the set are actually passed as environment

variables to the builder, so declaring an attribute

perl = perl;

will do the trick: it binds an attribute perl to the function argument which also hap-

pens to be called perl. However, this looks a bit silly, so there is a shorter syntax.

The inherit keyword causes the speciﬁed attributes to be bound to whatever variables

with the same name happen to be in scope.

Builder for the Hello component Figure 2.7 shows the builder for GNU Hello. To build

a component, Nix executes the builder. The attributes in the attribute set of the derivation

are passed as environment variables. Attribute values that evaluate to derivations (such as

perl, which below in Figure 2.8 is bound to a derivation) are built recursively. The paths

of the resulting components are passed through the corresponding environment variables.

The special environment variable out holds the intended output path. This is where the

builder should store its result.

The build script for Hello is a typical Unix build script: it unpacks the sources, runs a

conﬁgure script that detects platform characteristics and generates Makeﬁles, runs make to

compile the program, and ﬁnally runs make install to copy it to its intended location in the

ﬁle system. However, there are some Nix-speciﬁc points:

8 When Nix runs a builder, it initially completely clears all environment variables (ex-

cept for the attributes declared in the derivation). For example, the PATH variable is

empty

. This is done to prevent undeclared inputs from being used in the build pro-

cess. If for example the PATH contained /usr/bin, then the builder might accidentally

use /usr/bin/gcc.

Actually, it is initialised to /path-not-set to prevent Bash from setting it to a default value.

2.2. Nix expressions

The ﬁrst step is therefore to set up the environment so that the tools included in the

standard environment are available. This is done by calling the setup script of the

standard environment. The environment variable stdenv points to the location of the

standard environment being used. (It wasn’t speciﬁed explicitly as an attribute in the

Hello expression in Figure 2.6, but mkDerivation adds it automatically.)

9 Since Hello needs Perl, we have to make sure that Perl is in the PATH. The perl en-

vironment variable points to the location of the Perl component (since it was passed

in as an attribute to the derivation), so the shell expression $per l/bin evaluates to the

directory containing the Perl interpreter.

10 Now we have to unpack the sources. The src attribute was bound to the result of

fetching the Hello source distribution from the network, so the src environment vari-

able points to the location in the Nix store to which the distribution was downloaded.

After unpacking, we change to the resulting source directory.

11 GNU Hello is a typical Autoconf-based package. Autoconf [64, 172] is a tool that

allows a source-based package to automatically adapt itself to its environment by de-

tecting properties such as availability and paths of libraries and other dependencies,

quirks in the C compiler, and so on. Autoconf-based packages always have a script

called conﬁgure that performs these tests and generates ﬁles such as Makeﬁles, C

header ﬁles (e.g., conﬁg.h), etc. Thus, we ﬁrst have to run Hello’s conﬁgure script

(which was generated by Autoconf). The store path where the component is to be

stored, is passed in through the environment variable out. Here we tell conﬁgure to

produce Makeﬁles that will build and install a component in the intended output path

in the Nix store.

Finally we build Hello (make) and install it into the location speciﬁed by the out

variable (make install).

Composition of the Hello component Since the Nix expression for Hello in Figure 2.6

is a function, we cannot use it directly to build and install a Hello component. We need

to compose it ﬁrst; that is, we have to call the function with the expected arguments. In

the Nix Packages collection this is done in the ﬁle pkgs/system/all-packages-generic.nix,

where all Nix expressions for components are imported and called with the appropriate

arguments. This is a large ﬁle, but the parts relevant to Hello are shown in Figure 2.8.

13 This ﬁle deﬁnes a set of attributes, all of which in this case evaluate to concrete

derivations (i.e., not functions). In fact, we deﬁne a mutually recursive set of at-

tributes using the keyword rec. That is, the attributes can refer to each other

. This

is precisely what we want, namely, “plug” the various components into each other.

14 Here we import the Nix expression for GNU Hello. The import operation just loads

and returns the speciﬁed Nix expression. In fact, we could just have put the contents

While attributes can refer to values deﬁned in the same scope, derivations cannot be mutual dependencies:

such derivations would be impossible to build (see also Lemma 3 on page 130).

2. An Overview of Nix

...

rec {

hello = (import ../applications/misc/hello

{

inherit fetchurl stdenv perl;

};

perl = (import ../development/interpreters/perl) {

inherit fetchurl stdenv;

};

fetchurl = (import ../build-support/fetchurl) {

inherit stdenv; ...

};

stdenv = ...;

}

Figure 2.8.: pkgs/system/all-packages-generic.nix: Composition of GNU Hello and other

components

of Figure 2.6 in all-packages-generic.nix at this point; this is completely equivalent,

but it would make the ﬁle rather bulky.

Relative paths such as ../applications/misc/hello are relative to the directory of the Nix

expression that contains them, rather than the current directory or some global search

path. Since the expression in Figure 2.8 is stored in pkgs/system/all-packages-

generic.nix, this relative path resolves to pkgs/applications/misc/hello, which is the

directory of the Nix expression in Figure 2.6.

Note that we refer to ../applications/misc/hello, not ../applications/misc/hello/-

default.nix. The latter is actually equivalent to the former in the case of directories;

when importing a path referring to a directory, Nix automatically appends /default.nix

to the path.

15 This is where the actual composition takes place. Here we call the function imported

from ../applications/misc/hello with an attribute set containing the arguments that the

function expects, namely fetchurl, stdenv, and perl. We use the inherit construct

again to copy the attributes deﬁned in the surrounding scope (we could also have

written fetchurl = fetchurl;, etc.). Note that Nix function arguments are not positional;

Nix functions just expect an attribute set containing attributes corresponding to the

declared (formal) arguments.

The result of this function call is an actual derivation that can be built by Nix (since

when we ﬁll in the arguments of the function, what we get is its body, which is the

call to stdenv.mkDerivation in Figure 2.6).

16 Likewise, concrete instances of Perl, fetchurl, and the standard environment are con-

2.2. Nix expressions

structed by calling their functions with the appropriate arguments.

The attribute fetchurl actually does not evaluate to a derivation, but to a function that

takes arguments url and md5, as seen in Figure 2.6. This is an example of partial

parameterisation.

Thus, the value bound to the hello attribute in Figure 2.8 represents a derivation of an

instance of Hello, as well as of all its dependencies.

Surprisingly, describing not just components but also compositions is not a common

feature of the component description formalisms of deployment systems. For instance,

RPM spec ﬁles (see page 6) describe how to build components, but not how to build the

dependencies—even though the dependencies can have a profound effect on the build re-

sult. The actual composition is left implicit. Nominal dependency speciﬁcations are used

to require the presence of certain dependencies, but as we have seen, nominal dependency

speciﬁcations are insufﬁcient. Formalisms such as RPM spec ﬁles therefore woefully un-

derspecify components. Only a full set of RPM spec ﬁles closed under the dependency

relation fully describes a component build (disregarding, of course, the real possibility of

undeclared dependencies and other environmental inﬂuences).

A slightly more complex example Figure 2.9 shows a Nix expression that demonstrates

some additional language features. It contains a function that produces Subversion com-

ponents. Subversion [31, 137] is a version management system [159, 180]. Subversion

clients can access a central version management repository through various means: local

ﬁle system access, WebDAV (an extension to HTTP to support authoring and version-

ing [30]), and the specialised svnserve protocol. These various access methods lead to a

number of optional features, which in turn lead to optional dependencies

• Subversion has two storage back-ends for repositories: one that uses a built-in ﬁle

system based storage, and one that uses the external Berkeley DB embedded database

library. Use of the latter back-end induces a dependency on the Berkeley DB com-

ponent.

• Subversion can build an Apache module that allows it to act as a WebDAV server for

remote clients. This feature requires the Apache HTTPD server component.

• Subversion can access remote WebDAV repositories over HTTP by default. If access

over HTTPS (i.e., HTTP encrypted over the Secure Sockets Layer [76]) is desired,

then it requires the OpenSSL cryptographic library component.

• Data sent over HTTP connections can be compressed using the Zlib algorithm,

which requires the Zlib library component.

In other words, Subversion has quite a bit of build-time variability. Figure 2.9 shows

how such variability can be expressed in the Nix expression language. For instance, the

optional support for SSL encryption is expressed by giving the function a Boolean param-

eter sslSupport

17 that speciﬁes whether to build a Subversion instance with SSL support.

There are actually even more dependencies than listed here, such as bindings for various languages. These

have been omitted to keep the example short.

2. An Overview of Nix

{ bdbSupport ? false

, httpServer ? false

, sslSupport ? false

, compressionSupport ? false

, stdenv, fetchurl

, openssl ? null

, httpd ? null, db4 ? null, expat, zlib ? null

assert bdbSupport -> db4 != null;

assert httpServer -> httpd != null && httpd.expat == expat;

assert sslSupport -> openssl != null &&

(httpServer -> httpd.openssl == openssl);

assert compressionSupport -> zlib != null;

stdenv.mkDerivation {

name = "subversion-1.2.1";

builder = ./builder.sh;

src = fetchurl {

url = http://subversion.tigris.org/downloads/subversion-1.2.1.tar.bz2;

md5 = "0b546195ca794c327c6830f2e88661f7";

};

openssl = if sslSupport then openssl else null;

zlib = if compressionSupport then zlib else null;

httpd = if httpServer then httpd else null;

db4 = if bdbSupport then db4 else null;

inherit expat bdbSupport httpServer sslSupport;

}

Figure 2.9.: pkgs/applications/version-management/subversion/default.nix: Nix expression

for Subversion

This is distinct from the passing of the actual OpenSSL dependency 18 in order to separate

logical features from the dependencies required to implement them, if any. Many features

require no dependencies, or require multiple dependencies. The list of function arguments

also shows the language feature of default arguments, which are arguments that may be

omitted when the function is called. If they are omitted, the default is used instead.

A call to the Subversion function (e.g., in all-packages-generic.nix) to build a minimal

Subversion component will look like this:

subversion = (import .../subversion) {

inherit fetchurl stdenv expat;

};

On the other hand, the following call builds a full-featured Subversion component:

subversion = (import .../subversion) {

2.2. Nix expressions

inherit fetchurl stdenv openssl db4 expat zlib;

httpd = apacheHttpd;

localServer = true;

httpServer = true;

sslSupport = true;

compressionSupport = true;

};

There are some other language aspects worth noting in Figure 2.9. Assertions enable

consistency checking between components and feature selections. The assertion in 19

veriﬁes that if Berkeley DB support is requested, the db4 dependency should not be null.

The value null is a special constant that allows components to be omitted, which would be

wrong here. Note that the default for the db4 argument is in fact null, which is appropriate

for bdbSupport’s default of false. But if the function is called with bdbSupport set to true,

then the assertion protects the user against forgetting to specify the db4 parameter.

The next two assertions 20 express consistency requirements between components. The

ﬁrst one states that if the Apache WebDAV module is to be built, then Apache’s expat

dependency (Expat is an XML parsing library) should be exactly equal to Subversion’s

expat dependency, where “exactly equal” means that they should be in the same store path.

This is because the dynamic libraries of Apache and Subversion will end up being linked

against each other. If both bring in different instances of Expat, a runtime link error will

ensue. Hence the requirement that both Expats are to be the same. Likewise, if SSL

and WebDAV support are enabled, Apache and Subversion should use the same OpenSSL

libraries.

As in the Hello example, the dependencies must be passed to the build script. But we

only pass the optional dependencies if their corresponding features are enabled

21 . If they

are disabled, the value null is passed to the builder (null corresponds to an empty string in

the environment variables). This is to relieve the programmer from having to ﬁgure out

at the call site what speciﬁc dependencies should be passed for a given feature selection.

After all, that would break abstraction—such decisions should be made by the component’s

Nix expression, not by the caller. Thus, the caller can always pass all dependencies, and

the component’s Nix expression will ﬁgure out what dependencies to use. As we will see

in Section 4.1 (page 62), unused dependencies will not be evaluated, let alone be built,

due to the fact that Nix expressions are a lazy language (values are only computed when

needed).

Figure 2.10 shows the builder for the Subversion component, just to give an idea of what

is involved in getting a non-trivial piece of software to work in Nix. This builder is at a

higher level than the one we have seen previously for Hello (Figure 2.7), as it uses the

generic builder, which is a shell function deﬁned by the standard environment that takes

care of building most Unix components with little or no customisation. Most modern Unix

software has a standard build interface consisting of the following sequence of commands:

tar xvf ...sources.tar...

./configure

make

make install

2. An Overview of Nix

buildInputs="$openssl $zlib $db4 $httpd $expat"

source $stdenv/setup

configureFlags="--without-gdbm --disable-static"

if test "$bdbSupport"; then

configureFlags="--with-berkeley-db=$db4 $configureFlags"

if test "$sslSupport"; then

configureFlags="--with-ssl --with-libs=$openssl $configureFlags"

if test "$httpServer"; then

configureFlags="--with-apxs=$httpd/bin/apxs --with-apr=$httpd ..."

makeFlags="APACHE_LIBEXECDIR=$out/modules $makeFlags"

else

configureFlags="--without-apxs $configureFlags"

installFlags="$makeFlags"

genericBuild

Figure 2.10.: pkgs/applications/version-management/subversion/builder.sh: Builder for

Subversion

This is in essence what the generic builder does. For many components, a simple call

24 to

the generic builder sufﬁces. The generic builder is discussed in more detail in Section 7.1.2.

So the effort to build Subversion is quite modest: it is a matter of calling Subversion’s

conﬁgure script with the appropriate ﬂags to tell it where it can ﬁnd the dependencies. For

instance, the conﬁgure ﬂags to enable and locate Berkeley DB are set in 23

. Actually,

following the Autoconf philosophy of auto-detecting system capabilities, some of the op-

tional features are enabled automatically if the corresponding dependency is found. For

example, Zlib compression support is enabled automatically if conﬁgure ﬁnds Zlib in the

compiler and linker search path. The variable buildInputs contains a list of components that

should be added to such search paths; thus all components added in 22 are in the scope of

the compiler and linker.

This concludes the brief overview of the Nix expression language. Chapter 4 provides a

more in-depth discussion of the syntax and semantics of the language.

2.3. Package management

Installing Now that we have written Nix expressions for Hello, we want to build and

install the component. That is, we want to reach a state where the Hello component is in

some way “available” to the user. In a Unix system, this means that the application should

2.3. Package management

be in the user’s search path:

$ hello

Hello, world!

while in different settings it might mean that the application becomes available as an icon

on the desktop, or as a menu item in the Start Menu.

On Unix, operations such as installation, upgrading, and uninstallation are performed

through the command nix-env, which manipulates so-called user environments—the views

on “available” components (discussed below on page 36). For instance,

$ nix-env -f .../all-packages.nix -i hello

installing `hello-2.1.1'

evaluates the Nix expression in all-packages.nix, looks for a derivation with symbolic name

hello, builds the derivation, and makes the result available to the user. The building of the

derivation is a recursive process: Nix will ﬁrst build the dependencies of Hello, which

entails building their dependencies, and so on. However, if a derivation has been built

previously, it will not be rebuilt.

Queries It is of course possible to query the set of installed components:

$ nix-env -q

hello-2.1.1

This query prints the set of components in the current user environment. Likewise, it is

possible to query which components are available for installation in a Nix expression:

$ nix-env -f .../all-packages.nix -qa

a52dec-0.7.4

acrobat-reader-7.0

alsa-lib-1.0.9

ant-blackdown-1.4.2

ant-j2sdk-1.4.2

ant-j2sdk-1.5.0

...

It is usually not convenient to specify the Nix expression all the time. Therefore it is possi-

ble to specify a speciﬁc Nix expression as the default for package management operations:

$ nix-env -I .../all-packages.nix

All subsequent nix-env invocations will then use this ﬁle if no expression is explicitly

speciﬁed.

Upgrades and rollbacks When a new version of the set of Nix expressions that contains

Hello comes along, we can upgrade the installed Hello:

$ hello -v

hello - GNU hello 2.1.1

2. An Overview of Nix

$ nix-env -f .../all-packages.nix -i hello

upgrading `hello-2.1.1' to `hello-2.1.2'

$ hello -v

hello - GNU hello 2.1.2

This is standard procedure for a package management system, completely analogous to,

e.g., rpm -i hello and rpm -U hello. However, if it turns out that the new version of Hello

does not meet our requirements, we can roll back to the previous version:

$ nix-env --rollback

$ hello -v

hello - GNU hello 2.1.1

The reason that this is possible is that the new version of Hello resides in a different path

in the Nix store, since at least some inputs of the build process of the new Hello must be

different (e.g., its sources and its name attribute). The old one is still there. There is no

destructive upgrading: in the purely functional model, components never change after they

have been built.

User environments User environments are Nix’s mechanism for allowing different users

to have different conﬁgurations, and to do atomic upgrades and rollbacks. Clearly, we want

to abstract over store paths; it would not do for users to have to type

$ /nix/store/dpmvp969yhdq...-subversion-1.1.3/bin/svn

to start a program.

Of course we could set up the PATH environment variable to include the bin direc-

tory of every component we want to use, but this is not very convenient since changing

PATH doesn’t take effect for already existing processes. The solution Nix uses is to cre-

ate directory trees of symlinks to activated components (similar to systems such as GNU

Stow [78])

. These are called user environments and they are components themselves

(though automatically generated by nix-env), so they too reside in the Nix store. In Fig-

ure 2.11 the user environment /nix/store/0c1p5z4kda11...-user-env contains a symlink to

just Subversion 1.1.2 (dashed arrows in the ﬁgure indicate symlinks). This is what we

would have obtained if we had performed

$ nix-env -i subversion

on a set of Nix expressions that contained Subversion 1.1.2.

This does not in itself solve the problem, of course; a user would not want to type

/nix/store/0c1p5z4kda11...-user-env/bin/svn either. This is why there are symlinks outside

of the store that point to the user environments in the store, e.g., the symlinks default-42-

link and default-43-link in the example. These are called generations since whenever one

performs a nix-env operation, a new user environment is generated based on the current

one. For instance, generation 43 was created from generation 42 when the user executed

While this approach is rather Unix-speciﬁc, user environments can be implemented in many other ways, as is

discussed in Section 7.2.

2.3. Package management

/nix/store

5mq2jcn36ldl...-subversion-1.1.2

bin

svn

/nix/var/nix/profiles

default

default-42-link

default-43-link

g32imf68vvbw...-firefox-1.0.1

bin

firefox

0c1p5z4kda11...-user-env

bin

svn

3aw2pdyx2jfc...-user-env

bin

svn

firefox

dpmvp969yhdq...-subversion-1.1.3

bin

svn

switch

/home/alice

.nix-profile

/home/bob

.nix-profile

/home/carol

.nix-profile

carol

carol-23-link

PATH=~/.nix-profile/bin

Figure 2.11.: User environments

$ nix-env -i subversion firefox

on a set of Nix expressions that contained Firefox and a new version of Subversion.

Generations are grouped together into proﬁles so that different users don’t interfere with

each other (unless they share a proﬁle). For example, the links default-42-link and default-

43-link are part of the proﬁle called default. The ﬁle default itself is a symlink that points

to the current generation. When we perform a nix-env operation, a new user environment

and generation link are created based on the current one, and ﬁnally the default symlink is

made to point at the new generation.

Upgrades are atomic. This means that a user can never observe that an upgrade is “half-

way through”; the user either sees all the components of the old user environment, or all the

components of the new user environment. This is a major improvement over most deploy-

ment systems which work by destructive upgrading. Due to the purely functional model,

as we are building new components (including user environments, which are components

also), the components in the user’s current user environment are left entirely untouched.

They are not in any way changed. The last step in the upgrade—switching the current

generation symlink—can be performed in an atomic action on Unix.

Different users can have different proﬁles. The user’s current proﬁle is pointed to by the

symlink ~/.nix-proﬁle (where ~ denotes the user’s home directory). Putting the directory

~/.nix-proﬁle/bin in the user’s PATH environment variables completes the user environments

picture (Figure 2.11), for if we now resolve the chain of symlinks from ~/.nix-proﬁle/bin,

we eventually end up in the bin directory of the current user environment, which in turn

contains symlinks to the activated applications. A user can create or switch to a new proﬁle:

$ nix-env --switch-profile /nix/var/nix/profiles/my-new-profile

2. An Overview of Nix

which will simply ﬂip the ~/.nix-proﬁle symlink to the speciﬁed proﬁle. Subsequent nix-env

operations will then create new generations in this proﬁle.

Uninstalling and garbage collection A component can be removed from a user envi-

ronment:

$ nix-env -e hello

uninstalling `hello-2.1.1'

$ hello -v

bash: hello: No such file or directory

However, nix-env -e does not actually remove the component from the Nix store. This is

because the component might still be in use by the user environment of another user, or be

a dependency of a component in some environment. More importantly, if the component

were removed, it would no longer be possible to perform a rollback.

To reclaim disk space, it is necessary to periodically run the garbage collector, which

deletes from the store any components not reachable from any generation of any user

environment. (Reachability is deﬁned with respect to the runtime dependencies found by

the hash scanning approach discussed earlier, and in more detail in Section 3.4.) That is,

the generations are roots of the garbage collector.

This means that the Hello component cannot be garbage collected until the old gener-

ations that included it are gone. The following command removes any generations of the

user’s proﬁle except the current:

$ nix-env --remove-generations old

Note that this removes the ability to roll back to any conﬁguration prior to the invocation of

this command. Thus, the user should only perform this operation when sure that rollback

is not necessary.

After we have removed the old generations, the Hello component has become garbage,

assuming no generations of other proﬁles reach it. Thus an execution of the garbage col-

lector will remove the component from the Nix store:

$ nix-store --gc

deleting `/nix/store/bwacc7a5c5n3...-hello-2.1.1'

...

(The command nix-store performs “low-level” Nix store operations.)

The advantage of garbage collection of components is that users never need to consider

whether some dependency might still be needed. A common phenomenon in many de-

ployment systems is that users are asked to conﬁrm whether it is safe to remove some

component (e.g., “Should Windows delete foo.dll?”). Since Nix ensures reliable depen-

dency information, garbage collection can be safe and automatic. The act of uninstallation

is a logical action concerning only top-level components (i.e., applications), not dependen-

cies that should remain hidden to users.

2.4. Store derivations

outside the Nix store in the Nix store

Nix

expressions

translation

(nix-instantiate)

store

derivations

building

(nix-store --realise)

store derivation

outputs

Figure 2.12.: Two-stage building of Nix expressions

2.4. Store derivations

So how do we build components from Nix expressions? This could be expressed directly

in terms of Nix expressions, but there are several reasons why this is a bad idea. First, the

language of Nix expressions is fairly high-level, and as the primary interface for develop-

ers, subject to evolution; i.e., the language might change to accommodate new features.

However, this means that we would have to be able to deal with variability in the Nix

expression language itself: several versions of the language would need to be able to co-

exist in the store. Second, the richness of the language is nice for users but complicates

the sorts of operations that we want to perform (e.g., building and deployment). Third,

Nix expressions cannot easily be identiﬁed uniquely. Since Nix expressions can import

other expressions scattered all over the ﬁle system, it is not straightforward to generate an

identiﬁer (such as a cryptographic hash) that uniquely identiﬁes the expression. Finally, a

monolithic architecture makes it hard to use different component speciﬁcation formalisms

on top of the Nix system (e.g., we could retarget Makeﬁles to use Nix as a back-end).

For these reasons Nix expressions are not built directly; rather, they are translated to

the more primitive language of store derivations, which encode single component build

actions. This is analogous to the way that compilers generally do the bulk of their work

on simpler intermediate representations of the code being compiled, rather than on a full-

blown language with all its complexities. Store derivations are placed in the Nix store, and

as such have a store path too. The advantage of this two-level build process is that the paths

of store derivations give us a unique identiﬁcation of objects of source deployment, just as

paths of binary components uniquely identify objects of binary deployment.

Figure 2.12 outlines this two-stage build process: Nix expressions are ﬁrst translated

to store derivations that live in the Nix store and that each describe a single build action

with all variability removed. These store derivations can then be built, which results in

derivation outputs that also live in the Nix store.

The exact details of the translation of Nix derivation expressions into store derivations is

not important here; the process is fully described in Section 5.4. What matters is that each

derivation is translated recursively into a store derivation. This translation is performed

automatically by nix-env, but it can also be done explicitly through the command nix-

instantiate. For instance, supposing that foo.nix is a Nix expression that selects the value

hello from all-packages-generic.nix in Figure 2.8, then applying nix-instantiate will yield

the following:

$ nix-instantiate foo.nix

/nix/store/1ja1w63wbk5qrscwg4wmzk9cbri4iykx-hello-2.1.1.drv

That is, the expression foo.nix is evaluated, and the resulting top-level derivation is trans-

2. An Overview of Nix

{ output = "/nix/store/bwacc7a5c5n3...-hello-2.1.1" 25

, inputDrvs = { 26

"/nix/store/7mwh9alhscz7...-bash-3.0.drv",

"/nix/store/ﬁ8m2vldnrxq...-hello-2.1.1.tar.gz.drv",

"/nix/store/khllx1q519r3...-stdenv-linux.drv",

"/nix/store/mjdfbi6dcyz7...-perl-5.8.6.drv" 27

}

, inputSrcs = {"/nix/store/d74lr8jfsvdh...-builder.sh"} 28

, system = "i686-linux"

, builder = "/nix/store/3nca8lmpr8gg...-bash-3.0/bin/sh" 30

, args = ["-e", "/nix/store/d74lr8jfsvdh...-builder.sh"] 31

, envVars = { 32

("builder","/nix/store/3nca8lmpr8gg...-bash-3.0/bin/sh"),

("name","hello-2.1.1"),

("out","/nix/store/bwacc7a5c5n3...-hello-2.1.1"),

("perl","/nix/store/h87pfv8klr4p...-perl-5.8.6"), 33

("src","/nix/store/h6gq0lmj9lkg...-hello-2.1.1.tar.gz"),

("stdenv","/nix/store/hhxbaln5n11c...-stdenv-linux"),

("system","i686-linux"),

("gtk","/store/8yzprq56x5fa...-gtk+-2.6.6"),

}

Figure 2.13.: Store derivation for the hello attribute in Figure 2.8

lated to a store derivation which is placed in /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv. All

input derivations of hello are recursively translated and placed in the store as well.

The store derivation in /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv is shown in Fig-

ure 2.13

. It lists all information necessary to build the component, with references to

the store derivations of dependencies:

25 The output ﬁeld speciﬁes the path that will be built by this derivation, if and when it

is built

. It is computed essentially by hashing the store derivation with the output

ﬁeld cleared.

The inputDrvs ﬁeld contains the paths of all the input derivations. These have to be

built before we can build this derivation.

28 The inputSrcs ﬁeld contains the paths of all input sources in the Nix store. The

Hello Nix expression in Figure 2.8 referred to a local ﬁle builder.sh. This ﬁle has

been copied by nix-instantiate to the Nix store, since everything used by the build

process must be inside the Nix store to prevent undeclared dependencies.

This is a pretty-printed representation of the actual store derivation, which is encoded as an ATerm [166].

Figure 5.8 on page 105 shows the actual ATerm.

This ﬁeld exists in the extensional model described in Chapter 5, where the output path is computed in advance,

but not in the intensional model described in Chapter 6, where it is only known after the derivation has been

built.

2.4. Store derivations

29 The system ﬁeld speciﬁes the hardware and operating system system on which the

derivation is to be built. This ﬁeld is part of the hash computation, so derivations

for different systems (e.g., i686-linux versus powerpc-darwin) result in different store

paths.

30 The builder ﬁeld is the program to be executed. The reader might wonder about the

apparent mismatch between the builder declared in Figure 2.6 and the one speciﬁed

here. This is because the function stdenv.mkDer ivation massages the arguments it

receives a bit, substituting a shell binary as the actual builder, with the original shell

script passed to the shell binary as a command line argument.

31 The args ﬁeld lists the command line arguments to be passed to the builder.

32 The envVars ﬁeld lists the environment variables to be passed to the builder.

Note that inputDr vs and envVars specify different paths for corresponding depen-

dencies. For example, Hello’s Perl dependency is built by the derivation stored in

/nix/store/mjdfbi6dcyz7...-perl-5.8.6.drv 27 . The output path of this derivation is /nix/store/-

h87pfv8klr4p...-perl-5.8.6 33 .

Building store derivations Store derivations can be built by calling the command nix-

store --realise, which “realises” the effect of a derivation in the ﬁle system. It builds the

derivation (if the output path of the derivation does not exist already), and prints the output

path.

$ nix-store --realise /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

...

/nix/store/bwacc7a5c5n3...-hello-2.1.1

Nix users do not generally have to deal with store derivations. For instance, the nix-env

command hides them entirely—the user interacts only with high-level Nix expressions,

which is really just a fancy wrapper around the two commands above. However, store

derivations are important when implementing deployment policies. Their relevance is that

they give us a way to uniquely identify a component both in source and binary form,

through the paths of the store derivation and the output, respectively. This can be used to

implement a variety of deployment policies, as we will see in Section 2.5.

Closures revisited A crucial operation for deployment is to query the closure of a store

path under the dependency relation, as discussed in Section 2.1. Nix maintains this infor-

mation for all paths in the Nix store. For instance, the closure of the Hello component can

be found as follows:

$ nix-store -qR /nix/store/bwacc7a5c5n3...-hello-2.1.1/bin/hello

/nix/store/72by2iw5wd8i...-glibc-2.3.5

/nix/store/bg6gi7s8mxir...-linux-headers-2.6.11.12-i386

/nix/store/bwacc7a5c5n3...-hello-2.1.1

/nix/store/q2pw1vn87327...-gcc-3.4.4

2. An Overview of Nix

This means that when we deploy this instance of Hello, we must copy all four of the paths

listed here

. When we copy these paths, we have binary deployment. It should be pointed

out that /nix/store/h87pfv8klr4p...-perl-5.8.6 is not in the closure, since it is not referenced

from the Hello binary or any of the paths referenced by it. That is, Perl is exclusively a

build-time dependency.

Similarly, we can query the closure of the store derivation:

$ nix-store -qR /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

/nix/store/0m055mdm0v5z...-perl-5.8.6.tar.bz2.drv

/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

/nix/store/d74lr8jfsvdh...-builder.sh

...

(98 more paths omitted)

...

Copying these paths gives us source deployment, since the store derivations describe

build actions from source.

It is also possible to combine the two:

$ nix-store -qR --include-outputs \

/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

/nix/store/0m055mdm0v5z...-perl-5.8.6.tar.bz2.drv

/nix/store/h87pfv8klr4p...-perl-5.8.6

/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

/nix/store/bwacc7a5c5n3...-hello-2.1.1

/nix/store/d74lr8jfsvdh...-builder.sh

...

(155 more paths omitted)

...

The --include-outputs ﬂag causes the closures of the outputs of derivations to be in-

cluded. It is worth noting that this set is more than the union of the two sets above, since

there are many output paths of derivations that are not in the closure of the Hello binary.

For instance, though the Hello binary does not reference Perl, the Perl binary is included

here because its derivation is included.

2.5. Deployment models

What we have described above is not deployment—it is build management combined with

local package management. The calls to nix-env build components locally, and construct

user environments out of the build results. To perform actual software deployment, it is

necessary to get the Nix expressions to the client machines. Nix in itself does not provide

any speciﬁc way to do so. Rather, it provides the mechanisms to allow various deployment

policies. The commands nix-instantiate and nix-store provide the fundamental mechanisms.

(The command nix-env implemented on top of these already implements a bit of policy,

namely, user environments.)

Here are some examples of deployment models implemented and used in practice to

distribute the Nix expressions of the Nix Packages collection:

• Manual download. A user can manually download releases of the Nix Packages

collection as tar archives, unpack them, and apply nix-env to install components.

In case the reader is wondering: the quite unnecessary dependency on linux-headers is through glibc; see

Section 6.7. The dependency on gcc is entirely unnecessary and does not occur in the “production” version

of Hello, which uses the utility patchelf (page 179) to prevent this retained dependency.

2.5. Deployment models

The downside to this approach is that it is rather laborious; in particular, it is not

easy to stay up-to-date with new releases.

• Updating through a version management system. As a reﬁnement to the previous

model, the set of Nix expressions can be distributed through a version management

system. For example, Nixpkgs can be obtained from its Subversion repository. After

the initial checkout into a local working copy, staying up-to-date is a simple matter

of running an update operation on the working copy. An added advantage is that it

becomes easy to maintain local modiﬁcations to the Nix expressions.

• Channels. A more sophisticated model is the notion of channels. A channel is

a URL that points to a tar archive of Nix expressions. A user can subscribe to a

channel:

$ nix-channel --add http://server/channels/nixpkgs-stable

Then, the command

$ nix-channel --update

downloads the Nix expressions from the subscribed channels and makes them the

default for nix-env operations. That is, an operation such as

$ nix-env -i firefox

will install a derivation named ﬁrefox from one of the subscribed channels. In par-

ticular, it is easy to keep the entire system up-to-date:

$ nix-channel --update

$ nix-env -u '*'

where nix-env -u ’*’ updates all installed components in the user environment to the

latest versions available in the subscribed channels.

• One-click installations. When a user wants to install a speciﬁc program, by far

the easiest method is to simply install it by clicking on it on some distribution web

page. Of course, this should also install all dependencies. An example of such

one-click installations is shown in Figure 2.14 (along with direct download of the

Nix expressions, and channel subscription information). Users can click on links on

release web pages to install speciﬁc components into their user environments. The

links lead to small ﬁles containing the information necessary to install the component

(e.g., Nix expressions), while the installation of the component is performed by a

simple tool associated with the MIME type [74] of those ﬁles.

Chapter 7 looks at the wide variety of deployment policies in much greater detail.

2. An Overview of Nix

Figure 2.14.: Deployment policies: direct download, channels, and one-click installation

2.6. Transparent source/binary deployment

The Nix model as described above is a source deployment model. Nix expressions describe

how to build components from source

. A source deployment model is convenient for

deployers because they do not need to create binary packages explicitly. It also gives users

the freedom to make local modiﬁcations and feature selections.

However, building all dependencies is rather slow. For the Hello component in Nix

Packages

, it entails building 63 derivations, including the builds of

hello-2.1.1, stdenv-linux, gcc-3.4.4, bash-3.0, gnutar-1.15.1, curl-7.14.0,

binutils-2.16.1, gnupatch-2.5.4, gnused-4.1.4, patchelf-0.1pre2286, linux-

headers-2.6.11.12-i386, zlib-1.2.2, bzip2-1.0.3, ﬁndutils-4.2.20, gnugrep-

2.5.1a, coreutils-5.2.1, perl-5.8.6, gcc-3.4.4, gnumake-3.80, gzip-1.3.3, gawk-

3.1.4, pcre-6.0, glibc-2.3.5, diffutils-2.8.1, gcc-3.4.4,

the fetchurl downloads of the sources of those packages, and some derivations for the boot-

strapping of stdenv (e.g., the statically linked compiler used to compile gcc-3.4.4); with

As we will see in Section 7.1.4, it is possible to support binary-only packages by using a “trivial” builder

that just fetches a binary distribution of a component, unpacks it and copies it to $out. This is how the Nix

Packages collection supports components such as Adobe Reader.

Data based on nixpkgs-0.9pre3252 on i686-linux.

2.6. Transparent source/binary deployment

downloads of source archives totalling 130 megabytes; and with the build results occupy-

ing 357 megabytes of disk space (although most of it can be garbage collected).

Thus, source deployment is clearly awful for most end-users, who do not have the re-

sources or patience for a full build from source of the entire dependency graph.

However, Nix allows the best of both worlds—source deployment and binary deploy-

ment—through its transparent source/binary deployment model. The idea is that pre-built

derivation outputs are made available in some central repository accessible to the client

machines, such as a web server or installation CD-ROM. For instance, a component dis-

tributor or system administrator can pre-build components, then push (upload) them to a

server using PUT requests:

$ nix-push http://server/dist/ $(nix-instantiate foo.nix)

This will build the derivations in foo.nix, if necessary, and upload it and all its dependencies

to the indicated site. For instance, the path /nix/store/mkmpxqr8d7f7...-ﬁrefox-1.0 will be

archived and compressed into an archive yq318j8lal09...-ﬁrefox.nar.bz2 and uploaded to

the server. Such a ﬁle is called a substitute, since a client machine can substitute it for a

build. The server provides a manifest of all available substitutes. An example entry in the

manifest is

{ StorePath: /nix/store/mkmpxqr8d7f7...-firefox-1.0

NarURL: http://server/dist/yq318j8lal09...-firefox.nar.bz2

Size: 11480169 }

which states that to create store path /nix/store/mkmpxqr8d7f7...-ﬁrefox-1.0, Nix can down-

load and unpack URL http://server/dist/yq318j8lal09...-ﬁrefox.nar.bz2.

To use these substitutes, client machines must register the fact that they are available

using the command nix-pull:

$ nix-pull http://server/dist

This does not actually download any of the substitutes, it merely registers their availability.

When the user subsequently attempts to install a Nix expression, and Nix when building

some store path p notices that it knows a substitute for p, it will download the substitute

and unpack it instead of building from source.

Thus, from a user perspective, source deployment automatically optimises into binary

deployment. This is good: binary deployment can be considered a partial evaluation of

source deployment with respect to a speciﬁc platform, and such optimisations should be

“invisible”.

However, if the user has made modiﬁcations to parts of the Nix expression, it is possible

that some or all of the store paths will be different. In that case, the substitutes fail to

be applicable and Nix will build from source. Thus, binary deployment automatically

“degrades” back to source deployment.

The transparent source/binary model therefore combines the ﬂexibility of source de-

ployment systems such as Gentoo with the efﬁciency of binary deployment systems such

as RPM.

This is somewhat simpliﬁed. The full details of nix-push manifests are in Section 7.3.

2. An Overview of Nix

Of course, the deployment models described in Section 2.5 support transparent source/-

binary deployment. For instance, a Nix channel contains not just a set of Nix expressions,

but also the URL of a manifest that clients can pull when they update from the channel.

Thus, subscribers to the channel automatically get binary deployment for those derivations

in the channel that have been pre-built.

Part II.

Foundations

3. Deployment as Memory Management

The previous chapter introduced the Nix system. It obtains isolation and non-interference

between components using a Nix store that stores components under ﬁle names that contain

a unique cryptographic hash of the component. However, some of the underlying concepts

might seem a bit suspicious! For instance, storing components in an essentially ﬂat, rigid

“address space” of components is very different from the way software is typically stored

in most operating systems

. And the hash scanning technique to identify dependencies

clearly cannot work in all circumstances.

The purpose of this chapter is to “derive” the basic design of the Nix store by relating

the main issues in software deployment to those in memory management in conventional

programming languages. As we shall see, conventional deployment tools treat the ﬁle sys-

tem as a chaotic, unstructured component store, similar to how an assembler programmer

would treat memory. In contrast, modern programming languages impose a certain disci-

pline on memory, such as rigidly deﬁned object layouts and prohibitions against arbitrary

pointer formation, to enable features such as garbage collection and pointer safety. The

idea is that by establishing a mapping between notions in the two ﬁelds, solutions from

one ﬁeld carry over to the other. In particular, the techniques used in conservative garbage

collection serve as a sort of apologia for the hash scanning approach used to ﬁnd runtime

dependencies.

3.1. What is a component?

It is useful to ﬁrst say some things about what it is that we are deploying. Of course,

we deploy software (although in Chapter 9 on service deployment, we will also see some

things being deployed that cannot be called software proper), but what are the units of

deployment? That is, what do we call the smallest meaningful artifacts that are subject to

deployment? It is quite customary in some communities to call these packages, but this

term has the problem that it implies that the artifact is in packaged form. For instance, a

.RPM ﬁle is a package, but what do we call the corresponding installed object? In RPM

terminology, this is also a package, and so the term is ambiguous.

In the Component-Based Software Engineering (CBSE) community, the term compo-

nent is used for units of deployment. Like some other terms in computer science (such as

object), this term is overloaded almost to the point of uselessness. In fact, Czyperski [154,

Chapter 11] surveys fourteen different deﬁnitions. Some deﬁnitions are not meaningfully

distinguishable from the notion of a module, or are used to identify any code reuse mech-

anism.

Despite the multitude of deﬁnitions, I shall use the term anyway, because I believe that

the essential aspect that should be captured by a deﬁnition of components is deployment.

Indeed, no Linux distribution based on Nix will ever be compliant with the Linux Standards Base [82, 81]!

3. Deployment as Memory Management

Otherwise, the notion of component does not essentially add anything not already provided

by concepts such as module, class, method, function, and other organisational structures

provided by programming languages. Components in CBSE have many other concerns

than deployment aspects, such as interfaces, contracts, composition techniques, veriﬁca-

tion of performance and other extra-functional requirements, and so on. However, a dis-

cussion at CBSE-2005 revealed that packaging and deployment were the only aspects on

which there was a consensus that they should be provided by a component-based system.

In this thesis, I will use the following deﬁnition of “software component”:

• A software component is a software artifact that is subject to automatic composition.

It can require, and be required by, other components.

• A software component is a unit of deployment.

The ﬁrst property is of course entailed by the etymology of the word component, except

for the stipulation that a composition can be established automatically. Thus, source code

fragments printed in a book, while composable through human intervention, are not com-

ponents. The second property implies that the purpose of the notion of a component is to

be the principal object of deployment.

This deﬁnition has some important implications. For instance, it does not say any-

thing about whether components are in “source” or “binary” form, since such a distinction

is rather technology-speciﬁc (e.g., what does it mean for interpreted languages such as

Python?). But since it has to be automatically composable, it must come with all the nec-

essary infrastructure to support automatic composition. For instance, mere source code is

not a component, but source code with the necessary meta-information—such as Make-

ﬁles, conﬁguration and build scripts, deployment descriptors, and so on—is a component.

So a Python source ﬁle in itself is not a component. But it is when placed in an RPM pack-

age or combined with a Nix expression, since that meta-information makes the component

available for automatic composition.

A subtle point is what we mean by “software.” Of course, software is a speciﬁcation that

can be executed automatically by a computer (possibly with the aid of other software, such

as an interpreter), but not every constituent part of a software system is executable; some

of it (such as data) is an adjunct to execution. This means that not all software components

need to contain executable code themselves, but are intended for ultimate composition with

an executable component that uses them. For instance, in a software system with an online

help feature, it might be useful to split off the online help into a separate component so that

it can be installed optionally.

The deﬁnition above can be considered the greatest common denominator of a number

of inﬂuential deﬁnitions. Czyperski [154, Section 4.1.5] gives the following deﬁnition:

“A software component is a unit of composition with contractually speciﬁed

interfaces and explicit context dependencies only. A software component can

be deployed independently and is subject to composition by third parties.”

The important aspects of this deﬁnition are the following:

• A component is a unit of independent deployment. Since it is a unit, it does not

have to support partial deployment—the component must be deployed in its entirety.

3.1. What is a component?

Since it is independently deployable, it should not cause interference with its target

environment, such as other components.

• Thus it must have explicit context dependencies only, since otherwise it would place

requirements (such as component dependencies) on its target environment that are

not formally speciﬁed by its deployment metadata.

• It has contractually speciﬁed interfaces, which enable it to be subject to third-party

composition, i.e., it is possible for other people’s components to use the component.

• It has no externally observable state. Different uses of the component over time

should not interfere with each other. Note that this does not mean that the code of

the component must have no side effects, i.e., should be written in a purely func-

tional language. It is perfectly ﬁne for an execution involving the component, e.g.,

the classes and objects loaded from a component, to have side effects. But the com-

ponent should not modify its on-disk representation or other static resources in a way

that affects future uses of the component. This is exactly the case with Nix compo-

nents: they can be entirely impure dynamically, but their static representation—the

ﬁles on disk—never changes after they have been built.

These properties do not always hold for the things that one might want to deploy. For

instance, Mozilla Firefox 1.0 (unless specially prodded not to do so) wants to make modiﬁ-

cations to some of its own ﬁles when started for the ﬁrst time, violating the requirement of

having no externally observable state. This is bad because it makes security difﬁcult (the

ﬁles in question should be writable by all potential users), and hinders proper identiﬁcation

in the sense of conﬁguration management (since the “same” component has different con-

tents over time). Likewise, many components are not intended for third-party composition,

but are rather speciﬁcally tied to certain other components and do not have a well-deﬁned

interface. The reason why such components are separate, rather than combined into a sin-

gle component, is typically to allow optional functionality to be omitted, or for licensing

reasons (e.g., some clients not being entitled to access the full feature set of a system).

This thesis’s deﬁnition differs from Czyperski’s in a number of ways:

• It does not require explicit context dependencies, it merely states that components

can have dependencies. Many components have implicit dependencies—which is

what makes correct deployment so difﬁcult. Of course, to ensure correct deploy-

ment, it is necessary to make implicit dependencies explicit.

• It does not require the possibility of third-party composition. As explained above,

many components are not in fact usefully reusable by third parties.

• It does not require contractually speciﬁed interfaces. It is enough for components to

be composable automatically, which requires some kind of formal interface (since

it can be used automatically), but this can be as weak as a symbol table in an ELF

library [160], or the existence of a named function in a Perl module.

Czarnecki and Eisenecker [40] have an even more minimalistic deﬁnition: components

are simply the “building blocks” of software systems that should be composable as easily

and ﬂexibly as possible in order to maximise reuse and minimise duplication. However,

3. Deployment as Memory Management

this deﬁnition has the problem that it makes a component nothing more than a general term

for reuse mechanisms, as stated above. This is intentional: the authors feel that component

is a “natural concept” for which it is hard to give a rigid deﬁnition. But a deﬁnition that is

overly broad is not very useful.

Meyer [119] deﬁnes a component as a “modular unit” that can be used by other compo-

nents, possesses an “ofﬁcial usage description” enabling its use by other components, and

is not “tied to any ﬁxed set of clients”. The deﬁnition used here does not have the latter

two requirements (many components have poorly deﬁned or missing usage descriptions,

or are speciﬁc to a particular client, as argued above).

So the deﬁnition of component used in this thesis is quite minimal. I do not mean to

imply that a stronger notion of component is not useful or appropriate in many cases. For

instance, component technologies that require strong contracts from components are very

valuable. But such kinds of “high level” composition aspects are not the subject of this

thesis. This thesis is about the “low level” aspect of storage of components: how can we

store components so that they do not interfere with each other, that their dependencies are

known, and so on? That is, it is about components in the ﬁle system.

Components in the ﬁle system In this thesis, I shall further restrict the notion of a

software component to things that have an independent representation in the ﬁle system,

i.e., a component consists of a set of ﬁles. Of course components do not have to exist as

discrete objects in the ﬁle system. They might be stored in a database (as suggested, e.g.,

by the Java Language Speciﬁcation [79, Section 7.2.2]). But this is quite rare, so we will

not consider it.

Through the ﬁle system, components can require functionality from other components,

and they can provide it. Thus, components need to be plugged into each other, i.e., be

composed. There are many mechanisms by which a composition can be realised, such

as static linking, dynamic linking, inclusion of source ﬁles, and so on; and these are just

general classes of composition technologies. Concrete realisation mechanisms are legion:

Unix static libraries, ELF dynamic libraries, Java JAR ﬁles, C preprocessor directives,

.NET assemblies, Perl modules, conﬁguration ﬁles, etc. These mechanisms also have

various binding times: some are typically used at build time, others at runtime, with the

more nebulous notions of packaging time and installation time thrown in for good measure.

But what they all have in common is that the composition must be established in the

ﬁle system. That is, since the components exist in the ﬁle system, to compose them,

the composition mechanism must be able to ﬁnd them in the ﬁle system. This aspect of

composition—establishing the “physical” composition,” so to speak—is often overlooked

in the component-based software engineering literature as well as in programming sys-

tems. This is unfortunate, since the difﬁculty in correctly and efﬁciently establishing and

managing the composition in the ﬁle system is the major source of deployment problems.

3.2. The ﬁle system as memory

In this section I recast the deployment problems identiﬁed in the introduction in terms of

concepts from the domain of memory management in programming languages. That we

can do this is not surprising: after all, ﬁle systems on the one hand and RAM on the other

3.2. The ﬁle system as memory

hand are just two different levels in the storage hierarchy. Where programs manipulate

memory cells, deployment operations manipulate the ﬁle system. This analogy reveals that

the safeguards against abuse of memory applied by programming languages are absent in

deployment, and suggests that we can make deployment safe by trying to impose similar

safeguards.

Components, as deﬁned above, exist in the ﬁle system and can be accessed through

paths, sequences of ﬁle names that specify a traversal through the directory hierarchy,

such as /usr/bin/gcc. We can view a path as an address. Then a string representing a path

is a pointer, and accessing a ﬁle through a path is a pointer dereference. Thus, component

interference due to ﬁle overwriting can be viewed as an address collision problem: two

components occupy overlapping parts of the address space.

Furthermore, we can view components as representations of values or objects. Just as

objects in a programming language can have references to other objects, so components

can have references to other components. If dereferencing a pointer is not possible because

a ﬁle does not exist, we have the deployment equivalent of a dangling pointer.

A component A is dependent on a component B if the set of ﬁles that constitutes A

enables an execution involving A to dereference pointers to the ﬁles constituting B. It

enables an execution to dereference B (rather than that it necessarily must dereference B),

since A itself might not be executed or even be executable; e.g., when A is a conﬁguration

ﬁle containing a path to B, and this ﬁle is read by some other component C.

For example, suppose that we have an application that is dynamically linked against a

ﬁle /lib/libc.so, the GNU C library on Linux systems. This means that the executable image

of the application contains an instruction to the dynamic linker to load the ﬁle /lib/libc.so.

Thus, the application enables a pointer to that ﬁle, since execution of the application will

cause a dereference of that ﬁle when it is started. But if the ﬁle is missing, the deployed

application contains a dangling pointer. Such dangling pointers are the result of incomplete

deployment, and they must be prevented.

To prevent dangling pointers, we should consider the various ways through which a

component A can cause a dereference of a pointer to another component B. Since compo-

nents are analogous to objects, we can provide analogues to the ways in which a method

of a class can obtain and dereference a pointer to another object.

First, a pointer to B can be obtained and dereferenced by A at runtime. This is a form

of late binding, since the pointer is not passed in when it is built, but rather when it is

executed. This may happen through environment variables (such as the PATH program

search path on Windows and Unix), program arguments, function arguments (in the case

of a library), registry settings, user interaction, and so on. Conceptually, this is similar to

the following pseudo-Java:

class Foo {

Foo() {}

int run(Bar y) { return y.doIt(); }

}

We can consider the constructor method Foo() to be the build time of the objects of class

Foo, while the method run corresponds to the runtime. Thus any arguments passed to the

constructor are analogous to build-time dependencies, while arguments passed to run are

3. Deployment as Memory Management

analogous to runtime dependencies. In this case, an instance of Foo obtains a pointer to an

instance of Bar at runtime.

Second, a pointer to B can be obtained and dereferenced by A at build time. In this

case, a pointer is passed in at build-time and is completely “consumed”, meaning that B is

not needed subsequently. It can therefore no longer cause a dangling pointer. Examples

include pointers to static libraries or the compiler, which are not usually retained in the

result. This is comparable to a constructor that uses a pointer to another object to compute

some derived value but that does not store the pointer itself:

class Foo {

int x;

Foo(Bar y) { x = y.doIt(); }

int run() { return x; }

}

Third, a pointer to B can be obtained by A at build time and dereferenced at runtime.

In this case, a pointer to B is passed to and saved in the build process that constructed A,

and is dereferenced during program execution. This is the notion of retained dependencies

described on page 23. It happens often with Unix-style dynamically linked libraries: the

build of an application stores the full path to the library in the RPATH of the application

binary (see page 23), which is used to locate the library at program startup. This is equiv-

alent to the constructor of an object storing a pointer that was passed in, which is later

dereferenced:

class Foo {

Bar x;

Foo(Bar y) { x = y; }

int run() { return x.doIt(); }

}

Here, the execution of the constructor is similar to the construction of a component, and

the execution of method run() is similar to the use of a component.

Finally, and orthogonal to the previous methods, a pointer to B can be obtained using

pointer arithmetic. Since paths are represented as strings, any form of string manipulation

such as concatenation may be used to obtain new paths. If we have a pointer to /usr/bin,

then we may append the string gcc to obtain /usr/bin/gcc. Note that the names to be ap-

pended may be obtained by dereferencing directories, that is, reading their contents.

It follows from the above that it is hard to ﬁnd the set of pointers that can be dereferenced

by a component. First, pointers may already be present in the source. Second, pointers

passed in at build-time may or may not be stored in the component. Obviously, we do not

want to distribute a compiler along with an application just because it was used to build

it—but other build-time components, such as dynamic libraries, should be. Finally, pointer

arithmetic can be used to obtain new pointers in uncontrolled ways. For correct software

deployment it is essential that no dangling pointers can occur. Thus, we need a method to

prevent these.

Figure 3.1 summarises the mapping of concepts between the domain of memory man-

agement in programming language implementation on the one hand, and deployment on

the other hand. (The third set of correspondences is ﬁrst discussed in Section 3.4.)

3.3. Closures

Programming Language Domain Deployment Domain

Concepts

memory ⇔ disk

objects (values) ⇔ components

addresses ⇔ path names

pointer dereference ⇔ I/O

pointer arithmetic ⇔ string operations

dangling pointer ⇔ reference to absent component

object graph ⇔ dependency graph

Kinds of dependencies

calling already constructed object with

reference to other object

⇔ runtime dependency through late bind-

ing

calling constructor with reference to

other object, not stored

⇔ build-time dependency

calling constructor with reference to

other object, stored

⇔ retained dependency

Pointer discipline vs. deployment styles

languages with total absence of pointer

discipline (e.g., assembler)

⇔ typical Unix-style deployment

languages with enough pointer disci-

pline to support conservative garbage

collection (e.g., C, C++)

⇔ Nix

languages with full pointer discipline

(e.g., Java, Haskell)

⇔ an as-yet unknown deployment style

not enabled by contemporary operat-

ing systems

Figure 3.1.: Deployment / memory management analogies

3.3. Closures

As noted above, dangling pointers in components are a root cause of deployment failure.

Thus, to ensure successful software deployment, we must copy to the target system not just

the ﬁles that make up the component, but also all ﬁles to which it has pointers. Formally,

the set of ﬁles to be included in the distribution of a component is the closure of the set

of ﬁles in the component under the points-to relationship. Informally, a ﬁle that is part of

a component is characterised as a tuple (p,c) where p is the path where the ﬁle is stored,

and c is the contents required at that path. The contents of a path include not just the ﬁle

contents if p is a regular ﬁle, but also metadata such as access permissions. In addition,

the ﬁle may be a directory, in which case the contents include a mapping from directory

entry names to the contents of these directory entries. This will be made more precise in

Section 5.2.1.

Now the closure of a set of ﬁles C is the smallest set C

⊇ C satisfying

∀(p,c) ∈ C

: ∀p

re f

∈ references(c) : ∃(p

) ∈ C

: p

re f

= p

where references(c) denotes the set of pointers contained in the contents of c. That is, if a

path p is in the closure, then the paths to which it has pointers must be as well.

3. Deployment as Memory Management

By deﬁnition, a closure does not contain dangling pointers. A closure can therefore

be distributed correctly to and deployed on another system. Unfortunately, as mentioned

in Section 3.2, it is not possible in general to determine the set references(c) because of

retained dependencies and pointer arithmetic. In the next section, I propose a heuristic

approach to reliably determine this set.

3.4. A pointer discipline

In the previous section we saw that correct deployment without dangling pointers can be

achieved by deploying ﬁle system closures. Unfortunately, determining the complete set

of pointers is hard.

In the domain of programming languages, we see the same problem. Languages such as

C [106, 100] and C++ [153, 101] allow arbitrary pointer arithmetic (adding or subtracting

integers to pointers, or casting between integers and pointers). In addition, compilers for

these languages do not generally emit runtime information regarding record and stack lay-

outs that enable one to determine the full pointer graph. Among other things, this makes

it impossible to implement garbage collectors [104, 179] that can accurately distinguish

between garbage and live objects. Garbage collectors consist of two logical phases [179].

In the detection phase, the collector discovers which objects on the heap are “live,” i.e.,

reachable by following pointers from the stack, registers, and global variables. In the

reclamation phase, all objects that are not live (i.e., dead) are freed. To ﬁnd live objects in

the detection phase, it must be clear what the pointers are. For instance, the collector must

know what words on the stack are pointers, and not, say, integers.

Other languages address this problem by imposing a pointer discipline: the layouts

of memory objects (including the positions of pointers) are known, and programs cannot

manipulate pointers arbitrarily. For example, Java does not permit casting between integers

and pointers, or direct pointer arithmetic. Along with runtime information of memory

layouts, this enables precise determination of the pointer graph.

For ﬁle systems the problem is that arbitrary pointer arithmetic is allowed and that we

do not know where pointers are stored in ﬁles. Certainly, if we restrict ourselves to compo-

nents consisting of certain kinds of ﬁles, such as Java class ﬁles, we can determine at least

part of the pointer graph statically, for instance by looking at the classes imported by a Java

class ﬁle. However, this information is still not sufﬁcient due to the possibility of dynamic

class loading. Since the dependency information cannot be guaranteed to be complete and

because this technique is not generally applicable, this approach does not sufﬁce.

The solution to proper dependency identiﬁcation comes from conservative garbage col-

lection [15, 14]. This is a technique to provide garbage collection for languages that have

pointer arithmetic and no runtime memory layout information, such as C and C++. Conser-

vative garbage collection works by constructing a pointer graph during the detection phase

by assuming that anything that looks like a valid pointer, is a valid pointer. For instance,

if we see a bit pattern in memory corresponding to a pointer to some address p, and p is

in fact part of an allocated part of memory, the object containing p is considered live. Of

course, this assumption might be wrong: the bit pattern might simply encode an integer, a

ﬂoating point number, and so on. But conservative garbage collection errs on the side of

caution: a misclassiﬁcation causes a possibly dead object to be considered live, preventing

3.4. A pointer discipline

it from being reclaimed. The worst that can happen as a result is that we might run out of

memory. Since misclassiﬁcations are rare, such cases are unlikely to occur

Since there is a correspondence between objects in memory and ﬁles in a ﬁle system, we

would like to borrow from conservative garbage collection techniques. That is, we want

to borrow the concept of conservatively scanning for pointers from the detection phase.

Speciﬁcally, we want to scan the contents of ﬁles (of which we do not know the layout),

and if we see a string of characters that looks like a ﬁle name, we assume that the ﬁle being

scanned has a pointer to the ﬁle indicated by the name. But there are two problems with

this idea. First, there might be many false positives: the fact that the strings usr, bin and

gcc occur in a component, does not necessarily imply that the component is dependent

on /usr/bin/gcc). But more importantly, arbitrary pointer arithmetic makes it possible that

even ﬁles that are not referenced, might still be dereferenced at runtime through pointers

computed at runtime.

There is a reason that conservative garbage collection works in languages such as C—

namely, that they actually do have a minimal pointer discipline. For instance, pointer

arithmetic is not allowed to increment or decrement a pointer beyond the boundaries of

the object that it originally pointed to. Of course, the compiler has no way to verify the

validity of such arithmetic, but programmers are well aware of the fact that a C fragment

char * p = malloc(100);

...

char * q = p + 200; /* ouch! */

printf("%c", *q);

is quite illegal. Thus, correct C programs will work with a conservative garbage collector.

But even this minimal pointer discipline is absent in general component storage in the

ﬁle system! Tools can perform arbitrary pointer arithmetic on paths to produce pointers to

any ﬁle in the ﬁle system. Most tools don’t, but even in a limited sense it is a problem. For

instance, once the C compiler has a pointer to /usr/include, it will happily dereference that

directory and produce pointers to all the ﬁles in it. In a way, how we store components

in the ﬁle system in conventional deployment systems is reminiscent of programming in

assembler—there is perfect freedom, and because of that freedom, all bets are off and we

can make no statements about the structure of memory objects (see the bottom third of

Figure 3.1).

So we have to impose a discipline on the storage of components. There are two aspects

to this:

• Pointers have to be shaped in such a way that they are reliably recognisable. Short

strings such as bin offer too much opportunity for misdetection.

• Components have to be separated from each other. The constituent ﬁles of multiple

components should not be mixed. Otherwise a tool can “hop” from one component

to another through pointer arithmetic.

But note that a system using conservative garbage collection can be vulnerable to denial-of-service attacks

if an attacker has an opportunity to feed it data (e.g., strings) that contains many bit patterns that might be

misidentiﬁed as pointers.

3. Deployment as Memory Management

The Nix store introduced in Section 2.1 satisﬁes both requirements. Components are

stored in isolation from each other in paths such as /nix/store/bwacc7a5c5n3...-hello-2.1.1,

which include long, distinctive strings such as bwacc7a5c5n3.... The latter is suitable for

pointer scanning, contrary to, say, hello. Thus, if we ﬁnd the ASCII string bwacc7a5c5n3...

in a different component, we can conclude that there is a dependency; and if we don’t ﬁnd

the string, we can conclude that no such dependency exists between the two components.

As noted in Section 3.2, the build of a component may retain a pointer to another com-

ponent, thus propagating a dependency to later points on the deployment timeline. In con-

ventional deployment systems, these dependencies have to be speciﬁed separately, with all

the risks inherent in manual speciﬁcation. By analysing those ﬁles of a component that are

part of a distribution (in source or binary form) or of an installation, we can automatically

detect timeline dependencies, such as build-time and runtime dependencies.

Pointer hiding What are the risks of this approach, i.e., when does it fail to detect depen-

dencies? It has exactly the same limitation as conservative garbage collection in program-

ming languages, namely, the failure to detect pointers due to pointer hiding. Pointer hiding

occurs when a pointer is encoded in such a way that it is not recognised by the scanner (a

false negative). For example, in C we can do the following:

char * p = malloc(...);

unsigned int x = ~((unsigned int) p); /* invert all bits */

p = 0;

... /* run the collector */

p = (char *) ~x;

That is, we allocate a pointer, then cast it to an integer, ﬂip all the bits in the integer, and

clear the original pointer. Later, we invert the integer again, and cast it back to a pointer,

thus recovering the original pointer. However, if the garbage collector runs in between, it

will not be able to detect that the referenced memory object if alive, since x does not look

like a pointer to that object. Likewise, the collector can be fooled by writing a pointer to

disk and reading it back later.

With components in the ﬁle system, there are similar risks

. If the collector searches for

ASCII representations of the hash parts of component names, then anything that causes

those hash parts to be not represented as contiguous ASCII strings will break the detec-

tion. Obviously, ﬂipping the bit representation of the ASCII string will accomplish that,

but this is unlikely to happen in real software. More likely is that the ﬁle name will be

split in some way. Path components are a common place to split path names: /nix/store/-

bwacc7a5c5n3...-hello-2.1.1 might be represented as a list ["nix", "store", "bwacc7a5c5n3...-

hello-2.1.1"]. This is why we scan for just the hash part, not the whole store path. Tools

are unlikely to split inside path components since substrings of path components have no

general semantic meaning.

More threatening is the use of another representation than ASCII. For instance, if paths

are represented in UTF-16 [34, Section 2.5], Nix’s scanner will not ﬁnd them. Of course,

In all fairness, the risks are potentially a bit bigger in the ﬁle system than in C. The C standard leaves the

semantics of the code example above implementation-deﬁned. So most things that can cause pointer hiding

are not used by educated C programmers in general. On the plus side, the chance of a false positive is

practically non-existent.

3.5. Persistence

the scanner can easily be adapted to support such encodings. Another concern that is harder

to support is compressed executables. The best solution here is not to use such facilities if

they are available.

Ultimately, how well the scanning approach works is an empirical question. As we shall

see in Chapter 7, scanning works very well in practice, as evidenced by the large set of

components in the Nix Packages collection, which have not yielded a single example of

a hidden pointer. Admittedly, this result may be dependent on the characteristics of the

components being deployed; namely, they are Unix components, which do not typically

feature compressed executables, and store ﬁlenames almost invariably in plain ASCII.

3.5. Persistence

The technique of pointer scanning, as described in the previous section, solves the problem

of reliable identiﬁcation of component dependencies. Here I describe a solution to the

problem of component interference, which occurs when two components occupy the same

addresses in the ﬁle system. When we deploy software by copying a ﬁle system closure to

another system, we run the risk of overwriting other software. The underlying problem is

similar to that in the notion of persistence in programming languages, which is essentially

the migration of data from one address space to another (such as a later invocation of a

process). We cannot simply dump the data of one address space and reload them at the

same addresses in the other address space, since those may already be occupied (or may

be invalid). This problem is solved by serialising the data, that is, by writing it to disk in a

format that abstracts over memory locations.

Unfortunately, it is hard to apply an analogue of serialisation to software deployment,

because it requires changing pointers, which may not be reliably possible in ﬁles

. For

instance, a tempting approach is to rename the ﬁles in a closure to addresses not existing

on the target system. To do this, we also have to change the corresponding pointers in the

ﬁles. However, patching ﬁles in such a manner might cause the affected components to

break, e.g., due to internal checksums on ﬁles being invalidated in the process.

Thus, we should choose addresses in such a way as to minimise the chance of address

collision. To make our scanning approach work, we already need long, recognisable ele-

ments in these ﬁle names, such as the base-32 representations of 160-bit values used in Nix.

Not every selection of values prevents collisions: e.g., using a combination of the name

and version number of a component is insufﬁcient, since there frequently are incompatible

instances even for the same version of a component.

Another approach is to select random addresses whenever we build a component. This

works, but it is extremely inefﬁcient; components that are functionally equal would obtain

different addresses on every build, and therefore might be stored many times on the same

system. That is, there is a complete lack of sharing: equal components are not stored at

the same address.

Hence, we observe a tension between the desire for sharing on the one hand, and the

avoidance of collisions on the other hand. The solution is another notion from memory

management. Maximal sharing [166] is the property of a storage system that two values

But if we assume that we can rewrite pointers, then we can do something very similar to serialisation. The

implications of this assumption (which turns out to be quite reasonable) are explored in Chapter 6.

3. Deployment as Memory Management

occupy the same address in memory, if and only if they are equal (under some notion of

equality). This minimises memory usage in most cases.

The notion of maximal sharing is also applicable to deployment. We deﬁne two compo-

nents to be equal if and only if the inputs to their builds are equal. The inputs of a build

include any ﬁle system addresses passed to it, and aspects like the processor and operating

system on which it is performed. We can then use a cryptographic hash [145] of these

inputs as the recognisable part of the ﬁle name of a component. Cryptographic hashes

are used because they have good collision resistance, making the chance of a collision

negligible. In essence, hashes thus act as unique “product codes” for components. This

is somewhat similar to global unique identiﬁers in COM [16], except that these apply to

interfaces, not implementations, and are not computed deterministically. In summary, this

approach solves the problem of component interference at local sites and between sites

by imposing a single global address space on components. The implementation of this

approach is discussed in Chapter 5.

4. The Nix Expression Language

This chapter describes the Nix expression language, the formalism used to describe compo-

nents and compositions. It ﬁrst motivates why a lazy functional language is an appropriate

choice. It then gives a full description of the syntax and semantics of the language. Fi-

nally, it discusses some of the more interesting aspects of the implementation of the Nix

expression evaluator.

4.1. Motivation

The Nix expression language is a simple, untyped, purely functional language. It is not a

general purpose programming language. Its only purpose is to describe components and

compositions. A component is created through the derivation primitive operation, which

accepts all information necessary to build the component. This includes its dependencies,

which are other derivations or sources.

Purely functional language A purely functional language gives us a clean component

description formalism. The notion that components can be produced by functions that

accept values for the variation points in the component, is much more ﬂexible than, say, a

language that binds such variation points using global variables.

Consider the ATerm library [166], which is a small library for term manipulation. It has

several build-time options, in particular an option that speciﬁes whether the domain feature

“maximal sharing” is enabled. This setting is vitally important, as it completely changes

the semantics of the library. Thus a build with maximal sharing cannot be substituted by

a build without maximal sharing, and vice versa. Now suppose that we have a component

consisting of two programs, foo and bar, that require the ATerm library with and without

maximal sharing respectively. In a purely functional language, this is easy to model. We

just make a function for the ATerm library:

aterm = {maximalSharing}: derivation {...};

We can now call this function twice with different values, in the derivations of foo and bar

respectively:

foo = derivation { ...

buildInputs = [ (aterm {maximalSharing = true;}) ];

};

bar = derivation { ...

buildInputs = [ (aterm {maximalSharing = false;}) ];

};

4. The Nix Expression Language

Imagine that we had a formalism that used global variables to bind variation points

such as maximalSharing. If the language were a Makeﬁle-like declarative formalism, then

it would be quite hard to implement this example. Makeﬁles [56] are a formalism to

describe the construction of (small-grained) components. Variability in a component is

typically expressed by setting variables, e.g.,

MAX_SHARING = 1 # or 0

CFLAGS = -DMAX_SHARING=$(MAX_SHARING)

aterm: ...

foo: aterm

bar: aterm

It is not clear how to build foo and bar in both variants concurrently. In Make, it is typ-

ically very hard to build a component in multiple conﬁgurations that exist side-by-side.

If it is done at all, it is usually accomplished through ad hoc tricks such as calling Make

recursively with different ﬂags, giving the build result a different name for each call.

If the formalism were imperative, then it would be possible. Such a style might look

like this:

maxSharing = true;

foo(aterm());

maxSharing = false;

bar(aterm());

But this has the same problems that imperative programming has in general. Since the

order in which statements are executed matters, it is hard to see exactly what goes into a

build function such as aterm(). Of course, in a trivial example like this, it is doable. But

once we get many functions, variables, and potential control-ﬂow paths, it may become

very hard to comprehend what is happening. Note that this is the case even if variables are

not global. For instance, maxSharing could be a ﬁeld of the aterm object. But even then

the execution order frustrates comprehension.

Interestingly, most true component-level formalisms do not specify compositions at all,

just components. (With “component-level” I mean that they work at the level of large-

grained components, like RPM spec ﬁles do, rather than at the level of small-grained com-

ponents, like Makeﬁles.) An RPM spec ﬁle for instance speciﬁes a component build action,

which can include variables that can be set when the rpmbuild tool is called, but there is no

way for one RPM spec ﬁle to declare that it needs the result of another RPM spec ﬁle with

certain variables set to certain values. In any case RPM will not recursively build those

dependencies, let alone with the desired variable values (since it has no way to prevent

collisions between the various builds).

Other essentially functional build formalisms are Amake and Odin, discussed in Sec-

tion 10.3.

Laziness A lazy language only evaluates values when they are needed. This is a very

useful property for a component description language. Here are some concrete examples

of why this is useful.

It is possible to deﬁne large sets of components concurrently (i.e., in a single ﬁle or ex-

pression), without having to build them all. For instance, the Nix expression pkgs/system-

4.1. Motivation

/all-packages-generic.nix in the Nix Packages collection returns a large attribute set of

derivations:

rec {

stdenv = derivation { ... };

glibc = derivation { ... };

gcc = derivation { ... };

hello = derivation { ... };

subversion derivation { ... };

firefox = derivation { ... };

}

and hundreds more. Since the language is lazy, the right-hand sides of the attributes will

not be evaluated until they are actually needed, if at all. For instance, if we install any

speciﬁc component from this set, say, nix-env -i hello, then only the hello value will be

evaluated (although it may in turn require other values to be evaluated, such as stdenv).

Laziness also allows greater efﬁciency if only parts of a complex data structure are

needed. Take the operation nix-env -qa, which prints the name attribute of all top-level

derivations in a Nix expression:

$ nix-env -f ./pkgs/system/i686-linux.nix -qa

a52dec-0.7.4

acrobat-reader-7.0

alsa-lib-1.0.9

ant-blackdown-1.4.2

ant-j2sdk-1.4.2

ant-j2sdk-1.5.0

...

This operation is quite fast: it takes around 0.6 seconds on an Athlon XP 2600 on an

expression containing 417 top-level derivations. But the derivation calls for those 417

derivations potentially need to do a lot of work, as we will see in Section 5.4: they have to

copy sources to the Nix store, compute cryptographic hashes, and so on. But that work is

actually attached to two attributes returned from derivation, namely drvPath and outPath,

which compute the derivation and output store paths. As long as those attributes are not

used, they are not computed. Above, this was the case, since nix-env -qa only uses the

name attribute. But if we force the computation of either of those two attributes:

$ nix-env -f ./pkgs/system/i686-linux.nix -qa --drv-path

a52dec-0.7.4 /nix/store/spkk...-a52dec-0.7.4.drv

acrobat-reader-7.0 /nix/store/1fdl...-acrobat-reader-7.0.drv

alsa-lib-1.0.9 /nix/store/5a7h...-alsa-lib-1.0.9.drv

...

then the operation takes much longer: 2.7 seconds.

Laziness allows arguments to be passed to functions that may not be used. In the Subver-

sion example (Figure 2.9), the Subversion function passes dependencies such as openssl

and httpd that are only passed to the derivation if the corresponding domain feature is set,

e.g.,

4. The Nix Expression Language

{ ..., sslSupport, openssl, ... }:

derivation {

openssl = if sslSupport then openssl else null;

}

That is, the derivation’s openssl attribute is set to null if SSL support is disabled, regardless

of the value of the openssl function parameter. This allows us to unconditionally pass an

instance of openssl in the function call in all-packages-generic.nix:

subversion = import .../subversion {

inherit openssl;

sslSupport = false; # set to `true' if you want SSL support

};

Thanks to laziness, OpenSSL will only be built if SSL support is enabled, despite the fact

that it is always passed as an argument. This simpliﬁes the call sites. Without laziness, the

caller would have the responsibility to ensure that no unnecessary dependencies are passed

to the derivation, thus breaking abstraction.

4.2. Syntax

This section presents the syntax of the Nix expression language using the Syntax Deﬁnition

Formalism [175, 90], which has the following useful properties:

• It is scannerless: the lexical syntax and the high-level “context-free syntax” are

speciﬁed in the same formalism. Thus, no separate lexical scanner is necessary. The

result is a single context-free grammar.

• It supports generalised LR (GLR) parsing. Most context-free parsers (e.g., LL(k)

and LR(k) parsers, for ﬁxed k) suffer from bounded look-ahead: they must choose

between different productions on the basis of a ﬁxed number of tokens. This is

inconvenient for language implementors, since they must deal with the resulting

parse conﬂicts. For instance, the input fragment "{x" could be start of an attribute set

(e.g., {x=123;}), or a function deﬁnition (e.g., {x}: x)

. This requires the programmer

to introduce additional production rules to resolve the conﬂict. Since generalised LR

parsing has unbounded look-ahead, such grammar transformations are unnecessary.

Another advantage of GLR is that since it supports arbitrary context-free grammars

(including ambiguous ones), it allows grammars to be composed. This makes it easy

to extend programming languages [19].

• It is a declarative formalism that speciﬁes a grammar separate from any particular

programming language (contrary to, say, Yacc [103]). This is the main reason why

I use SDF in this chapter: it allows the entire language to be described in a concise,

readable and directly usable formalism, with little meta-syntactic overhead.

Actually, this is the only shift/reduce conﬂict in the language, so generalised LR parsing is not that important

here, except that it enables scannerless parsing.

4.2. Syntax

The current Nix implementation unfortunately does not use the SDF parser (sglr) for

performance reasons (scannerless parsing in particular is expensive). Rather, it uses Bi-

son [66], which is the GNU implementation of Yacc [103]. It recently added support for

GLR parsing, but still requires a separate lexical scanner. The Flex lexical scanner gener-

ator [130], an implementation of Lex [113], is used for lexical analysis.

The following is a brief overview of SDF necessary to understand the Nix grammar. A

full speciﬁcation of SDF can be found in [175]. An SDF grammar contains production

rules of the form

symbols → nt

where symbols is a sequence of terminals and non-terminals, and nt is the non-terminal pro-

duced by the production rule. This notation is somewhat untraditional: most context-free

grammar formalisms are written the other way around ( nt ← symbols or nt → symbols).

The left-hand side can contain various types of syntactic sugar:

• Regular expressions to denote lexical elements.

• The usual EBNF constructs: repetition (S*), choice (S?), and grouping ((S)).

SDF has two kinds of productions: those deﬁning the lexical syntax, and those deﬁning

the context-free syntax. Lexical syntax is used to deﬁne the tokens of the language. An

example of the ﬁrst is the deﬁnition of the non-terminal for identiﬁers:

lexical syntax

[a-zA-Z\_][a-zA-Z0-9\_\']* -> Id

This deﬁnes identiﬁers using a regular expression: they start with a letter or underscore,

and are followed by zero or more letters, digits, underscores, and accents.

Layout—whitespace and comments that can occur anywhere between tokens—is also

deﬁned using lexical syntax productions. For example, the production

[\ \t\n] -> LAYOUT

says that spaces, tabs and newlines are all layout. There is a built-in production that says

that sequences of layout are also layout. The non-terminal LAYOUT has a special status,

as it is automatically inserted between all symbols in the productions of the context-free

syntax.

For example, the following context-free syntax

rule deﬁnes a production for a simple

expression language:

context-free syntax

Expr "+" Expr -> Expr

This rule is (essentially) desugared to

LAYOUT? Expr LAYOUT? "+" LAYOUT? Expr LAYOUT? -> Expr

The term “context-free syntax” in SDF is a bit confusing, as both its “lexical syntax” and “context-free syntax”

are combined to a single context-free grammar.

4. The Nix Expression Language

SDF allows priorities and associativities to be deﬁned to disambiguate otherwise am-

biguous grammars. The following example deﬁnes left-associative productions for addi-

tion and multiplication in a simple expression language, and then deﬁnes that multiplica-

tion takes precedence over addition:

context-free syntax

Expr "+" Expr -> Expr {left}

Expr "*" Expr -> Expr {left}

context-free priorities

Expr "*" Expr -> Expr

> Expr "+" Expr -> Expr

Finally, there are typically all sorts of ambiguities between the lexical elements of a lan-

guage. For instance, the string if can be parsed as the identiﬁer "if", a sequence of identiﬁers

i and f, and the keyword if (assuming that the language has a production containing such a

keyword). The SDF features of rejects and follow restrictions enforce the correct parsing.

This declaration says that if should not be parsed as an identiﬁer:

lexical syntax

"if" -> Id {reject}

Technically, this deﬁnes a production that whenever it matches at some point with the

input, it causes any other Id production for those same characters to be rejected. To prevent

a sequence of identiﬁer characters from being interpreted as a sequence of identiﬁers, a

follow restriction can be used:

lexical restrictions

Id -/- [a-zA-Z0-9\_\']

This says that an Id cannot be followed by a character matching the given character class.

Thus, xy cannot be parsed as the identiﬁers x and y, since x is followed by a letter.

4.2.1. Lexical syntax

We start with the lexical syntax of Nix expressions. It is divided into two parts: the deﬁni-

tion of the layout, and of the actual information-carrying tokens of the language.

Figure 4.1 shows the SDF module that deﬁnes the layout of the language. (SDF gram-

mars are modular in order to allow the different aspects of a language to be deﬁned sepa-

rately, and to make it easier to combine grammars.) Layout consists of whitespace (spaces,

tabs and newlines) and comments. It is worth noting that the language supports two styles

of comments:

(x: x) # Single-line comments like this

"foo" + /* Multi-line

comments like this */ "bar"

Figure 4.2 deﬁnes the remainder of the lexical syntax of the language. The token types

deﬁned here are identiﬁers (which are exactly as described in the Id example above), and

various kinds of constants deﬁned using regular expressions. These are:

• Natural numbers (Int).

4.2. Syntax

module Nix-Layout

exports

sorts HashComment Asterisk Comment

lexical syntax

[\ \t\n] -> LAYOUT

HashComment -> LAYOUT

Comment -> LAYOUT

"#" ~[\n]* -> HashComment

"/*" ( ~[\*] | Asterisk )* "*/" -> Comment

[\*] -> Asterisk

lexical restrictions

Asterisk -/- [\/]

HashComment -/- ~[\n]

context-free restrictions

LAYOUT? -/- [\ \t\n]

Figure 4.1.: Layout in Nix expressions

• Strings (Str) enclosed between double quotes. Characters are escaped by preﬁxing

them with a backslash.

• Paths (Path) are essentially sequences of characters with at least one forward slash in

them. To be precise, they are sequences of path components separated by slashes. A

path component consists of one or more letters, digits, and some other characters

The initial path component, i.e., the one before the ﬁrst slash, may be omitted. If it

is omitted, we have an absolute path, e.g., /home/alice/.proﬁle. If it is present, we

have a relative path, e.g., ../../foo/builder.sh.

• URIs (Uri) are a convenience for the speciﬁcation of URIs in, e.g., calls to the func-

tion fetchurl. The advantage of having URIs as a language construct over simply us-

ing strings is that the user gets syntax checking, and can omit the enclosing quotes.

The regular expression for URIs used here follows [12, Appendix A].

The remainder of Figure 4.2 is concerned with rejects and follow restrictions. Note

that the keywords of the language (used in the context-free syntax below) must be given

twice, ﬁrst to reject them as identiﬁers, and second to prevent them from being followed

by identiﬁer characters (e.g., ifx should not be parsed as the keyword if followed by the

variable x). That we have to specify keywords twice is a current infelicity of SDF.

Also note that Booleans (true and false) are not specially deﬁned here. Rather, true and

false are parsed as identiﬁers and interpreted as built-in nullary functions during execution,

as discussed in the semantics below.

4.2.2. Context-free syntax

Figure 4.3 shows the context-free syntax of the language. A parse of a Nix expression

must match an Expr non-terminal 34 . Almost all productions here produce Exprs. In that

So currently there is no way to deﬁne paths that contain characters not in this set.

4. The Nix Expression Language

module Nix-Lexicals

exports

sorts Id Int Str Path Uri

lexical syntax

[a-zA-Z\_][a-zA-Z0-9\_\']* -> Id

"rec" | "let" | "if" | "then" | "else" |

"assert" | "with" | "inherit" -> Id {reject}

[0-9]+ -> Int

"\"" ~[\n\"]* "\"" -> Str

[a-zA-Z0-9\.\_\-\+]* ("/"[a-zA-Z0-9\.\_\-\+]+)+ -> Path

[a-zA-Z] [a-zA-Z0-9\+\-\.]* ":"

[a-zA-Z0-9\%\/\?\:\@\&\=\+\$\,\-\_\.\!\~\*\']*

-> Uri

lexical restrictions

Id -/- [a-zA-Z0-9\_\']

Int -/- [0-9]

Path -/- [a-zA-Z0-9\.\_\-\+\/]

Uri -/- [a-zA-Z0-9\%\/\?\:\@\&\=\+\$\,\-\_\.\!\~\*\']

"rec" "let" "if" "then" "else"

"assert" "with" "inherit" -/- [A-Za-z0-9\_\']

Figure 4.2.: Lexical syntax of Nix expressions

respect the Nix expression grammar is quite ﬂat: for instance, there is no separate gram-

matical level for concepts such as modules, function deﬁnitions, and so on. Everything is

an expression.

The ﬁrst production just injects the non-terminals for constants into Expr 35 . Of course,

expressions can be enclosed in parentheses to override the normal precedence rules 36 .

The language has two types of functions. The ﬁrst 37 takes a single argument. For

instance, the (anonymous) identity function can be deﬁned as follows:

x: x

Of course, this style of function is just a plain λ -abstraction from the λ -calculus [10], as we

will see in the discussion of the semantics below. Though this style only allows functions

with a single argument, since this is a functional language we can still deﬁne (in a sense)

functions with multiple arguments, e.g.,

x: y: x + y

which is a function taking an argument x that returns another function that accepts an

argument y.

The second style of function deﬁnition 38 , which we have seen in Section 2.2, is more

important in this language. It takes an attribute set and binds the attributes deﬁned therein

to local variables. Thus,

4.2. Syntax

module Nix-Exprs

imports Nix-Lexicals

exports

sorts Expr Formal Bind ExprList

context-free start-symbols Expr

context-free syntax

Id | Int | Str | Uri | Path -> Expr

"(" Expr ")" -> Expr

Id ":" Expr -> Expr

"{" {Formal ","}* "}" ":" Expr -> Expr

Id -> Formal

Id "?" Expr -> Formal

Expr Expr -> Expr

"assert" Expr ";" Expr -> Expr

"with" Expr ";" Expr -> Expr

"{" Bind* "}" -> Expr

"rec" "{" Bind* "}" -> Expr

"let" "{" Bind* "}" -> Expr

Expr "." Id -> Expr

Id "=" Expr ";" -> Bind

"inherit" ("(" Expr ")")? Id* ";" -> Bind

"[" ExprList "]" -> Expr

"" -> ExprList

Expr ExprList -> ExprList

"if" Expr "then" Expr "else" Expr -> Expr

Figure 4.3.: Context-free syntax of Nix expressions

4. The Nix Expression Language

{x, y}: x + y

declares a function that accepts an attribute set with attributes x and y (and nothing else),

and the expression

({x, y}: x + y) {y = "bar"; x = "foo";}

yields "foobar". The formal (expected) argument can have a default value, allowing it to be

omitted. So

({x, y ? "bar"}: x + y) {x = "foo";}

also evaluates to "foobar".

Following the tradition of most functional languages, function calls are written by juxta-

position, i.e., f x instead of, say, f(x). Whether the absence of an argument delimeter makes

function calls more or less readable is a matter of controversy. However, most calls in Nix

expressions pass argument sets, which are enclosed in curly braces and therefore delimited

anyway (i.e., f {attrs}).

Assertions have been discussed in Section 2.2 (page 33). With expressions 40 allow all

attributes in an attribute set to be added to the lexical scope. For example,

with {y = "bar"; x = "foo";}; x + y

evaluates to "foobar"; the variables x and y deﬁned in the attribute set are in scope in the

expression x + y. This construct is primarily useful in conjunction with import, allowing

values to be “included” from an attribute set deﬁned in a ﬁle:

with (import ./definitions.nix); x + y

In essence, this enables a “conventional” module system: commonly used deﬁnitions can

be placed in a separate ﬁle, and imported into the lexical scope of an expression in another

ﬁle. But note that contrary to most module systems, with and import enable a “policy-free”

module system: for instance, imports can occur in any expression, not just at top level.

The various types of attribute set deﬁnitions 41 —plain, recursive, and lets—are dis-

cussed in more detail below. All three contain lists of attributes enclosed in curly braces.

There are two kinds of attribute deﬁnitions 42 : plain name/value pairs, and inherits, which

copy the value of an attribute from the surrounding lexical scope or an optional expression.

Lists 43 are enclosed in square brackets, and the list elements are juxtaposed, not sepa-

rated by an explicit delimiter. For instance,

[1 2 3]

is a list of three elements. More importantly,

[a b c]

is also a list of three elements, not a call to function a with arguments b and c

Finally, the language has conditionals

44 . There is only an if...then...else construct and

no if...then, since the latter does not make sense in a functional setting as it leaves the

value of the expression undeﬁned in case of a false condition. Assertions already serve this

4.3. Semantics

Expr "==" Expr -> Expr {non-assoc}

Expr "!=" Expr -> Expr {non-assoc}

"!" Expr -> Expr

Expr "&&" Expr -> Expr {right}

Expr "||" Expr -> Expr {right}

Expr "->" Expr -> Expr {right}

Expr "//" Expr -> Expr {right}

Expr "~" Expr -> Expr {non-assoc}

Expr "?" Id -> Expr

Expr "+" Expr -> Expr {left}

Figure 4.4.: Context-free syntax of Nix expressions: Operators

purpose—they abort evaluation if the condition is false. Note that due to the absence of an

if...then, there is no “dangling else” problem.

Figure 4.4 continues the context-free syntax. It deﬁnes operators, along with their asso-

ciativities. The meaning of the operators is given below.

Figure 4.5 deﬁnes the relative precedences of all language constructs. Functions bind the

weakest, and thus the body of a function extends maximally to the right unless the function

is enclosed in parentheses. Assertions, with-expressions and conditions are the next level,

followed by the operators. Function application binds very strongly. These priorities cause

the following example expression:

{x}: assert true; f x != !g y

to be parsed as:

({x}: (assert (true); ((f x) != (!(g y)))))

4.3. Semantics

This section gives the formal semantics of the Nix expression language.

4.3.1. Basic values

The basic (data) values of the Nix expression language, in addition to natural numbers,

strings, paths, and URIs described above, are the following:

• Null values, denoted as the built-in nullary function null.

• Booleans, denoted as the built-in nullary functions true and false.

The reason that there is a ExprList non-terminal is actually for this very reason: we cannot write Expr* in the

production rule for lists, since then we are not able to give it a priority relative to function calls.

4. The Nix Expression Language

context-free priorities

Expr "." Id -> Expr

> Expr ExprList -> ExprList

> Expr Expr -> Expr

> Expr "~" Expr -> Expr

> Expr "?" Id -> Expr

> Expr "+" Expr -> Expr

> "!" Expr -> Expr

> Expr "//" Expr -> Expr

> Expr "==" Expr -> Expr

> Expr "!=" Expr -> Expr

> Expr "&&" Expr -> Expr

> Expr "||" Expr -> Expr

> Expr "->" Expr -> Expr

> "if" Expr "then" Expr "else" Expr -> Expr

> "assert" Expr ";" Expr -> Expr

> "with" Expr ";" Expr -> Expr

> Id ":" Expr -> Expr

> "{" {Formal ","}* "}" ":" Expr -> Expr

Figure 4.5.: Context-free syntax of Nix expressions: Priorities

• Subpaths are a somewhat obscure language feature that allows ﬁles in derivations to

be referenced from other derivations. This is useful if a derivation (as is typical) pro-

duces a directory tree, and we are interested in a particular ﬁle in that tree. Suppose

that we have a variable perl that refers to a derivation that builds Perl. The output

of this derivation is a directory at some path p. Suppose now that we want to use

the Perl interpreter as the builder of some other derivation. However, the interpreter

is a program stored not at p but at p + "/bin/perl". With subpaths, we can write the

following:

derivation { ...

builder = perl ~ /bin/perl;

}

The operator

is the constructor of subpath values. We need subpaths in order to

maintain the proper dependency relation between derivations. The above might have

been written as:

derivation { ...

builder = perl.outPath + "/bin/perl";

}

That is, we compute the path of the builder through ordinary string concatenation

(the outPath attribute contains the computed store path of the perl derivation). As

we shall see in Section 5.4, this derivation will not properly identify perl as one of

its dependencies. It will just have an opaque string value for its builder attribute

Currently, direct references to outPath are not disallowed, so this unsafe example is actually possible. It can be

disallowed by restricting access to the outPath and drvPath attributes described in Section 5.4.

4.3. Semantics

All path values are internally in canonical form, meaning that they are not relative to

the current directory (i.e., start with /), do not contain . or .. elements, do not contain

redundant separators (e.g., //), and do not end in a separator. Any relative paths in a Nix

expression are absolutised relative to the directory that contained the expression. For in-

stance, if the expression /foo/bar/expr.nix contains a path ../bla/builder.sh, it is absolutised

to /foo/bla/builder.sh. Sometimes Nix expressions are not speciﬁed in a ﬁle but given on

the command line or passed through standard input, e.g., in nix-env and nix-instantiate.

Such paths are absolutised relative to the current directory. In the semantic rules below we

will make use of a function canonicalise(p,d) (whose deﬁnition is omitted on grounds of

dullness) that yields a canonical representation of path p, absolutised relative to directory

d if necessary.

4.3.2. Compound values

The language has two ways to form compound data structures: lists and attribute sets.

Lists Lists are enclosed in square brackets, as described above. List elements are lazy, so

they are only evaluated when needed. Currently, the language has very limited facilities for

list manipulation. There is a built-in function map that applies a function to all elements in

a list. Lists can be included in derivations, as we will see in Section 5.4. Other than that,

there are no operations on lists.

Attribute sets The most important data type in the language is the attribute set, which is

a set of name/value pairs, e.g.,

{ x = "foo"; y = 123; }

Attribute names are identiﬁers, and attribute values are arbitrary expressions. The order of

attributes is irrelevant, but any atttribute name can occur only once in a set. Attributes can

be selected using the . operator:

{ x = "foo"; y = 123; }.y

This expression evaluates to 123.

Recursive attribute sets allow attribute values to refer to each other. They are constructed

using the rec keyword. Formally, each attribute in the set is added to the scope of the entire

attribute set. Hence,

rec { x = y; y = 123; }.x

evaluates to 123. If rec were omitted, the identiﬁer y in the deﬁnition of the attribute x

would refer to some y bound in the surrounding scope. Recursive attribute sets introduce

the possibility of recursion, including non-termination:

rec { x = x; }.x

A let-expression, constructed using the let keyword, is syntactic sugar for a recursive

attribute set that automatically selects the special attribute body from the set. Thus,

4. The Nix Expression Language

let { body = a ++ b; a = "foo"; b = "bar"; }

evaluates to "foobar".

As we saw above, when deﬁning an attribute set, attributes values can be inherited from

the surrounding lexical scope or from other attribute sets. The expression

x: { inherit x; y = 123; }

deﬁnes a function that returns an attribute set with two attributes: x which is inherited from

the function argument named x, and y which is declared normally. The inherit construct is

just syntactic sugar. The example above could also be written as

x: { x = x; y = 123; }

Note that the right-hand side of the attribute x = x refers to the function argument x, not to

the attribute x. Thus, x = x is not a recursive deﬁnition.

Likewise, attributes can be inherited from other attribute sets:

rec {

as1 = {x = 1; y = 2; z = 3;};

as2 = {inherit (as1) x y; z = 4;};

}

Here the set as2 copies attributes x and y from set as1. It desugars to

rec {

as1 = {x = 1; y = 2; z = 3;};

as2 = {x = as1.x; y = as1.y; z = 4;};

}

However, the situation is a bit more complicated for rec attribute sets, whose attributes

are mutually recursive, i.e., are in each other’s scope. The intended semantics of inherit is

still the same: values are inherited from the surrounding scope. But simply desugaring is

no longer enough. So if we desugar

x: rec { inherit x; y = 123; }

x: rec { x = x; y = 123; }

we have incorrectly created an inﬁnite recursion: the attribute x now evaluates to itself,

rather than the value of the function argument x. For this reason rec is actually internally

stored as two sets of attributes: the recursive attributes, and the non-recursive attributes.

The latter are simply the inherited attributes. We denote this internally used representation

of rec sets as rec {as

/as

}, where as

and as

are the recursive and non-recursive attributes,

respectively.

4.3. Semantics

subst(subs,x) =

(

e if (x e) ∈ subs

x otherwise

subst(subs,{as}) = {map(λ hn = ei.hn = subst(subs,e)i,as)}

subst(subs,rec {as

/as

}) = rec { map(λ hn = ei.hn = subst(subs

,e)i,as

)

/ map(λ hn = ei.hn = subst(subs,e)i,as

)}

where subs

= {x e|x e ∈ subs ∧ x 6∈ names(as)}

subst(subs,let {as

/as

}) = analogous to the rec case

subst(subs,x: e) = x: subst(subs

,e)

where subs

= {x

e|x

e ∈ subs ∧ x 6= x

}

subst(subs,{fs}: e) = {fs}: subst(subs

,e)

where subs

= {x e|x e ∈ subs ∧ x 6∈ argNames(fs)}

Figure 4.6.: subst: Substitutions

4.3.3. Substitutions

Variable substitution is an important operation in the evaluation process

. The substitution

function subst(subs,e) performs a set of substitutions subs in the expression e. The set

subs consists of substitutions of the form x e that replace a variable x with an expression

Here and in the semantic rules below, the variable e ranges over expressions, x over vari-

ables, p over paths, s over strings, and n over attribute names, with subscripts as appropri-

ate. The members of attribute sets are denoted collectively as as, and we write hn = ei ∈ as

to denote that the attribute set as has an attribute named n with value e. The arguments of

a multi-argument function are denoted as fs (for “formals”).

The auxiliary function names(as) yields the set of attribute names occurring in the left

hand side of a set of attributes as. Likewise, argNames(fs) yields the set of names of

formal arguments from a set of formal arguments (note that formal arguments can also

contain defaults, which are left out by this function).

The function subst replaces all free variables for which there is a substitution. A variable

is free in a subexpression if it is not bound by any of its enclosing expressions. Variables

are bound in functions and in recursive attribute sets. In recursive attribute sets, only the

recursive attributes (as

) bind variables; the non-recursive attributes (as

) do not. The

function subst is shown in Figure 4.6. For all cases not speciﬁcally mentioned here, the

substitution is recursively applied to all subexpressions. For instance, for function calls the

following substitution is applied:

subst(subs,e

) = subst(subs,e

) subst(subs,e

)

The term substitution in this section has no relation to Nix’s substitute mechanism.

4. The Nix Expression Language

It is assumed that the substitution terms (i.e., the expressions in subs) contain no free

variables, so subst does not have to perform renaming to prevent name capture; the style

of evaluation used below allows us to get away with this.

4.3.4. Evaluation rules

The operational semantics of the language is speciﬁed using semantic rules of the form

7→ e

that transform expression e

into e

. Rules may only be applied to closed terms,

i.e., terms that have no free variables. Thus it is not allowed to arbitrary apply rules to

subterms.

An expression e is in normal form if no rules are applicable. Not all normal forms are

acceptable evaluation results. For example, no rule applies to the following expressions:

123 x

assert false; 123

{x = 123;}.y

({x}: x) {y = 123;}

The predicate good(e) deﬁnes whether an expression is a valid evaluation result. It is true

if e is a basic or compound value or a function (lambda), and false otherwise. Since rules

are only allowed to be applied to an expression at top level (i.e., not to subexpressions),

a good normal form is in weak head normal form (WHNF) [133, Section 11.3.1]. Weak

head normal form differs from the notion of head normal form in that right-hand sides of

functions need not be normalised. A nice property of this style of evaluation is that there

can be no name capture [10], which simpliﬁes the evaluation machinery.

An expression e

is said to evaluate to e

, notation e

∗

7→ e

, if there exists a sequence

of zero or more applications of semantic rules to e

that transform it into e

such that

good(e

) is true; i.e., the normal form must be good. In the implementation, if the normal

form of e

is not good, its evaluation triggers a runtime error (e.g., “undeﬁned variable” or

“assertion failed”).

Not all expressions have a normal form. For instance, the expression

(rec {x = x;}).x

does not terminate. But if evaluation does terminate, there must be a single normal form.

That is, evaluation is conﬂuent [5]. The conﬂuence property follows from the fact that at

most one rule applies to any expression. The implementation detects some types of inﬁnite

recursion, as discussed below.

The semantic rules are stated below in the following general form:

RULE-NAME :

condition

e 7→ e

That is, we can conclude that e evaluates to e

(e 7→ e

), if the proposition condition holds.

If there are no conditions, the rule is simply written as RULE-NAME : e 7→ e

4.3. Semantics

Attribute sets The SELECT rule implements attribute selection. This rule governs suc-

cessful selection, i.e., it applies only if the given attribute name exists in the attribute set.

SELECT :

∗

7→ {as} ∧ hn = e

i ∈ as

e.n 7→ e

Note that there is no rule for failure. If attribute n is not in as, evaluation fails and a nice

error message is printed in the actual implementation.

A recursive attribute set is desugared to a normal attribute set by replacing all occur-

rences of references to the attributes with the recursive attribute set. For instance, if e =

rec {x = f x y; y = x;}, then e is desugared to

{ x = f (e.x) (e.y);

y = e.x;

}

or in full,

{ x = f ((rec {x = f x y; y = x;}).x)

((rec {x = f x y; y = x;}).y);

y = (rec {x = f x y; y = x;}).x;

}

This desugaring is implemented by the REC rule:

REC : rec {as

/as

} 7→ {subst(subs,{as

}) ∪ as

}

where

subs = {n (rec {as

/as

}).n | n ∈ names(as

∪ as

)}

Contrary to what the reader might expect, this substitution does not lead to a potential ex-

plosion in the size of expressions in our implementation, since we use ATerms that employ

maximal sharing to store equal subterms exactly once, as discussed in Section 4.4.

A let-expression is just syntactic sugar that automatically selects the attribute body from

a recursive set of attributes:

LET : let {as

/as

} 7→ (rec {as

/as

}).body

Function calls Function calls to single-argument functions (i.e., lambdas) are just plain

β -reduction in the λ -calculus [10].

β -REDUCE :

∗

7→ x: e

7→ subst({x e

},e

)

As we shall see in Theorem 2 (page 85), the expression e

contains no free variables.

Therefore, there is no danger of name capture in subst.

Multi-argument function calls, i.e., functions that accept and open an attribute set, are a

bit more complicated:

β -REDUCE’ :

∗

7→ {fs}: e

∧ e

∗

7→ {as} ∧ match

7→ subst(subs, e

)

4. The Nix Expression Language

where

match = (∀n ∈ names(as) : n ∈ argNames(fs)) ∧ (∀n ∈ fs : n ∈ names(as))

subs = {n arg(n) | n ∈ argNames(fs)}

arg(n) =

(

e if hn = ei ∈ as

e if n 6∈ names(as) ∧ n ? e ∈ fs

Note that a multi-argument function call is strict in its argument—the attribute set—but

not in the values of the attributes. The Boolean value match ascertains whether each actual

argument matches a formal argument, and whether each formal argument without a default

matches an actual argument (note that ∀n ∈ fs does not apply to the formal arguments with

defaults, which would be ∀n ? e ∈ fs). The function arg(n) determines the actual value to

use for formal argument n, taking it from as if it has an attribute n, and using the default in

fs otherwise.

Conditionals Conditional expressions ﬁrst evaluate the condition expression. It must

evaluate to a Boolean. (Evaluation fails if it is not.) The conditional then evaluates to one

of its alternatives.

IFTHEN :

∗

7→ true

if e

then e

else e

7→ e

IFELSE :

∗

7→ false

if e

then e

else e

7→ e

Assertions An assertion assert e

; e

evaluates to e

if e

evaluates to true. Otherwise,

evaluation fails.

ASSERT :

∗

7→ true

assert e

; e

7→ e

Withs With-expressions with e

; e

substitute the attributes deﬁned in the attribute set e

into the resulting expression e

WITH :

∗

7→ {as}

with e

; e

7→ subst({n e | hn = ei ∈ as},e

)

Operators The equality operators are deﬁned as follows.

OPEQ :

∗

7→ e

∧ e

∗

7→ e

∧ e

= e

== e

7→ true

∗

7→ e

∧ e

∗

7→ e

∧ e

6= e

== e

7→ false

Equality between expressions (e

= e

) is syntactic. Of course, inequality (e

!= e

) is

deﬁned as the negation of equality.

4.3. Semantics

The Boolean operators are straightforward, although it is worth noting that conjunction

and disjunction (“and” and “or”) are lazy, i.e., “short-circuited”.

OPNEG :

∗

7→ false

!e 7→ true

∗

7→ true

!e 7→ false

OPAND :

∗

7→ false

&& e

7→ false

∗

7→ true ∧ e

∗

7→ false

&& e

7→ false

∗

7→ true ∧ e

∗

7→ true

&& e

7→ true

OPOR :

∗

7→ true

|| e

7→ true

∗

7→ false ∧ e

∗

7→ true

|| e

7→ true

∗

7→ false ∧ e

∗

7→ false

|| e

7→ false

OPIMPL :

∗

7→ false

→ e

7→ true

∗

7→ true ∧ e

∗

7→ true

→ e

7→ true

∗

7→ true ∧ e

∗

7→ false

→ e

7→ false

The update operator e

// e

yields the right-biased union of the attribute sets e

and e

that is, the set consisting of the attributes in e

and e

, with attributes in e

overriding those

in e

if there are name conﬂicts.

OPUPDATE :

∗

7→ {as

} ∧ e

∗

7→ {as

}

// e

7→ {rightUnion(as

,as

)}

where

rightUnion(as

,as

) = as

∪ {hn = ei | hn = ei ∈ as

∧ n 6∈ names(as

)}

Addition is deﬁned between strings and between paths.

OPPLUSSTR :

∗

7→ s

∧ e

∗

7→ s

+ e

7→ s

OPPLUSPATH :

∗

7→ p

∧ e

∗

7→ p

+ e

7→ canonicalise(p

"/" +

)

where +

denotes string concatenation.

Finally, there is an operator ? to check attribute set membership. This is used to guard

the use of attributes that may not be deﬁned, e.g.,

builder = if args ? builder then args.builder else bash;

It is implemented as follows.

OPHASATTR+ :

∗

7→ {as} ∧ n ∈ names(as)

e ? n 7→ true

OPHASATTR- :

∗

7→ {as} ∧ n 6∈ names(as)

e ? n 7→ false

4. The Nix Expression Language

Primops Primitive operations, or primops, are functions that are built into the language.

They need not be deﬁned or imported; they are in scope by default. But since they are

normal identiﬁers, they can be overridden by local deﬁnitions.

The most important primop, indeed the raison d’être for the Nix expression language,

is the primop derivation. It translates a set of attributes describing a build action to a store

derivation, which can then be built. The translation process is performed by the function

instantiate, given in Section 5.4. What is relevant here is that instantiate(as) takes a set of

attributes as, translates them to a store derivation, and returns an identical set of attributes,

but with three attributes added: the attribute type with string value "derivation", and the

attributes drvPath and outPath containing the store paths of the store derivation and its

output, respectively.

So intuitively, derivation just calls instantiate:

DERIVATION :

∗

7→ {as}

derivation e 7→ {instantiate(as)}

However, as I argued at the beginning of this chapter (page 63), we want derivations to be

very lazy: the expensive call to instantiate should only be made if we actually need drvPath

and outPath. For this reason, derivation postpones the computation of those attributes by

using an indirection:

DERIVATION :

∗

7→ {as}

derivation e 7→ {rightUnion(as,as

)}

where

= {htype = "derivation"i,hdrvPath = e

.drvPathi,houtPath = e

.outPathi}

= derivation! {as}

The internal (not user-accessible) primop derivation! performs the actual instantiation:

DERIVATION! : der ivation! {as} 7→ {instantiate(as)}

Thus, when we evaluate attributes that result from a call to derivation, there is essentially

no runtime cost unless we evaluate the drvPath or outPath attributes.

The next primop is import. The unary operation import p reads the ﬁle p, parses it as a

Nix expression, applies certain checks, and evaluates and returns the expression. The valid-

ity checks are that the expression must be closed (i.e., not contain any unbound variables),

that no attribute set deﬁnition deﬁnes two attributes with the same name, and similarly

that no multi-argument function deﬁnition has two formal arguments with the same name.

The requirement that imported expressions are closed is rather important: it implies that

no imported expression can refer to the local scope of the caller. Also, any relative path

constants occurring in the expression are absolutised.

IMPORT :

∗

7→ p

import e 7→ loadExpr(p)

4.4. Implementation

If p denotes a directory, the path "/default.nix" is automatically appended. This is an

organisational convenience for people who want to store each component’s Nix expression

and builder in a separate directory (as in, e.g., Figure 2.5).

The function loadExpr takes care of loading, parsing and processing the ﬁle. The astute

reader will note that loadExpr depends on the contents of the ﬁle system, that is, it depends

on some mapping of paths to ﬁle system contents. This mapping is left implicit in the

evaluation rules. It may also appear to make the evaluation calculus impure, implying

that equational reasoning (an important property of purely functional languages) no longer

holds. For instance, it might appear that import p == import p does not always evaluate to

true. However, as we will see below, due to maximal laziness this is not in fact the case.

The call import p is only evaluated once for a given p.

The import primitive is overloaded to also work on derivations:

IMPORT’ :

∗

7→ {as} ∧ h"type" = e

i ∈ as ∧ e

∗

7→ "derivation"

∧h"drvPath" = e

i ∈ as ∧ e

∗

7→ p

import e 7→ loadExpr(build(p))

The function build builds a store derivation. It is deﬁned in Section 5.5 and returns the

output path of the derivation. Overloading import to work on derivations allows Nix ex-

pressions to generate and use other Nix expressions. We will use this feature in Chapter 10

to implement build management with Nix, as it allows the automatic generation of lists of

inputs from other inputs (e.g., the list of header ﬁles imported by a C source ﬁle).

Finally, the nullary primops tr ue, false, and null are constant symbols: they represent

themselves, and have no evaluation rules. There is also an ever-growing set of utility

functions (added in an admittedly ad hoc fashion) such as map, baseNameOf, and so on;

but the semantics of those is not very relevant to a discussion of the foundations of the Nix

expression language.

4.4. Implementation

Maximal laziness Nix expression evaluation is implemented using the ATerm library

[166], which is a library for the efﬁcient storage and runtime manipulation of terms. The

Nix expression parser yields an ATerm encoding of the term. For example, the expression

(x: x) 123

yields the ATerm

Call(Function1("x",Var("x"),Pos("(string)",1,3)),Int(123))

The Pos node indicates position information, which is stored for certain language con-

structs (notably functions and attributes) to provide better error messages.

A very nice property of the ATerm library is its maximal sharing: if two terms are

syntactically equal, then they occupy the same location in memory. This means that a

shallow pointer equality test is sufﬁcient to perform a deep syntactic equality test. Maximal

sharing is implemented through a hash table. Whenever a new term is created, the term is

looked up in the hash table. If the term already exists, the address of the term obtained from

4. The Nix Expression Language

eval(e) :

if cache[e] 6= ε :

if cache[e] = blackhole :

Abort; inﬁnite recursion detected.

return cache[e]

else :

cache[e] ← blackhole

← eval

(e)

cache[e] ← e

return e

Figure 4.7.: Evaluation caching

the hash table is returned. Otherwise, the term is allocated, initialised, and added to the

hash table. A garbage collector takes care of freeing terms that are no longer referenced.

Maximal sharing is extremely useful in the implementation of a Nix expression inter-

preter since it allows easy caching of evaluation results, which speeds up expression eval-

uation by removing unnecessary evaluation of identical terms. The interpreter maintains a

hash lookup table cache : ATer m → ATerm that maps ATerms representing Nix expressions

to their normal form. Figure 4.7 shows pseudo-code for the caching evaluation function

eval, which “wraps” the evaluation rules deﬁned in Section 4.3.4 in a caching layer. The

function eval

simply implements those evaluation rules. It is assumed that eval

calls back

into eval to evaluate subterms (i.e., every time a rule uses the relation

∗

7→ in a condition),

and that it aborts with an appropriate error message if e does not evaluate to a good normal

form. Thus we obtain the desired caching. The special value ε denotes that no mapping

exists in the cache for the expression

. Note that thanks to maximal sharing, the lookup

cache[e] is very cheap: it is a lookup of a pointer in a hash table.

The function eval also perform a trick known as blackholing [134, 110] that allows

detection of certain simple kinds of inﬁnite recursion. When we evaluate an expression e,

we store in the cache a preliminary “fake” normal form blackhole. If, during the evaluation

of e, we need to evaluate e again, the cache will contain blackhole as the normal form for

e. Due to the determinism and purity of the language, this necessarily indicates an inﬁnite

loop, since if we start evaluating e again, we will eventually encounter it another time, and

so on.

Note that blackholing as implemented here differs from conventional blackholing, which

overwrites a value being evaluated with a black hole. This allows discovery of self-

referential values, e.g., x = ... x ...;. But it does not detect inﬁnite recursions like this:

(rec {f = x: f x;}).f 10

since every recursive call to f creates a new value of x, and so blackholing will not catch

the inﬁnite recursion. In contrast, our blackholing does detect it, since it is keyed on

maximally shared ATerms that represent syntactically equal expressions. The example

In the ATerm library, this means that the table lookup returned a null pointer.

4.4. Implementation

above is evaluated as follows:

(rec {f = x: f x;}).f 10

(REC) 7→ {f = x: (rec {f = x: f x;}).f x;}.f 10

(SELECT) 7→ (x: (rec {f = x: f x;}).f x) 10

(β -REDUCE) 7→ (rec {f = x: f x;}).f 10

This ﬁnal expression is equal to the ﬁrst (which is blackholed at this time), and so an

inﬁnite recursion is signalled.

The current evaluation cache never forgets the evaluation result of any term. This obvi-

ously does not scale very well, so one might want to clear or prune the cache eventually,

possibly using a least-recently used (LRU) eviction scheme. However, the size of current

Nix expressions (such as those produced during the evaluation of Nixpkgs) has not com-

pelled me to implement cache pruning yet. It should also be noted that due to maximal

sharing, cached values are stored quite efﬁciently.

The expression caching scheme described here makes the Nix expression evaluator max-

imally lazy. Languages such as Haskell are non-strict, meaning that values such as func-

tion arguments or let-bindings are evaluated only when necessary. A stronger property is

laziness, which means that these values are evaluated at most once. (Even though the se-

mantics of Haskell only requires non-strictness, in practice all implementations are lazy.)

Finally, maximal laziness means that syntactically identical terms are evaluated at most

once. This is not the case with mere laziness: a compiler is not obliged (or generally capa-

ble) to arrange for the evaluation result of two terms occurring in different locations in the

program, or in different invocations of a function, to be shared. For instance, there is no

guarantee that a lazy Haskell implementation will evaluate the expression product [1..1000]

only once when evaluating variable z in the following program:

x = product [1..1000]

y = product [1..1000]

z = x + y

Maximal laziness simpliﬁes the implementation of the Nix expression evaluator. For

instance, it is not necessary to implement sharing of β -redexes. Without sharing, it is

typically necessary to implement the rule β -REDUCE as follows:

β -REDUCE :

∗

7→ x: e

7→ let x = e

in e

where we give let a special “destructive update” semantics so that the evaluation result of

x is written back into the right-hand side of the let-binding [51]. This is typically how

laziness is implemented in functional languages: x is a pointer that points to a piece of

memory containing code and environment pointers (the closure or thunk [134]), which

after evaluation is overwritten with the actual result.

Error messages If Nix expression evaluation fails, or we import a syntactically or oth-

erwise invalid Nix expression, a backtrace is printed of the evaluation stack to give the user

some clue as to the cause of the problem. For instance, if we try to install Subversion from

a Nix expression that calls the one in Figure 2.9 with compressionSupport set to true, but

with zlib set to null, we get the following error message:

4. The Nix Expression Language

$ nix-env -f ./system/populate-cache.nix -i subversion

error: while evaluating the attribute `subversion'

at `./system/all-packages-generic.nix', line 1062:

while evaluating the function

at `(...)/subversion-1.2.x/default.nix', line 1:

assertion failed

at `(...)/subversion-1.2.x/default.nix', line 19

This tells the user not only what assertion triggered the error, but also where the incor-

rect call to the Subversion function was made (namely, in all-packages-generic.nix at line

1062).

Backtraces in lazy languages are tricky, as a failing value may be ﬁnally evaluated in

a piece of code far removed from where the value was “produced” [55]. Consider for

example:

let {

f = b: {x = assert b; 123;};

body = (f false).x;

}

Its evaluation will yield the following backtrace:

error: while evaluating the file `foo.nix':

while evaluating the attribute `body' at `foo.nix', line 3:

while evaluating the attribute `x' at `foo.nix', line 2:

assertion failed at `foo.nix', line 2

Note the absense of a reference to the function f. In a strict language, f would appear in the

backtrace. But also note that since attributes are annotated with position information, we

still get the position of the deﬁnition of the attribute x in the body of f.

Another problem with backtraces in lazy languages is that tail-call optimisation [151]

may remove useful stack frames. However, the current Nix expression evaluator does not

optimise tail-calls.

In practice, however, the stack traces printed by Nix are very helpful in everyday use,

as the example above shows. The trace may not show every pertinent code location, and

it may not show them in a meaningful order, but the locations that it does give usually

give enough of a clue to ﬁgure out the cause of the problem. A useful improvement is

a kind of slicing for assertion failures (which are the most common type of errors, apart

from missing attributes or function arguments). Currently, the interpreter simply prints that

an assertion has failed, i.e., its condition expression evaluated to false. So it is useful to

print an explanation as to why it is false. For example, if a guard expression p && x != null

evaluates to false, Nix should print whether p is false, or x != null is false. If p is false, it

should give an explanation as to how p obtained that value. In an interpreter (as opposed

to a compiled implementation), this is not hard to implement since the original expression

is still known, not just its normal form.

Optimising substitution There is an important opportunity for optimisation in the im-

plementation of β -reduction: if we substitute a term, e.g., replacing free occurrences of

4.4. Implementation

x in e

by e

in the β -reduction of (x : e

) e

, we never need to substitute under the re-

placements of x. Consider the expression (x : y : e

) e

, where e

is a large expression.

With normal substitution, we ﬁrst replace all occurrences of x in e

with e

. Then, we

replace all occurrences of y in the resulting term with e

. This substitution also descends

into the e

replacements of x, even though those subterms are closed. Since e

is large, this

is inefﬁcient.

The optimisation is that we can mark replacement terms to indicate to the substitution

function that it need not descend into such subterms. This is possible because of the fol-

lowing theorems.

Lemma 1. Every term to which a semantic rule is applied during evaluation is closed.

Proof. The property follows by induction on the application of semantic rules. The base

case is the initial terms produced by the parser. Here the induction hypothesis hold trivially

since the parser checks that these terms are closed and aborts with an error otherwise.

For the inductive step, we must show two things.

First, we must show that each rule that takes a closed term produces a closed term. This

can be easily veriﬁed. For instance, β -REDUCE yields the body of the function, e

, with

its sole free variable x substituted by the argument. e

cannot have additional free variables

because the function call e

is closed, and function calls do not bind variables. Likewise,

the expression yielded by SELECT must be closed because the attribute set in which it was

contained is closed, and attribute sets do not bind variables.

Second, we must show that subterms evaluated in the conditions of rules are closed. This

can also be easily veriﬁed. For instance, β -REDUCE recursively evaluates the expression

in a function call e

. Since by the induction hypothesis e

is closed, it follows that

is also closed, as function calls do not bind variables.

Theorem 2. All substitution terms occurring during evaluation are closed.

Proof. This property follows by inspecting all calls to subst in the evaluation rules and

observing that the substitution terms (i.e., the expressions in subs) are always closed. For

instance, in the rule β -REDUCE, subs = {x e

}. The term e

is closed because by

Lemma 1 the call e

is closed, and function calls bind no variables. A similar argument

holds for the subst calls in REC, β -REDUCE’ and WITH.

Since substitution terms are always closed, we can adapt the variable case of the substi-

tution function subst in Figure 4.6 as follows:

subst(subs,x) =











closed(e) if (x closed(e)) ∈ subs

closed(e) if (x e) ∈ subs

x otherwise

That is, replacement terms e are placed inside a wrapper closed(e). (The ﬁrst case merely

prevents redundant wrapping, e.g., closed(closed(e)), which reduces the effectiveness of

caching and blackholing.) The wrapper denotes that e is a closed subterm under which

no substitution is necessary, since it has no free variables. To actually make use of this

optimisation, we also add a case to subst to stop at closed terms:

subst(subs,closed(e)) = closed(e)

4. The Nix Expression Language

Of course, during evaluation we must get rid of closed eventually. That’s easy:

CLOSED : closed(e) 7→ e

as a closed term is semantically equivalent to the term that it wraps.

5. The Extensional Model

In Chapter 2, we have seen that Nix offers many advantages over existing deployment

systems, such as reliable dependency information, side-by-side deployment of versions

and variants, atomic upgrades and rollbacks, and so on. And as explained in Chapter 3,

it obtains these properties by imposing a “pointer discipline” on ﬁle systems, thus treat-

ing the component deployment problem similar to memory management in programming

languages.

The current and subsequent chapter formalise the operation of the Nix system. This

includes the translation of Nix expressions into store derivations, building store derivations,

the substitute mechanism, garbage collection, the invariants of the Nix store that must hold

to ensure correct deployment, and so on.

There are actually two different variants or models of Nix. The original model is the

extensional model, described in this chapter. It is the model with which we have had the

most experience. It implements all of the deployment properties described in Chapter 2.

The next chapter describes the intensional model which is more powerful in that it allows

the sharing of a Nix store between mutually untrusted users; however, it also places slightly

stricter requirements on components. Most of the aspects of the extensional model carry

over without modiﬁcation to the intensional model.

This chapter discusses all aspects of the semantics of the extensional model: the basics of

cryptographic hashing (Section 5.1), the notion of File System Objects and the organisation

of the Nix store (Section 5.2), the operation of adding sources and other “atomic” values

to the store (Section 5.3), translating Nix expressions to store derivations (Section 5.4),

building store derivations (Section 5.5), and garbage collection (Section 5.6). It concludes

with an explanation of the notion of extensionality (Section 5.7), which sets up the next

chapter on the intensional model.

5.1. Cryptographic hashes

Since Nix heavily uses cryptographic hash functions in store paths and in other places,

this section introduces some properties and notations that will be used in the remainder of

Part II.

A hash function [36] is a function h : A → B that maps values from a possibly inﬁnite

domain A onto a ﬁnite range B. For instance, h(x ∈ N) = x mod 27 is a hash function that

maps natural numbers to the numbers [0..26]. Given value x ∈ A and y = h(x) ∈ B, the

value x is called the preimage and y is called the hash of x. Since A is typically larger than

B, there must necessarily exist values x

∈ A such that h(x) = h(y). These are collisions

of the hash function h. A good hash function will have the property that collisions can be

expected to occur with low probability given the sets of values that will be fed into the hash

function in typical usage patterns.

5. The Extensional Model

A cryptographic hash function [145] is a hash function that maps arbitrary-length byte

sequences onto ﬁxed-length byte sequences, with a number of special properties:

• Preimage resistance: given a value y ∈ B, it should be computationally infeasible to

ﬁnd an x ∈ A such that y = h(x). Computational infeasibility means that ideally the

most efﬁcient way to ﬁnd x is a brute-force attack, that is, trying all possible values

in A. This means computing |A|/2 hashes on average.

• Collision resistance: it should be infeasible to ﬁnd any x

∈ A such that h(x

) =

h(x

). Due to the birthday paradox [145], a brute-force attack will ﬁnd a collision

with probability

after computing approximately

|B| hashes.

• Second preimage resistance: given an x

∈ A, it should be infeasible to ﬁnd another

∈ A such that h(x

) = h(x

). The difference with collision resistance is that here

is given.

For our purposes collision resistance and second preimage resistance are what matters—

it should not be feasible to ﬁnd collisions. This is since we do not ever want two different

components in the Nix store to be stored in the same store path. Preimage resistance—

infeasibility of inverting the hash—is not important. We do not care if it is possible to ﬁnd

a component given a store path.

Prominent examples of cryptographic hash functions are MD5 [141] and SHA-1 [127].

Nix initially used MD5, which produces 128-bit hashes. However, MD5 has recently been

broken [178] in the sense that it is now possible to ﬁnd collisions with a work effort of a

few hours on conventional hardware. That is, collision resistance has been broken, but this

technique does not ﬁnd preimages or second preimages.

In the context of Nix, a collision attack such as the one against MD5 means that it

is possible to construct two components that hash to the same store path. Speciﬁcally, an

attacker could create two components, one benevolent and one malevolent (e.g., containing

a Trojan horse—a malicious component masquerading as legitimate software), with the

same MD5 hash. The attacker would then publish a Nix expression that uses fetchurl

to download the benevolent component from a server, and ﬁnally replace the benevolent

component with the malevolent one on the server. Since on download the hash continues

to match, the deception would not be noticed. Constructing such components is quite easy

(see, e.g., [156, Section 8.4.4], and [120, 105] for examples of the construction of harmless

and harmful executables with matching MD5 hashes). Thus collisions of the underlying

hash are a serious concern.

SHA-1 is a very widely used hash function. It produces 160 bit hashes, making it more

resistant to birthday attacks in theory as it requires 2

hashes on average in a brute-force

attack. However, recent attacks [177, 146] have reduced the effort of ﬁnding a collision to

less than 2

hashes. This is still a substantial effort, but it creates enough doubt regarding

the future security of SHA-1 to require a transition away from it (“Attacks always get

better; they never get worse.” [147]).

There are a number of strengthened variants of SHA-1 that produce larger hashes. These

are SHA-224, SHA-256, SHA-384, and SHA-512 [127]. While there are strong doubts

about the future security of these hashes, they represent the best choice until a next-

generation hash function becomes available (e.g., through a similar process as the AES

selection [126]).

5.1. Cryptographic hashes

The Nix system uses SHA-256 hashes. However, MD5 and SHA-1 are available in

functions such as fetchurl for compatibility. The use of these hashes is still safe if they

apply to components from a trusted source, since it remains infeasible for a third-party

attacker to ﬁnd second preimages. However, we should migrate away from these hashes to

SHA-256.

Notation Byte sequences are ﬁnite-length sequences of bytes. The size of byte sequence

c is denoted |c|. Given a byte sequence c of size |c|, we write c[i],0 ≤ i < |c| to denote

the (i + 1)’th byte in the sequence. In examples, we write byte sequences initialised from

ASCII characters between quotes, e.g., "Hello" is a byte sequence of length 5.

Given a byte sequence c, we write h = hash

from the cryptographic hash function algorithm t, where t ∈ {md5, sha1,sha256}.

The function printHash

returns a hexadecimal string representation of a byte sequence,

useful for printing hashes. It is deﬁned as follows:

printHash

∑

0≤i<|c|

(digits

[c[i] div 16] + digits

[c[i] mod 16])

where + and

∑

denote string concatenation, and digits

is the byte sequence of hexadeci-

mal ASCII characters, i.e., "0123456789abcdef". For example,

printHash

(hash

md5

("Hello World")) = "b10a8db164e0754105b7a99be72e3fe5"

There is also a function printHash

that returns a base-32 string representation of a byte

sequence. It produces representations of length d|c| × 8/ lg 32e = d|c| ×

e. Since this is

shorter than the representations produced by printHash

(which have length |c| × 2), it is

useful inter alia for the representation of hashes in store paths. It is deﬁned as follows:

printHash

∑

d|c|×

e>i≥0

digits

[(

∑

0≤ j<|c|

c[ j] × 256

)/32

mod 32]

That is, interpreting c as a base-256 encoding of some number, with digits from least

signiﬁcant to most signiﬁcant (i.e., a little-endian number), we print the number in base-

32 with digits from most signiﬁcant to least signiﬁcant. (Note that the outer summation

denotes string concatenation, while the inner summation denotes integer addition.) The set

of digits is

digits

= "0123456789abcdfghijklmnpqrsvwxyz"

i.e., the alphanumerics excepting the letters e, o, u, and t. This is to reduce the possibility

that hash representations contain character sequences that are potentially offensive to some

users (a known possibility with alphanumeric representations of numbers [11]). Here is an

example of printHash

printHash

(hash

sha1

("Hello World")) = "s23c9fs0v32pf6bhmcph5rbqsyl5ak8a"

It is sometimes necessary to truncate a hash to a speciﬁc number of bytes n. For instance,

below we will truncate 32-byte SHA-256 hashes to 20 bytes (of course, this reduces the

5. The Extensional Model

security of the hash). This truncation is done by cyclically XORing the input with an

initially zero-ﬁlled byte array of length n. Formally, the result of truncate(n,c) is deﬁned

by the equation

truncate(n,c)[i] =

0≤ j<|c|, j mod n=i

c[ j]

where ⊕ denotes the XOR operator.

5.2. The Nix store

5.2.1. File system objects

The Nix store is a directory in the ﬁle system that contains ﬁle system objects (FSOs)

, such

as ﬁles and directories. In the context of the Nix store, FSOs typically represent software

components, user environments and store derivations. As we shall see below, Nix must

perform various operations on FSOs, such as computing cryptographic hashes over their

contents, scanning for strings, rewriting strings, and so on. Thus, it is useful to formalise

the notion of FSOs.

Operating systems have different kinds of objects that live in a ﬁle system. All modern

operating systems at the very least have hierarchical directories and ﬁles that consist of

unstructured sequences of bytes (as opposed to, say, ﬁles consisting of records). Beyond

that, there are many extensions:

• Symbolic links on Unix.

• Permissions, ranging from MS-DOS’s read-only attribute, through Unix’s access

rights, to complex Access Control Lists (Windows NT, various Unixes).

• Extended attributes (OS/2, Windows NT) that allow arbitrary name/value pairs to be

associated with ﬁles.

• Streams (Windows NT) or resource forks (Mac OS) that extend the notion of a ﬁle

as a sequence of bytes to being sequences of bytes.

• Forests of directory hierarchies (e.g., as formed by drive letters in Windows).

• Hard links that turn ﬁle systems into directed acyclic graphs instead of trees.

• Device nodes, FIFOs, named pipes, sockets, and other types of ﬁles with special

semantics.

Fortunately, for our purposes—which is storing components in the Nix store—most of

these features are either not relevant (drive letters, device nodes, etc.), can be ignored (hard

links), or are never used in practice (streams). Some others, such as permissions, we will

ignore to make life simpler.

In the Unix tradition, the term ﬁle refers to any ﬁle system object (i.e., an FSO), while the term regular ﬁle

refers to what most people call a ﬁle. This thesis uses the term FSO to avoid the ambiguity.

5.2. The Nix store

data FSO = Regular Executable ByteStream

| Directory [(FileName, FSO)]

| SymLink ByteStream

data Executable= Executable

| NonExecutable

Figure 5.1.: Abstract syntax for ﬁle system objects

What we absolutely cannot ignore for deployment of Unix software is regular ﬁles,

directories, and symbolic links. Also, while we can get away with disregarding most ﬁle

permissions, the executable bit must be maintained

. This leads to the grammar for FSOs

shown in Figure 5.1. (The data type for FSOs shown in this ﬁgure uses the Haskell-like

notation described in Section 1.7.) A ByteStream is a sequence of 0 or more bytes, and

a FileName is a sequence of 1 or more bytes not equal to 0. For example, the ﬁle system

object

Directory [

("foo",Regular NonExecutable "Hello World"),

("bar",Directory [

("xyzzy",Symlink "../foo")

])

]

deﬁnes a directory with two ﬁles: a non-executable ﬁle foo with contents Hello World, and a

subdirectory bar containing a symlink called xyzzy to foo in the parent directory. Note that

the top-level directory is anonymous. Also note that we do not keep any meta-information

about ﬁles other than the executable bit; notably, time stamps are discarded.

The type of FSOs deﬁned in Figure 5.1 is not actually used in the implementation of Nix.

They are merely a device used in this chapter to formalise certain aspects of the operation

of Nix. In the pseudo-code algorithms in this chapter, I will use the imperative functions

fso ← readPath(p) to denote reading the FSO fso stored at path p, and wr itePath(p, fso)

to denote writing FSO fso to path p. For presentational simplicity, I ignore error handling

aspects.

To perform operations on FSOs such as computing cryptographic hashes, scanning for

references, and so on, it is useful to be able to serialise FSOs into byte sequences, which

can then be deserialised back into FSOs that are stored in the ﬁle system. Examples of

such serialisations are the ZIP and TAR ﬁle formats. However, for our purposes these

formats have two problems:

• They do not have a canonical serialisation, meaning that given an FSO, there can

be many different serialisations. For instance, TAR ﬁles can have variable amounts

of padding between archive members; and some archive formats leave the order

Nix should also support setuid executables, which are programs that are executed under a different user ID

than the caller. However, it is not entirely clear how to support these in a clean way.

5. The Extensional Model

of directory entries undeﬁned. This is bad because we use serialisation to compute

cryptographic hashes over FSOs, and therefore require the serialisation to be unique.

Otherwise, the hash value can depend on implementation details or environment

settings of the serialiser. The canonicity of archives will be particularly important in

Section 7.5, when we apply binary patches to archives.

• They store more information than we have in our notion of FSOs, such as time

stamps. This can cause FSOs that Nix should consider equal to hash to different

values on different machines, just because the dates differ.

• As a practical consideration, the TAR format is the only truly universal format in the

Unix environment. It has many problems, such as an inability to deal with long ﬁle

names and ﬁles larger than 2

bytes. Current implementations such as GNU Tar

work around these limitations in various ways.

For these reasons, Nix has its very own archive format—the Nix Archive (NAR) format.

Figure 5.2 shows the serialisation function serialise(fso) that converts an FSO to a byte

sequence containing a NAR archive. The auxiliary function concatMap( f , x s) applies the

function f to every element of the list xs and concatenates the resulting byte sequences.

The function sortEntries sorts the list of entries in a directory by name, comparing the

byte sequences of the names lexicographically. The function serialise is typically used in

conjunction with readPath, e.g.,

c ← serialise(readPath(p)).

Of course, there is also a deserialisation operation deser ialise(c) (typically used with

writePath) that converts a byte sequence c containing a NAR archive to an FSO. It is the

inverse of serialise and is not shown here. The following identity holds:

deserialise(ser ialise(fso)) = fso

Note that deserialise is a partial function, since most byte sequences are not valid NAR

archives.

5.2.2. Store paths

A store object is a top-level ﬁle system object in the Nix store, that is, a direct child of

the Nix store directory. Indirect subdirectories of the Nix store are not store objects (but

contained in FSOs that are store objects).

A store path is the full path of a store object. It has the following anatomy:

storeDir/hashPart-name

The ﬁrst part is the path of the Nix store, denoted storeDir. The second is a 160-bit cryp-

tographic hash in base-32 representation. Finally, there is a symbolic name intended for

human consumption. An example of a store path is:

/nix/store/bwacc7a5c5n3qx37nz5drwcgd2lv89w6-hello-2.1.1

This is no longer a big issue, since Nix nowadays canonicalises all FSOs added to the store by setting irrelevant

metadata to ﬁxed values (see page 112). For instance, it sets the time stamps on all ﬁles to 0 (1 Jan 1970,

00:00:00 UTC). However, metadata such as the last-accessed time stamp might still cause non-canonicity.

5.2. The Nix store

serialise(fso) = str("nix-archive-1") + serialise

(fso)

serialise

(fso) = str("(") + seralise

(fso) + str(")")

serialise

(Regular exec contents) =

str("type") + str("regular")

(

str("executable")+ str(""), if exec = Executable

"", if exec = NonExecutable

+str("contents") + str(contents)

serialise

(SymLink target) =

str("type") + str("symlink")

+str("target") + str(target)

serialise

(Director y entries) =

str("type") + str("directory")

+concatMap(serialiseEntry, sortEntries(entries))

serialiseEntry((name, fso)) =

str("entr y") + str("(")

+str("name") + str(name)

+str("node") + serialise

(fso)

+str(")")

str(s) = int(|s|) + pad(s)

int(n) = the 64-bit little endian representation of the number n

pad(s) = the byte sequence s, padded with 0s to a multiple of 8 bytes

Figure 5.2.: serialise: Serialising ﬁle system objects into NAR archive

On the other hand,

/nix/store/bwacc7a5c5n3qx37nz5drwcgd2lv89w6-hello-2.1.1/bin/hello

is not a store path (but it has a preﬁx that is a store path).

Usually, storeDir is /nix/store, and in this thesis we will tacitly assume that it is. For

binary deployment purposes (Section 5.5.3), it is important that Nix store locations on the

source and target machines match.

The symbolic name consists of one or more characters drawn from the set of letters (in

either case), digits, and the special characters in the set "+-._?=".

We will assume that store paths are always fully canonicalised (see page 73), i.e., they

are absolute, do not contain . or .. elements, and do not have redundant separators (/). The

store path above is in canonical form; a non-canonical example of the same path is:

/nix/../nix//./store/bwacc7a5c5n3qx37nz5drwcgd2lv89w6-hello-2.1.1/

5. The Extensional Model

No component of the store location storeDir is allowed to be a symbolic link. The

reason is that some builders canonicalise paths by resolving symlink components. These

paths may then be stored in derivation outputs as retained dependencies. Then, if storeDir

has different symlink components on different machines, builds on these machines might

not be interchangeable.

In the remainder, we denote the inﬁnite set of syntactically correct store paths as Path.

Syntactic correctness depends on the value of storeDir—the location of the Nix store—but

we leave that implicit. We will use the convenience function hashPart(p) to extract the

hash part of a store path. Likewise, namePart(p) yields the symbolic name part of p.

How do we compute the hash part of store paths? The hash computation has some

important properties. First, both the Nix store path and the symbolic name are part of the

hash. This means that Nix stores in different locations yield different hash parts, and that

sources or outputs that are identical except for their symbolic names have different hash

parts. This is important because the hash scanning approach to dependency determination

described in Section 3.4 only looks at the hash parts of paths, not the symbolic name.

Second, different types of store objects—notably sources and outputs—have different

hashes. This prevents users from adding sources that “impersonate” outputs. In unshared

Nix stores, this is not a major issue, but it is important for security in a shared store that

uses the technique described in Section 6.2 to enable sharing of locally built outputs.

These properties are achieved in the function makePath, which computes store paths as

follows:

makePath(type, descr, name) =

nixStore + "/" + printHash

(truncate(20, hash

sha256

(s))) + "-" + name

where

s = type + ":sha256:" + descr + ":" + nixStore + ":" + name

(Examples will be shown in the remainder of this chapter.) Thus, the hash part of the

store path is a base-32 representation of a 160-bit truncation of a SHA-256 hash of the

variable elements that must be represented in the hash, namely, the location of the Nix store

nixStore, the type of the store object (e.g., "source"), a unique descriptor string describing

the store object (e.g., a cryptographic hash of the contents of the FSO in case of sources),

and the symbolic name.

Why do we truncate the SHA-256 hash to 160 bits? Nix originally used base-16 (hex-

adecimal) representations of 128-bit MD5 hashes, giving a hash part of 128/ lg 16 = 32

characters. Since MD5 was recently broken (see Section 5.1), a decision was made to

switch to 160-bit SHA-1 hashes. The hexadecimal representation was at that time replaced

with a base-32 representation, which has the pleasant property of keeping the length at

160/lg 32 = 32 characters. (Some builders strip off hash parts from store paths using a

hard-coded constant of 32 characters (clearly a bad practice), so these did not need to be

changed.) However, as attacks against SHA-1 were also published, SHA-256 was used

instead. Unfortunately, hash parts of d256/ lg32e = 52 characters are a bit too long: to-

gether with the symbolic name, they exceed the maximum ﬁle name lengths of some ﬁle

systems. In particular, the Joliet extension to ISO-9660 has a maximum ﬁle name length

of 64 characters [38]. This would make it difﬁcult to burn Nix stores on a CD-ROM, a

5.2. The Nix store

potentially useful distribution method. For this reason the SHA-256 hash is truncated to

160 bits. The assumption is that this is no less secure than 160-bit SHA-1, and possibly

more secure due to its different structure. However, given that most modern hashes share

the same basic design, the security of all of them might be ephemeral.

5.2.3. Path validity and the closure invariant

Nix maintains meta-information about store paths in a number of database tables. This

information includes the following:

• The set of valid paths, which are the paths that are “ﬁnished”, i.e., whose contents

will no longer change. For instance, the output path of a derivation isn’t marked

as valid until the builder that produces it ﬁnishes successfully. The contents of an

invalid path are undeﬁned (except when the FSO is under construction, in which

case a lock is used to indicate this fact, as we shall see below).

• The references graph (i.e., the deployment analogue of the pointer graph in program-

ming languages). For each valid path, Nix maintains the set of references to other

valid paths discovered by scanning for hash parts.

• The substitutes that have been registered by tools such as nix-pull (Section 2.6).

• Traceability information: for each valid output path, Nix remembers the derivation

that built it.

The current Nix implementation stores this information in a transactional Berkeley DB

database. As we shall see, transactions are important to ensure that the store invariants

described below always hold, that interrupted operations can be restarted, and that multiple

operations can be safely executed in parallel.

The most important pieces of information in the database are the set of valid paths and

the references graph, which are deﬁned here. The other database tables are deﬁned below.

The type of a database table t is given as t : A → B, where A is the type of the key and B is

the type of the value corresponding to a key; t[x] denotes the value corresponding to key x.

The special value ε indicates that no entry occurs for a value in the given mapping. Setting

of table entries is denoted by t[x] ← y.

The mapping valid : Path → Hash serves two purposes. First, it lists the set of valid

paths—the paths that have been successfully built, added as a source, or obtained through

a substitute. It does not include paths that are currently being built, or that have been left

over from failed operations. Thus, valid[p] 6= ε means that path p is valid. The following

invariant states that valid paths should exist.

Invariant 1 (Validity invariant). If a path is valid, it exists in the Nix store:

∀p ∈ Path : valid[p] 6= ε → pathExists(p)

Second, the valid mapping stores a SHA-256 cryptographic hash of the contents of each

valid path. That is, when the path is made valid, the valid entry for the path is set as follows:

valid[p] ← "sha256:" + printHash

(hash

sha256

(serialise(readPath(p))))

5. The Extensional Model

(The name of the hash algorithm is incorporated to facilitate future upgrading of the hash

algorithm.) For instance, given a valid path /nix/store/bwacc7a5c5n3...-hello-2.1.1 with an

FSO whose serialisation has SHA-256 hash 085xkvh8plc0..., we have

valid["/nix/store/bwacc7a5c5n3...-hello-2.1.1"] = "sha256:085xkvh8plc0..."

The purpose of storing these content hashes is to detect accidental tampering with store

objects, which are supposed to be read-only. The command nix-store --verify --check-

contents checks that the contents of store paths are still consistent with their stored hashes.

That is, the following invariant must hold.

Invariant 2 (Stored hash invariant). The contents at valid store paths correspond to the

cryptographic hashes stored in the valid table:

∀p ∈ Path : valid[p] 6= ε →

valid[p] = "sha256:" + printHash

(hash

sha256

(serialise(readPath(p))))

The mapping references : Path → {Path} maintains the references graph, i.e., the set

of store paths referenced by each store object. For sources, the references are usually

empty. For store derivations, they are exactly the union of the store paths listed in its

inputSrcs and inputDrvs ﬁelds. For output paths, they are found through scanning using

the method outlined in Section 3.4. As we will see in Section 5.6.3, the references graph

is acyclic except for trivial cycles deﬁned by self-references, i.e., when p ∈ references[p].

Self-references occur when a builder stores its output path in its output.

An important property of the Nix store is that at all times the following invariant holds.

Invariant 3 (Closure invariant). The set of valid paths is closed under the references

relation:

∀p ∈ Path : valid[p] 6= ε → ∀p

∈ references[p] : valid[p

] 6= ε

The closure of a valid path p is the set of all paths reachable by traversing references

from p:

closure(p) = {p} ∪

[

∈references[p]

closure(p

)

As I explained in Section 2.1, the closure is vitally important for correct deployment, since

the closure of a component is the set of paths that might be accessed in an execution

involving the component. The closure is what must be copied in its entirety when we

deploy the component. Invariant 3 (closure) and Invariant 1 (validity) together give us

complete deployment: if a path p is valid, then all its potential runtime dependencies are

also valid; and all these paths exist in the ﬁle system.

The references graph is also maintained in the opposite direction. Nix maintains a map-

ping referers : Path → {Path} that deﬁnes the transpose of the graph deﬁned by the refer-

ences mapping

. Thus, the following invariant must hold.

The referers table should properly be spelled referrers. However, the Nix implementation unwittingly followed

the unfortunate tradition established by the HTTP standard, which has a misspelled Referer header ﬁeld [57].

5.3. Atoms

Invariant 4 (Integrity invariant). The graphs deﬁned by the references and referers map-

pings are each other’s transposes:

∀p, p

∈ Path : p ∈ references(p

) ↔ p

∈ referers(p)

We can also compute the closure of a valid path under the referers relation (the referrer

closure):

closure

(p) = {p} ∪

[

∈referers[p]

closure

)

This is the set of all paths that can reach path p in the references graph. Note that while

closure(p) is constant for a valid path (i.e., can never change due to the immutable nature

of valid objects), closure

(p) is variable, since additional referrers can become valid (e.g.,

due to building), and existing referrers can become invalid (due to garbage collection).

The main application of referrers is to allow users to gain insight in the dependency

graph. The command nix-store --query --referers allows users to query what paths refer to

a given path. For instance,

$ nix-store -q --referers /nix/store/wa2xjf809y7k...-glibc-2.3.5

/nix/store/vcwh8w6ryn37...-libxslt-1.1.14

/nix/store/vfps6jpav2h6...-gnutar-1.15.1 [...]

shows the components in the store that use a particular instance of Glibc. Likewise, the

command nix-store --query --referers-closure prints all paths that directly or indirectly refer

to the given path.

5.3. Atoms

A basic operation on the Nix store is to add atoms to the store. Atoms are store objects that

are not derived, that is, are not built by performing a store derivation. The foremost exam-

ples of atoms are sources referenced from derivations in Nix expressions (e.g., builder.sh

in Figure 2.6), and store derivations.

The operation addToStore(fso, refs, name) shown in Figure 5.3 adds an FSO fso to the

Nix store. The store path is computed from the cryptographic hash of the serialisation of

fso, and the symbolic name name. The references entry for the resulting path is set to refs.

The operation works as follows. It computes the cryptographic hash over the serialisa-

tion of the FSO, and uses that and name to construct the store path p 45 . If path p is not

valid 47 , the FSO is written to path p 48

. The path is marked as valid, with the references

of p set to refs 49 . The references are set by the helper function setReferences 50 , which

also updates the corresponding referrer mappings. The remainder of the code is concerned

with concurrency.

The hash part of path p is essentially a cryptographic hash of the atom being added.

Therefore there is a correspondence between the ﬁle name of the store object created

by addToStore, and its contents. As a consequence, knowing the contents of an FSO

(and its symbolic name) also tells one its store path. This property is known as content-

addressability. It does not hold for all store objects: derivation outputs are not content-

addressable, since their output paths are computed before they are built (as we will see in

5. The Extensional Model

addToStore(fso, refs, name) :

c ← serialise(fso)

h ← hash

sha256

(c)

p ← makePath("source", printHash

(h), name) 45

if valid[p] = ε : 46

lock ← acquirePathLock(p)

if valid[p] = ε : 47

if pathExists(p) :

deletePath(p)

writePath(p, fso) 48

in a database transaction : 49

valid[p] ← h

setReferences(p, refs)

releasePathLock(p, lock)

return p

setReferences(p, refs) 50

references[p] ← refs

for each p

∈ refs :

referers[p

]

∪

← {p}

Figure 5.3.: addToStore: Adding atoms to the Nix store

Section 5.4). This is in fact the deﬁning characteristic of the extensional model. In the

intensional model described in Chapter 6, content-addressability is extended to all store

objects.

Concurrency We have to take into account the remote possibility that parallel Nix pro-

cesses simultaneously try to add the same atom. If this happens, the operations should be

idempotent, and the contents of a path should never change after it has been marked as

valid. There should be no race conditions, e.g.:

1. Process A detects that the target path p is invalid, and starts writing the atom to p.

2. Process B detects that p is invalid, and wants to write the atom to the target path, but

there is an obstructing ﬁle at p. As it cannot make assumptions about invalid paths,

it deletes p and starts writing the atom to p.

3. While B is half-way through, A ﬁnishes writing and marks the path as valid. Note

that some of A’s writes have been deleted.

4. Before B ﬁnishes writing, A starts the builder of a derivation that has p as input. The

builder uses the incorrect contents of p.

There are several solutions to prevent races. The simplest one is to use a global mutex

lock on the Nix store for all operations that modify the store. These operations are the

5.3. Atoms

acquirePathLock(p) :

while true:

fd ← open(p + ".lock", O_WRONLY|O_CREAT, 0666)

fcntl(fd,F_SETLKW, ...F_WRLCK...) 51

if size of ﬁle fd = 0 : 52

return fd 53

close(fd)

releasePathLock(p, fd) :

deleteAndSignal(p + ".lock", fd)

deleteAndSignal(p, fd) :

deletePath(p)

write(fd, "deleted") 54

close(fd) 55

Figure 5.4.: acquirePathLock and releasePathLock: Locking on Unix

addition of atoms, the building of derivations (Section 5.5), and garbage collection (Sec-

tion 5.6). Every Nix process acquires the lock on startup, and releases it when ﬁnished.

If the lock is already held, processes block until it becomes available. This approach is

clearly unacceptable because of the unbounded latency it introduces for Nix operations.

A better solution is ﬁne-grained locking, where a process acquires a mutex lock on the

speciﬁc store path that it wants to add. The locking operations lock ← acquirePathLock(p)

and releasePathLock(p, lock) essentially acquire and release a lock on a temporary ﬁle

p.lock, i.e., p with the extension .lock appended.

Figure 5.4 shows pseudo-code for an implementation of these operations on Unix. It

uses the POSIX standard system call fcntl, which locks byte ranges in ﬁles [152]. The

advantage of POSIX locks is that they are released automatically when the process holding

them dies, either normally or abnormally. In contrast, the use of ﬁle existence as a mutex

makes it harder to recover gracefully from abnormal termination and system crashes.

A mostly aesthetic complication is that we don’t want the lock ﬁles to hang around

indeﬁnitely. They should be removed when they are no longer necessary, i.e., when p

has been registered as being valid. Deleting the lock ﬁle after we have released the lock

introduces a subtle race, however:

1. Process A creates the lock ﬁle p.lock and locks it.

2. Process B opens the lock ﬁle, attempts to acquire a lock, and blocks.

3. Process A sees that p is invalid, and begins to write to p. However, it is interrupted

before it can make p valid.

4. As part of its cleanup, process A releases the lock, closes and deletes the lock ﬁle,

and terminates. Note that B still has the ﬁle open; on Unix it is possible to delete

open ﬁles.

5. The Extensional Model

5. Process B acquires the lock.

6. Process B sees that p is invalid, and begins to write to it.

7. Process C creates the lock ﬁle p.lock and locks it. This is a different ﬁle than the one

currently opened by B—it has a different inode number. Thus C can acquire a lock.

8. Process C sees that p is invalid, and begins to write to it.

9. Now processes B and C are concurrently writing to p.

To prevent this race, we have to signal to process B that the lock ﬁle it has open has

become “stale”, and that it should restart. This is done by the function deleteAndSignal,

called by the deleting process: it writes an arbitrary string (the deletion token) to the deleted

ﬁle 54

. If another process subsequently acquires a lock 51 , but sees that the size of the ﬁle

is non-zero 52 , then it knows that its lock is stale.

On Windows, there is a much simpler solution: we just try to delete the ﬁle, which will

fail if it is currently opened by any other process. Thus, the ﬁle will be deleted by the last

process to release the lock, provided no new lockers have opened the lock ﬁle.

The dual checks for path validity are an optimisation ( 46 , 47 ). Strictly speaking, the

initial path validity check 46 can be omitted. However, the extra check prevents a lock

creation and acquisition in the common case where the path is already valid.

5.4. Translating Nix expressions to store derivations

Nix does most of its work on store derivations, introduced in Section 2.4. Store derivations

are the result of the evaluation of high-level Nix expressions. Nix expressions can express

variability, store derivations cannot—a store derivation encodes a single, speciﬁc, constant

build action.

The command nix-instantiate performs precisely this translation: it takes a Nix expres-

sion and evaluates it to normal formal using the expression semantics presented in Sec-

tion 4.3.4. The normal form should be a call to derivation, or a nested structure of lists and

attribute sets that contain calls to derivation. In any case, these derivation Nix expressions

are subsequently translated to store derivations using the method described in this section.

The resulting store derivations can then be built, as described in the next section. The

command nix-store --realise does exactly that. The high-level package management com-

mand nix-env combines translation and building as a convenience to users.

The abstract syntax of store derivations is shown in Figure 5.5 in a Haskell-like [135]

syntax (see Section 1.7). The store derivation example shown in Figure 2.13 is a value

of this data type. The outputHash and outputHashAlgo ﬁelds implement so-called ﬁxed-

output derivations and are discussed in Section 5.4.1. The other ﬁelds were explained in

Section 2.4.

So suppose that we have an expression e that evaluates to derivation e

, and e

evaluates

to an attribute set {as}. According to the DERIVATION and DERIVATION! rules (page 80),

e then (essentially) evaluates to the result of calling instantiate(as). instantiate is a function

that translates the argument attribute set of a call to derivation to a store derivation, writes

100

5.4. Translating Nix expressions to store derivations

data StoreDrv = StoreDrv {

output : Path,

outputHash : String,

outputHashAlgo : String,

inputDrvs : [Path],

inputSrcs : [Path],

system : String,

builder : Path,

args : [String],

envVars : [(String,String)]

}

Figure 5.5.: Abstract syntax of store derivations

as a side-effect the store derivation to the Nix store

, and returns the original attribute set,

with the following three attributes added:

• drvPath contains the path of the store derivation.

• outPath contains the output path of the store derivation, that is, d.output.

• type is set to the value "der ivation". This allows instantiate to detect that attribute sets

occurring in its arguments are actually subderivations. It also allows nix-instantiate

and nix-env to see whether evaluation results are actually derivations.

The function instantiate is shown in Figure 5.6. Its main job is to construct a store

derivation 56 which is written to the Nix store using addToStore 65 . The hard part is in

ﬁlling in the ﬁelds of the store derivation from the attributes as. In particular, we need to

compute the following bits of information:

• We need an output path (the ﬁeld output). The output path reﬂects all information

that goes into the derivation. Therefore we initially leave output empty 57 , then

compute a cryptographic hash over this initial derivation and use makePath to con-

struct the actual path 64 . The hash is computed using the function hashDrv, which

conceptually just computes a SHA-256 hash over a concrete syntax representation

of the store derivation. However, the notion of ﬁxed-output derivations complicates

matters a bit. The actual operation of hashDrv is discussed below. The ﬁnal output

path is also placed in the out environment variable, in order to communicate the path

to the builder.

• We need to compute the environment variable bindings envVars 63 . envVars is a set

of name/value tuples. Each attribute hn = ei in {as} is mapped to an environment

variable binding. This translation is done by the function processBinding shown in

Figure 5.7. It takes e and returns, among other things, a string representation of

Aﬁcionados of purely functional programming languages may be dismayed by the impurity of producing a

store derivation as a side-effect. However, the side-effect is not observable in the Nix expression evaluation,

and the idempotency of the operation ensures that equational reasoning still holds.

101

5. The Extensional Model

instantiate(as) :

name ← eval(as.name)

if name ends in ".drv" : abort

if name contains invalid characters : abort

d ← StoreDrv { 56

output = "" 57

outputHash = eval(as.outputHash) if attr. exists, "" otherwise

outputHashAlgo = ""

("r :" if eval(as.outputHashMode) = "recursive", "" otherwise) +

(eval(as.outputHashAlgo) if attr. exists, "" otherwise)

inputDrvs = {processBinding(e).drvs | hn = ei ∈ as} 59

inputSrcs = {processBinding(e).srcs | hn = ei ∈ as} 60

system = eval(as.system) // Must evaluate to a string.

builder = concSp(processBinding(as.builder).res)

args = map(λ e . processBinding(e).res, as.args) 62

envVars = {(n, concSp(processBinding(e).res)) | hn = ei ∈ as} 63

∪ {(out, "")}

}

d.output ← makePath("output:out", hashDrv(d), name) 64

d.envVars["out"] ← d.output

p ← addToStore(printDrv(d), d.inputDrvs ∪ d.inputSrcs, name + ".drv") 65

return {outPath = d.output, drvPath = p}

Figure 5.6.: instantiate: Instantiation of store derivations from a Nix expression

the normal form of e. processBinding is discussed in more detail below. However, it

should be pointed out here that processBinding returns a list of strings, since attribute

values can be lists of values. For instance, it is common to communicate a list of

paths of components to a builder:

buildInputs = [libxml2 openssl zlib];

Since the value of an environment variable is a plain string, the list of

strings returned by processBinding is concatenated with spaces separating the el-

ements; e.g., "/nix/store/ngp50703j8qn...-libxml2-2.6.17t/nix/store/8ggmblhvzciv...-

openssl-0.9.7ft/nix/store/kiqnkwp5sqdg...-zlib-1.2.1" where t denotes a space.

• Likewise, processBinding is used to set the command-line arguments from the args

attribute 62 , and the builder from the builder attribute 61 . Since args is a list, we do

not need to concatenate the strings returned by processBindings.

• The system ﬁeld is initialised from the system attribute.

• The set of input derivations inputDrvs (the build-time dependencies) is the set of

store paths of derivations that occur in the attributes

59 . These are discovered by

processBinding.

102

5.4. Translating Nix expressions to store derivations

processBinding(e) :

← eval(e)

if e

= true :

return {drvs = /0, srcs = /0, res = ["1"]}

if e

= false :

return {drvs = /0, srcs = /0, res = [""]}

if e

= a string s :

return {drvs = /0, srcs = /0, res = [s]}

if e

= null :

return {drvs = /0, srcs = /0, res = [""]}

if e

= a path p : 66

← addToStore(readPath(p), /0, baseName(p))

return {drvs = /0, srcs = {p

}, res = [p

]}

if e

= an attribute set {as} with as.type = "derivation" : 67

// Note that {as} is the result of a call to derivation.

return {drvs = {eval(as.drvPath)}, srcs = /0, res = [eval(as.outPath)]}

if e

= a list [ls] : 68

ps ← map(processBinding, ls)

return

{ drvs =

p∈ps

p.drvs, srcs =

p∈ps

p.srcs

, res = concatMap(λ p . p.res, ps)}

abort

Figure 5.7.: processBinding: Processing derivation attributes

• The set of input sources inputSrcs is the set of store paths of sources that occur in

the attributes 60 . These are also discovered by processBinding.

Processing the attributes Clearly, the core of the translation is the function process-

Binding, shown in Figure 5.7. It evaluates a Nix expression, inspects it, and returns three

pieces of information in a structure:

• A list of strings that represent the normal form of the expression (res).

• The set of store paths of derivations occurring in the expression (dr vs).

• The set of store paths of sources occurring in the expression (srcs).

Simple values like strings, Booleans, and null map to singleton string lists. For instance,

true is translated to ["1"], and false is translated to [""]. The latter is so that we can write in

shell scripts:

if test -n "$var"; then ... fi

to test whether Boolean variable $var is set.

103

5. The Extensional Model

For lists, processBinding is applied recursively to the list elements 68 . The resulting

string lists are concatenated into a single list. Thus, nested lists are allowed; they are

ﬂattened automatically.

For paths, processBinding copies the referenced FSO to the Nix store using addToStore

and returns the resulting path in srcs and res 66 . Thus all referenced sources end up in the

Nix store. The ﬁle name of the source (returned by the function baseName) is used as the

symbolic name of the store path.

For attribute sets, processBinding checks whether the type attribute of the derivation (if

present) is equal to derivation. If so, the value denotes the result of a call to derivation. This

is where instantiate recurses into subderivations: instantiate calls processBinding, which

calls eval, which eventually calls instantiate for derivations occurring in the input attributes.

Let us look at an example. Suppose that we have a derivation

args = ["-e" ./builder.sh python];

where python is a variable that evaluates to another derivation. Since this is a list, process-

Binding will recurse into the list elements. This yields the following results. For "-e", we

get

{drvs = /0,srcs = /0,res = ["-e"]}.

For ./builder.sh, the ﬁle builder.sh in the current directory is added to the Nix store using

addToStore. Supposing that the ﬁle has SHA-256 hash 49b96daeab53... in hexadecimal,

addToStore will call makePath with the following arguments:

makePath("source","49b96daeab53...","builder.sh").

makePath constructs the hash part by hashing and truncating the following string

source:sha256:49b96daeab53...:/nix/store:builder.sh

into 043577cf3vlb.... The store path thus becomes /nix/store/043577cf3vlb...-builder.sh, and

processBinding returns

{ drvs = /0,srcs = {/nix/store/043577cf3vlb...-builder.sh}

, res = ["/nix/store/043577cf3vlb...-builder.sh"] }.

For python, if we suppose that it evaluates to an attribute set with drvPath = /nix/store/-

mi4p0ck4jqds...-python-2.4.1.drv and outPath = /nix/store/s3dc4m2zg9bb...-python-2.4.1,

processBinding returns

{ drvs = {/nix/store/mi4p0ck4jqds...-python-2.4.1.drv},srcs = /0

, res = ["/nix/store/s3dc4m2zg9bb...-python-2.4.1"] }.

Note that the path of the store derivation is placed in drvs (and will therefore eventually end

up in the inputDrvs of the referring derivation), since the derivation is needed to recursively

build this dependency when we build the current derivation. On the other hand, the path of

the output of the derivation is placed in res (and will end up in an environment variable or

command-line argument), since that is what the builder is interested in.

104

5.4. Translating Nix expressions to store derivations

Derive(

[("out","/nix/store/bwacc7a5c5n3...-hello-2.1.1","","")],

[("/nix/store/7mwh9alhscz7...-bash-3.0.drv",["out"]),

("/nix/store/fi8m2vldnrxq...-hello-2.1.1.tar.gz.drv",["out"]),

("/nix/store/khllx1q519r3...-stdenv-linux.drv",["out"]),

("/nix/store/mjdfbi6dcyz7...-perl-5.8.6.drv",["out"])],

["/nix/store/d74lr8jfsvdh...-builder.sh"],

"i686-linux",

"/nix/store/3nca8lmpr8gg...-bash-3.0/bin/sh",

["-e","/nix/store/d74lr8jfsvdh...-builder.sh"],

[("builder","/nix/store/3nca8lmpr8gg...-bash-3.0/bin/sh"),

("name","hello-2.1.1"),

("out","/nix/store/bwacc7a5c5n3...-hello-2.1.1"),

("perl","/nix/store/h87pfv8klr4p...-perl-5.8.6"),

("src","/nix/store/h6gq0lmj9lkg...-hello-2.1.1.tar.gz"),

("stdenv","/nix/store/hhxbaln5n11c...-stdenv-linux"),

("system","i686-linux")] )

Figure 5.8.: ATerm representation of the Hello derivation

Finally, the results of the recursive calls are combined for the list, and we get

{ drvs = {/nix/store/mi4p0ck4jqds...-python-2.4.1.drv}

, srcs = {/nix/store/043577cf3vlb...-builder.sh}

, res = ["-e" "/nix/store/043577cf3vlb...-builder.sh"

"/nix/store/s3dc4m2zg9bb...-python-2.4."] }.

Writing the derivation When the ﬁnal output path of the derivation has been computed,

we write a string representation of the store derivation to the Nix store using addTo-

Store 65 . The function printDrv returns a byte sequence that represents the store derivation.

The contents of the byte sequence is a textual ATerm. ATerms [166] are a simple format

for the exchange of terms. For the Hello example, the ATerm representation is shown in

Figure 5.8

I will not formalise the translation to ATerms here, which is straightforward. However,

the reader may be wondering about the presence of the "out" strings in the ATerm encoding

of the inputDrvs ﬁeld, as these do not occur in the abstract syntax, and about the fact that

output is a list here, also containing the mysterious "out" string. This is actually for future

compatibility with a Nix semantics that supports multiple output paths. Every derivation

would produce a set of labelled outputs, the default output having label out. Multiple out-

puts allow more ﬁne-grained deployment. In Section 6.7, I show a semantics that supports

this.

Note that in the call to addToStore, we set the references of the new store object to the

union of the store derivations (inputDrvs) and sources (inputSrcs) used by the derivation.

Thus, the references graph is maintained for store derivations. This means that we can

query the closure of a derivation, which is the set of ﬁles necessary to do a build of the

Whitespace has been added to improve legibility. Actual textual ATerms as produced by the function ATwrite-

ToString() in the ATerm API do not contain whitespace. In other words, ATwriteToString() produces a canonical

representation of an ATerm, and thus of the store derivation it represents.

105

5. The Extensional Model

derivation and all its dependencies. As we saw in Section 2.4, this is useful for source

deployment.

On the other hand, the path in the output ﬁeld is not a reference of the store derivation,

since the output in general does not exist yet, and because we want garbage collection of

derivations and outputs to be independent.

5.4.1. Fixed-output derivations

The only remaining aspect of the translation is the computation of the output path by

hashDrv. As stated above, conceptually the output path is constructed from a SHA-256

hash over the ATerm representation of the initial store derivation (i.e., the one with output

set to the empty string 57 ). However, the matter is complicated by the notion of ﬁxed-

output derivations.

Fixed-output derivations are derivations of which we know the output in advance. More

precisely, the cryptographic hash of the output path is known to the Nix expression writer.

The rationale for ﬁxed-output derivations is derivations such as those produced by the

fetchurl function. This function downloads a ﬁle from a given URL. To ensure that the

downloaded ﬁle has not been modiﬁed, the caller must also specify a cryptographic hash

of the ﬁle. For example,

fetchurl {

url = http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz;

md5 = "70c9ccf9fac07f762c24f2df2290784d";

}

It sometimes happens that the URL of the ﬁle changes, e.g., because servers are reorganised

or no longer available. We then must update the call to fetchurl, e.g.,

fetchurl {

url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;

md5 = "70c9ccf9fac07f762c24f2df2290784d";

}

If a fetchurl derivation followed the normal translation scheme, the output paths of the

derivation and all derivations depending on it would change. For instance, if we were to

change the URL of the Glibc source distribution—a component on which almost all other

components depend—massive rebuilds will ensue. This is unfortunate for a change which

we know cannot have a real effect as it propagates upwards through the dependency graph.

Fixed-output derivations solve this problem by allowing a derivation to state to Nix that

its output will hash to a speciﬁc value. When Nix builds the derivation (Section 5.5), it

will hash the output and check that the hash corresponds to the declared value. If there is

a hash mismatch, the build fails and the output is not registered as valid. For ﬁxed-output

derivations, the computation of the output path only depends on the declared hash and hash

algorithm, not on any other attributes of the derivation.

Figure 5.9 shows the deﬁnition of the fetchurl function. The expression presented here

is somewhat simpliﬁed from the actual expression in Nixpkgs, which accepts SHA-1 and

SHA-256 hashes in addition to MD5 hashes. A ﬁxed-derivation output is declared by

deﬁning the following attributes:

106

5.4. Translating Nix expressions to store derivations

{stdenv, curl}: # The `curl' program is used for downloading.

{url, md5}:

stdenv.mkDerivation {

name = baseNameOf (toString url);

builder = ./builder.sh;

buildInputs = [curl];

# This is a fixed-output derivation;

# the output has to have MD5 hash `md5'.

outputHashMode = "flat";

outputHashAlgo = "md5";

outputHash = md5;

inherit url;

}

Figure 5.9.: pkgs/build-support/fetchurl/default.nix: ﬁxed-output derivations in fetchurl

• The attribute outputHashAlgo speciﬁes the hash algorithm: outputHashAlgo ∈ {

"md5","sha1","sha256"}.

• The attribute outputHash speciﬁes the hash in either hexadecimal or base-32 nota-

tion, as autodetected based on the length of the string.

• The attribute outputHashMode ∈ {"recursive","ﬂat"} states whether the hash is com-

puted over the serialisation of the output path p, i.e., hash(serialise(readPath(p))),

or over the contents of the single non-executable regular ﬁle at p, respectively. The

latter (which is the default) produces the same hashes as standard Linux commands

such as md5sum and sha1sum.

Recursive mode is useful for tools that download directory hierarchies instead of

single ﬁles. For instance, Nixpkgs contains a function fetchsvn that exports revisions

of directory hierarchies from Subversion repositories, e.g.,

fetchsvn {

url = https://svn.cs.uu.nl:12443/repos/trace/nix/trunk;

rev = 3297;

md5 = "2a074a3df23585c746cbcae0e93099c3";

}

For historical reasons, the outputHashMode attribute is encoded in the outputHash-

Algo ﬁeld 58 .

These three attributes are used in the output path computation. This means that, for

instance, the url attribute in fetchurl derivation is disregarded. Thus, instantiation of the

two fetchurl calls shown above produces two store derivations that have the same output

ﬁeld.

107

5. The Extensional Model

hashDrv(d) :

if d.outputHash 6= ""

return hash

sha256

("ﬁxed:out:" + d.outputHashAlgo

+":" + d.outputHash

+":" + d.output)

else :

← d { // I.e., d

is a modiﬁed copy of d

inputDrvs = {hashDrv(parseDrv(readPath(p))) | p ∈ d.inputDrvs} ) 69

}

return hash

sha256

(printDrv(d

))

Figure 5.10.: hashDrv: Hashing derivations modulo ﬁxed-output derivations

So how do we compute the hash part of the output path of a derivation? This is done

by the function hashDr v, shown in Figure 5.10. It distinguishes between two cases. If the

derivation is a ﬁxed-output derivation, then it computes a hash over just the outputHash

attributes

If the derivation is not a ﬁxed-output derivation, we replace each element in the deriva-

tion’s inputDr vs with the result of a call to hashDrv for that element. (The derivation at

each store path in inputDr vs is converted from its on-disk ATerm representation back to a

StoreDrv by the function parseDrv.) In essence, hashDrv partitions store derivations into

equivalence classes, and for hashing purpose it replaces each store path in a derivation

graph with its equivalence class.

The recursion in Figure 5.10 is inefﬁcient: it will call itself once for each path by which

a subderivation can be reached, i.e., O(V

) times for a derivation graph with V derivations

and with out-degree of at most k. In the actual implementation, memoisation is used to

reduce this to O(V + E) complexity for a graph with E edges.

5.5. Building store derivations

After we have translated a Nix expression to a graph of store derivations, we want to

perform the build actions described by them. The command nix-store --realise does this

explicitly, and nix-env and nix-build do it automatically after store derivation instantiation.

The postcondition of a successful build of a derivation is that the output path of the

derivation is valid. This means that a successful derivation is performed only once (at least

until the user runs the garbage collector). If the output is already valid, the build is a trivial

operation. If it is not, the build described by the derivation is executed, and on successful

completion the output path is registered as being valid, preventing future rebuilds.

The build algorithm is also where binary deployment using substitutes is implemented.

If we want to build a derivation with output path p, p is not valid, and we have a substitute

for p, then we can execute the substitute to create p (which the substitute will generally

...as well as the output ﬁeld. This ﬁeld is empty when hashDrv is called for “unﬁnished” derivations from in-

stantiate 64

, but it contains an actual path when hashDrv calls itself recursively for “ﬁnished” derivations 69 .

This ensures that the name attribute and the location of the store are taken into account.

108

5.5. Building store derivations

do by downloading a pre-built binary and unpacking it to p). Substitution is discussed

in Section 5.5.3; here we just assume that there exists a function substitute(p) : Bool that

returns true if path p is valid or it has managed to make p valid by executing a substitute,

and false otherwise.

This section shows how Nix builds store derivations. It ﬁrst presents a “simple” build

algorithm that recursively builds store derivations. Next, it introduces the concept of build

hooks, which are a mechanism to support distributed multi-platform builds in a transpar-

ent and centralised manner. Finally it shows a build algorithm that can perform multiple

derivations in parallel.

5.5.1. The simple build algorithm

The build algorithm is shown in Figure 5.11. The function build takes as its sole argument

the store path p

drv

of the derivation to be built. It returns the output path of the derivation

if the build succeeds, and aborts if it does not. It consists of the following main steps:

70 First, we need to ensure that the derivation p

drv

exists. If it does not, it might be

possible to obtain it through a substitute. It is a fatal error if p

drv

cannot be made

valid. In any case, when p

drv

is valid, it is read into d, a value of the type StoreDrv

shown in Figure 5.5.

While the primary purpose of substitutes is to speed up the construction of deriva-

tion outputs, they may also be used to distribute the store derivations themselves. In

general, what we distribute to client machines are Nix expressions, and the transla-

tion of those will cause the store derivations to be created locally. However, it is also

possible to forego Nix expressions entirely and work directly on store derivations.

For instance, nix-env allows one to install a store derivation directly:

$ nix-channel --update

$ nix-env -i /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

If /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv is not valid, it is possible that a sub-

stitute for it is available (e.g., obtained from a subscribed channel). The primary

advantage of skipping Nix expressions in this manner is that it makes one indepen-

dent of the evolution of the Nix expression language, removing a possible source

of incompatibility between the distributor and the clients. This is in fact one of the

main advantages of the two-step build process.

71 We try to substitute the output path of the derivation. If it is already valid (as stated

above, substitute checks for this trivial case), or it can be produced through a substi-

tute, we’re done right away.

72 Since the output is not valid and there is no substitute, we have to perform the build

action. We need to build all input derivations in inputDrvs, or, more precisely, we

need to ensure that the output paths of the input derivations are valid. So we recur-

sively build the input derivations.

Note that it is not necessary to ensure that the input sources in inputSrcs are valid.

This is because of the closure invariant: the inputSrcs are all in the references

109

5. The Extensional Model

build(p

drv

) :

if ¬ substitute(p

drv

) : abort 70

d ← parseDrv(readPath(p

drv

))

if substitute(d.output) : return d.output 71

inputs ← /0

for each p ∈ d.inputsDrvs : 72

build(p)

← parseDrv(readPath(p))

inputs

∪

← closure(d

.output) 73

for each p ∈ d.inputsSrcs :

inputs

∪

← closure(p) 74

lock ← acquirePathLock(d.output) 75

if valid[d.output] 6= ε :

releasePathLock(d.output, lock)

return d.output

if d.system 6= thisSystem : abort 76

if pathExists(d.output) : deletePath(d.output) 77

tmp

← createTempDir() 78

Run d.builder in an environment d.envVars

and with arguments d.args and in directory p

tmp

if the builder exits with exit code 6= 0 :

deletePath(d.output)

abort

canonicalisePathContents(d.output)

if d.outputHash 6= "" : 81

Extract mode m and hash algorithm t from d.outputHashAlgo

if m indicates ﬂat mode :

Check that d.output is a non-executable regular ﬁle

and that hash

(readFile(d.output)) matches d.outputHash

else :

Check that hash

(serialise(readPath(d.output))) matches d.outputHash

In a database transaction : 82

valid[d.output] ← hash

sha256

(serialise(readPath(d.output)))

setReferences(d.output, scanForReferences(d.output, inputs))

deriver[d.output] ← p

drv

releasePathLock(d.output, lock)

return d.output

Figure 5.11.: build: Building store derivations

110

5.5. Building store derivations

entry for p

drv

(cf. Figure 5.6), and thus validity of p

drv

implies validity of each

p ∈ d.inputSrcs. Likewise, we know for sure that we can load the input derivations,

since these too are references of p

drv

75 We acquire an exclusive lock on the output path. Once we have acquired the lock, we

know that there is no other Nix process building the output path (including through

substitutes, since as we will see, substitute also locks properly).

However, once we have acquired the lock, it is possible that another process got

there ﬁrst, i.e., saw that the output was invalid, acquire the lock, and did the build.

This can be detected by checking once more whether the output path is valid. If it

is, we’re done.

Thanks to locking and the transactional semantics of registering valid paths (step

82 described below), an interrupted operation (such as building a derivation) can

always be restarted, requiring no further recovery. It also ensures that all operations

are idempotent, i.e., can be repeated any number of times. Operations can also be

run in parallel. In particular, it is safe to run multiple build actions in parallel. Since

locking is ﬁne-grained, as opposed to having a global lock for the entire Nix store,

parallel execution can be used to speed up builds. If more than one CPU is available,

this gives better resource utilisation. However, even if there is only one CPU, we may

get a performance improvement if some of the builds are heavily I/O-bound [6].

76 It is only possible to build the derivation if the current machine and operating system

match the system ﬁeld of the derivation. A Nix installation on an i686-linux machine

cannot build a derivation for a powerpc-darwin machine. The platform type of the

running Nix instance is denoted by thisSystem. However, Section 5.5.2 describes an

extension to the build algorithm that enables distributed multi-platform builds.

77 It is possible that the output path already exists, even though it is not valid. Typically,

this is because a previous build was interrupted before the builder ﬁnished or before

the output could be registered as valid. Such a residual output can interfere with the

build, and therefore must be deleted ﬁrst.

78 Builds are executed in a temporary directory p

tmp

that is created here (typically, it is

a fresh subdirectory of /tmp). p

tmp

is deleted automatically after the builder ﬁnishes

(not shown here). The temporary directory provides a place where the builder can

write scratch data. For instance, the unpacking and compiling of the Hello sources

performed by the Hello builder in Figure 2.7 is done in the temporary directory.

The fact that the temporary directory is initially empty reduces the possibility of

interference due to ﬁles pre-existing in the builder’s initial current directory.

79 Here we actually perform the build by running the executable indicated by the builder

ﬁeld of the derivation, with the command-line arguments args and the environment

variables envVars, and with the current directory set to p

tmp

. Note that envVars

speciﬁes the complete environment; the builder does not inherit any environment

variables from the caller of the Nix process user. This prevents interference from

environment variables such as PATH.

111

5. The Extensional Model

In the real implementation some additional variables are initialised to speciﬁc val-

ues. The variable TMPDIR is set to p

tmp

so that tools that need temporary space write

to that directory, reducing the possibility of interference, left-over ﬁles, and security

problems due to /tmp races. The variable HOME, which points to the user’s home

directory, is set to the meaningless value /homeless-shelter. Many programs look

up the home directory corresponding to the current user ID in /etc/passwd or similar

if HOME is not set. Setting HOME prevents such lookups and reduces the possibil-

ity of interference from tools that read conﬁguration ﬁles from the home directory.

Likewise, PATH is set to the value /path-not-set, since the Unix shell initialises PATH

to an undesirable default such as /usr/local/bin:/bin:/usr/bin, which typically causes

interference. Finally, the variable NIX_STORE is set to storeDir. Some tools need

the location of the store for various purposes (see, e.g., Section 7.1.3).

80 If the build ﬁnishes successfully, the output is canonicalised by the function canon-

icalisePathContents. It sets all ﬁle system meta-information not represented in the

canonical serialisation to ﬁxed values. In particular, it does the following:

– The “last modiﬁed” timestamps on ﬁles are set to 0 seconds in the Unix epoch,

meaning 1/1/1970 00:00:00 UTC.

– The permission on each ﬁle is set to octal 0444 or 0555, meaning that the ﬁle

is readable by everyone and writable by nobody, but possibly has execute per-

mission. Removing write permission prevents the path from being accidentally

modiﬁed when it is used as an input to other derivations.

– Special permissions, such as set-user-ID and set-group-ID [152], are removed.

– A fatal error is ﬂagged if a ﬁle is found that is not a regular ﬁle, a directory, or

a symlink.

The actual implementation of canonicalisePathContents(p) is not shown here, but

in its effect is equal to deserialising a canonical serialisation of the path:

writePath(p, readPath(p))

where writePath takes care to create ﬁles with canonical values for meta-information

not represented in the canonical serialisation.

81 If d is a ﬁxed-output derivation (Section 5.4.1), then we have to check that the output

produced by the builder matches the cryptographic hash declared in the derivation.

Recall that there are two modes of ﬁxed-output derivation: ﬂat mode that requires the

output to be single non-executable ﬁle, and recursive mode that applies to arbitrary

outputs. Flat mode hashes the plain ﬁle contents (read by the function readFile),

while recursive mode hashes the serialisation of the entire FSO.

82 Since the build has ﬁnished successfully, we must register the path as valid and set its

references mapping. This must be done in a database transaction to maintain the in-

variants. Thanks to the ACID properties of the underlying Berkeley DB database, it

is always safe to interrupt any Nix operation, and Nix can always recover gracefully

112

5.5. Building store derivations

scanForReferences(p, paths) :

c ← serialise(readPath(p))

refs ← /0

for each p

∈ paths :

if ﬁnd(c, hashPart(p

)) 6= /0 :

refs

∪

← {p

}

return refs

Figure 5.12.: scanForReferences: Scanning for store path references in build results

from non-corrupting crashes, including power failures. In the case of operating sys-

tem or hardware crashes, this requires the ﬁle system containing the Nix store to be

data-journaled [142], or to otherwise guarantee that once data has been committed

to disk, no rollback to previous contents will occur.

Following the stored hash invariant (page 96), the valid entry for the output path is

set to the SHA-256 of the serialisation of the path contents.

The references entry is set to the set of store paths referenced from the output con-

tents. As described in Chapter 3, the references are found by scanning for the hash

parts of the input paths. The set of input paths is the union of the closures of the

input sources 74 and the outputs of the input derivations 73 . Note that it is quite

possible for the output to contain a reference to a path that is not in inputSrcs or

inputDrvs, but that is in the closure of those paths. Such a reference can be obtained

by the builder indirectly by dereferencing one of its immediate inputs. Given the

full set of input paths, the function scanForReferences returns the set of referenced

paths.

The deriver mapping maintains a link from outputs to the derivations that built them.

It is described below.

Scanning for references The function scanForReferences, shown in Figure 5.12, de-

termines the references in a path p to other store paths.

We only need to scan for store paths that are inputs, i.e., that are in the closure of in-

putSrcs and the outputs of inputDrvs. It is not necessary to scan for references to arbitrary

store paths, since the only way that a builder can introduce a reference to a path not in the

input set, is if it were to read the Nix store to obtain arbitrary store paths. While it is easy

to construct a contrived builder that does this, builders do not do so in practice. Indeed,

if this were an issue, we could block such behaviour by making the Nix store unreadable

but traversable (i.e., by clearing read permission but enabling execute permission on the

directory on Unix systems).

Thus scanForReferences also takes a set of potentially referenced paths. It then searches

for the hash part of each path in the serialisation of the contents of the FSO at p. To re-

iterate from Section 3.4, we only care about the hash part because that’s the most easily

identiﬁable part of a store path, and least likely to be hidden in some way. The search for

the hash part is done using an ordinary string searching algorithm, denoted as ﬁnd(s,t),

113

5. The Extensional Model

which yields the integer offsets of the occurrences of the byte sequence t in the byte se-

quence s (an empty set indicating no matches). If and only if there is a match, we conclude

that path p has a reference to the store path p

For the string search, the Boyer-Moore algorithm [18] is particularly suited because the

hash parts consist of a small subset of the set of byte values, allowing the algorithm to

jump through the input quickly in many cases. For instance, if we encounter any byte not

in the set of base-32 characters (see digits

in Section 5.1), we can immediately skip 32

characters. Especially for binary ﬁles this will happen quite often. Thus it is reasonable to

expect that the search achieves Boyer-Moore’s Θ(n/m) average running time in most cases

(where n is the length of the serialisation in bytes, and m is the length of the hash part).

If it is desirable to support encodings of the hash part other than plain ASCII (and to

date, it has not been necessary to do), scanForReferences can be adapted appropriately.

For instance, to detect UTF-16 encodings of the hash part, we should just scan for the byte

sequence s[0] 0 s[1] 0 s[2] 0 . .. s[31] 0, where s = hashPart(p

), p

∈ paths, and 0 denotes

a byte with value 0

Traceability For many applications it is important to query which derivation built a given

store path (if any). Section 11.1 gives the example of blacklisting, which allows users to

determine whether the build graph of a component contains certain “bad” sources. In

Software Conﬁguration Management, this is known as traceability—the ability to trace

derived artifacts back to the source artifacts from which they were constructed.

To enable traceability, Nix maintains a database table deriver : Path → Path that maps

output paths built by derivations to those derivations. The following invariant maintains

traceability.

Invariant 5 (Traceability invariant). The deriver of a valid path p, if deﬁned, must be a

valid store derivation with output path p.

∀p ∈ Path : (valid[p] 6= ε ∧ deriver[p] 6= ε) →

(valid[deriver[p]] 6= ε ∧ parseDrv(readPath(deriver[p])).output = p)

However, this invariant is actually optional in the current implementation, because not

all users want traceability. The disadvantage of full traceability is that the entire derivation

graph of each output path has to be kept in the store for the lifetime of those paths.

It is worth noting at this point that Nix in no way maintains a link between the build

graph (store derivations and sources) in the Nix store on the one hand, and the Nix expres-

sions from which they were instantiated on the other hand. If the latter are under version

management control, there is no way to trace store paths back to the artifacts in the version

management system from which they ultimately originated. This is most certainly a desir-

able feature, but it is not clear how such information can be maintained without coupling

Nix too closely to the version management system (as is the case in Vesta and ClearCase,

which integrate version management and build management, as discussed in Section 7.6).

This assumes a little-endian encoding of UTF-16 (i.e., UTF-16LE) [34]. For the big-endian variant (UTF-

16BE), the 0s should preﬁx the hash characters.

114

5.5. Building store derivations

let {

pkgsLinux = ...;

pkgsDarwin = ...;

fooC = derivation {

name = "fooC";

system = "i686-linux";

src = ./foo.tar.gz;

builder = ...;

inherit (pkgsLinux) stdenv ... tiger;

};

fooBin = derivation {

name = "foo";

system = "powerpc-darwin";

src = fooC;

builder = ...;

inherit (pkgsDarwin) stdenv ...;

};

body = fooBin;

}

Figure 5.13.: Example of a distributed, two-platform build

5.5.2. Distributed builds

The build algorithm in Figure 5.11 builds derivations only for the current platform (thisSys-

tem). Through a relatively simple extension called build hooks, Nix supports distributed

multi-platform builds.

Figure 5.13 shows a hypothetical example of a Nix expression that requires a distributed

multi-platform capable build system. The scenario is that we want to compile a package

foo, written in a hypothetical language called Tiger, for Mac OS X (powerpc-darwin). Un-

fortunately, the Tiger compiler does not run on that platform, but it does run on Linux

(i686-linux) and can compile to C code. Thus, we have two derivations. The ﬁrst derivation

compiles the foo package to C code on i686-linux, and produces an output consisting of C

source ﬁles. The second derivation compiles the C sources produced by the ﬁrst derivation

to native PowerPC code on powerpc-darwin. With the build hook mechanism, we can just

build the second derivation on any machine normally, e.g.,

$ nix-env -f foo.nix -i foo

and any derivation that cannot be executed on the local machine is forwarded to a machine

of the appropriate type. In this way, build hooks allow us to abstract over multi-platform

builds. It is not necessary for the user to interact explicitly with the various machines

involved in the build. In Section 9.4 we will see a powerful application of this: building

distributed multi-platform services.

Build hooks are useful not just for multi-platform builds, but also to accelerate large

single-platform builds. For instance, Nixpkgs contains hundreds of components that need

115

5. The Extensional Model

to be built for i686-linux, and many of those derivations can be scheduled in parallel since

they do not depend on each other. The build hook mechanism allows as many derivations

as possible to run in parallel on the available hardware of that type.

A build hook is a user-deﬁned program (typically written in a scripting language like

Perl) that is called by Nix to perform the build on behalf of Nix. The build hook mechanism

is policy-free, that is, it can do anything it wants, but a build hook typically does the

following things:

• It determines whether it knows a remote machine of the appropriate platform type

that can handle the derivation. If not, it refuses the job, and the calling Nix pro-

cess executes the job normally, which is to say that it runs the builder if thisSystem

matches d.system, and aborts otherwise.

• If an appropriate machine is known, the hook can either accept the job or tell the

calling Nix process to postpone the job. The latter is appropriate if the load on the

selected machine is too high: it is typical to allow no more jobs to run concurrently

on a machine than the number of CPUs. If the hook returns a “postpone” reply,

the calling Nix process will wait for a while, or continue with some other derivation

(this is not supported by the algorithm in Figure 5.11, but is supported by the parallel

algorithm in Section 5.5.4).

• If the hook has accepted the job, the calling Nix process will create a temporary

directory containing all information that the hook needs to perform the build, such

as the store paths of the inputs and output.

• The hook copies the FSOs denoted by the input paths (the inputs set computed in

Figure 5.11) to the Nix store on the target machine, and registers them as valid with

the appropriate reference graph mappings. It is important that the Nix stores on both

machines are in the same location in the ﬁle system. The derivation itself (at p

drv

) is

also copied.

• The hook builds the derivation on the remote machine, e.g., by using the Secure

Shell (ssh) protocol [61] to run nix-store --realise remotely.

• If the remote build succeeds, the output at d.output in the Nix store of the remote

machine is copied back to the local Nix store.

When the hook ﬁnishes successfully, Nix proceeds normally as if it had executed the

builder itself, i.e., by scanning for references in the output and registering path validity.

As stated above, this is a policy-free mechanism: it doesn’t care how the build hook

does its work. In general, the build hook will have some conﬁguration ﬁle listing remote

machines, authentication information to automatically log in to those machine, and so on.

An example of a concrete build hook is given in the context of build farms in Section 8.3.

Hook protocol A build hook is enabled by the user by pointing the environment variable

NIX_BUILD_HOOK at the hook. When Nix needs to perform a build, it ﬁrst executes

the hook to discover whether it is able and willing to perform the build. Thus the build

algorithm in Figure 5.11 is augmented as shown in Figure 5.14. Nix calls the hook 84 with

all information that the hook needs to determine whether it can handle the job, namely:

116

5.5. Building store derivations

build(p

drv

) :

(...elided...)

if the environment variable NIX_BUILD_HOOK is deﬁned :

tmp

← createTempDir()

Write inputs ∪ closure(p

drv

) to ﬁle p

tmp

/inputs

Write d.output to ﬁle p

tmp

/outputs

Write the references of inputs to ﬁle p

tmp

/references

(fd

read

,fd

write

) ← pipe()

Asynchronously start NIX_BUILD_HOOK in directory p

tmp

with fd

write

attached to ﬁle descriptor 3

and arguments [0,thisSystem, d.system, p

drv

]

reply ← read a line from fd

read

if reply = "accept" then :

Wait until the hook ﬁnishes

if the hook exited with exit code 6= 0 : abort

Scan and register the path as valid, as in Figure 5.11

return

else if reply = "postpone" then :

Sleep a bit, then retry; see Section 5.5.4 for a better implementation.

else if reply 6= "decline" then : abort

Fall through to normal handling...

if d.system 6= thisSystem : abort

(...elided...)

Figure 5.14.: Augmentation of build in Figure 5.11 to add support for build hooks

• A Boolean value signifying whether the calling Nix process can execute more build

jobs right now. This is applicable to the parallel build algorithm shown in Sec-

tion 5.5.4 below, where Nix can execute a user-deﬁned number of jobs in parallel.

• The value of thisSystem.

• The value of d.system. The hook needs this to determine whether it knows a remote

machine of the appropriate type.

• The store path of the derivation (p

drv

In addition, Nix creates a temporary directory containing the following ﬁles:

• inputs contains the store paths of all inputs, along with the closure of the store deriva-

tion itself.

• outputs contains the store path of the output. It is outputs, plural, to support multiple

output paths as described in Section 6.7.

• references contains a textual representation of the references mapping of the inputs.

After copying the inputs to the target Nix store, the command nix-store --register-

117

5. The Extensional Model

validity is used to register the validity of the copied paths on the target machine. This

command accepts a description of the references graph in exactly this format.

A Unix pipe is created to communicate with the hook 83 . Its write side is attached to

ﬁle descriptor 3 in the hook child process. The hook should print one the following strings

on ﬁle descriptor 3 (followed by a linefeed character):

• postpone indicates that the caller should wait, typically until another job has termi-

nated, hopefully decreasing the load on the intended target machine(s).

• decline means that the caller should try to build the derivation itself.

• accept means that the hook has accepted the build job.

If the hook accepts the job, it is responsible for producing output in d.output. If it

ﬁnishes successfully, the output path is registered as valid, with the references determined

by scanForReferences, just as in Figure 5.11.

5.5.3. Substitutes

Substitutes are the mechanism by which Nix enables its transparent source/binary deploy-

ment paradigm, as we saw in Section 2.6. A substitute is an invocation of some arbitrary

program that creates a store object through some other means than normal building. It is

primarily used to obtain the output of a derivation by downloading a pre-built FSO from a

server rather than building the derivation from source.

The database mapping substitutes : Path → {(Path,[String],Path)} stores the substitutes

that have been registered by user commands. The left-hand side is the store path to which

the substitutes apply. The right-hand side is a set of substitutes. Each substitute is a

three-tuple (substituter, args,deriver), where substituter is the path of the program to be

executed to perform the substitution, args are the command-line arguments to be passed

to that program, and deriver is the store path of the derivation that produced the FSO on

the originating machine. The latter is necessary to maintain the deriver link for traceability

(on installations where this is enabled). An example entry in the substitutes mapping is:

substitutes["/nix/store/bwacc7a5c5n3...-hello-2.1.1"] = {

( "download-url.sh"

, ["http://example.org/1pc7y48cs3qn....nar.bz2"]

, "/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv"

)

}

which establishes a single substitute: if we want to make path /nix/store/bwacc7a5c5n3...-

hello-2.1.1 valid, we can do so by executing the program download-url.sh with argument

http://example.org/1pc7y48cs3qn....nar.bz2. Here download-url.sh is presumably a pro-

gram that downloads a ﬁle from the given URL, and uncompresses and unpacks it to store

path /nix/store/bwacc7a5c5n3...-hello-2.1.1.

This is another example of a policy-free mechanism: Nix does not care how the sub-

stitute produces the store object. It can download from a server, interactively prompt the

118

5.5. Building store derivations

user to insert an installation CD, and so on. Indeed, there is no guarantee that a substitute

produces correct output, which is a problem for secure deployment. The intensional model

discussed in Chapter 6 addresses this issue. Section 7.3 discusses the implementation of a

concrete binary deployment scheme based on the substitute mechanism.

The FSOs that we obtain through substitutes typically have references, of course. For

instance, the Hello binary in /nix/store/bwacc7a5c5n3...-hello-2.1.1 has a reference to

/nix/store/72by2iw5wd8i...-glibc-2.3.5. To maintain the closure invariant, we must ensure

that the latter path is valid before we download the former. Thus, the references mapping

for a path must be known beforehand. This means that when a substitute is registered,

the references for the path in question must also be registered. Formally, this gives us the

following invariant.

Invariant 6 (Reference invariant). A path that is valid or substitutable is called a usable

path. For all usable paths, the set of references must be known:

∀p ∈ Path : (valid[p] 6= ε ∨ substitutes[p] 6= ε) → references[p] 6= ε

Note that the references entry for a valid or substitutable path can be the empty set ( /0),

it just cannot be undeﬁned (ε).

The command nix-store --register-substitutes registers a set of substitutes, along with the

references of the paths to which the substitutes apply. This command is used by concrete

deployment policies such as nix-pull (Section 2.6). For example:

$ nix-store --register-substitutes

/nix/store/bwacc7a5c5n3...-hello-2.1.1

/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

download-url.sh

http://example.org/1pc7y48cs3qn....nar.bz2

/nix/store/72by2iw5wd8i...-glibc-2.3.5

/nix/store/bwacc7a5c5n3...-hello-2.1.1

/nix/store/q2pw1vn87327...-gcc-3.4.4

Here a substitute is registered for the path /nix/store/bwacc7a5c5n3...-hello-2.1.1 that takes

one argument and has three references. The substitute speciﬁcation on standard input lists

the store path being substituted, its deriver (if any), the substituter program, the command-

line arguments, and the references. The latter two are listed one per line, each list preceded

by a line specifying its length.

The function substitute(p) that tries to make a usable path p valid through substitutes is

shown in Figure 5.15. It returns a Boolean indicating whether it succeeded in making p

valid. If the path is already valid, we are done right away 85 . Likewise, if the path is not

valid but there are no substitutes, we are also done, in a negative sense 86 .

Otherwise, the path is not valid but we do have substitutes, so we can hopefully build the

path using one of them. However, to maintain the closure invariant, we must ﬁrst ensure

that all the paths that will be referenced by p are also valid. By the reference invariant, we

already know the references of p. Thus, we call substitute recursively on all references 87 .

To prevent interference with potential other Nix processes while running the substitutes,

a lock is ﬁrst acquired on the output path 88 . As in the build algorithm in Figure 5.11, it

119

5. The Extensional Model

Bool substitute(p) :

if valid[p] 6= ε : return true 85

if substitutes[p] = ε : return false 86

for each p

∈ references[p] :

if ¬ substitute(p

) : return false

lock ← acquirePathLock(p) 88

if valid[p] 6= ε : return true

for each (program,args, deriver) ∈ substitutes[p] : 89

if pathExists(p) : deletePath(p)

if execution of program program with

arguments [p] + args exits with exit code 0 : 90

assert(pathExists(p))

canonicalisePathContents(d.output) 91

valid[p] ← hash

sha256

(serialise(readPath(p))) 92

deriver[p] ← deriver

releasePathLock(p, lock)

return true

releasePathLock(p, lock)

if fall-back is disabled : abort

return false

Figure 5.15.: substitute: Substitution of store paths

is possible that the output path has become valid between the initial validity check and the

acquisition of the lock. Thus, a ﬁnal validity check is done.

Next, we try each substitute until one succeeds 89 . The substitute program is called

with as command-line arguments the target path p, and the arguments registered for the

substitute

90 . The substitute program is supposed to produce an FSO at path p and return

exit code 0. If so, the path’s metadata is canonicalised 91 , the path is registered as valid,

and the deriver is set 92 . If not, the next substitute is tried until no more substitutes remain.

Generally, it is a fatal error if all substitutes for a path fail. When substitute is called

from a derivation build action, Nix does not automatically “fall back” to building from

source 93 . The reason is that we presumably registered the substitute for a reason, namely,

to avoid the overhead of a source build. If the substitute fails for a transient reason—such

as a network outage—the user in general does not want a source build instead, but prefers

to wait or repair the cause of the problem. However, Nix commands do accept a ﬂag to

enable automatic fall-back:

$ nix-env -i hello --fall-back

...

downloading `/nix/store/bwacc7a5c5n3...-hello-2.1.1'

from http://example.org/foo

curl: `example.org': no route to host

falling back...

building `/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv'

120

5.5. Building store derivations

...

Relation to build hooks So what is the difference between a substitute and a build hook,

as they both short-circuit the build process? A build hook is used to “out-source” the

building of a derivation, while a substitute is used to reproduce an already existing build

result. That is, build hooks operate at build time, while substitutes operate at deployment

time. Speciﬁcally, the substitution of the output of a derivation only needs the references

of the output (i.e., runtime dependencies) to be valid, while a build hook requires that all

inputs of the derivation (i.e., build-time dependencies) are valid.

For example, the GNU Hello derivation has build-time dependencies on GCC and sev-

eral other packages, but these are not referenced from the output. Thus, if we use a sub-

stitute to download a Hello binary, dependencies such as GCC will not be installed. But if

we used build hooks, Nix would ﬁrst install them. So while the substitutes can in principle

be subsumed by build hooks, substitutes are much more efﬁcient for deployment purposes

(apart from the fact that making the build-time dependencies available may simply be un-

acceptable in many cases, such as closed-source deployment).

Also, substitutes allow us to forego store derivations altogether. For instance, nix-env

can install output paths directly:

$ nix-env -i /nix/store/bwacc7a5c5n3...-hello-2.1.1

This command will make /nix/store/bwacc7a5c5n3...-hello-2.1.1 valid using a substitute (if

it wasn’t already valid). The difference with installing from a store derivation is that this

mode of installation cannot fall back to a build from source if the substitutes fail.

5.5.4. The parallel build algorithm

The build algorithm described above does not always make optimal use of the available

hardware. When multiple CPUs are available, it is a good idea to execute multiple deriva-

tions in parallel. When substitutes are used, it may well be the case that each substitute

only uses a fraction of the available download bandwidth, i.e., executing multiple down-

loads in parallel might speed things up. More importantly, when build hooks are used to

distribute build jobs to remote machines, we certainly shouldn’t wait for one job to ﬁnish

when there are idle machines.

This section brieﬂy describes a parallel build algorithm that can execute multiple builds

and substitutions simultaneously. The parallel algorithm, rather than the one described

above, is what is implemented in the current Nix implementation.

The parallel build algorithm works on sets of goals. There are two types of goals:

• A derivation goal attempts to make the output of a derivation valid.

• A substitution goal attempts to make a path valid through substitutes.

Each type of goal is implemented by a ﬁnite state machine. The nodes of the ﬁnite state

machine represent states. Each state has associated with it an action that is to be performed

when the state is reached. Typically, the action is to start other goals or processes. The goal

is then “put to sleep” until the condition associated with one of the out-going edges of the

121

5. The Extensional Model

run(topGoals) :

awake ← topGoals

while true :

for each goal ∈ awake :

Remove goal from awake

Execute the action associated with the current state of goal

if goal has reached a ﬁnal state :

Add goals waiting on goal to awake

Remove goal from topGoals

if topGoals = /0 : return

Wait for termination of child processes, add corresponding goals to awake

Figure 5.16.: run: Goal execution (sketch)

state becomes true. At this point, the goal is woken up, the transition to the corresponding

new state is made, and the action associated with the new state is performed. This continues

until the goal reaches a ﬁnal state.

There is a top-level function run that executes a set of initial top-level goals (all being

in their initial state). It is shown in Figure 5.16. It maintains a set of “awake” goals,

e.g., those goals that have made a transition (meaning that some event has occured that is

relevant to the goal). Initially, the top-level goals are all awake, so their initial actions can

be executed. If a top-level goal ﬁnishes, it is removed from the set of top-level goals, and

the runner continues until there are no more top-level goals.

The “events” that can wake up a goal are the termination of a child process (i.e., a

builder, a substituter, or a build hook), or the termination of a subgoal. A subgoal of a goal

g is a goal that g is waiting on. Subgoals can be shared. Suppose that we start the parallel

building of top-level derivations Hello and Firefox, which both depend on the same Glibc.

One top-level derivation will create a subgoal for Glibc and its other input derivations. It

then goes to sleep, waiting for the subgoals to ﬁnish. When the Firefox goal is executed, it

will see that a subgoal for Glibc already exists, and simply add itself to the list of “waiters”.

When the Glibc goal ﬁnishes, it will wake up both the Hello and Firefox goals, allowing

them to make progress (as soon as their other subgoals have ﬁnished too, that is).

Figure 5.17 shows the state machine for derivation goals. When a goal starts, it

creates a substitution subgoal for its derivation store path (which corresponds to the

call substitute(p

drv

) in the simple build algorithm in Figure 5.11). If the subgoal suc-

ceeds, it starts a substitution subgoal for the output path (corresponding to the call

substitute(d.output)). If that subgoal does not succeed, we have to perform the build. So

ﬁrst the input derivations must be built, and thus derivation subgoals are created for each

of those (corresponding to the recursive calls to build). If each is successful, we can run the

build hook, or if that yields a decline response, the builder. We limit however the number

of child processes that can run in parallel

. If this limit is reached, no more child processes

can be started until one ﬁnishes (the available “build slots” are said to be exhausted), so the

goal goes to sleep. When the hook or builder ﬁnishes successfully, the usual postprocess-

This limit can be speciﬁed using the --max-jobs N ﬂag.

122

5.5. Building store derivations

INIT

start substitution goal for store derivation

FAILED

substitution goal failed

HAVE STORE DERIVATION

start substitution goal for output path

substitution goal succeeded

SUCCESS

substitution goal succeeded

NO OUTPUT

start derivation goals for all input derivations

substitution goal failed

one or more subgoals failed

TRY TO BUILD

run the build hook / builder

all subgoals succeeded

hook or builder failed

hook declined/postponed, or no build slot;

wait until a child finishes

FINISHED

check output, scan, register

hook or builder succeeded

no output success

Figure 5.17.: State machine for derivation goals

INIT

SUCCESS

path is already valid

INVALID

start substitution goals for all references

path is not yet valid

FAILED

one or more subgoals failed

TRY NEXT

pop next substitute

all subgoals succeeded

no more substitutes

TRY TO BUILD

run the substitute program

have a substitute

substitute succeeded

substitute failed

no build slot;

wait until a child finishes

Figure 5.18.: State machine for substitution goals

123

5. The Extensional Model

ing is performed: the references are determined, the content hash of the path is computed,

and the path is registered as valid. (The transitions from state FINISHED are not actual

events, but simply the outcomes of the action associated with that state’s action.)

Figure 5.18 shows the state machine for substitution goals. A substitution goal ﬁrst

checks that the path is not already valid (again, the edges from INIT are not actual events).

If it is invalid, subgoals are started to make the references of the path valid. Then, each

substitute is tried until one succeeds.

Normally, when a subgoal fails, its waitees (the goals waiting on it) are immediately

woken up and fail as well (except when failure is normal, e.g., in the substitution of an

output path). These waitees may have other subgoals that are still busy. Each goal has a

reference count, and when the waitees kill themselves, they decrement the reference counts

of all remaining subgoals. If the reference count of a goal reaches 0, it is also killed.

Associated child processes are terminated. Thus, failure of a single subgoal propagates

upwards, killing all other goals along the way.

This is usually the desired behaviour, but not always. For instance, in the Nix build farm

described in Chapter 8, the job for Nixpkgs contains hundreds of more-or-less independent

derivations. The failure of one should not terminate the others, since the build results of

the successful derivations can be reused in subsequent build jobs. For this reason, there

is a ﬂag --keep-going that has the following effect: when a subgoal fails, the waitees are

not woken up until all their other subgoals have ﬁnished as well. At that point, the waitees

notice the subgoal failure, and make the transition to the FAILED state as well

5.6. Garbage collection

Nix never deletes components from the Nix store on uninstallation or upgrade actions.

Rather, to reclaim disk space, it is necessary to routinely run a garbage collector.

Garbage collection in Nix is pretty much like garbage collection in programming lan-

guages (except for one unique property discussed below). There is a pointer graph deﬁned

by the references relation, and there is a set of roots. The closure of the set of roots under

the references relation is the set of live store objects—these are FSOs that are potentially

in use, and therefore must not be deleted. All store objects that are not live are dead—these

can be deleted safely.

The garbage collector therefore has the following phases:

• Determine the set of root paths.

• Compute the live paths as the closure of the set of root paths.

• Delete the dead paths, which are those that exist in the Nix store but are not in the

set of live paths.

This section ﬁrst discusses how root paths are registered. Then it shows two garbage

collection algorithms: a plain “stop-the-world” collector that blocks all other Nix actions

while it is running, and a concurrent collector that can run in parallel with other Nix actions.

This behaviour is similar to GNU Make’s [68] -k ﬂag.

124

5.6. Garbage collection

5.6.1. Garbage collection roots

The set of roots of the garbage collector are deﬁned as symlinks to store paths in the

directory /nix/var/nix/gcroots. Conceptually, if we for example have a store path /nix/store/-

bwacc7a5c5n3...-hello-2.1.1 that we want to be preserved, we can register it as a root as

follows:

$ nix-store -r $(nix-instantiate foo.nix)

/nix/store/bwacc7a5c5n3...-hello-2.1.1

$ ln -s /nix/store/bwacc7a5c5n3...-hello-2.1.1 \

/nix/var/nix/gcroots/my-root

It is also possible to register store derivations as roots, e.g.,

$ nix-instantiate foo.nix

/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

$ ln -s /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv \

/nix/var/nix/gcroots/my-root

This causes the store derivations and its dependencies—other store derivations and

sources—to be preserved by the collector.

However, there is a race condition in this method of root registration. If the garbage

collector is ran just between the call to nix-store or nix-instantiate on the one hand, and the

creation of the symlink on the other hand, the store object and some or all of its dependen-

cies may be garbage collected. It is therefore possible to let the Nix tool in question create

the root:

$ nix-instantiate --add-root /nix/var/nix/gcroots/my-root foo.nix

/nix/var/nix/gcroots/my-root

Through locking, Nix ensures that there is no race condition.

Having to store links in /nix/var/nix/gcroots is often inconvenient. Consider the Nix

utility nix-build that instantiates and builds a Nix expression, and leaves a symlink result

pointing to the output path of the derivation in the current directory:

$ nix-build ./foo.nix

$ ls -l result

lrwxr-xr-x ... result -> /nix/store/bwacc7a5c5n3...-hello-2.1.1

$ ./result/bin/hello

Hello, world!

We want to implement nix-build so that its result is registered as a root. One way to do this

is to have nix-build generate a fresh symlink in /nix/var/nix/gcroots, and have ./result be a

symlink to that. I.e., we solve the problem through an extra indirection. However, there is

a problem: if we delete ./result, the root persists. So the user would always need to make

sure to delete the target of the result symlink when deleting result.

125

5. The Extensional Model

A better solution is the notion of indirect roots, which is in a sense the opposite solution.

We let ./result be a direct symlink to the store path, and create a symlink to result in the

designated directory /nix/var/nix/gcroots/auto. For example,

$ nix-store -r --add-root ./my-root --indirect \

$(nix-instantiate foo.nix)

./my-root

$ ls -l my-root

lrwxr-xr-x ... result -> /nix/store/bwacc7a5c5n3...-hello-2.1.1

$ ls -l /nix/var/nix/gcroots/auto

lrwxr-xr-x ... 096hqdxcf6i2... -> /home/eelco/test/my-root

$ ./my-root/bin/hello

Hello, world!

The name of the symlink in auto is a cryptographic hash of the root location (e.g., /home/-

eelco/test/my-root). This ensures that subsequent root registrations with the same location

will overwrite each other (which is good, since it minimises the number of redundant

symlinks in auto).

When the garbage collector runs, it chases the symlinks deﬁned in /nix/var/nix/gcroots/-

auto. If it encounters a dangling symlink (i.e., pointing to a non-existent path), the root

pointer is automatically deleted. Thus, when result is removed, the root in the auto direc-

tory will be removed as well on the next run of the garbage collector.

Figure 5.19 shows the algorithm for ﬁnding the garbage collector roots. It returns a set

of store paths. The closure of this set under the references relation is the set of live paths.

By default there is a symlink in /nix/var/nix/gcroots to the directory /nix/var/nix/proﬁles/

that contains the user environment generation links. Thus all user environment generations

are roots of the garbage collector.

Discussion Why don’t we scan the entire ﬁle system outside of the Nix store for roots?

This is certainly possible, but I have opted for a more disciplined root registration protocol

instead for performance reasons—scanning the entire ﬁle system might take too long. Just

scanning for symlink targets is not overly expensive, since it is easy to recognise symlinks

that point to the Nix store. However, to be fully general we should also scan the contents

of regular ﬁles, just as we do with the output of a build. Since we do not know anything

about the internal structure of those ﬁles, we have to apply scanForReferences to the entire

ﬁle system with paths set to all current store paths. Since there can be many store paths—

millions in large installations—this becomes prohibitively expensive.

A possible extension to root discovery is to automatically use open store ﬁles as roots.

In the scheme described above, it is possible that a currently executing program is garbage

collected. For instance, the command sequence

$ nix-env -i firefox

$ firefox & # start Firefox in the background

$ nix-env -e firefox

$ nix-env --remove-generations old

$ nix-store --gc

126

5.6. Garbage collection

ﬁndRoots() :

return ﬁndRoots

("/nix/var/nix/gcroots", false)

ﬁndRoots

(p, recurseSymlinks) :

if p is a directory :

roots ← /0

for each directory entry n in p :

roots

∪

← ﬁndRoots

(p + "/" + n, recurseSymlinks)

return roots

else if p is a symlink :

target ← targetOf(p)

if target is a store path :

return {p}

else : // target is not a store path

if recurseSymlinks :

if pathExists(target) :

return ﬁndRoots

(target, false)

else :

deletePath(p)

Figure 5.19.: ﬁndRoots: Root discovery

will cause Firefox to be (probably) garbage collected, even though it is still running.

There is no portable method to discover the set of open ﬁles, but on most operating

systems there are methods to do so. For instance, on Linux the open ﬁles are the targets of

the symlinks /proc/*/fd/*

A more extreme approach is to scan for roots in memory. On Linux, the device /dev/mem

provides access to the system’s physical memory. This has the same efﬁciency problem

as scanning in regular ﬁles: since we do not know anything about the internal structure

of the memory contents, we have to scan for hash parts. Furthermore, since /dev/mem

maps the physical address space, hashes may be fragmented in memory if they cross page

boundaries; i.e., if character n of a hash is stored at address m, then character n+1 need not

necessarily be at address m + 1. The scanner may fail to detect references if this happens.

So memory scanning only works well for memory devices that map virtual address spaces,

e.g., /proc/*/mem.

5.6.2. Live paths

The live store paths are those that are reachable from the roots through the references

relation. However, it is useful to extend the notion of reachability a bit. For instance,

for some users it is desirable to keep the build-time dependencies of a component: if a

user (e.g., a developer) often builds components from source, it is nice if a build-time

On Linux, the /proc directory contains subdirectories for each live process containing information about them,

e.g., /proc/7231, each having a subdirectory fd that holds symlinks to the open ﬁles of the process.

127

5. The Extensional Model

dependency like the compiler is retained by the garbage collector.

As another example, consider the traceability afforded by the deriver mapping, which

tells us which derivation built a path. Some techniques, such as the blacklisting approach

described in Section 11.1 to enforce upgrades of insecure or otherwise “bad” components,

rely (in part) on knowledge of the build-time dependency graph.

For this reason, the Nix garbage collector has two options that can be set by the admin-

istrator of a Nix installation. They both extend the notion of liveness. The options are

deﬁned in the /nix/etc/nix/nix.conf.

• gc-keep-outputs: If true, the garbage collector will keep the outputs of non-

garbage derivations. That is, if derivation p is live, then its output path

parseDrv(readPath(p

drv

)).output is also live (if it is valid). If false (which is the

default), derivation outputs are not followed, so they will be deleted unless they are

roots themselves (or reachable from other roots).

Note that this is stronger than registering the output of a derivation as a separate root,

since it is transitive. That is, it keeps not just the output of the top-level derivation,

but also the outputs of all its subderivations. For instance, if we registered the Hello

derivation /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv as a root, then not only would

the Hello output /nix/store/bwacc7a5c5n3...-hello-2.1.1 be kept (if valid), but also the

source distribution downloaded by fetchurl, the C compiler, and so on.

• gc-keep-derivations: If true (the default), the garbage collector will keep the deriva-

tions from which non-garbage store paths were built. That is, if store path p is live,

then its deriver deriver[p] is also live (if deﬁned and valid).

Thus, the ﬁrst option causes liveness to ﬂow from derivations to outputs, while the sec-

ond causes liveness to ﬂow from outputs to derivations. Typically, developers turn both

options on to ensure that all build-time dependencies are kept, even when only the outputs

are registered as roots.

Figure 5.20 shows the computation of the set of live paths from the set of roots. It

performs a simple traversal through the graph of store paths with edges deﬁned by the

union of the references relation, and the deriver and output links.

5.6.3. Stop-the-world garbage collection

After we have computed the live paths, we can garbage-collect by deleting all paths that

are not live. However, life is not quite so simple:

• Some non-live paths are actually about to be “born”, i.e., they are currently being

built by some derivation.

• Paths that are garbage at the beginning of the collection might be “resurrected”.

This is a big difference with garbage collection in general [179]. In programming

language implementations, garbage memory objects can never become live again

because there are no pointers to them, and pointers are only created through allo-

cation from the free space. Here, pointers (paths) are computed deterministically,

128

5.6. Garbage collection

livePaths(roots) :

todo ← roots

done ← /0

while todo 6= /0

p ← remove one path from todo

if p 6∈ done :

done

∪

← {p}

todo

∪

← closure(p)

if gc-keep-outputs = true ∧ p ends in ".drv" :

d ← parseDrv(readPath(p))

if valid[d.output] 6= ε :

todo

∪

← {d.output}

if gc-keep-derivations = true ∧ deriver[p] 6= ε ∧ valid[deriver[p]] 6= ε :

todo

∪

← {deriver[p]}

return done

Figure 5.20.: livePaths: Computing the set of live paths

so they can be recreated. For example, suppose that the Hello output /nix/store/-

bwacc7a5c5n3...-hello-2.1.1 is dead at the start of the collection run. During the

collection, a user may run nix-env -i hello and so make it reachable and live again.

The simplest solution by far is to use “stop-the-world” garbage collection, which means

that there are no other Nix operations while the collector is running. A stop-the-world

collector is shown in pseudo-code in Figure 5.21.

Stop-the-world semantics is implemented by using a global lock. All normal Nix pro-

cesses acquire a read (shared) lock when they start, and release it when they terminate

The garbage collector, on the other hand, acquires a write (exclusive) lock on startup

94 .

Thus the collector can only proceed when there are no other processes modifying the store

or the database. While the collector is running, new processes block until the collector

ﬁnishes.

The collector then ﬁnds the roots and computes the set of live paths. Next, it ﬁnds the

set of dead paths by reading the entries in the Nix store and tossing out those that are in

the live set 95 . The dead paths should then be deleted. However, they cannot be deleted

in arbitrary order, since that violates the closure invariant. This does not matter if the

collector runs to completion, since at that time the closure invariant holds again. But if the

collector is interrupted prematurely, we can have a store that violates the closure invariant.

Consider a store with the Hello output and one of its dependencies, Glibc. Suppose that

both are dead and about to be deleted. The collector deletes Glibc but is interrupted before

it can delete Hello. Then Hello is still valid, and so if the user were to run nix-env -i hello,

an incomplete Hello is installed into the user environment. (Note that the build algorithm

This is not shown here, but such a lock is acquired in POSIX using the system call fcntl(fd,F_SETLKW, ...) and

passing a lock type F_RDLCK. In contrast, the exclusive lock acquired by acquirePathLock in Figure 5.4 uses

F_WRLCK.

129

5. The Extensional Model

gc() :

lock ← acquirePathLock("/nix/var/nix/global") 94

roots ← ﬁndRoots()

live ← livePaths(roots)

dead ← /0

for each directory entry n in the Nix store (storeDir) :

p ← storeDir + "/" + n

if p 6∈ live : 95

dead

∪

← {p}

dead

← topological sort of dead under the references relation

such that p < q if q ∈ references[p] 96

for each p ∈ dead

(in topologically sorted order) :

invalidatePath(p)

deletePath(p)

releasePathLock("/nix/var/nix/global", lock)

invalidatePath(p) :

In a database transaction: 97

valid[p] ← ε

for each p

∈ references[p] :

referers[p

] ← referers[p

] − {p}

references[p] ← ε

deriver[p] ← ε

Figure 5.21.: Stop-the-world garbage collector

through substitute in Figure 5.15 only checks the validity of the top-level path, and not

its references, since the closure invariant makes such a check redundant. So it is actually

difﬁcult to recover from an inconsistent store; we cannot just rebuild all derivations.)

To prevent this, we need to make sure that a path is deleted before its references are

deleted. That is, if p refers to q, then p must be deleted before q. This deﬁnes a partial

ordering on the set of dead paths. A correct ordering on the set of dead paths that preserves

the closure invariant can therefore be found by topologically sorting the dead paths under

the relation <, where p < q if q ∈ references[p] 96 .

Before each store object is deleted, its metadata in the database should be cleared, which

is done by the function invalidatePath 97 . The cleanup must be done in a transaction to

maintain database integrity. It must also be done before path deletion to maintain the

validity invariant (Invariant 1).

It is easy to see that the collector in Figure 5.21 maintains the closure invariant.

Lemma 3 (No cycles in references graph). The references graph contains no non-trivial

cycles. A cycle is trivial if it is a self-reference, i.e., p ∈ references[p].

Proof. This property follows trivially from the fact that a path’s references must exist prior

to the creation of the path (since the references, determined by scanForReferences, are a

130

5.6. Garbage collection

subset of the inputs in Figure 5.11). A cycle means that there must exist a path p

that

references a path p

that did not exist at the time p

was created.

Theorem 4 (GC correctness). The garbage collector in Figure 5.21 maintains the closure

invariant at all times.

Proof. Assume that the invariant holds at the start of the garbage collector run. By con-

tradiction, assume that the invariant is then violated at some point. This means that there

exists a valid path p and an invalid path q, such that q ∈ references[p]. However, then

p < q under the ordering deﬁned above, and since by Lemma 3 there are no cycles in the

references graph, p < q implies ¬(q < p). This means that p must have been deleted and

made invalid before q. Thus p is both valid and invalid, a contradiction.

5.6.4. Concurrent garbage collection

Stop-the-world garbage collection, while simple to implement, has very bad latency. While

the collector runs, no other Nix processes can start (that is, they block until the collector

ﬁnishes). This is unacceptable, since the collector might run for a very long time. For

instance, in our Nix-based build farm (Chapter 8), where we run the collector once every

few months, it can take a few hours to delete some 100 gigabytes of garbage.

Worse, there is a liveness issue: the collector might never start. If there always is a

process holding a read lock, then the collector cannot acquire a write lock. Of course,

any such process should terminate eventually, but by that time another process may have

acquired a read lock as well. In a build farm or a Nix store with a similarly high utilisation,

there might never be a point in time in which there is no process holding a read lock. If

locks are strongly fair [3], this is not a problem: the kernel will just block further attempts

to acquire a read lock while the garbage collector is waiting for a write lock. However, this

is not the case in general

Therefore we need a concurrent garbage collector—one that can run even as other Nix

operations are executing. This section shows ﬁrst a partially concurrent collector that ad-

dresses the liveness issue, i.e., that can run in parallel with already existing Nix processes.

In particular, builds that are in progress (and that might take a long time to ﬁnish) do not

prevent the collector from starting. We can then improve the latency problem by making

this collector run incrementally.

The main issues to consider are the following:

• Some paths may be in use but may not yet be reachable from a root, e.g., during a

build done by nix-env.

• New roots may be created while the collector is running. Paths reachable from those

roots should not be deleted.

The solution to the ﬁrst problem is to have running processes write a log of all store paths

that they are about to access or create to a log ﬁle, which is read by the collector. Such a log

ﬁle is per-process, e.g., /nix/var/nix/temproots/15726, where 15726 is the Unix process ID.

The store paths in those ﬁles are called temporary roots. Function addTempRoot, shown in

For instance, strong fairness is not speciﬁed for POSIX’s fcntl operations.

131

5. The Extensional Model

addTempRoot(p) :

if this is the ﬁrst call to addTempRoot in this process :

do :

Acquire a read lock on "/nix/var/nix/gc" 98

fd ← open("/nix/var/nix/temproots/" + pid,

O_WRONLY|O_CREAT|O_TRUNC, 0666) 99

Release the read lock on "/nix/var/nix/gc"

Acquire a read lock on fd

while ﬁle fd is not empty 100

Upgrade the read lock on fd to a write lock 101

Write p to fd

Downgrade the write lock on fd to a read lock

Figure 5.22.: addTempRoot: Registering a temporary garbage collection root

Figure 5.22, shows how a process registers a store path p as a temporary root. The idea is

that to register a root, the process must acquire a write lock on its per-process log ﬁle 101 .

The garbage collector, shown below, will acquire a read lock on each per-process log ﬁle.

Thus, no process can proceed after the garbage collector has acquired a read lock on its log

ﬁle.

There is a mechanism in place to clean up stale log ﬁles in /nix/var/nix/temproots. When

the log ﬁle is ﬁrst created 99 , a read lock is acquired to enable the collector to detect

whether the log ﬁle is stale. If the collector can acquire a write lock on a log ﬁle, then

apparently the corresponding process has died without removing its own log ﬁle, and thus

the collector can remove the log ﬁle. However, to prevent a race with ﬁle creation, i.e., the

collector removing a newly created log ﬁle before the process can acquire a lock on it, we

use deleteSignal (page 98) to signal and detect this condition 100 .

There are several instances where addTempRoot must be called:

• In addToStore (Figure 5.3), after the call to makePath and before the ﬁrst valid-

ity check. This ensures that the results of Nix expression translation are kept until

process termination.

• In substitute (Figure 5.15), before the validity check. This ensures that the results of

builds are kept until process termination.

On the other hand, we don’t register the derivation path p

drv

in build (Figure 5.11) as a tem-

porary root, even though p

drv

is going to be accessed. The reasoning is that registering p

drv

is someone else’s responsibility. For instance, in an execution of nix-env, any derivation

path will have been registered in the same process by addToStore in the Nix expression

translation. Likewise, in a call to nix-store --realise, the user is responsible for registering

the argument as a root (e.g., by using nix-instantiate --add-root).

The second problem—new roots being created while the collector is running—is ad-

dressed by blocking the creation of new roots while the collector is running. To reg-

ister a permanent root, a process must ﬁrst acquire a read lock on the global lock ﬁle

"/nix/var/nix/gc". The collector on the other hand acquires a write lock and holds it during

132

5.6. Garbage collection

gc() :

Acquire a write lock on "/nix/var/nix/gc" 102

roots ← ﬁndRoots() 103

live ← livePaths(roots)

temp ← /0

for each log ﬁle l in "/nix/var/nix/temproots/" : 104

fd ← open log ﬁle l

if we can acquire a write lock on fd :

deleteAndSignal(l, fd)

else :

Acquire a read lock on fd

temp

∪

← the set of paths read from fd

live

∪

←

p∈temp

closure(p)

dead

← topological sort of dead under the references relation

for each p ∈ dead

(in topologically sorted order) :

if p ends in ".lock" : 105

lock

← open lock p

if we cannot acquire a write lock on fd

lock

: skip

invalidatePath(p)

deletePath(p)

if p ends in ".lock" : deleteAndSignal(p, fd

lock

)

Release the write lock on "/nix/var/nix/gc"

Figure 5.23.: Concurrent garbage collector

its execution. Since adding a lock takes a small, bounded amount of time, the collector

will quickly acquire the write lock

Figure 5.23 shows the concurrent garbage collector. It ﬁrst acquires the global lock

"/nix/var/nix/gc"

102 . This prevents new permanent roots from being created. Thus, the

set of permanent roots determined in 103 cannot increase while the collector runs. It also

blocks new Nix processes since these acquire a read lock 98

It then reads the set of temporary roots from the per-process ﬁles in /nix/var/nix/-

temproots/ 104 . It acquires a read lock on each ﬁle, thus blocking the creation of new

temporary roots by the corresponding processes. Stale log ﬁles are removed here, as de-

scribed above.

After this, no new roots—permanent or temporary—can be created. The garbage col-

lector can therefore proceed normally. There are some boring details

105 regarding the

deletion of lock ﬁles, which we only want to delete if they are stale (left over from an in-

terrupted operation). Staleness is discovered by attempting to obtain a write lock (without

blocking). If this succeeds, then the lock is either stale, or a process just created it but has

not acquired a lock on it yet. The latter case is handled by writing a deletion token to it

Strictly speaking, this is not strongly fair: it is in principle possible that there is always at least one process

holding a read lock. However, given the typical frequency of root creation and the time it takes to do so, this

is extremely unlikely to occur in practice.

133

5. The Extensional Model

using deleteAndSignal. This token is detected by addTempRoot, which will restart upon

seeing it.

The correctness of the algorithm follows from the observation that no Nix process will

access any store path that is not in the live set, since all temporary roots are known to the

collector through the log ﬁles, and the creation of new temporary or permanent roots is

blocked.

The collector described here is still not perfectly concurrent: though it can run in parallel

with previously executing processes (in particular, with builders), it blocks new processes

and new temporary roots. Thus collector execution is strongly fair but still has bad latency.

However, this is easily ﬁxed by making the collector incremental. The collector can simply

periodically restart until no more garbage exists (indeed, the user is free to interrupt the

collector if desirable). Restarting will release the locks, allowing waiting processes to

proceed. An even more concurrent scheme is one where processes inform the collector

of new temporary roots by writing them to a socket and waiting synchronously for an

acknowledgement from the collector.

5.7. Extensionality

So why do we call this model extensional? The reason is that we make an assumption of

extensional equality. Build operations in general are not pure: the contents that a builder

stores in the output path can depend on impure factors such as the system time. For in-

stance, linkers often store a timestamp inside the binary contents of libraries. Thus, each

build of a derivation can produce a subtly different output.

However, the model is that such impure inﬂuences, if they exist, do not matter in any

signiﬁcant way. For instance, timestamps stored in ﬁles generally do not affect their opera-

tion. Hence extensional equality: two mathematical objects (such as software components)

are considered extensionally equal if they behave the same way for any operation on them.

That is, we do not care about their internal structure.

Thus, while any derivation can have a potentially inﬁnite set of output path contents

that can be produced by an execution of its builder, we view the elements of that set as

interchangeable.

But this is a model—an assumption about builders. If a builder yields a completely

different component when invoked at a certain time of day, it is beyond the scope of the

model. In reality, also, we can always observe inequality by observing the internal encod-

ing of the component, since that is a permissible operation. But the model is that any such

observation does not take place or does not have an observable effect.

134

6. The Intensional Model

The previous chapter discussed the extensional Nix store model. This model works very

well, but it has one substantial limitation (as well as a few smaller ones): it does not allow

sharing of a Nix store between arbitrary users. If we allow just anyone with an account on

the system to run arbitrary Nix operations, we run into serious security problems:

• A user could modify the Nix store directly and inject a Trojan horse (a malicious

component masquerading as legitimate software) into a component used by other

users, allowing their accounts to be compromised when they execute the component.

• Even if direct access to the store were restricted, i.e., if all builds were performed by

a system process on behalf of other users, it is still possible for users to “crack” the

Nix store by starting a build that interferes with the builds of other users.

• And even if that were not possible, the substitute mechanism is dangerous in that we

must trust that an FSO obtained through a substitute is actually a real build result of

the derivation from which it is claimed to have been built.

This chapter shows a different model for the Nix store, called the intensional model, that

allows these problems to be solved. The basic idea is to make the entire Nix store content-

addressable, meaning that each store path has a name that corresponds to the hash of the

contents of that path. This invariant implies that if a store path is trusted, then its contents

are trusted as well, an improvement over the extension model in the previous chapter.

Of course, whether to trust a store path is a different problem, but that assessment can be

made by each individual user instead of being made globally for all users. If different users

obtain substitutes from different sources, with different levels of trustworthiness, they do

not interfere with each other.

The intensional model is a more recent development than the extensional model, and it

has only been implemented as a rough prototype. Thus some of the results in this chapter

are somewhat more tentative in nature. However, the crucial assumption of the intensional

model—that the technique of hash rewriting works on common components—has been

validated on the Nix Packages collection.

6.1. Sharing

This section motivates why sharing of a Nix store between multiple users is a desirable

property. Sharing here means that multiple users have “write access” to the Nix store,

i.e., can build and install software, create user environments, and so on. This immediately

raises issues of trust: if not all users mutually trust each other, how can they share a store?

It should be noted that many deployment systems do not support sharing in this sense.

For instance, almost all Unix deployment systems allow only the administrator (root) to

135

6. The Intensional Model

install software. (An exception is Zero Install [112], which allows arbitrary users to install

software, with duplicate installations of the same component shared between users, i.e.,

downloaded and stored on disk only once.) So lack of sharing is not necessarily a terrible

limitation for a deployment system, as in many environments it is perfectly ﬁne if only

the administrator can make installation decisions. Such environments include single-user

workstations (where the user is the administrator), small multi-user systems where all users

trust each other, and typical multi-user environments where it is in fact a feature if the

software environment can be determined exclusively by the system administrators.

Why do we want to share Nix stores? But many environments are somewhere in be-

tween the extremes of a single user on the one hand, and multiple users with a single

deployment authority on the other hand. Imagine a university shell server that allows any

student with an account to log in. It is clearly terribly insecure to give “root” access to

any student in order to install software. But on the other hand, it is also desirable to allow

students to install software. Other examples are computational grids and hosting environ-

ments. If the facilities of the deployment system are only available to the administrator,

the users have to “manually” install software, e.g., by compiling from source, or by down-

loading and unpacking pre-built binaries. But in either case, the advantages of using a

deployment system—such as dependency tracking—are gone and unavailable to the end-

users. This can lead to correctness problems. For instance, if a user compiles a component

that has a runtime dependency on a “globally” installed component, that is, one managed

by the deployment system, then the deployment system will have no knowledge of this de-

pendency. Thus, the deployment system may consider it safe to uninstall the dependency.

Also, if many users need a piece of software that has not been globally provided by the

administrator, and each of them installs it separately (e.g., in his or her account), this will

cause a considerable amount of resource wastage in terms of disk space, memory, CPU

time lost due to redundant dynamic link resolution, and so on.

Another scenario where sharing is extremely valuable is in build management, i.e., in

the domain of tools such as Make and Ant. A typical development model in a multi-

developer project is that developers each have a working copy of the source code, obtained

from a version management repository. Each developer then builds the project in his or her

own working copy. However, with large projects, the overhead of building can be quite

substantial, sometimes in the order of hours or days. Thus it would be great if the artifacts

created by one user could be automatically “recycled” by other users; i.e., if a user has

to build something that has already been built previously by another user, the previous

build result is re-used automatically. Some build systems have this feature, notably Vesta’s

evaluator and ClearCase’s ClearMake (discussed in Section 10.3). However, these tools

depend on non-portable techniques to implement dependency identiﬁcation, as discussed

in Chapter 10.

Also, as we will see in Chapter 9, Nix is very useful for service deployment, e.g., a

concrete web server with all its software components, static conﬁguration and static data

ﬁles, management scripts, and so on. It is easy to imagine a shared system, such as a

hosting environment, where many different users instantiate Nix services. Many of those

services will overlap, e.g., because they partially use the same software components (say,

Apache). This common subset should be stored only once.

136

6.1. Sharing

Finally, sharing is especially important in Nix because derivation outputs are typically

not relocatable, since most outputs contain references to their own output path or to other

paths in the Nix store. Any user can have a private Nix store, e.g., /home/alice/nix/store.

However, pre-built binaries assume a particular location for the Nix store; almost always,

this is /nix/store. Since the location of the Nix store is often stored in outputs, pre-built

binaries are simply not applicable to a Nix store in another location in the ﬁle system. In

fact, this is precisely why the location of the Nix store is part of the store path computation

(page 94): different hashes are computed for the same store derivations in different Nix

stores to prevent any possible mix-up through the substitute mechanism.

So for these reasons it is important for a Nix store—the “heap” or address space of

installed components in the system—to be shareable among users. This means that users

have not only “read” access to the components in the store, but can also run package

management operations themselves, i.e., build, install, uninstall and upgrade components,

both from source and from binaries through the substitute mechanism.

The trust model Of course, sharing is not hard—secure sharing is. The difﬁculty lies

in providing certain trust properties. Given a derivation d, how can a user trust that the

contents of d.output are actually the build result of this derivation, and not, say, a Trojan

horse placed there by another user?

The basic trust problem is as follows. A user Alice

has obtained a set of Nix expressions

from a trusted source, Dave (the Distributor). We do not deﬁne how this trust relation

is established. The idea is that Alice trusts the producer of the Nix expressions to create

expressions that build software that is useful, does not contain Trojans or spyware, does not

destroy data, and so on. This type of trust cannot be established formally; the determination

can only be made by a combination of knowledge of the producer’s history, reputation, and

other social factors, and possibly code review (although this is beyond most users). For

instance, a Nix user will typically trust the Nix Packages collection.

We also assume that the set of expressions has been received unmodiﬁed by Alice, mean-

ing that they have not been modiﬁed by a “man in the middle”. This can be ensured by

using appropriate cryptographic techniques, e.g., verifying a digital signature on the set

of expressions [145], or fetching them from Dave through a secure channel (e.g., HTTPS

with a valid certiﬁcate).

Given this trusted set of Nix expressions, the trust property that we seek to ensure is that

if Alice installs a derivation from this Nix expression, the resulting component is one that

has been produced by that derivation. I.e., if foo.nix is trusted, and Alice runs

$ nix-env -f foo.nix -i bar

to install a derivation named bar, then the derivation output that is added to Alice’s user

environment is an actual derivation output of that derivation. It should not be possible for

other users to replace the derivation output with something else, or to interfere with the

build of the derivation. To put it precisely, the property that we want is:

The trust property: in a shared Nix store, if a user A installs or has installed

a trusted Nix expression, then the closures resulting from building the store

We follow the convention of the security and cryptographic literature to denote the several roles in a protocol

or exchange as Alice (A), Bob (B), Carol (C), Dave (D), and so on.

137

6. The Intensional Model

derivations instantiated from that expression are equal to what may have been

produced if user A had exclusive write permission to the Nix store.

Thus, for each user, a shared Nix store must produce the same results as if the store had

been a single-user store owned by that user. Note a subtlety in the phrasing: I say that the

result of building must match what may have produced in a single-user store, not what they

would have been. This is due to indeterminism; a single derivation may produce different

results (e.g., due to time stamps). In essence, each derivation has a possibly inﬁnite set

of possible build results, its output equivalence class. But we do not have a direct way to

test set membership in this class, i.e., to verify that a particular output was produced by a

derivation. Testing this set membership is undecidable in general.

Informally, the trust property can be stated as follows:

If the source of a component is trusted, then the binary that is installed in the

user’s user environment is also trusted.

Breaking the trust property So under what circumstances can the trust property be vi-

olated? The principal scenarios were sketched at the beginning of this chapter. Clearly, if

users can modify the contents of the Nix store directly, then there is no security whatsoever.

So let’s assume that all operations on the Nix store are executed by a Nix server process

(daemon) that performs Nix operations, such as instantiation, building and adding garbage

collector roots, on behalf of users; and that only this server process and its children have

write access to the store and the Nix database.

Even if direct access to the store is prevented in this way, malicious users can still “hack”

the store by creating and building a derivation whose builder modiﬁes other paths than its

output path, e.g., overwrites an already existing store object with a Trojan horse. This

violates the trust property, as the store object no longer corresponds to an actual output of

its deriver.

Then there is the substitute mechanism. Recall from Section 5.5.3 that a substitute is

an arbitrary program that builds a store path, typically by downloading its contents from

somewhere. How can we trust that a substitute for a derivation output path actually pro-

duces contents that correspond to an actual output of the derivation? So the substitute

mechanism can be used to trivially violate the trust property.

Sharing in the extensional model In the extensional model of the previous chapter,

sharing a Nix store is only possible if all users of the Nix store trust each other not to violate

the trust property. For instance, suppose that user Alice has obtained a Nix expression

E from a trusted source, and pulls substitutes from machine X , where X is a malicious

machine that provides Trojaned binaries for the output paths of the derivation produced by

E. For instance, it provides a manifest with the following substitute deﬁnition:

{ StorePath: /nix/store/bwacc7a5c5n3...-hello-2.1.1

106

NarURL: http://haxxor.org/dist/f168bcvn27c9...-hello.nar.bz2

Size: 35182 }

where the NAR archive f168bcvn27c9...-hello.nar.bz2 unpacks to a version of Hello 2.1.1

containing a Trojan horse that installs a back door on Alice’s machine. (Manifests were

brieﬂy discussed in Section 2.6, and will be covered in detail in Section 7.3.) When Alice

executes the Hello binary, her account may become compromised.

138

6.2. Local sharing

Suppose that subsequently Bob wants to install the same expression E. He ﬁrst pulls

from a different, trusted machine Y , obtaining a bona ﬁde substitute:

{ StorePath: /nix/store/bwacc7a5c5n3...-hello-2.1.1

107

NarURL: http://nix.org/dist/1hnv0817f168...-hello.nar.bz2

Size: 34747 }

However, this correct substitute will not avail him. When he installs Hello from the

trusted Nix expression (nix-env -i hello), Alice’s Trojaned binary will be added to his

user environment. After all, the Hello derivation’s output path, /nix/store/bwacc7a5c5n3...-

hello-2.1.1, is already valid, and by the substitute mechanism (Figure 5.15) that is sufﬁcient

to ﬁnish the build of the derivation.

The basic problem is that both substitutes want to occupy the same location in the ﬁle

system (cf. 106 and

107 ); but this is of course not possible. Nix makes an assumption

of extensional equality (Section 5.7): any substitute for the same path is behaviourally

equivalent. But it has no way to verify that this assumption is valid.

Thus, source X may claim that its substitute is an output of some derivation d; but this

may be a lie, leading to a violating of the trust property. We can therefore only register

substitutes from source X if all users that share the store trust source X.

The remainder of this chapter ﬁrst shows that the trust property actually can be provided

in the extensional model for locally built derivations. Thus, users need not have mutual

trust relations. However, secure sharing becomes impossible once substitutes are used.

This leads to the development of the intensional model that enables sharing of a Nix store

even in the presence of substitutes.

6.2. Local sharing

As we have seen above, sharing between machines is only possible in the extensional

model if all users have the same remote trust relations. For locally-built derivations on the

other hand (i.e., when not using substitutes), we can allow mutually untrusted users. The

trick is in preventing a user from inﬂuencing the build for some derivation d in such a way

that the result is no longer a legitimate output of d.

For instance, if Alice has direct write access to the Nix store, she can start a build of

derivation d, then overwrite the output path with a Trojan horse. Similarly, even if builds

are done through a server process that executes builds on behalf of users but running under

a different user ID (uid), Alice can interfere with the build of d by starting a build of a

specially crafted derivation d

, the builder of which writes a Trojan horse to the output

path of d.

To ensure that the trust property holds for locally-built derivations, we therefore need to

ensure that the following specialised property holds.

The execution of the builder of a derivation d (d.builder) modiﬁes no path in

the Nix store other than d.output.

We can ensure this property as follows. First, users no longer have direct write access

to the Nix store. As suggested above, this is the absolute minimum measure necessary

to obtain secure sharing. All builds are performed by a Nix server process on behalf of

users. The server runs builders under user IDs that are distinct from those of ordinary

139

6. The Intensional Model

user or system processes (e.g., nix-build-{1,2,...}). Also, no two concurrent builds can

have the same uid. This prevents one builder from interfering with the output of another,

as illustrated above. Thus, the server maintains a “pool” of free and in-use uids that can

be used for building. How a builder can create its intended output path is discussed on

page 157.

When a build ﬁnishes, prior to marking the output path as valid, we do the following:

• Ensure that there are no processes left running under the uid selected for the builder.

On modern Unix systems this can be done by performing a kill(-1, SIGKILL) opera-

tion while executing under that uid, which has the effect of sending the KILL signal

to all processes executing under uid [99]. Older versions of the POSIX standard

however did not mandate this behaviour [98]: on such systems, killing all processes

running under a certain uid is tricky as it is fraught with race conditions.

• Change ownership of the output to the global Nix user.

• Remove write permission and any set-user-ID and set-group-ID bits (which are spe-

cial permission bits on ﬁles that cause them to be executed with the rights of a

different user or group—a possible security risk).

Note that the latter two steps have a subtle race condition. For instance, if we change

ownership ﬁrst, we have the risk of inadvertently creating a setuid binary owned by the

global Nix user

. If however we remove write and setuid permission ﬁrst, a left-over

process spawned by the builder could restore those permissions before the ownership is

changed. This is why the ﬁrst step is important. Also, on Unix, if a left-over process

opened a ﬁle before the ownership changes, it can still write to it after the change, since

permissions are only checked when a ﬁle is opened.

In conclusion, in the extensional model we can do secure source-based deployment with

sharing. Thus, the trust property is guaranteed in the absence of substitutes. Of course,

this is not enough: we also want transparent binary deployment through the substitute

mechanism.

6.3. A content-addressable store

As we saw in Section 6.2, we can have secure sharing of locally built derivation outputs,

but not of remotely built outputs obtained through the substitute mechanism. All users have

to trust that the contents produced on another machine purportedly from some derivation d

is indeed from derivation d. As stated above, such a trust relation must be global for a Nix

installation. In this section we develop a substantially more powerful model in which this

is not the case. We do this by moving to a fully content-addressable Nix store for all store

objects.

The property of content-addressability means that the address of an object is determined

by its contents. Equivalently, if we know the contents of an object, we know its address.

In the context of Nix, addresses are of course store paths. We have seen in Section 5.3

Exactly to prevent this race, Linux removes setuid permission when changing ownership. However, this is not

standard behaviour [99].

140

6.3. A content-addressable store

(page 97) that some FSOs in the extensional model have this property. In particular, store

derivations and sources added to the store by instantiate have store paths that are computed

from their contents. The function addToStore (Figure 5.3) computes the store path as

follows:

p ← makePath("source", printHash

(h), name)

Thus the store path depends entirely on the cryptographic hash h of the contents of the

FSO being added, and its symbolic name. (So strictly speaking, these paths can only be

considered “content-addressing” if the symbolic name is considered part of the content.)

Since we assume that collisions of the cryptographic hash functions used in Nix do

not occur in practice (since they are supposedly infeasible to produce), we have a one-to-

one correspondence between paths and contents. This gives us some concrete properties.

Given a particular FSO fso, there is at most one path whose contents are equal to fso. Also,

assuming infeasibility of ﬁnding hash collisions and given two Nix stores both containing

a valid path p, the FSOs stored at p must be equal (for if they are not, we have found a

hash collision).

Content-addressability is an extremely useful property. Recall the example on page 138,

where Alice and Bob had different trust relations, so there were substitutes with different

contents for a single store path. This problem does not exist in a content-addressable

store, since in such a system path equality (between different substitutes) implies content

equality.

But not all store objects in the extensional model are stored in a content-addressed way.

Derivation outputs are not; the output path of a derivation is computed from the input

derivation (Figure 5.6), not from the actual contents produced by the builder. Below we

will show how we can achieve content-addressability even for derivation outputs. As we

shall see, this is not trivial due to self-referential components.

Once we have the property of content-addressability, different users can have different

trust relations: for each user we can have a different derivation to output path mapping.

This is the intensional model—equality is deﬁned by internal contents, not by observable

behaviour. This model is much stronger than the extensional model, since it doesn’t make

the simplifying but unenforcible assumption of builder purity. Rather, intensionality is an

inherent property of the system.

6.3.1. The hash invariant

The crucial property of the intensional model is that the Nix store is now content-

addressable: if we know the contents of a store object, we know its store path.

Ideally, this would mean that if an FSO has hash h, then it is stored in the Nix store at

nixStore/h−name, where nixStore is the location of the Nix store and name is the symbolic

name for the FSO (e.g. hello-2.1.1). For instance, if a Hello FSO has truncated SHA-256

hash lm61ss8nz7hl..., then its store path would be /nix/store/lm61ss8nz7hl...-hello-2.1.1.

However, this is not sufﬁcient, since the symbolic name must be taken into account in

the hash part of the store path as well. If the same FSO were added multiple times, but

with different symbolic names, the resulting store paths must not have the same hash part,

since our scanning approach to ﬁnding references uses only the hash part to distinguish

141

6. The Intensional Model

dependencies. Therefore, in the intensional model, the function makePath that computes

store paths from a content hash h and a symbolic name name is deﬁned as follows:

makePath(h, name) =

nixStore + "/" + printHash

(truncate(20, hash

sha256

(s))) + "-" + name

where

s = "sha256:" + printHash

(h) + ":" + name

Note that contrary to the deﬁnition of makePath in the extensional model (page 94), this

function produces a store path that depends only on a cryptographic hash h of the contents

of the FSO to be stored, and its symbolic name name.

The hash h is the hash of the contents of the FSO being added. It is almost, but not quite,

the SHA-256 hash over the serialisation of the FSO, i.e., hash

sha256

(serialise(fso)). As we

shall see below, hashing is a bit more complicated due to self-referential components.

But let’s ignore that for now. Suppose that we want to add a store object to the store that

has SHA-256 hash 73b7e0fc5265... and symbolic name hello-2.1.1. Then,

s = "sha256:" + "73b7e0fc5265..." + ":" + "hello-2.1.1"

= "sha256:73b7e0fc5265...:hello-2.1.1"

and since

printHash

(truncate(20,hash

sha256

(s))) =

"wf9la39rq7sx5hj0jry0mn5v48w8cmwi",

we obtain the store path

/nix/store/wf9la39rq7sx5hj0jry0mn5v48w8cmwi-hello-2.1.1

The hash part wf9la39rq7sx... is determined entirely by the contents and the symbolic

name. There can never be another object in the store with the same contents and symbolic

name, but with a different store path. So there is a correspondence between a store path’s

hash part, and the contents at that store path. This is called the hash invariant, and it is

what sets the intensional model apart from the extensional model.

Invariant 7 (Hash invariant). The hash part of a valid store path p is determined by the

cryptographic hash of p:

∀p ∈ Path : valid[p] 6= ε → p = makePath(valid[p],namePart(p))

Recall that namePart(p) returns the symbolic name component of path p, e.g. for

/nix/store/bwacc7a5c5n3...-hello-2.1.1 it returns hello-2.1.1. Also recall that valid[p] con-

tains the cryptographic hash of the serialisation of the FSO at p, stored when p was made

valid. Due to the stored hash invariant (page 96), this hash must match the actual contents at

p. Also note that while we compare the entirety of p to makePath(valid[p],namePart(p)),

142

6.3. A content-addressable store

it is only the hash part of p that actually matters. The equality of the name part is tautolog-

ical, since we feed namePart(p) into the call to makePath.

So how does content-addressability help us to achieve secure sharing in multi-user Nix

stores? The answer is that users can now independently install software, i.e., build deriva-

tions. If the results of those independent builds are the same, they get sharing; if results

differ due to impurity, they do not. This applies not just to local builds but more signiﬁ-

cantly to substitutes.

In the example on page 138, when Alice installs a derivation for Hello using a Trojaned

substitute from a malicious machine, the result will end up in some path, say /nix/store/-

5n5drxv693s3...-hello-2.1.1. If Bob installs the same derivation but using a legitimate sub-

stitute, the contents will differ and thus the result will necessarily be in a different store

path, e.g. /nix/store/4d6hb6vxh388...-hello-2.1.1. This path will be added to his user envi-

ronment. Thus, he is insulated from Alice’s bad remote trust relation.

6.3.2. Hash rewriting

The property of content-addressability is easily stated but not so easily achieved. This is

because we do not know the content hash of a component until after we have built it, but

we need to supply an output path to the builder beforehand so that it knows where to store

the component. In order to achieve content-addressability for derivation output, we must

rename them after they have been built to the appropriate content-addressed name in the

store. Roughly speaking, we ﬁrst build the derivation in a temporary location, compute the

hash over the result, and rename the temporary path to the content-addressed ﬁnal location.

So the ﬁrst step is to perform a build in a store path p with a randomly generated hash

part, i.e.,

p = makePath(randomHash(), name)

The function randomHash() produces a sufﬁciently long pseudo-random base-32 number

of the same length as a normal hash part, i.e., 32 characters. For instance, if we are building

Hello, we might produce the temporary path

p = makePath("2c8d367ae0c4...", "hello-2.1.1")

= /nix/store/2jxsizriq3al...-hello-2.1.1

Thus, the builder is executed with the environment variable out set to the store path

/nix/store/2jxsizriq3al...-hello-2.1.1.

If the build ﬁnishes successfully, we compute the SHA-256 hash h over the serialisation

of the FSO at p. We then rename path p to p

, computed as follows:

= makePath(h, name)

Thus, the temporary path p, which did not obey the hash invariant, is changed to one that

does obey the hash invariant. Suppose that after the build of Hello we compute content

hash 5c769ad74cac.... Then the ﬁnal path p

becomes:

p = makePath("5c769ad74cac...", "hello-2.1.1")

= /nix/store/jj8d7j9j6scc...-hello-2.1.1

143

6. The Intensional Model

So the temporary build result /nix/store/2jxsizriq3al...-hello-2.1.1 is renamed to /nix/store/-

jj8d7j9j6scc...-hello-2.1.1.

There is a snag, however: simply renaming the temporary path doesn’t work in case of

self-references, that is, if the binary image of a ﬁle refers to its own store path. This is

quite common. For instance, the RPATH of a Unix executable (page 23) frequently points

to its own directory so that related library components can be found. If we rename the

temporary path p to p

in such a case, the references to p will become dangling references,

and the component probably will not work anymore.

We solve this problem through the technique of hash rewriting. The basic idea is that

we copy p to p

, but at the same time replace all occurrences of the string hashPart(p) in

the FSO with hashPart(p

). (Recall that hashPart(p) yields the hash part of path p.)

However, this changes the contents of the component, thereby invalidating the hash!

So we have a circular problem: we want to change the contents of an FSO to match its

hash, but that changes its hash, so the contents must be changed again, and so on. With

cryptographic hashes it is not feasible to compute a “ﬁxed point”: a string that contains a

representation of the hash to which the string hashes. I.e., it is not computationally feasible

to compute a string s such that

s = s

+ printHash(hash

(s)) + s

where s

and s

are an arbitrary preﬁx and sufﬁx to the occurrence of the hash representa-

tion.

We solve this problem by computing hashes modulo self-references. Essentially, this

means that we ignore self-references when computing the hash. So the hash h is not com-

puted as the SHA-256 hash of the serialisation of the FSO:

h = hash

sha256

(serialise(readPath(p)))

Instead, the hash is computed over the serialisation of the FSO, with all occurrences of

hashPart(p) zeroed out. That is,

h = hashModulo(serialise(readPath(p)), hashPart(p))

where

hashModulo(c, hp) = hash

sha256

(

∑

i∈ﬁnd(c,hp)

(i + ":") + ":" + replace(c, {hp 0}))

The function ﬁnd(c,h) (ﬁrst seen in Section 5.5) yields the offsets of the occurrences of the

substring h in the string s. The constant 0 denotes a string consisting of binary 0s of the

same length as the hash part hp, 32 characters. The function replace(c, r) applies a set of

textual substitutions r to the string s. Textual substitutions in r are denoted as x y, i.e.,

occurrences of the string x are replaced with the string y.

The function hashModulo computes a hash not just over c with the hash part hp cleared

out, but also takes the offsets of hp into account. It is necessary to encode the offsets of

the occurrences of hp into the hash to prevent hash collisions for strings that are equal

except for having either hp or 0-strings at the same location. The colons simply act as

disambiguators, separating the offsets and the contents.

144

6.4. Semantics

For example, suppose we have two FSOs A and B that are equal except that at one point,

A has a self-reference while B has zeroes, e.g.,

A = "xxxxHxxxHxxHxxx"

B = "xxxxHxxx0xxHxxx"

where H is a self-reference and 0 is a string of 32 zeroes. If we do not take self-reference

locations into account, A and B will hash to the same value.

The hash h computed using hashModulo is used to produce the new store path p

described above, i.e., using makePath. Then, we copy p to p

, rewriting all occurrences of

hashPart(p) in the contents of p with hashPart(p

writePath(p

, deserialise(c

))

where

= replace(serialise(readPath(p)), hashPart(p) hashPart(p

)).

Note that even though in case of self-references in general

hash

sha256

(serialise(readPath(p))) 6= hash

sha256

(serialise(readPath(p

))),

we do have

hashModulo(serialise(readPath(p)), hashPart(p))

= hashModulo(serialise(readPath(p

)), hashPart(p

)).

That is, the hash modulo the randomly generated hash part does not change after hash

rewriting.

6.4. Semantics

We can now formalise the intensional model. The main difference with the extensional

model is that output paths are no longer known a priori. But because of this, we cannot

prevent re-building a derivation by checking (as was done in substitute in Figure 5.15)

whether its output path is already valid. The same applies to checking for substitutes,

which are also keyed on output paths.

Also, due to impurity, a single derivation can now result in several distinct components

residing at different store paths, if the derivation is built multiple times (e.g., by different

users). That is, a derivation actually deﬁnes an equivalence class of store paths within the

Nix store, the members of such classes all having been produced by the same derivation.

Therefore, we make the following change to the abstract syntax of store derivations

deﬁned in Figure 5.5 (page 101). The ﬁeld output is removed; we don’t know the output

path in advance. But we add a ﬁeld eqClass : EqClass that identiﬁes the equivalence class

of the output.

So what is an equivalence class (i.e., what is the type EqClass)? In fact, the equiva-

lence class eqClass is exactly the same as the original output ﬁeld! It is computed in the

145

6. The Intensional Model

same way, namely, by hashing the derivation with its eqClass ﬁeld set to the empty string

(callout 64 on page 102). Note that the hash parts produced there cannot conﬂict with

those computed by makePath in the previous section, assuming that there are no hash col-

lisions (since the former hash preimages start with sha256:output:out:, while the latter hash

preimages start with sha256: followed by a hash). So for instance, when instantiating our

running example, the Hello derivation, we obtain d.eqClass = "/nix/store/bwacc7a5c5n3...-

hello-2.1.1". In the extensional model, we had d.output = "/nix/store/bwacc7a5c5n3...-hello-

2.1.1". Since eqClass is just a (fake) store path, the type EqClass is just an alias for the

type Path introduced for legibility.

So we have really just renamed the ﬁeld output to eqClass. However, there is an im-

portant difference in the meaning of eqClass. The equivalence class path is “virtual”: it is

never built. For example, in the intensional Nix store, we will never actually build a path

/nix/store/bwacc7a5c5n3...-hello-2.1.1.

The reason for using store paths for equivalence classes is that it gives us an easy way

to refer to the output of a derivation from other derivations. For instance, the envVars ﬁeld

of a derivation that depends on Firefox must in some way refer to the path of the Firefox

component, even though this path is not known in advance anymore. When we build a

derivation d depending on derivation d

, we simply rewrite in d.envVars all occurrences of

hashPart(d

.eqClass) to a trusted member of the equivalence class denoted by d

.eqClass.

Since we must remember for each derivation what output paths were produced by

it and who built or substituted them, we deﬁne a database mapping eqClassMembers :

EqClass → {(UserId, Path)}. The type UserId denotes a user identity. Here we will simply

take them to denote Unix user names, e.g., alice. More elaborate notions of identity are

also possible, e.g., using groups or classes of users.

Through the table eqClassMembers, an equivalence class maps to a set of concrete store

paths along with the names of the users that built or substituted them. For instance, if

both Alice and Bob have obtained Hello from their respective substitutes, the entry for

/nix/store/bwacc7a5c5n3...-hello-2.1.1 might be as follows:

eqClassMembers[/nix/store/bwacc7a5c5n3...-hello-2.1.1] =

{ ("alice", /nix/store/bpv6czrwrmhr......-hello-2.1.1)

, ("bob", /nix/store/mqgi8xail9vp......-hello-2.1.1)

}

Of course, a store path can occur multiple times for different users. This happens if multi-

ple builds have yielded exactly the same result (the ideal!), or if users have installed from

the same substitute.

The members of an equivalence class (i.e., the right-hand side of an eqClassMembers

entry) must be usable paths; that is, they must either be valid or have at least one substitute.

We can now deﬁne the set of trusted paths in the equivalence class of a derivation output

as the set of valid or substitutable paths for some user:

trustedPaths(eqClass, user) = {p | (user, p) ∈ eqClassMembers[eqClass]}

Equivalence classes and closures A path can be a member of multiple equivalence

classes. This is easy to see: we can conceive of any number of different derivations that

146

6.4. Semantics

produce the output "Hello World" in various ways. So we cannot unambiguously say to

what equivalence class a path belongs. However, as we shall see below, there are times

when we need to know this. In particular, when we compute the closure of a path p, we

want to know to what equivalence class each reference belongs. Such a query is only

meaningful given a certain context, i.e., when we compute the closure of a path p in an

equivalence class e.

To this end, a new database mapping refClasses : (Path,Path) → {(EqClass,EqClass)}

is needed. This table allows us to determine equivalence classes when we are following the

references graph. It has the following meaning: if (e,e

) ∈ refClasses[(p, p

)], then there is

an edge in the references graph from path p in equivalence class e to path p

in equivalence

class e

The function closure

(p, e) computes the closure of a path p in equivalence class e. It

returns a set of pairs (p,e) : (Path,EqClass). Its deﬁnition is as follows:

closure

(p, e) = {(p, e)} ∪

[

∈references[p]

followRef(p, e, p

)

The auxiliary function followRef(p, e, p

) yields the closure of p

coming from path p in

equivalence class e:

followRef(p, e, p

) =

(

closure

, ε) if es

= /0

∈es

closure

, e

) otherwise

where

= {e

| (e,e

) ∈ refClasses[(p, p

)]}

The case where es

= /0 is to handle paths that are not in any equivalence class. This is

quite normal: paths that are not outputs of derivations (such as sources) need not be in

equivalence classes. For such paths, the special value ε is used to denote the absence of

an actual equivalence class. It should be noted that the left-hand sides of the pairs (the

paths) in the set returned by closure

(p, e) are the same for all e given a certain p. Only

the right-hand sides (the equivalence classes) can differ.

6.4.1. Equivalence class collisions

The fact that a derivation can resolve to any number of output paths due to impurity, leads

to the problem that we might end up with a closure that contains more than one element

from the output equivalence class of a derivation.

Figure 6.1 shows an example of this phenomenon. (As in Figure 2.3, an edge from node

A to B denotes that A is a build-time input of B, i.e., the output of B can have a reference to

the output of A.) Suppose that Alice has built gtk and pkgconﬁg locally. These both depend

on glibc. She has also registered Bob’s remotely built libIDL as a substitute. It also depends

on glibc. However, though Bob’s glibc was built from the same derivation, due to impurities

the build result is different. Thus, the equivalence class for the Glibc derivation has at least

two members. This is not a problem in itself. However, suppose that Alice next tries to

build ﬁrefox, which depends on gtk, pkgconﬁg, and libIDL. We then end up with a Firefox

147

6. The Intensional Model

Equivalence class glibc

glibc

pkgconﬁg

gtk

glibc

libIDL

ﬁrefox

Figure 6.1.: Equivalence class

collision

Equivalence class glibc

glibc

pkgconﬁg

gtk

libIDL

′

glibc

libIDL

ﬁrefox

Figure 6.2.: Resolution of the equivalence class

collision

binary that links against two glibcs. This might work, or it might not—depending on the

exact semantics of dynamic linking. In any case, it is an observable effect—it inﬂuences

whether a build succeeds and whether the build result works properly.

Thus, we need to prevent that any closure ever contains more than one path from an

equivalence class. This is expressed by the following invariant.

Invariant 8 (Equivalence class unicity invariant). No two paths in a closure can be in the

same equivalence class.

∀p ∈ Path : valid[p] 6= ε →

∀e ∈ EqClass :

∀(p

) ∈ closure

(p, e) : ∀(p

) ∈ closure

(p, e) :

6= ε ∧ e

= e

) → p

= p

That is, for any two elements (p

) and (p

) in any closure of a valid path p, if the

equivalence classes are the same (e

= e

), then the paths must also be the same (p

= p

The condition e

= ε is to handle paths that are not in any equivalence class (such as

sources).

So when we build a derivation, we have to select from the paths in the union of input

closures a single element from each equivalence class. However, we must still maintain

the closure invariant (page 96). For instance, in Figure 6.1, we cannot simply select the

set {glibc

,gtk

,pkgconﬁg

,libIDL

}, since libIDL

depends on glibc

which is not in this

set. (In this running example, for clarity I do not use store paths but symbolic constants,

e.g., glibc

for an output and glibc

for its equivalence class. The subscripts have no

signiﬁcance; they just denote different store paths.)

Once again, hash rewriting comes to the rescue. We can rewrite libIDL

so that it refers

to glibc

instead of glibc

. That is, we compute a new path libIDL

with contents

replace(serialise(readPath(libIDL

)), {hashPart(glibc

) hashPart(glibc

)})

(Of course, self-references in libIDL

must also be rewritten as described in Section 6.3.2.)

This is shown in Figure 6.2. The dotted edge denotes a copy-with-rewriting action.

148

6.4. Semantics

Path selection An interesting problem is which paths to select from each equivalence

class such that the number of rewrites is minimised. For instance, if we select glibc

, then

we have to rewrite one path (namely libIDL

), while if we select glibc

, we have to rewrite

two paths (gtk

and pkgconﬁg

). I do not currently know whether there exists an efﬁcient

algorithm to ﬁnd an optimal solution. A heuristic that works ﬁne in practice is to do a

bottom-up traversal of the equivalence class dependency graph, picking from each class

the path that induces the least number of rewrites.

However, picking an optimal solution with respect to the current derivation is not par-

ticularly useful in any case, since it ignores both the state of the Nix store as a whole, and

future derivations. In our example Alice might in the future install many additional com-

ponents from Bob’s remote repository (e.g., because Bob is a primary distribution site).

Thus, globally there are many more paths referring to glibc

than to glibc

. In this case it

is better to select glibc

and rewrite Alice’s locally built components. Possible heuristics

include:

• Select the path that has the largest referrer closure in the Nix store. Recall from Sec-

tion 5.2.3 that the referrer relation is the transpose of the references relation. Thus,

the referrer closure of p is the set of all paths that directly or indirectly reference p.

• Select the path that is also trusted by the system administrator (e.g., a special user

named root).

• Select the path that was obtained from a substitute referring to a distribution site with

some special status.

Example In the description of the resolution algorithm below, we will use the example

from Figure 6.1. Thus, we have the following paths in the combined input closures of

Firefox, paired with the equivalence classes to which they belong:

{ (glibc

,glibc

),(glibc

,glibc

),(gtk

,gtk

)

, (pkgconﬁg

,pkgconﬁg

),(libIDL

,libIDL

) }

Note that glibc

and glibc

share the same equivalence class glibc

, thus

eqClassMembers[glibc

] = {("alice",glibc

), ("alice",glibc

)}.

We have the following references:

references[gtk

] = {glibc

}

references[pkgconﬁg

] = {glibc

}

references[libIDL

] = {glibc

}

Also, the refClasses table tells us what equivalence classes correspond to those references,

e.g.,

refClasses[(gtk

,glibc

)] = {(gtk

,glibc

)}

and so on.

149

6. The Intensional Model

resolve(paths) :

// Find the conﬂicts.

sources ← /0

for each (p,e) ∈ paths :

if e = ε : sources

∪

← {p} 108

else : conﬂicts[e]

∪

← {p} 109

// Select one path for each equivalence class.

selected ← selectPaths(conﬂicts) 110

paths

← /0

for each (p,e) ∈ selected :

111

paths

∪

← {maybeRewrite(p, e, selected)}

return paths

∪ {(p,ε) | p ∈ sources} 112

Figure 6.3.: resolve: Collision resolution algorithm

The resolution algorithm Figure 6.3 shows the resolution algorithm. The function re-

solve accepts a set paths : {(Path,EqClass)} of pairs (p,e) denoting a path p in equivalence

class e

. The elements in paths must be closed under the references relation but may vio-

late the equivalence class unicity invariant; the idea is that paths is the union of a number

of calls to closure

, deﬁned above. The function resolve yields a closed set of paths that

does obey the invariant.

The resolution algorithm works as follows. First, the set of conﬂicts is determined

109 .

The mapping conﬂicts : EqClass → {Path} maps the equivalence classes in paths to the

corresponding paths in paths. Note that if any equivalence class in conﬂicts has more than

one member in paths, there is a conﬂict. For our example, the conﬂicts set is as follows:

conﬂicts[glibc

] = {glibc

,glibc

} conﬂicts[gtk

] = {gtk

}

conﬂicts[pkgconﬁg

] = {pkgconﬁg

} conﬂicts[libIDL

] = {libIDL

}

Thus, there is a conﬂict in the glibc

equivalence class. (Some paths are in the “fake”

equivalence class ε 108 , returned by closure

to indicate paths that are not in an equivalence

class, such as sources. These are not subject to rewriting, but they are added to the result

returned by resolve to ensure the closure property.)

Second, the function selectPaths returns a mapping selected from equivalence classes

to paths, i.e., EqClass → Path. That is, it chooses an element from the members of each

equivalence class in conﬂicts 110

. By deﬁnition, this deﬁnes a set that is free of equivalence

class collisions. We do not specify here how selectPaths chooses an element from each

class. As discussed above, it must implement some policy of selecting a path from each

Again, it is necessary to specify the intended equivalence class because a path may be in multiple equivalence

classes. In a previous implementation, a table eqClasses : Path → {EqClass} was used to map paths to the

equivalence classes to which they belong. This led to a subtly ﬂawed semantics: since we didn’t know which

particular equivalence class for a path p was intended, we had to use all of them. The result was that a path

could be replaced with a path in an entirely different equivalence class.

150

6.4. Semantics

maybeRewrite(p, e, selected) :

newRefs ← /0

eqRefs ← /0

rewrites ← /0

for each p

ref

∈ references[p] : 113

if p = p

ref

∨ ¬(∃e

ref

: (e,e

ref

) ∈ refClasses[(p, p

ref

)]) : 114

newRefs

∪

← {p}

else :

Set e

ref

such that (e,e

ref

) ∈ refClasses[(p, p

ref

)]

repl

← selected[e

ref

] 115

repl

) ← maybeRewrite(p

repl

, e

ref

, selected) 116

// Note that e

ref

= e

repl

newRefs

∪

← {p

repl

}

eqRefs

∪

← {(p

ref

,e,e

ref

)}

rewrites

∪

← {hashPart(p

ref

) hashPart(p

repl

)} 117

if newRefs = references[p] : return (p,e) 118

in a database transaction :

new

← addToStore

(readPath(p),

references[p], eqRefs, namePart(p), rewrites, hashPart(p)) 119

eqClassMembers[e]

∪

← {(curUser, p

new

)}

return (p

new

,e)

Figure 6.4.: maybeRewrite: Collision resolution algorithm (cont’d)

equivalence class. For our example, there are only two possibilities, namely

selected[glibc

] = glibc

selected[gtk

] = gtk

selected[pkgconﬁg

] = pkgconﬁg

selected[libIDL

] = libIDL

and

selected[glibc

] = glibc

selected[gtk

] = gtk

selected[pkgconﬁg

] = pkgconﬁg

selected[libIDL

] = libIDL

Note that the paths of neither set are closed under the references relation. Thus, they are

not a valid set of paths that can be used as an input to a build. Indeed, the builder might

traverse each path to arrive at its references; if we pick the ﬁrst set, glibc

might still be

reached by following libIDL

Therefore, the paths in selected must be rewritten to a new set of paths that is closed

under the references relation, and still obeys the equivalence class unicity invariant. This

is done by the helper function maybeRewrite shown in Figure 6.4, which resolve calls for

151

6. The Intensional Model

each selected path 111 . maybeRewrite takes a path p in equivalence class e, and returns a

new store path p

new

in the same equivalence class e.

The basic idea is that the path p

new

is a copy of p produced by rewriting all references

of p that are not in selected, to equivalent paths that are in selected. Thus, we walk over

all references 113 , and for each reference p

ref

in equivalence class e

ref

that is not trivial

(i.e., a self-reference to p) or a source (there is no e

ref

) 114

, the equivalent path p

repl

selected[e

ref

] is chosen 115 . That is, the reference is replaced with the path that was selected

for its equivalence class. All these p

ref

s together form the set of new references newRefs.

If the new set of references is not equal to the old 118 , a set of rewrites is applied to the

contents of p that replaces the hash part of each p

ref

with the hash part of the corresponding

repl

117 . This rewriting is done by the function addToStore

(explained below), and yields

a new store path p

new

119

There is a ﬁnal complication, though; the selected replacement reference p

repl

may itself

not be closed with respect to selected, and therefore in need of rewriting. Thus, p

repl

itself

is rewritten using maybeRewrite 116 , and it is the resulting path p

repl

that is actually used

in the new set of references and for the hash rewrites.

The pseudo-code of maybeRewrite shown in Figure 6.4 will typically perform many

redundant recursive calls. This is safe, since the implementation of addToStore

is idem-

potent. In the actual implementation, simple memoisation is used to remember for each

path p in equivalence class e the corresponding rewritten path p

new

In essence, maybeRewrite performs a bottom-up rewrite on the graph of selected paths.

Let us illustrate this using our example. Suppose that we have selected the set {(glibc

glibc

), (gtk

,gtk

), (pkgconﬁg

,pkgconﬁg

), (libIDL

,libIDL

)}. We ﬁrst call may-

beRewrite on (glibc

,glibc

). This path has no references, so (glibc

,glibc

) is returned.

Next, we call maybeRewrite on (gtk

,gtk

). This path does have a reference, namely,

glibc

in the equivalence class glibc

. We replace this reference with the path in selected

in equivalence class glibc

, which happens to be glibc

itself. Recursively applying may-

beRewrite to (glibc

,glibc

) yields (glibc

,glibc

). Thus, nothing changes in the refer-

ences, and (gtk

,gtk

) is returned. The same happens with (pkgconﬁg

,pkgconﬁg

However, when we call maybeRewrite on (libIDL

,libIDL

), things get more interesting

since we do need a rewrite. This is because its sole reference, glibc

is replaced with

glibc

. Therefore, a copy is made of libIDL

, in which the occurrences of hashPart(glibc

)

are replaced with hashPart(glibc

). This yields a new path, libIDL

, which has a reference

to glibc

instead of glibc

. In summary,

maybeRewrite(glibc

, glibc

, selected) = (glibc

,glibc

)

maybeRewrite(gtk

, gtk

, selected) = (gtk

,gtk

)

maybeRewrite(pkgconﬁg

, pkgconﬁg

, selected) = (pkgconﬁg

,pkgconﬁg

)

maybeRewrite(libIDL

, libIDL

, selected) = (libIDL

,libIDL

)

Thus the set returned by resolve for our running example is {(glibc

,glibc

), (gtk

,gtk

(pkgconﬁg

,pkgconﬁg

), (libIDL

,libIDL

)}. Note that this set is closed under the refer-

ences relation since libIDL

references glibc

, not glibc

It may not be obvious at ﬁrst glance that resolve is correct, i.e., that it produces a set of

paths that obeys the equivalence class unicity invariant, and is closed. The ﬁrst property is

152

6.4. Semantics

trivial, since selectPaths by deﬁnition yields a set selected that obeys the invariant, and we

call maybeRewrite once for each (p,e) ∈ selected, yielding a path in the same equivalence

class. The second property is more subtle, and is proved as follows.

Theorem 5 (Closure of resolved set). The paths in the set paths

returned by the function

resolve in Figure 6.3 are closed under the references relation.

Proof. The proof is by structural induction. Note that maybeRewrite is called for each

(p,e) in selected. The induction hypothesis is that the closure of every path returned by a

call to maybeRewrite for a path p in equivalence class e is in paths

There is one base case: if a path (p,e) ∈ selected has only trivial or source references,

maybeRewrite returns (p, e) unmodiﬁed, so (p,e) ∈ paths

. Any non-trivial references are

source references and explicitly included in paths

at 112 . Thus the hypothesis holds.

The inductive step is as follows. If a path (p,e) ∈ selected has non-trivial references,

then it is rewritten to a new path (p

new

,e). Each non-trivial, non-source reference p

ref

of p in equivalence class e

ref

is rewritten by a recursive call to maybeRewrite on p

repl

selected[e

ref

], i.e., the selected replacement for (p

ref

). Note that (p

repl

ref

) ∈ selected,

which by the induction hypothesis means that the closure of p

repl

is in paths

Since this is the case for all p

repl

, the closures of all references of the path p

new

produced

by the call to addToStore are in paths

, and since p

new

is returned and added to paths

, the

closure of p

new

is also in paths

6.4.2. Adding store objects

Figure 6.5 shows the intensional version of the function addToStore that adds an atom

(an FSO) to the Nix store. It is merely a wrapper around a more powerful function,

addToStore

, that writes an FSO to the Nix store at its proper content-addressed location.

addToStore

is the only primitive operation that adds valid paths to the store in the inten-

sional model. This is in contrast to the extensional model, where the functions build and

substitute also produce valid paths. It takes six arguments:

• fso is the FSO to be written to the store.

• refs is the set of referenced store paths of the FSO.

• eqRefs is a set of tuples (p

ref

,e,e

ref

) that describe elements to be added to the ref-

Classes mapping. Each tuple signiﬁes that, given a resulting path p, the entry (e,e

ref

)

should be added to refClasses[(p, p

ref

)].

• name is the symbolic name to be used in the construction of the store path.

• rewrites is a set of hash rewrites (i.e., h

) to be applied to the FSO and to the

references refs and eqRefs.

• selfHash is the hash part of the path from which the FSO is being copied. As dis-

cussed in Section 6.3.2, self-references must be zeroed out when computing the

cryptographic hash of the FSO, and must be replaced with the hash part of the new

store path of the FSO. For atoms (such as sources), which originate from outside of

153

6. The Intensional Model

addToStore(fso, refs, name) :

return addToStore

(fso, refs, /0, name, /0, ε)

addToStore

(fso, refs, eqRefs, name, rewrites, selfHash) :

c ← replace(serialise(fso), rewrites) 120

if selfHash 6= ε :

h ← hashModulo(c, selfHash)

121

else :

h ← hash

sha256

(":" + c) 122

p ← makePath(h, name) 123

if valid[p] = ε :

lock ← acquirePathLock(p)

if valid[p] = ε :

if selfHash 6= ε :

c ← replace(c, {selfHash hashPart(p)}) 124

rewrites

∪

← {selfHash hashPart(p)}

writePath(p, deserialise(c))

in a database transaction :

valid[p] ← h

refs ← {replace(p, rewrites) | p ∈ refs}

125

setReferences(p, refs)

releasePathLock(p, lock)

for each (p

ref

,e,e

ref

) ∈ eqRefs :

refClasses[(p,replace(p

ref

, rewrites))]

∪

← {(e,e

ref

)} 126

return p

Figure 6.5.: addToStore: Adding store objects to a content-addressable store

the Nix store, no such self-reference is present in the FSO; the dummy value ε must

then be provided for selfHash. But when copying store paths, such as in the collision

resolution algorithm (Figure 6.3) or in the build algorithm (Figure 6.6), selfHash is

set to the hash part of the source path.

The function addToStore

works as follows. First, the hash rewrites are applied to the

serialisation of the FSO 120 . Next, the hash modulo self-references is computed using

the function hashModulo (page 144) 121 . However, this is only applicable when copying

inside the store, that is, when selfHash is not ε. If it is ε, the hash is computed normally

over the serialisation, but with a colon prepended 122 . This exactly matches a hashModulo

computation over contents that contain no self-references, since in that case we have

hashModulo(c, hp) = hash

sha256

(

∑

i∈ﬁnd(c,hp)

(i + ":") + ":" + replace(c, {hp 0}))

= hash

sha256

(

∑

i∈/0

(i + ":") + ":" + replace(c, {hp 0}))

= hash

sha256

(":" + c)

154

6.4. Semantics

and so selfHash is not needed.

The hash and symbolic name are used to construct a store path p 123 . Then, just as in

the extensional version of addToStore, we check if p is already valid, and if not, acquire

an exclusive lock on p and recheck for validity. If the path is still not valid, selfHash

is rewritten to hashPart(p) (if applicable) 124 , and the FSO is written to p. The path

is then registered as valid. But note that the hash rewrites that have been applied to the

contents c are also applied to the set of references 125

. This ensures, for instance, that the

old references passed in maybeRewrite 119 are changed to the new references. Likewise,

appropriate refClasses entries are added as described above 126 .

6.4.3. Building store derivations

Figure 6.6 shows the build algorithm for the intensional model. As in Section 6.2, we

assume that all operations on the Nix store are done by a privileged user on behalf of the

actual users, who do not have write access themselves.

The most important differences with the build algorithm for the extensional model (Fig-

ure 5.11) are as follows. Since there is no longer a single output that can be substituted or

checked for validity, we have to consider all paths in the output equivalence class. Thus,

we query the set of trusted paths in the equivalence class d.eqClass 127 . We then check

whether any of those paths are valid; if so, we’re done 128 . If not, we try to create one

through substitutes 129 .

If all of the substitutes fail, we have to perform the build. We ﬁrst recursively build

all input derivations. Just as in the extensional build algorithm, the set inputs of input

paths is computed. However, this set may contain equivalence class collisions. Thus,

those conﬂicts must be resolved 130 . The resulting set inputs is a closure that obeys the

equivalence class unicity invariant.

Recall that the envVars, args and builder ﬁelds of the derivation cannot refer to the output

paths of the actual input derivations, because those are not yet known at instantiation time.

Rather, these ﬁelds refer to the output equivalence classes of the input derivations. Now

that we have actual paths for the inputs, we can rewrite the hash parts of the equivalence

classes to the hash parts of the actual paths 131 .

The build is performed with the environment variable out set to a temporary path in

the Nix store 132 . The temporary output path is computed by replacing the hash part of

the output equivalence class with a random hash part of the same length, as described in

Section 6.3.2.

After the build, we copy and rewrite the temporary path to its ﬁnal, content-addressable

location in the Nix store 133 . The temporary path can now be deleted. The new store path

is registered as being inside the equivalence class d.eqClass 134 .

It is worth noting that the intensional build algorithm does not perform locking. It does

not have to. The random temporary path can be assumed to be unique if the random-

number generator is sufﬁciently cryptographically strong, or we can simply check whether

it already exists, and if so, pick another path. Since the temporary path is unique, there can

be no interference from other Nix processes building the same derivation. The absence of

locking has the advantage that it simpliﬁes the implementation and prevents a malevolent

user from causing a livelock (e.g., by registering a substitute for a path that does not termi-

nate). On the other hand, it may also create work duplication if multiple processes perform

155

6. The Intensional Model

build(p

drv

) :

if ¬ substitute(p

drv

) : abort

d ← parseDrv(readPath(p

drv

))

// Note: curUser is the invoking user.

trusted ← trustedPaths(d.eqClass, curUser) 127

for each p ∈ trusted : if valid[p] 6= ε : return p 128

for each p ∈ trusted : if substitute(p) : return p 129

// Gather all trusted input closures, then resolve.

inputs ← /0

for each p ∈ d.inputsDrvs :

← parseDrv(readPath(p))

build(p)

for each p

∈ trustedPaths(d

.eqClass, curUser) :

inputs

∪

← closure

, d

.eqClass)

for each p ∈ d.inputsSrcs :

inputs

∪

← closure

(p, ε)

inputs ← resolve(inputs)

130

// Rewrite equivalence classes to real paths.

mapping ← /0

for each (p,e) ∈ inputs :

mapping

∪

← {hashPart(e) hashPart(p)}

Apply the rewrites mapping to d.envVars, d.args and d.builder 131

// Build in a temporary path.

output ← replace(d.eqClass, {hashPart(d.eqClass) randomHash()})

d.envVars["out"] ← output 132

Run d.builder in an environment d.envVars

and with arguments d.args and in a temporary directory p

tmp

Canonicalise output & perform ﬁxed-output derivation tests as in Figure 5.11

// Copy to content-addressed location and register as valid.

refs ← scanForReferences(output, {p | (p, e) ∈ inputs})

in a database transaction :

output

← addToStore

(readPath(output), refs,

{(p

ref

,d.eqClass, e

ref

) | (p

ref

) ∈ inputs ∧ p

ref

∈ refs},

namePart(d.eqClass), /0, hashPart(output)) 133

eqClassMembers[d.eqClass]

∪

← {(curUser, output

)} 134

return output

Figure 6.6.: build: Building store derivations in the intensional model

156

6.4. Semantics

exactly the same build. This is safe, however: even if multiple builds produce exactly the

same outputs, the idempotency and transactional semantics of addToStore

ensure that the

resulting path is not written to twice.

Builder write permission A trivial but annoying problem is how we can allow builders

to create their output path, i.e., the path speciﬁed in the environment variable out. After

all, the builder is not supposed to have write permission to the Nix store! Thus a builder

should not be allowed to perform:

mkdir $out

since that implies that it can also create other paths in the store. The prototype implemen-

tation of the intensional model does not yet solve this problem (builders still have arbitrary

write permission to the store). There are a number of possible solutions. On Unix, the Nix

store directory could be given the sticky permission bit (S_ISVTX) [152], which prevents

users from deleting or renaming directories not owned by them. Under the secure local

build scheme from Section 6.2, the builder runs under a unique uid that differs from the

uid that owns the valid FSOs, and from the uids of other concurrently running builders.

Thus, the builder can create other paths than its expected output, but these can be deleted

afterwards, and it cannot modify valid paths.

Another solution is to have the calling Nix process create the output path on behalf of

the builder. The builder then does not need write permission to the store directory itself.

However, Nix must know the type of the output path (i.e., regular ﬁle, directory, or symlink)

ahead of time. Finally, the builder could ask the calling Nix process to create the output

path of the appropriate type, e.g., by talking to it through a socket.

6.4.4. Substitutes

Just as the table eqClassMembers ties equivalence class membership to users, we extend

the substitutes mapping (ﬁrst deﬁned in Section 5.5.3) with a UserId for each substitute to

indicate which user registered it, i.e., substitutes : Path → {(UserId, Path,[String],Path)}.

This means that a substitute is a 4-tuple consisting of the ID of the user who registered the

substitute, the path of the substitute program, its command-line arguments, and the deriver

(page 114). For example,

substitutes[/nix/store/bpv6czrwrmhr......-hello-2.1.1] =

{ ( "alice", "download-url.sh"

, ["http://example.org/0n4laxf6kq44....nar.bz2"]

, "/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv" )

, ( "bob","download-url.sh"

, ["http://example.org/r35w1y8xll12....nar.bz2"]

, "/nix/store/1ja1w63wbk5q...-hello-2.1.1.drv" )

}

Adding a UserId is not strictly necessary to ensure the trust property. After all, due to

content-addressability, if a substitute for path p fails to produce a path with a content hash

157

6. The Intensional Model

Bool substitute(p) :

if valid[p] 6= ε : return true

subs ← trustedSubs(p, curUser) 135

if subs = /0 : return false

for each p

∈ references[p] :

if ¬ substitute(p

) : return false

for each (user, program, args, deriver) ∈ substitutes[p] :

tmp

← replace(p, {hashPart(p) randomHash()}) 136

if execution of program with arguments [p

tmp

] + args exits with exit code 0 :

assert(pathExists(p

tmp

))

← addToStore

(readPath(p

tmp

), references[p], /0

namePart(p

tmp

), /0, hashPart(p

tmp

)) 137

if p = p

: return true 138

if fall-back is disabled : abort

return false

Figure 6.7.: substitute: Substitution of store paths in the intensional model

that corresponds with hashPart(p), the substitute can simply be considered to have failed,

and its output in p can be deleted. However, we do not want users to trigger other users’

substitutes, as that can lead to privacy violations. For instance, Alice could register a

substitute that as a side effect informs her when other users try to install certain paths.

Another necessary change is that when a substitute for path p is registered, refClasses

entries for the references of p must also be registered. This means that the command nix-

store --register-substitutes (Section 5.5.3) must be extended to accept a list of (p

ref

,e,e

ref

)

entries that can be added to refClasses.

Figure 6.7 shows the substitution function substitute(p). The main difference with the

extensional version (Figure 5.15) is that the substitute program must produce its output in a

temporary path p

tmp

136 which is then copied (using addToStore

) to its content-addressed

location 137 . Note that it can simply pass /0 for the eqRefs argument of addToStore

, since

the refClasses entries for p have already been set when the substitute was registered.

If the resulting path p

does not match with the path p that the substitute is supposed to

produce, the substitute is ignored and the next substitute is tried 138 . Also, only substi-

tutes registered by the current user are considered 135 . Analogously to trustedPaths, the

function trustedSubs(p, user) yields from substitutes[p] those substitutes register by user.

Security of references When a substitute is registered, the caller must also provide a

list of references and refClasses entries for the path. However, that information may be

wrong—the user could supply garbage. For instance, the references supplied by the user

could be incomplete. This could cause incomplete deployment: a path p realised through

a substitute could have dangling pointers. However, this is only a problem if the path is

used by other users. Those users presumably ﬁrst register their own substitutes before they

158

6.5. Trust relations

build a derivation that produces p. As part of the substitute registration, bona ﬁde users

must supply a full set of references. A simple solution is therefore as follows: when users

be realised through substitutes immediately, then added to the references of p.

Likewise, the registered references can contain bogus paths, i.e., paths that are not ac-

tually references. This can lead to an unsubstitutable path, e.g., where a user cannot sub-

stitute path p because another user registered a bogus (unsubstitutable) reference p

ref

. A

simple solution is to substitute p before we substitute its references (but not before mak-

ing it valid, of course), then scan p to check what declared references actually occur. The

bogus references can then be tossed out.

6.5. Trust relations

The intensional model described in the previous section gives us a Nix store that can be

shared by mutually untrusted users, or users who have different remote trust relations. Due

to content-addressability, we get sharing between multiple builds of a derivation if each

build produces exactly the same binary result, that is, if there is no impurity in the build. If

there is impurity, then each build result will end up under a different store path.

Between untrusted users, this is exactly what we want. If Alice obtains substitutes from

a malicious machine, it does not affect Bob. Note that Alice and Bob do automatically get

sharing if they happen to get their substitutes from the same remote machine.

However, we wish to re-enable sharing in common scenarios. For instance, users gen-

erally trust components installed by the administrator. Thus, if Alice is an administrator,

then Bob should be able to use the output paths already installed by Alice. In general, users

should be able to specify trust relations between each other.

We can achieve this through a simple extension of the intensional model called the mixed

model. For each user, we maintain a mapping trustedUsers : UserId → {UserId} that spec-

iﬁes for each user a set of trusted users. E.g., trustedUsers["bob"] = {"alice","bob"}. (The

mapping should be reﬂexive, i.e., u ∈ trustedUsers[u].) We then augment the function

trustedPaths:

trustedPaths(eqClass, user) =

{p | ∃u ∈ trustedUsers[user] : (u, p) ∈ eqClassMembers[eqClass]}

Otherwise, this is exactly the intensional model. Of course, sharing between users in-

creases the possibility of equivalence class collisions, but that is handled through the reso-

lution algorithm in Section 6.4.1. The name “mixed model” does not imply that we back

away from intensionality—the store is still fully content-addressed. We just gain back the

ability to have sharing between users. The crucial difference with the extensional model is

that sharing is now selective and ﬁne-grained.

6.6. Related work

Nix’s transparent source/binary model is a unique feature for a deployment system. Rela-

tive to binary-only or source-only deployment models, it adds the complication that we do

159

6. The Intensional Model

not only need to authenticate binaries but also the fact that they are a bona ﬁde result of

certain sources. However, caching and sharing between users of build results is a feature

of some SCM systems such as Vesta [92].

As claimed in the introduction, deployment systems tend to have monolithic trust mod-

els. Typical Unix package management systems such as RPM [62], Debian APT, or Gentoo

Linux [77] allow installation of software by the administrator only; software installed by

individual users is not managed by those tools. On the other hand, Mac OS X application

bundles may be installed and moved around by any user, but the system does not track

dependencies in any way.

Security aspects of deployment have typically focused on ensuring integrity of compo-

nents in transit (e.g., by using signatures), and on assessing or constraining the impact of

a component on the system (e.g., [173]). We have not addressed the issue of ensuring

that remote substitutes have not been tampered with (e.g., by a man-in-the-middle). Obvi-

ously, such problems can be solved by cryptographically signing substitutes—or rather the

manifests, since the fact that substitutes themselves have not been tampered with is easily

veriﬁed by comparing their cryptographic hashes to their names.

Microsoft’s .NET has a Global Assembly Cache that permits the sharing of compo-

nents [154]. It is however not intended for storage of components private to an applica-

tion. Thus, if multiple users install an application having such private components, du-

plication can occur. Also, .NET has a purely binary deployment model, thus bypassing

source/binary correspondence trust issues.

In [80] a scenario is described in which components impersonate other components.

This is not possible in a content-addressable ﬁle system with static component composition

(e.g., Unix dynamic libraries with RPATHs pointing to the full paths of components to link

against, as happens in the Nix Packages collection).

Content-addressability is a common property of the distributed hash schemes used in

peer-to-peer ﬁle-sharing and caching applications (e.g., Pastry [144]). It is also used in the

storage layer of version management tools such as Monotone [95].

6.7. Multiple outputs

Apart from secure sharing, a major advantage of the intensional model is that it allows

derivations to have multiple outputs. Derivations as we have deﬁned them up till now

produce a single output, i.e., just one store path in the Nix store. However, in many cir-

cumstances it is very useful to allow a derivation to produce multiple outputs, i.e., multiple

store paths. This allows ﬁner-grained deployment, reducing the size in bytes of the closures

that need to be deployed to and stored on client machines.

For instance, the GNU C Library component, Glibc, contains parts that are not actually

needed at runtime by the components that depend on it. Among these parts is its subdirec-

tory /include that contains 2.8 MiB of C header ﬁles and is only needed at build time. If

we perform a binary deployment of a dependent component to clients, it is quite wasteful

that these header ﬁles are copied also. In fact, the Glibc header ﬁles contain a symlink to

a different component, the Linux Kernel headers, which take up 7.6 MiB. So the unneces-

sary Glibc header ﬁles drag along another entirely unnecessary component! Incidentally,

this is why the Hello example in Chapter 2 had a retained dependency on linux-headers

160

6.7. Multiple outputs

(page 41).

Multiple outputs are not yet implemented in Nix, so there is no concrete syntax for

specifying them in Nix expressions. A possible syntax is as follows. A derivation with

multiple outputs speciﬁes a list of symbolically named outputs in the attribute outputs.

The Glibc derivation would specify the following:

derivation { ...

outputs = ["out" "dev" "bin"]

}

This declares three outputs. The output out contains the bulk of the component, i.e., the

C library. The output dev contains development-only parts, such as the header ﬁles men-

tioned above. The output bin contains utility programs shipped with the C library, which

like the header ﬁles are not necessary for most users.

The effect of the outputs attribute is that the resulting store derivation speciﬁes not one

output equivalence class, but rather several, identiﬁed by their symbolic names, e.g.,

glibc

.eqClasses = { ("out", "/nix/store/jzhnqh7ffq72...-glibc-2.3.5")

, ("dev", "/nix/store/34z5jyqqs1sf...-glibc-2.3.5-dev")

, ("bin", "/nix/store/76zsvfw4zizs...-glibc-2.3.5-bin")}

Conversely, a way is needed for dependent components to refer to speciﬁc outputs. A

component foo can specify that it needs the out and dev outputs of Glibc, but (by omission)

not bin:

{glibc}: derivation { ...

glibc = glibc.out; # can be shortened to just "glibc"

glibcHeaders = glibc.dev;

}

To represent this in store derivations, the elements of the inputDrvs ﬁeld become tuples that

specify which output of an input derivation is needed for a build, e.g.,

foo

.inputDr vs = { (p

d,glibc

,{"out", "drv"}), ...}

where p

d,glibc

is the store path of the derivation containing d

glibc

. In fact, the on-disk ATerm

representation of store derivations already supports this extension (see Figure 5.8).

So why are multiple outputs impossible in the extensional model? Imagine that we

install Hello from source, and add it to a user environment. Hello has a reference to the

out output of Glibc, but not the bin and dev outputs. Thus, when we run the garbage

collector, those outputs will be deleted (if they were present to begin with). Suppose that

we subsequently build a derivation that has a dependency on (say) the bin output of Glibc.

Unless we have a substitute that builds that path, we now have an unsolvable problem. The

only way that we can obtain the path without a substitute is to build Glibc. But the Glibc

build would overwrite its own out output, which is already valid, and thus should never be

modiﬁed. (If the overwriting is atomic or otherwise unobservable, there is no problem, but

that is not the case in general.) So we need to delete the out output ﬁrst. But to delete it

violates the closure invariant, as that path is reachable from a user environment. So the Nix

161

6. The Intensional Model

store is “jammed”: we have reached a state where a build simply cannot be done because

it is blocked by the validity of some of its outputs.

In the intensional model, this problem does not exist. A build can always be done,

because the outputs go to temporary paths that are constructed on the ﬂy. We can simply

perform a build to obtain all of its outputs in temporary paths, copy the missing outputs to

content-addressed locations, and discard the outputs that we already have. In the example

above, we perform a full build of Glibc. The outputs end up in temporary paths p

out

, p

dev

and p

bin

. We ﬁnalise the previously missing outputs p

dev

and p

bin

using addToStore

, and

discard p

out

as we already have it. The FSOs in p

dev

and p

bin

might have references to

out

, but those can be rewritten to the path of the out output that we already had.

The outputs of a derivation can have references to each other, and in fact this is quite

common. For instance, it can be expected that the programs in the bin output of Glibc

depend on the libraries in the out output. This means that the out output is in the closure

of the bin output, but not vice versa. But what happens when there are mutually recursive

references, e.g., when out also refers to bin? These must be forbidden, since the hash

rewriting scheme from Section 6.3.2 cannot handle them. For instance, when we copy out

and bin to their content-addressable locations, we must rewrite in both FSOs the hashes

of both paths. The function hashModulo only handles direct self-references, and it can do

so because the hashes to be disregarded in the hash computation are encoded into the ﬁle

name.

Fortunately, banning mutually recursive outputs is not a very onerous restriction, since

they are pointless. After all, mutual recursion between output paths requires them to be

deployed and garbage collected as a unit, negating the granularity advantages that multiple

outputs are intended to achieve.

6.8. Assumptions about components

It is useful to make explicit at this point what assumptions Nix makes about compo-

nents, i.e., the requirements imposed on components in order to allow them to be deployed

through Nix. These assumptions are:

• The component can be built with and installed at an arbitrary preﬁx. Counterexample

of a component that violates this assumption: some Linux system components have

Makeﬁles that contain “hard-coded” paths such as /lib. (Such Makeﬁles are easily

patched, though.)

• Likewise, the component does not expect that its dependencies are installed at certain

ﬁxed locations, and the locations of the dependencies can instead be speciﬁed at

build or runtime. Counterexample: the GNU C Library, Glibc, has a hard-coded

dependency on the POSIX shell in /bin/sh.

• The component can be built automatically, i.e., with no user interaction whatsoever.

Counterexample: the Unreal Tournament 2004 Demo has an interactive installer

that among other things asks the user to approve a license agreement. (Nevertheless

it is in Nixpkgs, since we can bypass the installer by unpacking the TAR archive

embedded in the installer’s executable.)

162

6.8. Assumptions about components

• The build process of the component is essentially pure (i.e., deterministic), mean-

ing that the output of the build process is fully determined by the declared inputs.

Counterexample: any builder that depends on a mutable network resource. This is

why fetchurl checks the downloaded ﬁle against a pre-speciﬁed cryptographic hash.

Another counterexample: the archiving tool for object ﬁles (ar) on Mac OS X stores

timestamps in archives, and these must not be newer than the timestamp on the

archive itself; otherwise, the linker will refuse to use the archive. (Since Nix canoni-

calises timestamps to 1970 through canonicalisePathContents (page 112), i.e., back

in time, this is not a problem.)

• The component does not need to be modiﬁed after it has been built, e.g., at runtime.

Counterexample: Firefox 1.0 needs to be run interactively with write permission

to its installation directories so that it can create some additional ﬁles. (However,

there is an obscure, poorly documented solution for this that allows these ﬁles to be

generated non-interactively.)

• The component has no environment dependencies. For instance, the component

should not require global registry settings. It is however ﬁne to use automatically

created per-user conﬁguration ﬁles or state. In other words, it should not be nec-

essary to run a program to realise the required environment; the closure in the Nix

store should be the only dependency. Counterexample: system daemons (server pro-

cesses) that require the existence of certain user accounts.

• Retained dependencies can be found in the component through scanning by scan-

ForReferences. There is no pointer hiding.

The intensional model makes one additional assumption:

• Hash rewriting does not change the semantics of the component. For instance, the

component has no internal checksums that are invalidated by hash rewriting.

If these assumptions hold for a component, then it can be deployed through Nix. Of

course, this raises the obvious question of whether these assumptions are likely to hold for

components in practice. In the next chapter, we will see that this is indeed the case.

163

6. The Intensional Model

164

Part III.

Applications

165

7. Software Deployment

The purpose of the Nix system is to support software deployment. In Part II we have

seen the low-level mechanisms necessary to support deployment. This chapter deals with

the higher-level mechanisms and actual policies that enable support for a wide range of

deployment scenarios. It also serves as validation for the Nix approach, since as we have

just seen in Section 6.8, Nix makes a number of assumptions about components that may

not always hold in practice. To discover to what extent these assumptions are valid, we

have applied Nix to the deployment of a large set of pre-existing software components, the

Nix Packages collection.

This chapter ﬁrst discusses the Nix Packages collection and the design principles that

went into it, as these are instructive for the use of Nix in general.

Second, it shows concrete “high-level” deployment mechanisms. The nix-pull mecha-

nism is built on top of the low-level substitute mechanism to enable a variety of deployment

policies, such as channels and one-click installations. User environments are “pseudo-

components” that are constructed automatically by nix-env to activate components.

Third, Nix’s purely functional model creates some unique challenges in supporting the

evolution of software installations over time. For instance, users will want to periodi-

cally upgrade software components. In a binary deployment policy, it is undesirable if all

changed components must be re-downloaded in their entirety, particularly because in the

purely functional model all dependent components change as well. Section 7.5 shows that

this problem can be ameliorated using binary patching.

Finally, Section 7.6 compares Nix to other deployment systems, and to techniques that

might be used to implement deployment.

7.1. The Nix Packages collection

As we have seen in Chapter 2, the Nix Packages collection (Nixpkgs) is a set of Nix ex-

pressions that evaluate to derivations for a large set of pre-existing, third-party software

components. Figure 7.1 shows 278 components for which Nixpkgs contains expressions

For some of the components, Nixpkgs provides multiple versions. In general, we endeav-

our to standardise on a single version of a component within a given release of Nixpkgs,

but this is not always possible. For instance, several versions of the GNU C Compiler are

necessary because there is no single version of the compiler that can build all components.

Also, some library components have major revisions that are not backwards compatible

(e.g., versions 1 and 2 of the GUI library GTK). Of course, the great thing about the Nix

expression language is that it makes it so easy to deal with this kind of variability.

The components in Nixpkgs are a reasonable cross section of the universe of Unix soft-

ware components, though with a bias towards open source components. This is because

This set is based on nixpkgs-0.9pre3415.

167

7. Software Deployment

a52dec acrobat-reader alsa-lib ant-blackdown ant-j2sdk apache-httpd aterm aterm-dynamic aterm-

java atk audioﬁle autoconf automake bash bibtex-tools binutils bison bittorrent blackdown boehm-

gc bsdiff bzip2 cdparanoia chmlib cil cksfv clisp coreutils cua-mode curl db4 diffutils docbook-

xml docbook-xml-ebnf docbook-xsl e2fsprogs eclipse-sdk ed emacs enscript esound exif expat

ﬁle ﬁndutils ﬁrefox ﬂac ﬂashplayer ﬂex fontconﬁg freetype f-spot gail gawk gcc gcc-static GConf

gdk-pixbuf generator getopt gettext ghc ghostscript glib glibc gnet gnome-desktop gnome-icon-

theme gnome-keyring gnome-mime-data gnome-panel gnome-vfs gnugrep gnum4 gnumake gnu-

patch gnuplot gnused gnutar gperf gqview graphviz grub gtk+ gtkhtml gtkmozembed-sharp gtk-

sharp gtksourceview gtksourceview-sharp guile gwydion-dylan gzip happy harp haskell-mode he-

lium hello help2man hevea intltool iputils j2sdk jakarta-bcel jakarta-regexp jakarta-tomcat jclasslib

jdk jetty jikes jing-tools jjtraveler jre kaffe lame lcms lcov less libart_lgpl libbonobo libbonoboui

libcdaudio libdvdcss libdvdplay libdvdread libexif libglade libgnome libgnomecanvas libgnomeprint

libgnomeprintui libgnomeui libgphoto2 libgtkhtml libICE libIDL libjpeg libmad libogg libpng lib-

sigsegv libSM libtheora libtiff libtool libvorbis libwnck libX11 libXau libXaw libXext libXft libXi libX-

inerama libxml2 libXmu libXp libXpm libXrender libxslt libXt libXtrans libXv libXxf86vm lynx mesa

mingetty mjpegtools mktemp modutils mono monodevelop mono-dll-ﬁxer monodoc mpeg2dec

MPlayer mplayerplug-in mysql mythtv nano nasm ncurses net-tools nix nmap nxml-mode ocaml

octave openssh openssl ORBit2 pan pango panoramixext par2cmdline patchelf pcre perl perl-

BerkeleyDB perl-DateManip perl-HTML-Parser perl-HTML-Tagset perl-HTML-Tree perl-libwww-

perl perl-LocaleGettext perl-TermReadKey perl-URI perl-XML-LibXML perl-XML-LibXML-Common

perl-XML-NamespaceSupport perl-XML-Parser perl-XML-SAX perl-XML-Simple perl-xmltv perl-

XML-Twig perl-XML-Writer pkgconﬁg popt postgresql postgresql-jdbc procps pygtk python qt

quake3demo rcs readline RealPlayer renderext rte ruby saxon saxonb screen scrollkeeper sdf2-

bundle SDL shadow shared-objects sqlite stdenv-linux strace strategoxt strategoxt-utils subversion

swig sylpheed sysvinit tetex texinfo thunderbird uml uml-utilities unzip ut2004demo util-linux valgrind

vim vlc wget which wxGTK wxPython xchm xextensions xf86vmext xfree86 xine-lib xine-ui xlib xpf

xproto xsel zapping zdelta zip zlib zoom zvbi

Figure 7.1.: Components in the Nix Packages collection

these components are most readily available, and allow third-party use and conﬁrmation

of our results. However, some binary-only components are also included, such as Adobe

Reader, Real Player, Sun’s Java 2 SDK, and the Unreal Tournament 2004 Demo. How we

support such components is discussed in Section 7.1.4.

In terms of application areas, the set of components spans a wide spectrum:

• Fundamental tools and libraries (Glibc, GCC, Binutils, Bash, Coreutils, Make, ...).

• Other compilers and interpreters (Perl, Python, Mono, Sun’s Java 2 JDK, ...).

• GUI libraries (X11 client libraries, GTK, Qt, ...).

• Other libraries (Berkeley DB, libxml, SDL, and many more).

• Developer tools (Eclipse, Emacs, Valgrind, Subversion, Autoconf, Bison, ...).

• Non-developer tools (teTeX, NMap, ...)

• Servers (Apache httpd, PostgreSQL, MySQL, Jetty, ...).

168

7.1. The Nix Packages collection

• End-user applications (Firefox, Xine, Gaim, UT2004, ...).

Likewise, several programming languages are represented, including C, C++, Java, C#,

Haskell, Perl, Python, and OCaml.

Nixpkgs does not contain any of these components itself; it only contains Nix expres-

sions and builders. The source or binary distribution ﬁles for the components are obtained

using fetchurl calls. This is a policy decision; it is perfectly possible to include these ﬁles

in the Nixpkgs distribution itself—but that would make a Nixpkgs download several hun-

dreds of MiBs large (not to mention that it might violate several licenses).

Nixpkgs is severely biased towards Linux-based systems. Many of the components in

Nixpkgs also build on other Unix-based operating systems, such as FreeBSD and Mac

OS X, but as we shall see in Section 7.1.2, the standard environment on Linux is far supe-

rior to those on other platforms as it produces components that do not require any external

(non-Nix) components to build or run. For instance, it uses its own copy of the GNU C

Library (Glibc). In addition, to build the standard environment on Linux, no external com-

ponents are needed, i.e., it is fully self-contained. Signiﬁcantly, this means that Nixpkgs

on Linux is almost perfectly insulated from the “host” operating system. A component that

builds and runs on, say, Red Hat Enterprise Linux, will also build and run on SuSE Linux.

This bias towards Linux is not the result of any fundamental aspect of Nix, but rather is

triggered by two important considerations:

• The fundamental components of Linux systems

are all open source. This makes

them much more useful for experimentation and adaptation than closed-source com-

ponents. In contrast, in Mac OS X, many fundamental components (such as the GUI

libraries) are closed source, which means that we cannot build them in the Nix store.

• Linux is highly component-based, much more than any other system, in the sense

that the components that make up a Linux system are all separately maintained,

built, and distributed. This is probably bad for users, as it creates a lack of vision

and coordination between the various parts of the system, but it is very good for

us. Consider by contrast FreeBSD, which is also open source, but only available

as a uniﬁed whole. That is, components such as the kernel, C library, compiler,

basic Unix tools, system administration tools, documentation, etc., are all part of a

single distribution. This means that we cannot easily take out, say, the C library, and

use it in our standard environment. Also, since Linux components must support a

wide variability in the environments in which they are used, they are typically quite

conﬁgurable. For instance, Glibc can be installed at a non-standard preﬁx. So Linux

is a truly component-based system in the sense of components in Section 3.1.

But again, other Unix systems are also supported, if in a somewhat less perfect state.

The open source nature of Linux enables us to explore an “ideal”, uncompromising envi-

ronment on Linux, where everything is deployed through Nix. On other systems, we have

to allow a certain measure of impurity. For instance, components must use system libraries

that are not built or deployed through Nix.

Of course, we cannot really speak of Linux systems in general, as Linux is just a kernel. There is no single

component other than the kernel that is common to all Linux-based systems (not even Glibc!). However,

there is a fuzzy set of components that make up typical Linux environments, e.g., Glibc, GCC, Bash, GTK,

Qt, Gnome, KDE, and so on.

169

7. Software Deployment

So what about non-Unix systems, in particular Microsoft Windows? I claimed in the

introduction that Nix does not require any operating system speciﬁc extensions, such as

virtual ﬁle systems (as used by, e.g., Zero Install [112]). But Nix does make one Unix-

centric assumption: that paths abstract over storage devices. For binary deployment to

work, it is important that the store is in the same path between systems. That’s why we

more-or-less insist on using /nix/store. On Unix, this ﬁxed path is not a problem, because

the directory /nix can be mounted on any physical device. But on Windows, if we use a

path such as C:\nix\store, we encode the device on which the store resides (e.g., C:) into

the store location. If another user uses D:\nix\store, e.g., because her D: has sufﬁcient

free space, she cannot pull substitutes from a cache that assumes C:\nix\store. Thus we

run afoul of Windows’s CP/M heritage. (This does not apply to Cygwin [88] or similar

environments, which provide a Unix-like ﬁle system view on Windows.) However, it is a

little-known feature that Windows also provides mount points

, so this is in fact not a fatal

problem.

7.1.1. Principles

There are a number of design principles guiding the construction o f the components in

Nixpkgs. The most important of these is the following:

⇒ “Static compositions are good.”

A static composition is one that is established at component build time. A dynamic com-

position is one that is established at runtime, i.e., after the components have been built and

deployed.

Suppose that we have a component that needs to call a program foo, provided by another

component. Dynamic composition, or late binding, means that we expect that at runtime,

we can ﬁnd foo somewhere in the PATH environment variable, and so the composition

mechanism in C might be:

execlp("foo",

args

);

In a static composition, on the other hand, the path of foo is already known at build time:

execl("/nix/store/4sqiwbf0kj22...-foo-1.3.1/bin/foo",

args

);

Of course, the path /nix/store/4sqiwbf0kj22...-foo-1.3.1 would not be hard-coded into the

source. Rather, the foo component should be given as an input to the derivation, and the

path should be ﬁlled in by the preprocessor:

execl(FOO_PATH "/bin/foo",

args

);

So why is the latter better? The answer is that it gives a more exact closure and more

predictable runtime behaviour. The component will always be able to call foo, as it is

contained in the component’s closure and so is always present. With dynamic composition,

foo might be missing, or it might be a wrong version.

An important example of the preference of static composition over dynamic composition

is the style of linking used in Nixpkgs. Of course, the ultimate form of static composition is

static libraries (which are linked into the images of the executables that use them), but those

are inefﬁcient. They waste disk space, memory, disk cache space, and memory cache lines.

In fact, MS-DOS 3.1 already provided a mount-like command called JOIN.

170

7.1. The Nix Packages collection

So are we doomed to the dynamic composition risks of the alternative: dynamic linking?

In fact, as we have seen previously, this is not the case with Unix (ELF) style dynamic

libraries, as these also support static compositions through the RPATH, which speciﬁes a

search path for the dynamic linker.

For instance, the Hello component in Chapter 2 contains in its executable image a direct

reference to the path of Glibc. Likewise, the Subversion component contains the paths of

its OpenSSL, Expat, Berkeley DB, Zlib, and Glibc dependencies. Thus, the Subversion

program can never fail to ﬁnd its dependencies, and it can never accidentally link to the

wrong libraries at runtime. The latter is actually a substantial risk; it happens quite often

that users (when they manually compile software) have different versions of libraries in

different places (e.g., /usr/lib and /usr/local/lib) and that the library used at runtime does not

match with what was detected at build time.

Of course, static composition is at odds with the desire to change compositions later

on. An advantage of dynamic linking over static linking is that dynamic libraries can be

replaced with “better” versions, e.g., to ﬁx bugs; and all dependent applications will then

automatically use the new version. However, this provides two conﬂicting features:

• The ability to ﬁx dependent applications all at once.

• The ability to break dependent applications all at once.

Clearly there is some tension between these two.

In the purely functional model, destructive upgrading is impossible by deﬁnition, and

so this “all at once” semantics is simply not available. To deploy a new version of some

shared component, it is necessary to redeploy Nix expressions of all installed components

that depend on it. This is expensive, but I feel that it is a worthwhile tradeoff against

correctness. Section 7.5 will show how this kind of upgrading can be done with reasonable

efﬁciency.

⇒ “Static compositions are good. Late static compositions are better.”

A late static composition is when composition is delayed to allow components to be

reused between compositions. Consider for instance the Mozilla Firefox component, a

web browser. Firefox can be extended through several means. For instance, plugins are

dynamically linked libraries that allow Firefox to handle additional MIME types, e.g.,

Macromedia Flash ﬁles through the Flash Player plugin. We can establish the composition

between Firefox and its plugins when we build Firefox. So its Nix expression will look

like this:

{stdenv, fetchurl, ..., plugins ? []}:

stdenv.mkDerivation { ...;

inherit plugins;

}

and its builder must somehow embed the paths of the plugins into the Firefox executable.

Thus the call

import .../firefox.nix { ...

plugins = [flashplayer realplayer MPlayer java];

}

171

7. Software Deployment

flashplayer

firefox

realplayer MPlayer java

Figure 7.2.: Static Firefox composition

firefox

firefoxWrapper

flashplayer realplayer MPlayer java

Figure 7.3.: Late static Firefox composi-

tion

yields the composition shown in Figure 7.2 (an arrow A → B denotes that A is an input of

B). Now if we want to change the composition, e.g., add a new plugin, the entire Firefox

component must be rebuilt, which takes an hour or so on current hardware.

Figure 7.3 shows a better way through late static binding: the plugins are not composed

with the Firefox component directly, but rather the composition is delayed and established

in a Firefox wrapper component. A wrapper component is typically just a shell script that

sets some environment variables, then calls the original (wrapped) program. Thus, the

ﬁrefoxWrapper component consists entirely of a shell script generated by its builder

#! .../sh

export MOZ_PLUGIN_PATH=

realplayer-path

/lib/plugins:...

exec

original-firefox-path

/bin/firefox "$@"

Since the wrapper component can be generated very quickly, changing the composition

is very cheap.

Note that dynamic binding of the Firefox plugins requires the user to set the environment

variable MOZ_PLUGIN_PATH manually prior to invoking the Firefox program. In the late

static binding example, the variable is set by the wrapper script, which resides in the Nix

store and is therefore part of the closure installed in the user environment.

The terms (late) static binding and dynamic binding are orthogonal to the usual notions

of early binding and dynamic or late binding in component-based development [154]. They

are not about binding time relative to the deployment process, for either can be done at the

deployer side and on the client side. They are about whether the composition is controlled

(i.e., described by a self-contained store derivation that results in a self-contained closure)

or uncontrolled (i.e., performed by means outside of the scope of Nix).

⇒ “User environments are not a composition mechanism.”

User environments can be used as a composition mechanism, since they bring applica-

tions together in the PATH of the user, where they can ﬁnd each other through dynamic

composition. But that is not how they should be used in general. Before we know it, users

will be given installation instructions like this: “Application foo needs application bar, so

you should run nix-env -i foo bar.” This is an abuse of user environments. If there is a

dependency, it should be expressed in a Nix expression that for instance builds a wrapper

around foo that allows it to ﬁnd bar.

There are pragmatic exceptions. We do not currently have a good handle on crosscutting,

conﬁgurable dependencies. An example is the EDITOR environment variable, honoured

by many Unix programs, that speciﬁes the editor preferred by the user. (For instance,

Nixpkgs contains code to generate wrappers automatically.

172

7.1. The Nix Packages collection

Subversion calls the program speciﬁed by this variable to allow the user to write commit

messages.) Wrapper scripts are not currently a feasible solution, since if the user wishes

to change his editor, he must rebuild all wrappers. Another example is fonts for X11

applications, which we want to conﬁgure globally; we don’t want to have a set of fonts as

a dependency of each X11 application.

The following is not so much a current principle as a hope for a better future:

⇒ “Scoped composition mechanisms are good.”

A mechanism is scoped if it limits visibility of symbols. For instance, if A imports B, and

B imports C, then the interface of C should not be visible to A (unless A explicitly imports

C). For very many composition mechanisms this is not the case. For instance, consider the

Subversion example again (Figure 2.9). Its Nix expression is a function that takes many

optional dependencies:

{ ..., bdbSupport ? false, httpServer ? false, ...

, openssl ? null, httpd ? null, db4 ? null, expat, zlib ? null

Suppose that we call this function with httpServer = true and bdbSupport = false. Then

Subversion should not be built with Berkeley DB support. Unfortunately, there is a good

chance that it will be, even though the db4 derivation attribute is empty. The reason is

that Apache may be built with Berkeley DB support, and Subversion’s build process will

sneakily ﬁnd Berkeley DB through Apache. For instance, Subversion’s conﬁgure script

asks the Apache subcomponent apr-util what compiler and linker ﬂag it needs to properly

use apr-util, and gets the following (approximate) results:

$ apu-config --includes

-I/nix/store/h7yw7a257m1i...-db4-4.3.28/include

$ apu-config --ldflags

-L/nix/store/h7yw7a257m1i...-db4-4.3.28/lib

If Subversion’s conﬁgure script then dutifully adds these ﬂags to its own list of compiler

and linker ﬂags, it will ﬁnd Berkeley DB despite it not having been passed in as an explicit

dependency.

This does not violate Nix’s property of preventing undeclared dependencies in a techni-

cal sense. After all, Berkeley DB is in the input closure of the Subversion build process,

since it is in the closure of the Apache output. But in a logical (or “moral”) sense, it is an

undeclared dependency.

Unscoped compositions mechanisms are not a Nix-speciﬁc problem; far from it. For

instance, every conscientious C programmer who cares about header ﬁle hygiene has ex-

perienced that the C header ﬁle mechanism is unscoped: if we have a C ﬁle that includes

header ﬁle X, and X in turn includes header ﬁle Y , then our ﬁle gets all the deﬁnitions in

Y . This makes it virtually impossible to validate whether the C ﬁle will compile under

all circumstances. Consider the case where X is included conditionally, but we uncondi-

tionally need the deﬁnitions from Y . (A result is that programmers tend to include more

dependencies than perhaps required, leading to increased build times [41].) Likewise, the

only way that we can validate whether the derivations involving Unix libraries produced

by a Nix function will build for all function parameters, is to build all possible derivations.

173

7. Software Deployment

A build can prove the presence of a dependency problem, not its absence, to paraphrase

Dijkstra [44].

⇒ “Fine-grained components are better than large-grained components.”

For the purpose of software deployment, ﬁne-grained components are preferable to

large-grained components since the latter tend to lead to unnecessary dependencies and

thus to unnecessarily large closures. This was already observed in [43]. Large-grained

components also reduce software reuse [42], although that is not a central concern here.

Consider for example the Python interpreter package. This package contains not just an

interpreter for the Python language, but also many Python modules. Some of these are op-

tional, as they can only be built if certain dependencies are present, e.g., wrapper modules

for database libraries, the NCurses and Readline libraries, and so on. Thus, if we want to

build a Python instance that supports all possible uses, we have to make it dependent on

all these external libraries, even if most components that use Python don’t actually need

most or all of the optional functionality. Thus, the closure size of all components that use

Python increases.

In contrast, the Perl interpreter package has far fewer optional dependencies, and Perl

wrapper modules for the aforementioned modules are distributed as separate components.

For instance, support for the Berkeley DB 4 database is provided in a separate package.

Thus, a Perl component that does not require DB 4 support will not end up with the Berke-

ley DB 4 component and the Perl wrapper module in its closure.

In Nixpkgs, a strong preference is therefore given to small-grained components. For

instance, Nixpkgs initially used the “XFree86” package to provide X11 client libraries

for GUI applications. However, XFree86 is a large, monolithic distribution that contains

not only the X11 client libraries, but also the X11 server, drivers, and utilities. Further-

more, many client libraries are not actually needed by most X clients. So it was a dis-

tinct improvement when we were able to replace XFree86 with the X.org modular X li-

braries [182], which provide each individual library as a separately deployed entity. Thus,

each X application in Nixpkgs has in its closure only the libraries that it actually needs.

Of course, small-grained components also have a downside: they increase composition

effort. So one certainly should not feel compelled to make components as small as possible,

just to improve reuse. Rather, from the deployment perspective, the presence of external

dependencies should guide decomposition of large-grained components. Also, the quality

of small components is not necessarily better than that of large components [89].

7.1.2. The standard environment

Of all the components in Nixpkgs, the standard environment (stdenv) has a special signiﬁ-

cance as it is used as an input by almost all other components. It is in essence an aggregate

of a number of components that are required by the build processes of almost all other

components. It is very inconvenient to specify them separately for each derivation that

uses them, and so they are combined into stdenv. The included components are:

• GCC, the GNU C Compiler, conﬁgured with C and C++ support. It has been patched

to help ensure purity, as described below in Section 7.1.3

174

7.1. The Nix Packages collection

• GNU coreutils, which contains a few dozen standard Unix commands.

• GNU ﬁndutils, which provides ﬁnd.

• GNU diffutils, which provides diff.

• The text manupulation tools GNU sed, GNU grep, and GNU awk.

• GNU tar, necessary for unpacking sources in TAR format.

• The compression utilities gzip and bzip2, necessary for uncompressing sources.

• GNU Make, an implementation of the make command.

• Bash, an implementation of the POSIX shell. It provides many extensions that the

standard environment does depend on.

• Patch, a tool for applying deltas produced by diff.

In addition, the standard environment provides a shell script called setup that initialises

the environment so that all these tools are in the builder’s path. Thus, the line

source $stdenv/setup

at the top of a builder sufﬁces to bring them into scope.

Furthermore, the inputs speciﬁed in the attribute buildInputs are brought into scope,

where “scope” depends on the build tools being used. By default, the bin subdirectory of

components is added to PATH, the include subdirectory is added to the C compiler’s search

path, and the lib subdirectory is added to the linker’s search path. But this can be extended

through setup hooks that components can optionally provide. For instance, if Perl is in

scope, then all lib/site_perl subdirectories of components will be added to Perl’s module

search path. For instance, the attribute

buildInputs = [perl perlXMLWriter];

will cause the bin directory of the perl derivation’s output to be added to PATH, and the

lib/site_perl directory of perlXMLWriter (the Perl module XML::Writer) to be added to Perl’s

search path.

The generic builder The setup script also provides the generic builder, which is a shell

function genericBuild that can automatically build many software components. An exam-

ple of its use was shown in Figure 2.10. A full description of the generic builder and the

ways in which it can be customised is given in the Nix manual [161], and its details are not

relevant here. Brieﬂy, the generic builder has a number of phases:

• The unpack phase unpacks the sources denoted by the attribute src (which may be a

list). The current directory is switched to the top-level directory of the source tree.

• The patch phase applies the patches denoted by the attribute patches. This is useful

to maintain Nixpkgs-speciﬁc modiﬁcations to a component’s source.

175

7. Software Deployment

• The conﬁgure phase brings the source into a buildable state by running the compo-

nent’s conﬁgure script with the argument --preﬁx=$out.

• The build phase builds the component by running make.

• The check phase runs make check to perform any tests (such as unit tests) included

in the component.

• The install phase installs the component into the store path denoted by the environ-

ment variable out by running make install.

Each of these phases can be overridden by the caller to support components whose build

system does not follow the general pattern. In addition, the default implementations of

the phases can be customised in many ways (e.g., additional ﬂags can be speciﬁed for the

conﬁgure and make invocations).

Logging A builder can write arbitrary text to its standard output and standard error ﬁle

descriptors [152]. Nix combines these streams and writes the text to its own standard error,

as well as to a per-derivation log ﬁle in the directory /nix/var/nix/log/drvs.

A practical problem in Unix software development that seems to become more painful

with each passing year is the log output generated by build processes. Gone are the days

that an invocation of make only printed easily-understood lines such as

cc -c foo.c

and if anything else appeared, it was a relevant error message. The use of tools such as

Automake and Libtool [172] and Pkgconﬁg [75] routinely leads to single Make actions that

are thousands of characters and dozens of lines long, and in which it is almost impossible to

see what is being compiled and how. Also, the size of the log output can be overwhelming:

the Firefox builder produces a log of 6.4 MiB. This makes it often difﬁcult to pinpoint the

cause of a build failure.

For this reason, the generic builder produces structured or nested log output. This means

that the log output becomes an ordered tree of log lines (i.e., the children of each node are

ordered) rather than a list. For instance, the output of each phase of the generic builder is a

subtree of the root. Furthermore, I patched GNU Make to produce nested output. The log

messages of recursive invocations of Make form a subtree, and the output of each Make

action is placed in a clearly labelled subtree.

Subtrees are delimited by ECMA-48 control sequences [53]. This allows nesting to be

easily retro-ﬁtted onto “legacy” tools. For instance, the byte sequence 1b 5b 33 70 (in

hexadecimal) or "\e[3p" (as a C string) begins a subtree with priority 3 (higher priority

numbers mean that the subtree contains less important information), while the byte se-

quence 1b 5b 71 or "\e[q" ends a subtree. The sequence 1b 5b is the ECMA-48 Control

Sequence Introducer (CSI), which according to the standard must be followed by any num-

ber of Parameter Bytes in the hexadecimal range 30–3f, and a single Final Byte in the range

40–7e. An ECMA-48 conformant terminal will not display the control sequences used for

log nesting since it does not know how to interpret them. Thus, when the log output is

echoed to a terminal, it appears as if no control sequences are present.

176

7.1. The Nix Packages collection

<nest><head>unpacking...</head>

<line>...</line> ...

</nest>

<nest><head>building...</head>

...

<nest><head>Entering directory `src'</head>

<nest><head>building foo.o</head>

</nest>

</logfile>

Figure 7.4.: Structured log output converted to XML

The structured log output (containing escape sequences) can be processed in a variety

of ways. One such way is the tool log2xml (included in Nix), which converts a log ﬁle

into an XML ﬁle [20], an example of which is shown in Figure 7.4. This XML ﬁle can

then be transformed into something useful using, e.g., XSLT [27]. For instance, there is

an XSL stylesheet log2html that produces a nicely formatted HTML page of the log, using

JavaScript and Dynamic HTML to allow parts of the tree to be collapsed or expanded. Only

subtrees that contain error messages are expanded by default, allowing the developer to ﬁnd

relevant parts of the log easily. An example produced by the Nix build farm (Chapter 8) is

shown in Figure 7.5.

Bootstrapping the standard environment The standard environment is used by the

build processes of almost all other components in Nixpkgs, which raises the question:

how do we build the standard environment itself? To build the components in stdenv, we

need a C compiler, standard Unix tools, a shell, and so on. In other words, we need a

stdenv to build stdenv. Thus, we have a classic bootstrapping problem.

The approach initially used, and still used to build stdenv on all platforms except stdenv-

linux, is to build stdenv using the “native” tools of the system, e.g., the C compiler in

/usr/bin/gcc. This is of course impure: it depends on inputs outside of Nix’s control. Thus,

a build of stdenv may not always succeed (e.g., if the C compiler is missing). However,

once stdenv has been built, we can build components with no outside interference. The

build process of stdenv has several stages:

• First a trivial initial standard environment is built (using /bin/sh for the builder) that

constructs a setup script that sets the PATH to /usr/bin, etc.

• The initial stdenv is used to build GCC.

• A second stdenv is then built, identical to the ﬁrst one, except that GCC is added to

the PATH.

• This stdenv is used to build all other stdenv components.

177

7. Software Deployment

Figure 7.5.: Structured log output

• From these components the ﬁnal stdenv is constructed. It has a “pure” PATH.

The stdenv thus produced has a compiler and linker that produces binaries that link

against the system’s C library. This is still impure, but on many systems we do not have

the ability to build our own copy of the C library. If we do, as on Linux, the build process

has a few extra steps. We ﬁrst use the trivial stdenv to build the C library. Then we build

GCC such that it links dynamically against the C library, since we want all programs in

the ﬁnal stdenv to use our own C library instead of the system C library. A stdenv is then

constructed that has environment variables set up so that the compiler and linker use our

own C library.

The ideal however is a completely pure stdenv: one that requires no external components

to build and run, and produces components that need no external components to run. Such

a stdenv has been implemented on i686-linux. Bootstrap tools such as a shell, compiler,

and so on, are provided as “sources”, or more properly, atoms (see Section 5.3). These

programs are all statically linked executables so that they do not need external components.

Roughly, the builder for stdenv looks like this:

derivation {

name = "stdenv-linux";

system = "i686-linux";

builder = ./bash; # statically linked Bash

args = [./builder.sh]; # build script

tar = ./tar; # statically linked GNU Tar

bzip2 = ./bzip2; # statically linked BZip2

tools = ./tools.tar.bz2; # everything else, statically linked

}

178

7.1. The Nix Packages collection

Thus, the statically linked executables for Bash et al. are part of the Nixpkgs source tree.

(In reality, tools.tar.bz2 is downloaded from the Internet, like sources obtained through

fetchurl, since it is rather large. Therefore, a statically linked curl download program is

also included.) These executables have of course been built themselves in some way, but

they have been “re-injected” into the build process as sources—things that, as far as Nix

can see, have not been derived.

Removing unnecessary dependencies Since we prefer static compositions over dy-

namic compositions, the standard environment tries to ensure that any executables pro-

duced by it are statically composed. That means that if an ELF (Unix) executable or

library [160] refers to another library, the directory of the latter must appear in the RPATH

of the former. To ensure that this is the case, stdenv’s linker adds the ﬂag -rpath path

to the linker ﬂags for every library directory mentioned through -L ﬂags. Thus, a linker

invocation

$ ld ... -L/nix/store/abcd...-foo/lib -lfoo

is transformed into

$ ld ... -L/nix/store/abcd...-foo/lib -lfoo \

-rpath /nix/store/abcd...-foo/lib

However, we do not know in advance whether library foo is actually used by the linker.

Regardless, the path /nix/store/abcd...-foo/lib is added to the RPATH of the output. Thus,

the component gets a retained but unnecessary dependency on /nix/store/abcd...-foo.

For this reason, the standard environment after the install phase applies a tool called

patchelf to all ELF executables and libraries in the output. This utility has the ability to

“shrink” the RPATH of arbitrary ELF executables, i.e., remove any directories from the

RPATH that do not contain any referenced library. It is effective in preventing many unnec-

essary retained dependencies. For instance, it prevents Hello’s unnecessary dependency on

GCC that we saw back on page 41.

7.1.3. Ensuring purity

If we build derivations in an environment that contains software components that are not

under Nix control, there is a danger of impurity since the builder might make use of inputs

outside of the Nix store. This is the main threat to the validity of the Nix approach. For

instance, there is no way that we can prevent a builder from calling /usr/bin/gcc. Thus,

while Nix’s hashing scheme ensures isolation between components in the store, it does not

ensure isolation between components in the store on the one hand and component outside

the store on the other hand.

However, we can use some “countermeasures” to reduce the risk of “infection” by non-

Nix components. One such countermeasure—the clearing of the environment—is built

into Nix. Nixpkgs implements a range of additional measures. These include:

• The GNU C Library on i686-linux does not have any default search path for libraries

(such as /usr/lib).

179

7. Software Deployment

• GCC has been patched not to search for header ﬁles in standard locations such as

/usr/include.

• The linker, ld, has been patched not to search for libraries in standard locations such

as /usr/lib.

• In addition, GCC and ld ignore or reject any command-line arguments that refer to a

non-store location (the only exception being the builder’s temporary directory). For

instance, a GCC ﬂag like -I/opt/include is ignored; a ld argument like /usr/lib/crt1.o

causes a fatal error. In practice, these measures prevent conﬁgure scripts, which

frequently scan for optional dependencies in a variety of “well-known” system di-

rectories, from ﬁnding dependencies outside of the store.

It should be noted that the threat of impurity described here can only occur if Nix is used

in a “hosted” environment, i.e., under an existing operating system that does not use Nix

for its component storage. If all components are deployed through Nix, that is, if we have

a pure environment, then this danger disappears. This leads to the as-yet unrealised ideal

of a “NixOS,” a fully Nix-based Unix distribution (discussed in Section 11.1).

7.1.4. Supporting third-party binary components

Most of the components in Nixpkgs are open source or free software components, i.e.,

their builders compile them from source. However, it is possible to deploy third-party

closed-source components as well, provided that the Nix expression for the component

can automatically obtain a binary distribution from somewhere (e.g., through fetchurl).

The builder for such a component is in a way “trivial”: it does not compile anything; it

essentially just unpacks the input binary distribution and copies it to out. Note that this

is perfectly ﬁne: Nix requires no semantics from builders other than that they produce an

FSO in the path denoted by out.

Of course, many closed-source components do not come in a form that allows us to write

a builder that installs them, since they have their own (frequently interactive) installers.

Sometimes we can trick our way past this. Sun’s Java SDK has an interactive installer

that requires the user to approve a license by pressing “Y”. By simply piping that character

into the installer process, the installer becomes non-interactive. Unfortunately we cannot

expect such tricks to be possible in general.

Even if we can unpack the component, there is the issue of the component’s dependen-

cies. It may have runtime dependencies on external programs or libraries. However, con-

trary to open source components, distributors of closed-source components cannot hard-

code paths to dependencies into programs since they cannot make assumptions about the

target environment where the component will run. Thus, paths to dependencies are usu-

ally conﬁgurable through environment variables. These dependencies can therefore be met

through late static composition using wrapper scripts, as described above.

An exception on Linux is the dependency on the GNU C library, and possibly some other

well-known libraries in /lib and /usr/lib. Worst of all is the path of the ELF dynamic linker,

which is hard-coded into every ELF executable. When a dynamically linked executable is

invoked on Linux, the kernel will load the dynamic linker speciﬁed by the executable. The

dynamic linker is almost always /lib/ld-linux.so.2. This is an impure external dependency,

180

7.1. The Nix Packages collection

and should not be used. Rather, the ld-linux.so of Nixpkgs’s Glibc should be used. But

there is no way to accomplish this through environment variables

The utility patchelf, which we already saw above for shrinking RPATHs, can also deal

with this situation. It can change the dynamic linker path embedded in an ELF executable.

Technically speaking, this is done by changing the contents of the INTERP segment of the

executable, which speciﬁes the dynamic linker (or “ELF interpreter”) [160]. We have suc-

cessfully deployed applications such as Sun’s JDK 5.0, Eclipse and Adobe Reader in this

way. The purity of the resulting components has been validated in the pure environment

provided by NixOS (discussed in Section 11.1).

7.1.5. Experience

The main reason for the development of Nixpkgs was to have a substantial body of real

world software components that could serve as a validation for the Nix approach. A par-

ticularly important goal was to see to what extent the assumptions from Section 6.8 hold.

Did these assumptions hold in practice?

Let’s evaluate the assumptions one by one:

• “The component can be built with and installed at an arbitrary preﬁx”: this assump-

tion holds for virtually all components, since in general they cannot assume that a

single ﬁxed path is acceptable to all users. A handful of components did require

a ﬁxed location, but these were easily patched. Also, some groups of components

require a shared preﬁx, i.e., they want to be installed in the same location. This

was true for some components in the modular X.org tree [182]. No closed-source

components were encountered that have a ﬁxed preﬁx.

• “The component does not expect that its dependencies are installed at certain ﬁxed

locations, and the locations of the dependencies can instead be speciﬁed at build

or runtime”: again, since most components cannot assume that users have installed

dependencies at a ﬁxed preﬁx, this assumption holds almost always. An exception is

ﬁxed dependencies on /bin/sh. This path is very widely hard-coded into shell scripts

due to the “hash-bang” convention [152] (i.e., specifying the script’s interpreter on

the ﬁrst line of the script). This is a source of impurity.

As a result, it is not clear whether a pure Nix environment can dispense with provid-

ing /bin/sh unless a substantial effort is undertaken to patch components. (Probably

most of this patching can be performed automatically, e.g., after the unpack phase in

the generic builder.)

• “The component can be built automatically, i.e., with no user interaction whatso-

ever”: no decent software component requires an interactive build process, as that

ﬂies in the face of good software engineering practice (e.g., preventing the use of a

build farm as in Chapter 8). But as mentioned above, some closed-source compo-

nents provide only interactive installers.

It is however possible to invoke the dynamic linker directly, specifying the program as an argument, e.g.,

/nix/store/72by2iw5wd8i...-glibc-2.3.5/lib/ld-linux.so.2 acroread. But this changes the calling convention of the

program.

181

7. Software Deployment

• “The build process of the component is essentially pure”. There are two major

sources of impurity. First, the component uses ﬁle system inputs that are not explic-

itly speciﬁed as inputs, i.e., are not in the set inputs in Figure 5.11. These impurities

can in turn be decomposed into undeclared dependencies on FSOs inside the Nix

store and outside the Nix store. The former has not been observed at all. Thus, iso-

lation works very well. This is good news for a pure Nix-based environment. The

latter does occur, as shown above, and cannot be prevented in general; though, as we

have seen in Section 7.1.3, the threat can be mitigated.

The second source of impurity is a reliance on non-ﬁle system information sources,

such as the system time or the network. In fact, only the system time is a real pos-

sibility; the network is seldom used by builders. However, virtually all components

in practice are pure in the sense of the equivalence relations in Chapter 6. That is, if

there are impurities, they do not affect the operation of the component in an observ-

able way. One exception was observed: static libraries on Mac OS X (page 163).

• “The component does not need to be modiﬁed after it has been built, e.g., at run-

time”: this assumption holds for almost all Unix components, since in a multi-user

environment a component cannot assume to have write permission to itself. Most

development components do not maintain state, and Unix application components

maintain state in the user’s home directory. A notable exception is IBM’s Eclipse

development environment, which in some pre-releases required write permission to

its own preﬁx. Unless such components are patched to write state to another loca-

tion, they will not work in Nix. But such problems generally will be ﬁxed, since

requiring write permission is just as unacceptable to other deployment systems such

as RPM.

It is possible that we will encounter problems with system components (which are

underrepresented currently) that maintain global, as opposed to user-speciﬁc, state.

For instance, the password maintenance utility passwd might (hypothetically) re-

quire that the password ﬁle is stored under $out/etc/passwd.

• “The component has no environment dependencies”: we have encountered no prob-

lems in practice with this assumption. Again, development components tend to have

no external state, and applications keep state in the user’s home directory, which they

automatically initialise on ﬁrst use. Thus, no special actions are necessary to make

these components work, other than assuring the presence of their closures in the ﬁle

system.

But there certainly are counter-examples. A web server component might require

the existence of a special user account under which the component is to execute.

However, such environment dependencies are usually distinct from the component

as a code-level entity. For instance, the same web server component might be used

for several actual servers on the same machine, running under different user accounts

as speciﬁed by the servers’ respective conﬁguration ﬁles. We will deal with this type

of state in Chapter 9.

• “Retained dependencies can be found in the component through scanning”: we have

not encountered a single instance of “pointer hiding”. Of course, that does not mean

182

7.1. The Nix Packages collection

that it cannot happen; it is trivial to construct a component that hides references to

retained dependencies. But Nixpkgs provides strong evidence that it does not occur

in practice.

• “Hash rewriting does not change the semantics of the component” (intensional

model only): the prototype implementation of the intensional model was applied to

a number of large closures, including Firefox and MonoDevelop, a C#-based graph-

ical development environment. The experiment included equivalence class collision

resolution as a result of combining builds from several users. No failures were en-

countered in any component

In summary, we can conclude the following:

• The hashing scheme presents no problems and works very well.

• Fixed dependencies on /bin/sh and a handful of other paths are a source of impurity.

This impurity does not occur in a pure Nix environment, but in that case components

need to be adapted to build and work at all. In impure environments, the counter-

measures of Section 7.1.3 prevent most “large” dependencies, but cannot prevent

direct calls to external paths. Thus vigilance is necessary; build logs for third-party

components should be inspected to verify that no external tools are being used.

There is a caveat to these results: since the components were not selected randomly,

but rather “on-demand” to suit user needs, there may be a selection bias. For instance,

development components may be over-represented. Also, since users are generally more

inclined to add small components than large components or entire systems consisting of

many components (e.g., KDE), there is a bias towards smaller components. Nevertheless,

large components (such as Firefox) and systems (such as a substantial part of Gnome) have

been added.

Enforcing purity Is it possible to enforce purity? We cannot do this in general, but

operating system-speciﬁc tricks, or even extensions, might be employed for this purpose

in the future. For instance, impurity due to non-Nix components can be prevented using

appropriate access control rights that prevent the builder from accessing them in the ﬁrst

place.

Impurity due to the system time can be prevented by disallowing builders from querying

the current time or timestamps on ﬁles. This can be accomplished by modifying operating

system calls to not return this information (e.g., always return a timestamp of 0). However,

impurity may even be present in the instruction set of the processor. For example, the Intel

Pentium has an instruction RDTSC to read the number of clock ticks elapsed since the last

restart [37]. However, this instruction can be disabled by the kernel.

And this is literally a historical footnote: in [52], we wrote that “patching ﬁles [by rewriting hashes] is unlikely

to work in general, e.g., due to internal checksums on ﬁles being invalidated in the process.” It turns out that

this assessment was too pessimistic.

183

7. Software Deployment

7.2. User environments

Section 2.3 introduced user environments as the mechanism through which end-user com-

ponents (i.e., applications) are activated. They are components that are automatically gen-

erated by nix-env to activate other components. There are many ways through which a

component can be activated, i.e., made available to the user. The sets of symlinks to pro-

grams in bin directories of components shown in Figure 2.11 are just one example of how

user environments can be implemented. One can imagine several other policies as to how

components can be activated:

• On Windows and similar GUI environments, components can be activated by adding

them to the Start menu. Thus, a user environment would consist of a hierarchy of

*.lnk ﬁles (on Windows) or *.desktop ﬁles (on KDE) that specify the location of a

program, its icon, associated media types, and so on.

• Similarly, the objects on the desktop in many GUI environments can be synthesised.

However, even in the Unix user environments described in Section 2.3, there are a num-

ber of policy decisions. One is the question of what constituent parts of installed compo-

nents are placed in the user environment, i.e., to which ﬁles in the installed components we

create symlinks. There are several options.

• Everything. The user environment is a hierarchy of symlinks that mirrors the union

of the directory hierarchies of the installed components. This is what the current

implementation does. However, this approach creates very large user environments,

often consisting of hundreds or even thousands of symlinks. The creation of such

user environments can take several seconds, which is too long. The disk space con-

sumption may also be considerable, depending on the implementation details of the

underlying ﬁle system.

• Programs only, i.e., the executables in the bin subdirectory of each installed com-

ponent. This is much more efﬁcient. However, components typically also have

non-code artifacts that we wish to activate, such as documentation. For instance,

the man subdirectory of each component may contain Unix manual pages in Troff

format. Users will expect those manual pages to be found by the man command.

• Allow the user to specify which parts of installed components should be symlinked.

This requires an extension to the nix-env installation syntax, e.g.,

$ nix-env -i firefox --activate bin,man

Another issue is how to deal with collisions, which occur when two installed compo-

nents have one or more ﬁles with identical relative paths. Such collisions are detected by

the builder of the user environment component. There are several possibilities:

• The builder can print an error message and abort, thus causing the nix-env operation

to fail. In this case no new generation is produced and the current generation symlink

does not change.

184

7.3. Binary deployment

• If the collision is caused by multiple versions of a component, the builder can give

precedence to the newest version (under some ordering of versions). Also, the older

version could be made available under a different name. If we have Firefox 1.0.5

and 1.0.6 installed concurrently in a user environment, the symlink to the former

might be called bin/ﬁrefox-1.0.5, and the latter bin/ﬁrefox.

• The builder can give precedence to the component that was added most recently.

(Currently, however, the builder does not have this information.)

• All conﬂicting ﬁles can be renamed, e.g., to bin/ﬁrefox-1.0.5 and bin/ﬁrefox-1.0.6.

These are all policy decisions regarding the computation of the user environment. The

current Nix implementation does not have an easy mechanism to change the built-in policy

(other than editing the user environment builder, a simple Perl script). However, it would

be fairly trivial to add an option for users to override the user environment builder, thus

allowing different policies.

7.3. Binary deployment

As we have seen previously, Nix (through Nix expressions) is at its core a source deploy-

ment system, but the substitute mechanism (Section 5.5.3) allows binary deployment as

an optimisation of source deployment, yielding a transparent source/binary deployment

model (Section 2.6). But the substitute mechanism is just that—a mechanism. It does not

implement any particular binary deployment method. It just consists of database registra-

tions of the fact that an FSO for a certain store path can be produced by executing some

program.

This section shows a speciﬁc implementation of binary deployment using the substitute

mechanism. It consists of three fairly simple tools:

• nix-push, executed on the deployer’s side, which computes the closure of a given

store path, packs the store paths in the closure into archives, and places them on a

web server.

• nix-pull, executed on the clients, which obtains information about the archives avail-

able on a web server, and registers the program download-using-manifests as a sub-

stitute for each corresponding store path.

• download-using-manifests, called by the substitute function (Figure 5.15), which

downloads and unpacks the requested FSO.

We shall now look at the various parts in detail. Suppose that we have a Nix expression

foo.nix that evaluates to a set of store derivations that we want to build and put on a web

server http://server/nix-cache from where the binaries can be fetched by clients. We can do

this as follows:

$ nix-push \

http://server/nix-cache \

http://server/nix-cache/MANIFEST \

$(nix-instantiate ./foo.nix)

185

7. Software Deployment

Recall that nix-instantiate translates a Nix expression to store derivations, and prints out

their store paths; and that the shell syntax $(...) causes those paths to be passed as argu-

ments to nix-push. The ﬁrst URL is the location to which the archives should be uploaded.

The second URL is the location to which the manifest of the binary deployment is up-

loaded. The manifest (as we brieﬂy saw in Section 2.6) contains information about every

uploaded archive.

Given a call nix-push archiveURL manifestURL paths, the steps that nix-push performs

are as follows:

• The closure of the set paths is computed using the command nix-store -qR --include-

outputs paths. As we saw on page 42, this performs a combined source/binary

deployment, i.e., it includes the store derivations and their inputSrcs, but also the

outputs of the store derivations. Each derivation is also built.

• For each path p in the closure, the canonical serialisation serialise(readPath(p))

(Section 5.2.1) is compressed using the bzip2 compression program. This archive

is uploaded to the URL archiveURL + "/" + printHash

(hash

sha256

(c)) + ".nar.bz2",

where c is the compressed serialisation (this means that archives are stored un-

der content-addressable names). The archive is uploaded using an HTTP PUT re-

quest [57], but only if a HEAD request reveals that the archive does not already exist

on the server

. This makes it useful to use the same archiveURL between nix-push

operations for multiple software releases, since common subpaths will be uploaded

and stored only once.

• A manifest is then created. For each path p that we have packed into a compressed

archive c and uploaded to url, it contains an entry of the following form:

{

StorePath:

NarURL:

url

Hash: sha256:

printHash

(hash

sha256

(c))

NarHash: sha256:

printHash

(hash

sha256

(serialise(readPath(p))))

Size:

|c|

References:

references[p]

Deriver:

deriver[p]

}

The meaning of such an entry is that if we want to produce a store path p, we can

do so by downloading a compressed serialisation from url and unpacking it into p.

The ﬁeld Hash is the hash of the compressed serialisation of p, while NarHash is

the hash of the uncompressed serialisation of p. The former allows clients to verify

that the downloaded archive has not been modiﬁed, while the latter will be of use in

the binary patch deployment scheme described in Section 7.5. The References and

Deriver ﬁelds are omitted if their corresponding database entries are equal to ε.

The actual nix-push program takes an additional URL to be used for the HEAD requests. This allows the PUT

and HEAD requests to use different URLs. E.g., the ﬁrst can refer to a CGI script [124] that handles uploads,

and the second is the actual location from which archives can be downloaded.

186

7.4. Deployment policies

• The manifest is uploaded using a PUT request to manifestURL.

On the client side, the availability of the pre-built binaries created by nix-push can be

registered using nix-pull manifestURL, e.g.,

$ nix-pull http://server/nix-cache/MANIFEST

The command nix-pull downloads the manifest and stores it in the directory /nix/var/nix/-

manifests. For each store path entry p in the manifest, a substitute is registered using the

command nix-store --register-substitutes (page 119). Note that this also sets the references

database entry for p so that the closure invariant can be maintained. The substitute program

is download-using-manifests, and there are no declared command-line arguments. But

recall from Figure 5.15 that the substitute program is always called with p as command-

line argument. Since all other necessary information for the download (such as the URL)

is stored in the manifests in /nix/var/nix/manifests, no further command-line arguments are

necessary.

The script download-using-manifests, when it is called by the function substitute in Fig-

ure 5.15 to produce a path p, reads all manifests in /nix/var/nix/manifests. Note that there

may be multiple manifest entries for p. The script picks one arbitrarily. (An alternative

implementation is to try each until one succeeds.) It downloads the archive from the URL

speciﬁed by the NarURL ﬁeld of the selected manifest entry, uncompresses it, deserialises

it to path p, and checks that the cryptographic hash speciﬁed in the NarHash ﬁeld of the

entry matches the actual hash. In Section 7.5, we will see an extension to this script that

supports binary patching.

7.4. Deployment policies

Since Nix provides basic mechanisms to support deployment, it is fairly easy to deﬁne

all sorts of speciﬁc deployment policies. In general, a deployment policy consists of two

elements:

• A way to get Nix expressions to the clients; this deﬁnes source deployment.

• A way to get binaries to the clients as an optimisation of source deployment through

substitutes; this deﬁnes binary deployment.

However, either can be omitted. Clearly, if the latter is omitted, we have pure source de-

ployment. If the former is omitted, clients cannot install through Nix expressions (since

they don’t have them), but they can install store paths directly, as we have seen in Sec-

tion 5.5, e.g.,

$ nix-env -i /nix/store/1ja1w63wbk5q...-hello-2.1.1.drv

$ nix-env -i /nix/store/bwacc7a5c5n3...-hello-2.1.1

This section describes a number of concrete policies that have been implemented.

187

7. Software Deployment

Direct download The most primitive deployment mechanism is to have users download

Nix expressions, e.g.,

$ wget http://example.org/nixpkgs.tar.bz2

$ tar xvfj nixpkgs.tar.bz2

$ nix-env -f ./nixpkgs/pkgs/system/all-packages.nix -i ...

Of course, this source deployment policy can be made into a binary deployment policy by

having clients perform a nix-pull of a manifest corresponding to the set of Nix expressions.

While this policy is primitive, it is quite useful in many environments. For instance, it

can be used in many typical networked environments to automatically distribute compo-

nents to (say) all desktop machines. The clients could periodically execute the displayed

command sequences (a pull model), or the commands could be executed automatically by

remote login from a script executed centrally by the system administrator (a push model).

Distribution through a version management system A slight modiﬁcation of the pre-

vious scheme is to obtain the Nix expressions from a version management repository in-

stead of a direct download. The following obtains a working copy of a set of Nix expres-

sions from a Subversion repository:

$ svn co http://example.org/nixpkgs

$ nix-env -f ./nixpkgs/pkgs/system/all-packages.nix -i ...

The command svn up can then be used to keep the Nix expressions up to date.

An advantage of this policy is that it makes it easy to maintain local modiﬁcations to

the set of Nix expressions. The version management tool ensures that updates from the

repository are merged with local modiﬁcations.

Again, nix-pull can be used to turn this into a binary deployment scheme. However, with

local modiﬁcations, the deployer can no longer ensure that the expressions that the client

installs have been pre-built. Of course, that is the whole point of transparent source/binary

deployment: it enables binary deployment to “degrade” automatically to source deploy-

ment.

Channels Channels are a convenient way to keep a set of software components up to

date. A channel is a URL u such that there is an archive containing Nix expressions at

u +"/nixexprs.tar.bz2" and a manifest at u + "/MANIFEST". The archive ﬁle nixexprs.tar.bz2

must unpack to a single directory that must contain (at the very least) a Nix expression de-

fault.nix. The derivations produced by the evaluation of that expression are the derivations

provided by the channel. The manifest provides optional binary deployment.

Channels are managed on the client side through a simple tool called nix-channel. Users

can “subscribe” to a channel:

$ nix-channel --add http://example.org/foo-channel

This causes the speciﬁed URL to be added to a list of channel URLs maintained per user.

Next, the client can obtain the latest Nix expressions and manifests from all subscribed

channels:

$ nix-channel --update

188

7.4. Deployment policies

The update operation, for each subscribed channel u, performs a nix-pull on u +

"/MANIFEST", and downloads and unpacks u + "/nixexprs.tar.bz2". It then makes the union

of the derivations in all Nix expressions the default for nix-env operations (through the

command nix-env -I). So after the update, components can be installed from the channels

using nix-env:

$ nix-env -i foo

Thus, a typical usage scenario is the following command sequence:

$ nix-channel --update

$ nix-env -u '*'

which updates all components in the user’s proﬁle to the latest versions available from the

subscribed channels. A typical higher-level policy is to perform this sequence automati-

cally at certain times (e.g., from a Unix cron job).

One-click installations A one-click installation is a convenient mechanism to install a

speciﬁc component. For many users it is the easiest way to install a component. One-

click installations are based on Nix packages, which are plain-text ﬁles that contain just

the following bits of information about a component:

• Its symbolic name.

• The store path of the store derivation.

• The store path of the output, i.e., the output ﬁeld of the store derivation.

• Its platform identiﬁer (e.g., i686-linux) to prevent the component from being installed

on inappropriate systems.

• The URL of a manifest.

An example of the full contents of a package ﬁle is the following:

NIXPKG1

http://nix.cs.uu.nl/dist/nix/nixpkgs-0.9pre3530/MANIFEST

firefox-1.0.6

i686-linux

/nix/store/cbqkqc039srl48yswgzx1gs5ywnkbidp-firefox-1.0.6.drv

/nix/store/66zq88a8g36kmjl6w0nm5lfi2rvjj566-firefox-1.0.6

Nix packages are intended to be associated in web browsers (through the MIME media

type [74] application/nix-package) with the package installer program nix-install-package.

Thus, by clicking on links in web pages to packages, the package is installed (hence the

term “one-click installation”). An example was shown in Figure 2.14 (page 44). The

installer nix-install-package may one day be a fancy graphical user interface showing in-

stallation progress, but is currently just a simple script that performs the following actions:

• It asks the user to conﬁrm the installation.

• It performs a nix-pull on the manifest.

189

7. Software Deployment

• It installs the store path of the derivation output p directly by executing nix-env -i p.

This sufﬁces to perform a binary installation of the component and all its runtime depen-

dencies. A slight variation install the store path of the store derivation, rather than the

output. In that case, source deployment is also possible.

Note that this deployment method completely bypasses Nix expressions on the client

side. The store path p is installed directly. The closure of p follows from the References

ﬁelds in the manifest. If the Nix expression language changes in incompatible ways, the

users of Nix packages are not affected. This demonstrates one of the advantages of sepa-

rating the Nix build model into high-level Nix expressions and low-level store operations.

7.5. Patch deployment

As we have seen, Nix has a purely functional deployment model. This provides impor-

tant advantages, such as side-by-side deployment of versions and variants, safe upgrades,

the ability to roll back, and so on. However, there is a cost in that the purely functional

model makes it hard to efﬁciently deploy upgrades to clients. In particular, upgrades to

“fundamental” dependencies are expensive.

Consider the dependency graph of Firefox shown in Figure 1.5 (page 10). The compo-

nent at the top of the graph, used by almost all other components, is the GNU C Library

(Glibc). Suppose that we discover a bug in Glibc, and wish to deploy a new version. The

most straightforward way to do this is to update the Nix expression in Nixpkgs that builds

Glibc to use the new version, and rebuild and reinstall the top-level components (e.g., Fire-

fox) that depend on it. That is, we deploy to the clients a new set of Nix expressions for all

installed components, and the client can then simply perform

$ nix-env -i firefox

$ nix-env -u '*' --leq

to upgrade all installed components with versions that are higher or equal (e.g., simply

built with newer dependencies).

But this is expensive! Recall from Figure 2.3 (page 22) that a change to an input (such

as Glibc) propagates through the dependency graph. This has two effects. First, all af-

fected components must be rebuilt. This is exactly what we want, since the change to the

dependencies may of course affect the result of the builds. This is the case even if inter-

faces haven’t changed. Consider statically linked libraries, smart cross-module inlining,

changes to the compiler affecting the binary interface, and so on. By far the easiest way to

produce outputs that are consistent with the Nix expressions is to build them from scratch.

Thanks to transparent source/binary deployment, this rebuild needs to be done only on the

distributor side; the clients just do an appropriate nix-pull.

Unfortunately, the second effect is that all affected components must be re-deployed

to the clients, i.e., the clients must download and install each affected component. This

creates a huge scalability problem, which this section attempts to address. If we want to

deploy a 100-byte bug ﬁx to Glibc, almost all components in the system must be down-

loaded again, since at the very least the RPATHs of dependent binaries will have changed

190

7.5. Patch deployment

to point at the new Glibc. Depending on network characteristics, this can take many hours

even on fast connections. Note that this is much worse than the ﬁrst effect (having to re-

build the components), since that can be done centralised and only needs to be done once.

The re-deployment, on the other hand, must be done for each client.

This shows the price to be paid for a purely functional deployment model. By contrast,

consider deployment models that use destructive upgrading, i.e., that overwrite the ﬁles of

components with newer versions. For instance, in RPM [62], we just install a new Glibc

RPM package that overwrites the old one.

This however prevents side-by-side deployment of variants (what if some component

needs the old Glibc because it is incompatible with the new one?), makes rollbacks much

harder (essential in server environments), and is generally bad from an SCM perspective

(since it becomes much harder to identify the current conﬁguration). Also, such destructive

upgrades only work with dynamic linking and other late-binding techniques; if the com-

ponent has been statically linked into other components at build time, we must identify all

affected components and upgrade them as well. This was a major problem when a security

bug was discovered in the ubiquitous Zlib compression library [1].

If we were to use destructive upgrading in Nix, it would violate the crucial deployment

invariant that the hash of a path uniquely describes the component. (This is similar to

allowing assignments in purely functional programming languages such as Haskell [135].)

From a conﬁguration management perspective, the hashes identify the conﬁguration of the

components, and destructive updates remove the ability to identify what we have on our

system. Also, it destroys the component isolation property, i.e., that an upgrade to one

component cannot cause the failure to another component. If an upgrade is not entirely

backwards compatible, this no longer holds.

One might ask whether the relative difﬁculty (in terms of hardware resources, not de-

veloper or user effort) of deploying upgrades doesn’t show that Nix is unsuitable for large-

scale software deployment. However, Nix’s advantages in supporting side-by-side vari-

ability, correct dependency, atomic rollbacks, and so on, in the face of a quasi-component

model (i.e., the huge base of existing Unix packages) not designed to support those fea-

tures, make it compelling to seek a solution to the upgrade deployment problem within the

Nix framework.

Upgrading through wrappers One way to efﬁciently deploy upgrades that works in

many instances of late static composition is to use wrapper components. These were al-

ready discussed in Section 7.1.1. The idea is to deploy a new set of Nix expressions that

describe a new derivation graph that is exactly the same as the original one (thus previously

installed components do not need to be rebuilt or redownloaded), except that at top-level

a wrapper component is added. This wrapper must ensure that at runtime the upgraded

component is used, rather than the old one.

Figure 7.6 shows an example for Mozilla Firefox, which has a dependency on Glibc,

both directly and indirectly through GTK; this situation is shown on the left. Suppose that

we want to deploy an upgraded version of Glibc. We deploy a Nix expression that evaluates

to the derivations shown on the right. The wrapper component depends on Firefox (which

it wraps and forwards to), and the new Glibc (glibc’). The wrapper script for Firefox calls

the original Firefox like this:

191

7. Software Deployment

glibc

gtk

firefox

glib

Original derivation graph

glibc

gtk

firefox

firefoxWrapper

glib

glibc’

New derivation graph

Figure 7.6.: Upgrading through wrappers

LD_LIBRARY_PATH=

new-glibc-path

:$LD_LIBRARY_PATH

exec

original-firefox-path

/bin/firefox "$@"

This causes the dynamic linker to use the new Glibc instead of the old one

. The old Glibc

is still part of the closure, but it won’t be used.

Clearly, the LD_LIBRARY_PATH is highly speciﬁc to dynamic linking on certain Unix

platforms, although similar tricks are available for many other types of composition (e.g.,

juggling the PATH variable for composition through program calls). We also often cannot

be certain that the override is used in all places. Finally, unscoped composition mecha-

nisms make it quite hard to see what the effect of an override will be, leading to buggy

wrappers.

A general solution: binary patching Thus, upgrading through wrappers is not a gen-

eral solution. A general solution to the huge binary redeployment caused by a change

to a fundamental component such as Glibc is by transparently deploying binary patches

between component releases. For instance, if a bug ﬁx to Glibc induces a switch from

/nix/store/72by2iw5wd8i...-glibc-2.3.5 to /nix/store/shfb6q9yvk0l...-glibc-2.3.5-patch-1, then

we compute the delta (the binary patch) between the contents of those paths and make the

patch available to clients. We also do this for all components depending on it. Subsequently

the clients can apply those patches to the old version to produce the new version. As we

shall see in Section 7.5.4, patches for components affected by a change to a dependency

are generally very small.

A binary patch describes a set of edit operations that transforms a base FSO stored at

path p

src

in the Nix store into a target FSO stored at path p

dst

. Thus, if a client needs path

dst

and it has path p

src

already installed, then it can speed up the installation of p

dst

downloading the patch from p

src

to p

dst

, copying p

src

to p

dst

in the Nix store, and ﬁnally

applying the patch to p

dst

This ﬁts nicely into Nix’s substitute mechanism (Section 7.3) used to implement trans-

parent binary deployment. We just extend its download capabilities: if a patch is available,

It’s a bit more tricky in reality, since the path of the old Glibc is hard-coded into the RPATHs of the Firefox and

GTK binaries. The dynamic linker has an option to override the RPATH, however.

192

7.5. Patch deployment

rather than doing a full download, we download the patch instead. Manifests are extended

to specify the availability of patches. For instance, the manifest entry

patch {

StorePath: /nix/store/lhix54krdqkp...-firefox-1.0

NarURL: http://server/0iihh1l7vvpa...-firefox-1.0.nar-bsdiff

Hash: sha256:...

NarHash: sha256:...

Size: 357

BasePath: /nix/store/mkmpxqr8d7f7...-firefox-1.0

BaseHash: sha256:...

}

describes a 357-byte patch from the Firefox component shown in the previous section

(stored at BasePath) to a new one (stored at StorePath) induced by a Glibc upgrade. If

a patch is not available, or if the base component is not installed, we fall back to a full

download of the new component; or even a local build if no download is available.

NarURL is the URL of the patch, Hash is the cryptographic hash of the contents of the

patch, and NarHash is the cryptographic hash of the serialisation of the resulting FSO (just

as in normal downloads). BaseHash is the cryptographic hash of the serialisation of the

FSO at the base path to which the patch applies. In the extensional model, due to impure

builders, the contents of an FSO at a given path need not always be the same. If the local

contents differ from the contents on which the patch is based at the deployer’s machine,

then the patch cannot be applied. In the intensional model with its content-addressable Nix

store, this is not an issue: if due to builder impurity a derivation has produced different

output on the local machine than on the deployer’s, the patch’s base path simply will not

exist.

7.5.1. Binary patch creation

There are many off-the-shelf algorithms and implementations to compute binary deltas

between two arbitrary ﬁles. Such algorithms produce a list of edit operations such as

copying a sequence of bytes from a position in the original ﬁle to a possibly different

position in the new ﬁle, inserting a new sequence of bytes at a some position in the new

ﬁle, and so on. I used the bsdiff utility [132] because it produces relatively small patches

(see Section 7.6).

However, the components in the Nix store are arbitrary directory trees. How do we pro-

duce deltas between directories trees? A “simple” solution is to compute deltas between

corresponding regular ﬁles (i.e., with the same relative path in the components) and dis-

tribute all deltas together. The full contents of all new ﬁles in the target should also be

added, as well as a list describing ﬁle deletions, changes to symlink contents, etc. Files not

listed can be assumed to be unchanged.

This method is both complicated and has the severe problem of not following renames.

For instance, the Firefox component stores most of its ﬁles in a subdirectory lib/ﬁrefox-

version. The method described above fails to patch, e.g., lib/ﬁrefox-0.9/libmozjs.so into

lib/ﬁrefox-1.0/libmozjs.so since the path names do not correspond; rather, the latter ﬁle is

stored in full in the patch. Hence, with this method patching is not very effective in the

presence of renames.

193

7. Software Deployment

A/B

c0 0f 85 2b fc

ff ff 8b 45 d8

85 c0 0f f3 ..

C/D

c0 0f 85 2b fc

ff ff 8b 45 d8

85 c0 0f f3 ..

copy

Base

Target

Figure 7.7.: Deltas between archives handle renames, deletions, and additions

There is however a much simpler and more effective solution: we take the patch between

the serialisations of the components. That is, we use the function serialise (Section 5.2.1)

to produce NAR archives for both the old and the new FSO, and compute the binary delta

between those two ﬁles. Here we see why it is important that serialise produces a canonical

representation of the FSO: it prevents a patch from failing to apply due to implementation

differences in the archiving tool, or system dependencies (e.g., different orderings of ﬁles

in directories). Formats such as TAR and ZIP do not have a canonical form.

Computing deltas between archives automatically takes renames, deletions, ﬁle type

changes, etc. into account, since these are just simple changes within the archive ﬁles.

Figure 7.7 shows an example of why this is the case: the original ﬁle A/B has been moved

and renamed to C/D, which furthermore is stored in a different part of the NAR archive.

However, the ﬁle contents are the same (or mildly changed). The binary delta algorithm

will just emit an edit operation that changes the ﬁrst ﬁle name into the second, followed by

the appropriate edit operations for the ﬁle contents. It does not matter whether the position

of the ﬁle in the archive has changed: contrary to delta algorithms like the standard diff

tool, bsdiff can handle re-orderings of the data.

To apply a patch, a client creates an archive of the base component, applies the binary

patch to it, and unpacks the resulting archive into the target path.

7.5.2. Patch chaining

It is generally infeasible to produce patches between every pair of releases of a set of com-

ponents. The number of patches would be O(n

m), where n is the number of releases and

m is the number of components. As an example, consider the Nix Packages collection. Pre-

releases of Nixpkgs are made automatically by a build farm (Chapter 8) on every commit to

its version management repository, which typically is several times a day. The pre-releases

are made available in a channel. Since we do not want to impose full redownloads on

developers whenever something changes, we want to make patches available.

Unfortunately, since pre-releases appear so often, we cannot feasibly produce patches

between each pair of pre-releases. So as a general policy we only produce patches be-

tween immediately succeeding pre-releases. Given releases 0.7pre1899, 0.7pre1928 and

0.7pre1931, we produce patches between 0.7pre1899 and 0.7pre1928, and between 0.7pre-

1928 and 0.7pre1931. This creates a problem, however: suppose that a user has Firefox

194

7.5. Patch deployment

... 0.6pre1741 ... 0.7pre1928 ... 0.70.5 0.6 0.7pre1777 0.7pre1785

Figure 7.8.: Patch sets created between Nixpkgs releases

from 0.7pre1899 installed, and Firefox changed in both succeeding releases, then there is

no patch that brings the user up-to-date.

The solution is to automatically chain patches, i.e., using a series of available patches

src

→ ... → p

→ p

dst

to produce path p

dst

. In the example above, we have a Firefox

component installed that can be used as the base path to a patch in the 0.7pre1928 release,

to produce a component that can in turn serve as a base path to a patch in the 0.7pre1931

release.

However, such patch sequences can eventually become so large that their combined size

approaches or exceeds the size of full downloads. In that case we can “short-circuit” the

sequence by adding patches between additional releases. Figure 7.8 shows an example of

patches between Nixpkgs releases thus formed. Arrows indicate the existence of a patch

set between pairs of releases. Here, patch sets are produced by directly succeeding pre-

releases, and between any successive stable releases. An additional “short-circuit” patch

set between 0.7pre1785 and 0.7pre1928 was also made.

In the presence of patch sets between arbitrary releases, it is not directly obvious which

sequence of patches or full downloads is optimal. To be fully general, the Nix substitute

downloader runs a shortest path algorithm on a directed acyclic graph that, intuitively, rep-

resents components already installed, available patches between components, and available

full downloads of components. Formally, the graph is deﬁned as follows:

• The nodes are the store paths for which pre-built binaries are available on the server,

either as full downloads or as patches, plus any store paths that serve as bases to

patches. There is also a special start node.

• There are three types of edges:

– Patch edges between store paths that represent available patches. The edge

weight is the size of the patch (in bytes). In the extensional hash, edges from

nodes representing valid store paths are only added if the cryptographic hash

of the serialisation of the store path matches the patch’s BaseHash ﬁeld, since

the patch is inapplicable otherwise.

– Full download edges from start to a store path for which we have a full down-

load available. The edge weight is the size of the full download.

– Free edges from start to a valid store path, representing an FSO that is already

available on the system. The edge weight is 0.

We then ﬁnd the shortest path between start and the path of the requested component

using Dijkstra’s shortest path algorithm. This method can ﬁnd any of the following:

195

7. Software Deployment

start

l9z62mg66xq3...-firefox-1.0.3

11158372

(full download)

zncs96rbsf9q...-firefox-1.0.1

(present)

3ll5pr8hasqr...-firefox-0.9

(present)

k7psi6ill5qb...-firefox-1.0.2

154179

(patch)

789234

(patch)

244869

(patch)

Figure 7.9.: Finding the optimal set of downloads

• A sequence of patches transforming an already installed component into the re-

quested component.

• A full download of the requested component.

• A full download of some component X which is then transformed using a sequence

of patches into the requested component. Generally, this will be longer than imme-

diately doing a full download of the requested component, but this allows one to

make only patches available for upgrades.

Figure 7.9 shows the graph for an instance of Firefox 1.0.2. It is available as a large full

download, and through a chain of patches. There are two Firefox instances that are valid

base paths for patches. (There might be other valid Firefox instances in the local store, but

these do not serve as base paths for patches.) There are 3 patches: one from Firefox 0.9 to

1.0.2, one from 1.0.1 to 1.0.2, and one from 1.0.2 to 1.0.3. The patch sequence that applies

the 1.0.1 to 1.0.2 and 1.0.2 to 1.0.3 patches to the valid 1.0.1 instance is the shortest path

from the start node to the desired store path of 1.0.3, and is therefore selected.

Above, edge weight was deﬁned as the size of downloads in bytes. We could take other

factors into account, such as protocol/network overhead per download, the CPU resources

necessary to apply patches, and so on. For instance, on a reasonably fast connection, a full

download might be preferable over a long sequence of patches even if the combined byte

count of those patches is less than the full download.

7.5.3. Base selection

To deploy an upgrade, we have to produce patches between “corresponding” components.

This is intuitively simple: for instance, to deploy a Glibc upgrade, we have to produce

patches between the old Glibc and the new one, but also between the components depend-

ing on it, e.g., between the old Firefox and the new one. However, a complication is that

the dependency graphs might not be isomorphic. Components may have been removed

or added, dependencies moved, component names changed (e.g., Phoenix to Firebird to

Firefox to mention a real-world example), and so on. Also, even disregarding component

196

7.5. Patch deployment

renames, simply matching by name is insufﬁcient because there may be multiple compo-

nent instances with the same name (e.g., builds for different platforms).

When deploying a set of target store paths Y, the base selection problem is to select

from a set of base store paths X a set of patches (X,Y ) ∈ (X × Y) such that the probability

of the Xs being present on the clients is maximised within certain resource constraints.

Clearly, we could produce patches between all Xs and Y s. This policy is “optimal”

in the sense that the client is always able to select the absolutely shortest sequence of

patches. However, it is infeasible in terms of time and space since producing a patch

takes a non-negligible amount of time, and most such patches will be large since they

will be between unrelated components (patching Adobe Reader into Firefox is obviously

inefﬁcient—though possible!).

Therefore, we need to select some subset of (X × Y). The solution currently imple-

mented is pragmatic: we use a number of properties of the components to guess whether

they “match” (i.e., are conceptually the “same” component). Indeed, the selection prob-

lem appears to force us to resort to heuristics for two reasons. First, there can be arbitrary

changes between releases. Second, we cannot feasibly produce all patches to select the

“best” according to some objective criterion.

Possible heuristics include the following:

• Same component name. This is clearly one of the simplest and most effective criteria.

However, there is a complication: there can be multiple components with the same

name. For instance, Nixpkgs contains the GNU C Compiler gcc at several levels

in the dependency graph (due to bootstrapping). Also, it contains two components

called ﬁrefox—one is the “real thing”, the other is a shell script wrapper around the

ﬁrst to enable some plugins. Finally, Nixpkgs contains the same components for

multiple platforms.

• The weighted number of uses can be used to disambiguate between components

at different bootstrapping levels such as GCC mentioned above, or disambiguate

between certain variants of a component. It is deﬁned for a component at path p as

follows:

w(p) =

∑

q∈users(p)

d(q,p)

where users(p) is the set of components from which p is reachable in the build-time

dependency graph, i.e., the components that are directly or indirectly dependent on

p; where d(q, p) is the unweighted distance from component q to p in the build-time

dependency graph; and where r ≥ 1 is an empirically determined value that causes

less weight to be given to “distant” dependencies than to “nearby” dependencies.

For instance, in the Nixpkgs dependency graph, there is a “bootstrap” GCC and a

“ﬁnal” GCC, the former being used to compile the latter, and the latter being used

to compile almost all other packages. If we were to take the unweighted number of

uses (r = 1), then the bootstrap GCC would have a slightly higher number of uses

than the ﬁnal GCC (since any component using the latter is indirectly dependent

on the former), but the difference is too small for disambiguation—such a difference

could also be caused by the addition or removal of dependent components. However,

197

7. Software Deployment

if we take, e.g., r = 2, then the weighted number of uses for the ﬁnal GCC will be

almost twice as large. This is because the bootstrap GCC is at least one step further

away in the dependency graph from the majority of components, thus halving their

contribution to its w(p).

Thus, if the ratio between w(p) and w(q) is greater than some empirically deter-

mined value k, then components p and q are considered unrelated, and no patch

between them is produced. A good value for k is around 2, e.g., k = 1.9.

• Size of the component. If the ratio between the sizes of two components differs more

than some value l, then the components are considered unrelated. A typical value is

l = 3; even if components differing in size by a factor of 3 are related, then patching

is unlikely to be effective. This trivial heuristic can disambiguate between the two

Firefox components mentioned above, since the wrapper script component is much

smaller than the real Firefox component.

• Platform. In general, it is pointless to create a patch between components for dif-

ferent platforms (e.g., Linux and Mac OS X), since it is unlikely that a client has

components for a different platform installed.

7.5.4. Experience

The binary patch deployment scheme described above has been implemented in the Nix

system. To get some experimental results regarding the efﬁciency of binary patching, I

used it to produce patches between 50 subsequent releases and pre-releases of the Nix

Packages collection. Base components were selected on the basis of matching names,

using the size and weighted number of uses to disambiguate between a number of com-

ponents with equal names. The use of patches is automatic and completely transparent to

users; an upgrade action in Nix uses (a sequence of) patches if available and applicable, and

falls back to full downloads otherwise. The results below show that the patching scheme

succeeds in its main goal, i.e., reducing network bandwidth consumption in the face of

updates to fundamental components such as Glibc or GCC to an “acceptable” level.

I computed for each pair of subsequent releases the size of an upgrade using full down-

loads of changed components, versus the size of the required patches to changed com-

ponents. Also, the average and median sizes of each patch for the changed components

(or full download, if no patch was possible) were computed. New top-level components

(e.g., applications introduced in the new release) were disregarded. Table 7.1 summarises

the results for a number of selected releases, representing various types of upgrades. File

sizes are in bytes unless speciﬁed otherwise. Omitted releases were typically upgrades of

single leaf components such as applications. An example is the Firefox upgrade in revision

0.6pre1702.

Efﬁcient upgrades or patches to fundamental components are the main goal of this sec-

tion. For instance, release 0.7pre1980 upgraded the GNU C Compiler used to build all

other components, while releases 0.7pre1820 and 0.7pre1977 provided bug ﬁxes to the

GNU C Library, also used at build time and at runtime by all other components. The

patches resulting from the Glibc changes in particular are tiny: the median patch size is

around 440 bytes. This is because such patches generally only need to modify the RPATH

198

7.5. Patch deployment

Release

Comps.

changed

Full

size

Total

patch

size

Savings

Avg.

patch

size

Median

patch

size

Remarks

0.6pre1069 27 31.6M 162K 99.5% 6172 898 X11 libraries update

0.6pre1489 147 180M 71M 60.5% 495K 81K Glibc 2.3.2 to 2.3.3,

GCC 3.3.3 to 3.4.2,

many other changes

0.6pre1538 147 176.7M 364K 99.8% 2536 509 Standard build environ-

ment changes

0.6pre1542 1 9.3M 67K 99.3% 67K 67K Firefox bug ﬁx

0.6pre1672 26 38.0M 562K 98.6% 22155 6475 GTK updates

0.6pre1702 3 11.0M 190K 98.3% 63K 234K Firefox 1.0rc1 to 1.0rc2

0.7pre1820 154 188.6M 598K 99.7% 3981 446 Glibc loadlocale bug ﬁx

0.7pre1931 1 1164K 45K 96.1% 45K 45K Subversion 1.1.1 to 1.1.2

0.7pre1977 153 196.3M 743K 99.6% 4977 440 Glibc UTF-8 locales

patch

0.7pre1980

154 197.2M 3748K 98.1% 24924 974 GCC 3.4.2 to 3.4.3

Table 7.1.: Statistics for patch sets between selected Nixpkgs releases and their immediate

predecessors

in executable and shared libraries. The average is higher (around 4K) because a handful of

applications and libraries statically link against Glibc components. Still, the total size of

the patches for all components is only 598K and 743K, respectively—a fairly trivial size

even on slow modem connections.

On the other hand, release 0.6pre1489 is not small at all—the patch savings are only

60.5%. However, this release contained many signiﬁcant changes. In particular, there was

a major upgrade to GCC, with important changes to the generated code in all components.

In general, compilers should not be switched lightly. (If individual components need an

upgraded version, e.g., to ﬁx a code generation bug, that is no problem: Nix expressions,

being written in a functional language, can easily express that different components must

be built with different compilers.) Minor compiler upgrades need not be a problem; release

0.7pre1980, which featured a minor upgrade to GCC, has a 98.1% patch effectiveness.

Patch generation is a relatively slow process. For example, the generation of the patch

set for release 0.7pre1820 took 49 minutes on a 3.2 GHz Pentium 4 machine with 1 GiB

of RAM running Linux 2.4.26. The bsdiff program also needs a large amount of memory;

its documentation recommends a working set of at least 8 times the base ﬁle. For a large

component such as Glibc, which takes 46M of disk space, this works out to 368M of RAM.

In fact, we cannot currently compute patches for teTeX (a T

X distribution), regardless of

how much physical memory or swap space is available, because the address space of 32-bit

machines is not large enough to compute patches between teTeX’s NAR archives, which

are around 230 MiB large. The solution is a more efﬁcient bsdiff implementation, or simply

to migrate to a 64-bit system for patch computation.

A ﬁnal point not addressed previously is the disk space consumption of upgrades. A

First release since 0.6pre1398.

199

7. Software Deployment

change to a component such as Glibc will still cause every component to be duplicated on

disk, even if they do no longer have to be downloaded in full. However, after an upgrade, a

user can run the Nix garbage collector that safely and automatically removes unused com-

ponents. Nonetheless, as an optimisation, we observe that many ﬁles in those components

will be exactly the same (e.g., header ﬁles, scripts, documentation, JAR ﬁles). Therefore,

I implemented a tool that “optimises” the Nix store by ﬁnding all identical regular ﬁles

in the store, and replacing them with hard links [152] to a single copy. On typical Nix

stores (i.e., subject to normal evolution over a period of time) this saved between 15–30%

of disk space. While this is useful, it is not an order of magnitude change as is the case

with the amount of bandwidth saved using patches. Section 11.1 discusses as future work

the possibility of using ﬁle systems that support delta storage to solve this problem.

7.6. Related work

This section compares Nix’s suitability for deployment to other tools. Section 1.2 already

touched on some of these. Most deployment tools are based on a destructive upgrade

paradigm. Thus non-interference is not guaranteed at all, rollbacks are not easily sup-

ported, and upgrading is certainly not atomic. Every system depends on the deployer to

specify correct runtime dependencies, although some systems have (optional) provisions

to determine build time dependencies. All systems have a fairly strict separation between

source and binary deployment, if both are supported at all. Interestingly, in terms of sup-

porting correct deployment, the tools that come nearest to the ideal are not classical de-

ployment systems (e.g., package managers) but “developer side” SCM systems such as

Vesta, as discussed below.

Traditional Unix deployment systems What follows is a list of previous, “conven-

tional” approaches to deployment in the Unix community, in more or less chronological

order. I then contrast these approaches to Nix.

The Depot [116] is a deployment scheme for heterogeneous environments based on

sharing components through Sun’s Network File System (NFS) [23] or a similar technol-

ogy. Such sharing is trivial in homogeneous environments, as each client machine can

mount and use exactly the same server ﬁle system. In a heterogeneous environment, how-

ever, it may be necessary to build components for many different platforms. The Depot

stores each component in its own isolated directory hierarchy (e.g., /depot/.primary/anApp),

with subdirectories for sources, platform-independent ﬁles (e.g., .../include), and platform-

dependent ﬁles (e.g., .../arch.sun4-os4). On the client machines, a platform-speciﬁc view

on these components is synthesised. For instance, the component may be made available

under /depot/anApp, with the platform-speciﬁc directory (e.g., /depot/.primary/anApp/-

arch.sun4-os4) mounted on /depot/anApp/arch. Thus, each client has the same logical

view of the component; e.g., /depot/anApp/arch/bin/foo always denotes the foo program for

the current platform. Of course, the client view can also be implemented through symlinks

or copying.

The Depot paper does not describe how components are built from sources. That is, it

does not appear to have an analogue to Nix expressions. Indeed, an interesting aspect of

200

7.6. Related work

Depot-like approaches is that they support deployment, but are lacking in certain conﬁg-

uration management aspects. For instance, most Depot papers leave unspeciﬁed how the

components in the Depot are created. Presumably, these are “manually” built by the system

administrator by unpacking sources in the appropriate directory, building the component,

and installing it to the appropriate target location. This means that the build is not under

CM control, so there is neither reproducibility nor traceability.

Another system called Depot is described in [32]. It uses symbolic links to activated

components, like Nix’s user environments. This is a very common technique; e.g., GNU

Stow [78] uses it as well.

Depot-Lite [143] improves on Depot. It has a primitive scanning approach for ﬁnding

retained dependencies: component directories are scanned for the names of other com-

ponent directories (e.g., tk-3.3). However, such names are not very unique (contrary to

cryptographic hashes) and may lead to many false positives. Many aspects of the tool are

rather ad hoc. For instance, “safe” uninstallation is performed by making the component

unreadable, then making it readable again if users complain during a certain time window.

Similarly, Depot-Lite allows normal users to install software, but the security model is that

components are made read-only after a “stabilisation period” of 21 days.

Depot and its descendents (including Store [26], Local Disk Depot [181], SEPP [125]

and GNU Stow [78]) are based on a notion of isolation between components, i.e., storing

components in their own private directory trees in the ﬁle system. Many other deployment

tools work within the traditional Unix ﬁle system organisation, i.e., installing components

into directories such as /usr/bin.

The FreeBSD Ports Collection [73] is an example of such a deployment system. As

described in Section 1.2, it automatically fetches and builds components from sources,

recursively installing dependencies as well. Thus the build process is at least mostly repro-

ducible. As [84] points out, a problem with the FreeBSD Ports Collection is that it embeds

dependency information into Makeﬁles, which makes it hard to query dependencies. A

more recent source deployment system is Gentoo Linux’s Portage [77], which places par-

ticular emphasis on customisability of components through its USE ﬂags mechanism.

The Red Hat Package Manager (RPM) [62] is also designed to support the conventional

Unix ﬁle system layout, but it makes the chaos of that layout manageable by tracking com-

ponent metadata in a database. This metadata allows an administrator to query what ﬁles

belong to what packages, and so on—an invaluable capability sorely lacking in many other

systems. The knowledge provided in the database also allows RPM to prevent operations

that would violate correctness constraints (e.g., two components occupying the same paths

in the ﬁle system, or removal of a component that is required by another component).

RPM operates at the package storage level, rather than the deployment level. That is, it

has no way to fetch missing dependencies automatically. Deployment systems such as the

FreeBSD Ports Collection, yum [174], and Debian’s venerable apt (Advanced Packaging

Tool) [149] maintain knowledge of repositories of remotely available packages and can

fetch these on demand. Thus they are complete deployment systems.

So how do all these systems compare to Nix? All deployment tools that use the tradi-

tional Unix ﬁle system layout must necessarily be based on destructive upgrading. Thus

they cannot support atomic upgrades or rollbacks, unless the underlying ﬁle system sup-

ports transactions, that is, when an arbitrarily long sequence of ﬁle system operations can

be wrapped into a single atomic transaction. (No widely used ﬁle system provides this

201

7. Software Deployment

capability.) Note that it is not enough for the ﬁle system to use transactions in its own

implementation, e.g., to ensure recoverability; transactions should be made available to

user space processes. Thus the operating system should have APIs to make transactions

available to user space (conventional ﬁle system APIs do not). An example is the Inversion

ﬁle system [128]. Deployme [129] is a system that uses unspeciﬁed existing deployment

tools to implement automatic rollback on failure of the new components (i.e., when an au-

tomatic test set fails). However, automatic rollback on failure was found to be undesirable

by users.

In a destructive model, side-by-side deployment of versions and variants is only possible

if the package author speciﬁcally arranges it (by making sure that ﬁles in different versions

or variants have non-overlapping paths).

None of the systems described above have any fundamental means to prevent undeclared

dependencies. Some use speciﬁc tricks such as looking at ELF executable headers to ﬁnd

library dependencies, or non-portable methods such as tracking all ﬁle system accesses

performed by builds (but note that this fails to discover retained dependencies!).

Automatic garbage collection of unused components is a rare feature, since it requires

a notion of “root” components. In systems such as RPM, even though the system knows

the dependency graph between components, it does not know what components constitute

roots (e.g., applications) and so can never delete components automatically.

Nix’s transparent source/binary deployment is a fairly unique feature. Gentoo does have

a primitive form of transparent source/binary deployment, but correspondence between

sources and binaries is only nominal. That is, dependencies, build ﬂags, and so on are not

taken into account. It is not widely used for general deployment, only for mass deployment

in speciﬁc environments.

A neat feature that Nix does not support is automatic, on-demand installation of appli-

cations. There are few deployment systems that do: Zero Install [112] is an exception.

When a user starts a program, the ﬁle system access is trapped and Zero Install automat-

ically downloads the program from a remote repository. Of course, the problem with on-

demand installation is that the remote component repository might not be available when

it is needed.

Some environments such as Windows and Mac OS X dispense with having a system-

wide, standard deployment system altogether

. That is, they do not have a package man-

agement system such as RPM or Nix that maintains a global view of all components in the

system along with their interdependencies. Rather, actions such as upgrading and uninstal-

lating are the responsibility of each application. This has important consequences. First,

it is impossible for administrators to do queries about components in a generic way. For

instance, they cannot ask to what component a given ﬁle belongs. Second, since each

application is essentially independent, there can be no dependencies. As a result, appli-

cations in these environments are all but required to be monolithic: they must contain all

their dependencies, i.e., each application must be its own closure.

Integrated CM systems An interesting class of systems that might support deploy-

ment is integrated conﬁguration management systems such as Vesta [92, 93, 94],

Recent versions of Windows do contain the Windows Installer, which provides a standardised method to install

packages. However, its use is not yet universal.

202

7.6. Related work

ClearCase [109], and Camera [114, 58, 59]. These provide more than mere source ver-

sion management: they also provide build management. All three use a virtual ﬁle system

(implemented through the NFS protocol) to track ﬁle system accesses, allowing detection

of build-time dependencies. However, they are classical CM systems in that they support

the development process, but are not intended to extend further along the software time-

line, i.e., to deployment on end-user machines (though ClearCase can be used to support

deployment in conjunction with IBM’s Tivoli Conﬁguration Manager [83]). That is, they

are used to build software artifacts such as executables, which are then shipped to the

clients through some unrelated mechanism, outside the scope of the CM system. It is a

fairly recent insight that CM tools should extend to deployment [167, 168].

However, perhaps we can use these tools to support deployment already, i.e., it is just a

matter of focus rather than underlying technology? This is certainly possible, but Nix has

some important differences:

• Since hashes are explicit in paths, we can scan for retained dependencies. The sys-

tems mentioned above can only detect dependencies when they are accessed, e.g.,

at build time. So we cannot compute a closure in advance, and it is possible that

we discover too late that the deployment is incomplete. This makes them unsuitable

for general deployment—that is, more restrictive constraints on components than the

assumptions in Section 6.8 must be made.

• Nix focuses on deployment. Quite a few aspects that have been carefully designed

in Nix to support deployment are absent in integrated CM systems, such as a no-

tion of user environments, various deployment models (source, binary, transparent

source/binary), secure sharing of a store, and so on.

• Nix does not rely on non-portable operating system extensions, i.e., a virtual ﬁle

system. Such extensions may be ﬁne for development systems (though this is ques-

tionable), but they are a barrier to use as a deployment tool, as each user must install

the extension.

• Likewise, integration of version management and build management is a barrier to

use, as it potentially forces developers to switch both their version and build man-

agement tools to the integrated system, even when they are only interested in one of

its aspects.

A nice property of Camera is that builds are performed in a chroot environment [114,

Section 8.3.2]. A chroot is a Unix feature that allows a process’s ﬁle system accesses to be

conﬁned to a part of the ﬁle system by setting its root directory to a speciﬁc directory [152].

This allows Camera to prevent undeclared dependencies.

.NET and Java Web Start Section 1.2 brieﬂy discussed Microsoft’s .NET [17, 154],

which can store components in a Global Assembly Cache. Assemblies are essentially

.NET’s units of deployment. Assemblies have unique strong names that allow versions or

variants to coexist. While this is a big step forward from previous deployment technologies

in the Windows environment, it differs in important ways from the work described in this

thesis:

203

7. Software Deployment

• It covers only executable resources, i.e., the assemblies in the GAC. It cannot see

dependencies caused by, e.g., paths speciﬁed in conﬁguration ﬁles.

• Most components are stored in unmanaged parts of the ﬁle system, i.e., outside of

the GAC.

• There is no connection between a strong name and the contents of a component.

Thus it is possible for multiple components produced by a vendor to have the same

strong name. In practice this is not a problem.

• It is bound to a speciﬁc component technology.

The latter criticism also applies to technologies such as Sun’s Java Web Start [117], which

enables web-based deployment of Java applications.

Nix on the other hand deals with components at the most fundamental level: their storage

in the ﬁle system. Thus it is agnostic with respect to particular component technologies.

Software updaters Many recent software products come with their own built-in package

management system (Vendor Product Updaters in the terminology of [102]), primarily

to support automatic updating and management of extensions. Such applications include

Mozilla Firefox, Eclipse, Microsoft Windows XP, and many others. These built-in updaters

have a terrible problem: they interact very badly with system-wide package management

systems such as Nix or RPM. A package manager assumes that it alone modiﬁes the set

of installed components. If a component modiﬁes itself or other components, the package

manager’s view of the system is no longer consistent with reality.

Furthermore, package managers provide useful functionality that product-speciﬁc up-

daters must reimplement—poorly. Indeed, they are often implemented in an ad hoc fash-

ion and ignore many important issues. Dependency management, side-by-side versioning,

rollbacks, and traceability are almost never properly supported. In particular, operation

in a multi-user environment—where the running component typically does not have write

permission to its own installation ﬁles—is almost always neglected. For instance, Firefox’s

automatic updater simply will not work in such environments.

The deployment lifecycle In [168, 24] the deployment process is structured into a life-

cycle consisting of various activities such as release, install, activation, deactivation, adapt,

update, adaptation, de-install and de-release. All of these activities are explicitly present

in Nix and the policies described in Section 7.4, except for the following. Adaptation

(modifying the conﬁguration of installed components) is not an explicit operation. Rather,

adaptation is typically done by modifying a Nix expression and re-installing it (cf. the Fire-

fox wrapper in Section 7.1.1). Also, components are not de-installed explicitly but rather

are removed implicitly by the garbage collector. Finally, the various policies do not have

a well-deﬁned way to de-release a component (i.e., make it unavailable). Of course, we

can simply remove the Nix expressions, manifests, and binaries from the distribution site.

However, this removal does not automatically propagate to clients; for instance, clients

might have manifests that refer to removed binaries.

Surveys of the extent to which various deployment tools support the deployment lifecy-

cle can be found in [24, 102].

204

7.6. Related work

The Software Dock The Software Dock [85, 84] is a distributed, agent-based deploy-

ment framework. It consists of two types of “docks”: release docks that represent com-

ponent producers and maintain information about available releases, and ﬁeld docks that

represent clients and maintain information about the state of the local machines (e.g., al-

ready installed software). An important aspect of the Software Dock is that it supports the

full lifecycle mentioned above. For instance, release docks can inform ﬁeld docks about

new releases. This is implemented through the Siena event service [25]. Software com-

ponents, when deployed to a client, are accompanied by agents, which are programs that

implement component-speciﬁc behaviour for lifecycle activities such as upgrades. An im-

portant part of the Software Dock is the Deployable Software Description (DSD), which

is a standard language for describing software system families; i.e., it allows variability in

related sets of components to be described.

The most important difference between the Software Dock and the present work is that

Nix addresses low-level building and storage of components. That is, it enforces isolation,

complete dependencies, and so on, by imposing a discipline on the storage of components.

In the Software Dock, it is up to the various agents that perform deployment activities how

to store components in the ﬁle system, whether to support multiple versions or variants

side-by-side, whether to support rollbacks, and so on.

The Software Dock’s main strengths lie in its high-level policies that support the entire

lifecycle of large systems, while this thesis emphasises the low-level aspects of compo-

nents. For instance, none of the deployment policies in Section 7.4 are truly “push” based.

Channels crudely can serve as a push model by having clients poll at short intervals, but

this is clearly a poor approach. Thus, closer cooperation between producers and clients,

such as enabled by the Software Dock’s event model, would allow more deployment poli-

cies to be supported. However, this is something that can be implemented in policies; it

does not affect the basic Nix mechanisms.

Release management Van der Hoek et al. [169] list a number of requirements on de-

ployment systems. It is instructive to determine to what extent Nix meets these require-

ments.

• “Dependencies should be explicit and easily recorded.” Enforcing correct depen-

dencies was one of the main drivers behind the present research. As we have seen in

Section 7.1.5, Nix is successful in preventing undeclared dependencies.

• “A system should be available through multiple channels.” The policy-freeness of

the substitute mechanism allows Nix to support a wide variety of distribution chan-

nels, such as HTTP, installation CD-ROMs, etc.

• “The release process should involve minimal effort on the part of the developer.”

This is somewhat hard to quantify, but Nix has several features that reduce devel-

oper effort. Nix expressions allow a system to be rebuilt automatically when the

developer changes some part of it. Transparent source/binary deployment removes

the burden of having to create binary packages explicitly (other than executing a nix-

push command on the source “package”). Isolation and correct dependencies reduce

maintenance effort by making components more robust.

205

7. Software Deployment

• “The scope of a release should be controllable,” i.e., developers should be able to

restrict releases to certain parties. Access control of releases is an orthogonal issue

to the low-level Nix mechanisms. For instance, if components are deployed through

a channel accessed through HTTP, then the web server can be conﬁgured with ap-

propriate access controls.

• “A history of retrievals should be kept.” Again, this is a feature that can be provided

by, e.g., a web server.

• “Sufﬁcient descriptive information should be available” to the user. This is currently

missing: there is nothing in Nix expressions to describe a component, other than its

name attribute. However, additional attributes can be easily added (e.g., a description

attribute) and presented to the user through commands such as nix-env or on web

pages for one-click installations.

• “Physical distribution should be hidden,” i.e., the user should not have to be aware

of the physical locations of components. Nix expressions frequently involve many

physical locations (e.g., calls to fetchurl), but these are all handled automatically and

do not concern the user (unless, of course, one of the locations becomes unavailable).

• “Interdependent systems should be retrievable as a group.” Dependent components

(e.g., servers that access each other) can always be composed into a wrapper compo-

nent that starts and stops them as appropriate (examples of this are shown in Chap-

ter 9). Assertions in Nix expressions can express consistency requirements between

components.

• “Unnecessary retrievals should be avoided.” This is certainly the case: once a store

path is valid, it does not need to be built or downloaded again.

Deployment languages Hall et al. [86] list a number of requirements for a software

deployment language, i.e., a formalism that contains the necessary information to support

the deployment process. They identify the following bits of information that must be

expressible in such a language:

• Assertion constraints express consistency requirements between components. The

Nix expression language provides assertions for this purpose. However, many con-

straints of this type are actually unnecessary in the purely functional model, since

constraints cannot be invalidated after a component has been built. For instance, the

paper gives an example of a Java application that requires exactly version 1.0.2 of

the Java Virtual Machine. This constraint can be true at installation time, but become

invalid if the JVM is updated later on. In Nix, if the JVM is part of the closure of the

application, this situation cannot happen.

• Dependency constraints are a more ﬂexible kind of constraint that assertion con-

straints.

• Artifacts are the things that make up the components in the system. Nix expressions

list all artifacts necessary to build components from source. However, they do not

206

7.6. Related work

list the artifacts in the resulting derivation outputs. This means, for instance, that

there currently is no way to query what component provides a certain command

(e.g., bin/ﬁrefox) since that information is only available after components have been

built—it does not follow from the Nix expression.

• Conﬁguration information describes the variability provided by components. Func-

tions in Nix expressions serve this role.

• Activities are the types of deployment actions that a component or system supports,

such as installation, upgrading, and uninstallation. But there might be many others,

such as starting or stopping a server component, so the language must be ﬂexible.

The Nix expression language has no direct provisions for supporting such activities,

but we can support them as a policy on components. For instance, Chapter 9 shows

how server components can support start and stop actions simply by providing a

script called control in their output paths.

Binary patching Binary patching has a long history, going back to manual patching of

binaries on mainframes in the 1960s, where it was often a more efﬁcient method of ﬁxing

bugs than recompiling from source. Binary patching has been available in commercial

patch tools such as .RTPatch, and interactive installer tools such as InstallShield. Most

Unix binary package managers only support upgrades through full downloads. Microsoft

recently introduced binary patching in Windows XP Service Pack 2 as a method to speed

up bug ﬁx deployment [39]. Recent releases of SuSE Linux provide Delta RPMs, which

are binary patches between serialisations of the contents of RPM packages in cpio archive

format. This is quite similar to the method of patch computation described in Section 7.5.1.

An interesting point is that the cpio archive format does not have a well-deﬁned canonical

form. So it is possible that in the future patches may fail to apply due to cpio implementa-

tion differences. Also, no automatic patch chaining is provided.

A method for automatically computing and distributing binary patches between

FreeBSD releases is described in [131]. It addresses the additional complication that

FreeBSD systems are often built from source, and the resulting binaries can differ even

if the sources are the same, for instance, due to timestamps being stored in ﬁles. In the

Nix patching scheme we guard against this possibility by providing the MD5 hash of the

archive to which the patch applies. If it does not apply, we fall back to a full download. In

general, h owever, this situation does not occur because patches are obtained from the same

source as the original binaries.

In Nix, the use of patches is completely hidden from users, who only observe it as a

speed increase. In general, deployment methods often require users to ﬁgure out what ﬁles

to download in order to install an upgrade (e.g., hotﬁxes in Windows). Also, if sequences

of patches are required, these must be applied manually by the user, unless the distributor

has consolidated them into a single patch. The creation of patches is often a manual and

error-prone process, e.g., ﬁguring out what components to redeploy as a result of a security

bug like [1]. In our approach, this determination is automatic.

The bsdiff program [132] that Nix uses to generate patches is based on the qsufsort

algorithm [108]. In our experience bsdiff outperformed methods such as ZDelta [162] and

207

7. Software Deployment

VDelta [97, 107], but a comparison of delta algorithms is beyond the scope of this thesis.

An overview of some delta algorithms is given in [97].

The problem of keeping derivates consistent with sources and dependency graph speci-

ﬁcations occurs in all build systems, e.g., Make [56]. To ensure correctness, such systems

must rebuild all dependent objects if some source changes. If a source is fundamental,

then a large number of build actions may be necessary. So this problem is not unique in

any way to Nix. However, the problems of build systems affect developers, not end-users,

while Nix is a deployment system ﬁrst and foremost. This is why it is important to ensure

that end-users are not affected by the use of a strict update propagation semantics.

208

8. Continuous Integration and Release

Management

This chapter shows that the Nix system can be applied to the problem of running a build

farm, which is a tool to support continuous integration and automated release manage-

ment. The former is the practice of automatically building and testing revisions of software

components during the development process. The latter is the process of automatically pro-

ducing releases of components that can be downloaded and installed by users. The two are

related: if a software revision builds correctly and passes the tests, the build can be made

available as a release. In fact, such a release can be made available to Nix users through

the mechanisms described in Section 7.4, such as channels or one-click installations.

8.1. Motivation

Continuous integration [72] is a good software engineering practice. The idea is that each

software development project should have a fully automated build system. Then we can

run the build system automatically to continuously produce the most recent version of the

software. Every time a developer commits a change to the project’s version management

system, the continuous integration system checks out the source of the project, runs its

automated build process, and creates a report describing the result of the build. The latter

might be presented on a web page and/or sent to the developers through e-mail.

Of course, developers are supposed to test their changes before they commit. The added

advantage of a continuous integration system (apart from catching developers who don’t

test their changes) is that it allows much more in-depth testing of the component(s) being

developed:

• The software may need to be built and tested on many different platforms (i.e., porta-

bility testing). It is infeasible for each developer to do this before every commit.

• Likewise, many projects have very large test sets (e.g., regression tests in a compiler,

or stress tests in a DBMS) that can take hours or days to run to completion.

• It may also be necessary to build many different variants of the software. For in-

stance, it may be necessary to verify that the component builds with various versions

of a compiler.

• Developers will typically use incremental building to test their changes (since a full

build may take too long), but this is often unreliable with many build management

tools (such as Make). That is, the result of the incremental build might differ from a

full build.

209

8. Continuous Integration and Release Management

• It ensures that the software can be built from the sources under revision control. In

systems such as CVS [60] and Subversion [137], developers commonly forget to

place source ﬁles under revision control.

• The machines on which the continuous integration system runs ideally provides a

clean, well-deﬁned build environment. If this environment is administered through

proper SCM techniques, then builds produced by the system can be reproduced.

In contrast, developer work environments are typically not under any kind of SCM

control.

• In large projects, developers often work on a particular component of the project,

and do not build and test the composition of those components (again since this

is likely to take too long). To prevent the phenomenon of “big bang integration”,

where components are only tested together near the end of the development process,

it is important to test components together as soon as possible (hence continuous

integration).

A continuous integration system typically sits in a loop building and releasing software

components from a version management system. These are its jobs. For each job, it

performs the following tasks:

1. It obtains the latest version of the component’s source code from the version man-

agement system.

2. It runs the component’s build process (which presumably includes the execution of

the component’s test set).

3. It presents the results of the build (such as error logs) to the developers, e.g., by

producing a web page.

A continuous integration system can also produce releases automatically. That is, when

an automatic build succeeds (and possibly when it fails!), the build result can be packaged

and made available in some way to developers and users. For instance, it can produce

a web page containing links to the packaged source code for the release, as well as bi-

nary distributions. The production of releases ﬁts naturally in the actions described above:

the build process of the component should produce the desired release artifacts, and the

presentation of the build result will be the release web page.

The machines on which the continuous integration system runs are sometimes referred

to as a build farm [91], since to support multi-platform projects or large sets of projects, a

possibly large number of machines is required. (As a “degenerate case”, a build farm can

also be a single machine.)

However, existing build farm tools (such as CruiseControl, Anthill, and Tinderbox) have

various limitations, discussed below.

Managing the environment One of the main problems in running a build farm is its

manageability. Build farms scale poorly in terms of administrative overhead. The jobs that

we want to run in the build farm require a certain environment (such as dependencies).

Thus we have to make sure that the proper environment exists on every machine in the

210

8.1. Motivation

build farm. If this is done manually, the administration effort of the build farm scales

linearly in the number of machines (Θ(n)).

Suppose that we want to build a component that requires a certain compiler X. We

then have to go to each machine and install X . If we later need a newer version of X,

the process must be repeated all over again. An ever worse problem occurs if there are

conﬂicting dependencies. A real-life example is that some components require different,

mutually exclusive versions of the Autoconf package [64]. That is, simply installing the

latest version is not an option. Of course, we can install these components in different

directories and manually pass the appropriate paths to the build processes of the various

components. But this is a rather tiresome and error-prone process.

Of course, Nix nails this problem: since Nix expressions describe not just how to build

a single component but also how to build all its dependencies, Nix expressions are an

excellent way to describe build jobs. Also, the problem of dealing with variability in the

environment (such as conﬂicting dependencies), are automatically resolved due to Nix’s

hashing scheme: different dependencies end up in different paths, and Nix takes care of

calling builders with the appropriate paths to dependencies. Finally, Nix’s support for

distributed and multi-platform builds (through the build hook mechanism of Section 5.5.2)

addresses the scalability problem: as we will see below, a conﬁguration change needs to

be made only once (to the Nix expression), and Nix through the build hook will take care

of rebuilding the new conﬁguration on all platforms.

Distributed and multi-platform builds Another problem in supporting multi-platform

projects is how to actually perform the build. Should each machine in the build farm

independently perform jobs? If yes, then it is hard to generate a combined multi-platform

release page. Therefore a centralised model is preferable, in which a single designated

machine selects jobs to build and forwards build actions to particular machines in the build

farm. This is far from trivial, since it entails copying build inputs and outputs to and from

machines. In fact, a build action on one machine can depend on the result of a build action

on another (an example of which will be shown below).

Again, the build hook mechanism enables a solution to this problem, since a single Nix

expression can describe derivations for different platforms (i.e., different values for the

system attribute). Derivations with different system values can also depend on each other.

The build hook described below takes care of copying the appropriate closures between

machines.

Building multiple conﬁgurations It should be easy to build components in many dif-

ferent variants, as mentioned above. A special case is building many different composi-

tions, as this can reveal information useful from the perspective of continuous integration.

Consider the real-life example shown in Figure 8.1 of two compilers: Stratego [176], a

language based on strategic term rewriting; and Tiger, an implementation of the Tiger lan-

guage [4] in Stratego. Suppose that both compilers have stable releases (e.g., Stratego 0.16

and Tiger 1.2) but are also under active development. Then there are various kinds of in-

formation that the Stratego and Tiger developers might want to obtain from the continuous

integration system:

• The Tiger developers want to know whether the most recent development version of

211

8. Continuous Integration and Release Management

Stratego 0.16

Tiger 1.2 Tiger HEAD

Stratego HEAD

Figure 8.1.: Multiple ways of combining component revisions

Tiger (its HEAD revision) still builds. Here it is appropriate to build Tiger against a

stable release of Stratego (i.e., 0.16), since when a build failure occurs the developers

can then be reasonably certain that the cause is in Tiger, not Stratego.

• However, the Tiger developers may also want to know whether their current source

base is synchronised with possible changes in the Stratego compiler; i.e., whether

the HEAD revision of Tiger builds against the HEAD revision of Stratego.

• Likewise, the Stratego developer may want to build the stable release of Tiger against

the HEAD revision of Stratego, so that Tiger can act as a large regression test for

Stratego.

This pattern is quite common in large projects where development is split among several

development groups who every so often make releases available to other groups. It is quite

easy to implement building all these variants, since we can just turn the Nix expressions

that build the various components into functions that accept their dependencies as param-

eters (as in Section 2.2).

Scheduling policies A build farm should support a variety of different scheduling poli-

cies. For instance, rather than building after each commit, the build can be performed on

a ﬁxed schedule, e.g., at 02:00 at night every day. This is often referred to as a “daily

build” [118]. It is however generally preferable to build as quickly as possible to ensure

speedy feedback to developers. This is sometimes infeasible. In a build farm used by sev-

eral jobs, a particular job may take a very long time to build, starving the other jobs. In that

situation it may be better to schedule the offending job to run (say) at most once a week,

or to run at a ﬁxed time of day.

More elaborate policies are also possible in a build farm that cleanly supports building

of different variants of a component. Consider for instance a compiler project that must

be built on many machines and has a huge regression test set. We can schedule two jobs

for this project: a continuously running quick one that builds the compiler on only one

platform, and runs only a few tests; and a weekly scheduled slow one that builds on all

platforms and runs all tests.

8.2. The Nix build farm

This section gives a sketch of the build farm that we implemented using Nix. The build

farm at present is not much more than a set of fairly simple scripts to run jobs, to build

212

8.2. The Nix build farm

<input id='job' type='svn'

url='https://.../repos/trace/release/trunk' />

139

<input id='patchelfHead' type='svn'

url='https://.../repos/trace/patchelf/trunk />

<job-script>generic-dist/build+upload.sh</job-script>

140

<arg>./jobs/nix/patchelf.nix</arg>

141

<arg>patchelfHeadRelease</arg>

142

<arg>http://nix.cs.uu.nl/dist/stratego</arg>

143

<notify-address>somebody@example.org</notify-address>

144

</job>

Figure 8.2.: A build job (simpliﬁed)

the desired release products such as various kinds of binary distributions, and to produce

release pages. The “heavy lifting” of managing the environment is provided by the Nix

technology described in previous chapters. Thus, the process of adding a job to the build

farm consists essentially of writing Nix expressions that describe components and their

dependencies.

The current Nix build farm consists of a number of components. At the highest level,

there is a supervisor script (supervisor.pl) that reads build jobs from a ﬁle (jobs.conf) and

executes them in circular order. The jobs ﬁle is in XML format. Figure 8.2 shows an ex-

ample of the declaration of a build job for the HEAD revision of the PatchELF component

(a small component that we saw on page 179). It speciﬁes the locations of the inputs to the

build (as URLs of Subversion repositories) 139 , the name of the script that performs the

job 140 , and its command-line arguments 141 . Each input is fetched from its Subversion

repository. The path of the job script is relative to the input that has ID job.

The supervisor sends e-mail notiﬁcation if a job fails (or if it succeeds again after it has

failed previously) to the address speciﬁed in the job 144 . To prevent a ﬂood of repeated

e-mail messages for a failing build, after a job fails, the supervisor will not schedule it

again until a certain time interval has passed. This interval increases on every failure using

a binary exponential back-off method.

Note that the supervisor is completely Nix-agnostic: it does not care how jobs are per-

formed. It is up to the job script to perform the build in some arbitrary way.

The job script build+upload.sh actually uses Nix to perform a build. It instantiates and

builds a Nix expression speciﬁed as a command-line argument (e.g., ./jobs/nix/patchelf.nix

at 141 ). This Nix expression is actually a function that takes as arguments the paths of

the inputs declared in the XML job declaration (i.e., at

139 ), and the target URL of the

release page (speciﬁed at 143 ). The job script expects the top-level derivation in the Nix

expression to produce a release page, which is an HTML page describing the release, plus

an arbitrary set of ﬁles associated with the release (such as source or binary distributions,

manuals to be placed online, and so on).

Figure 8.3 shows the Nix expression ./jobs/nix/patchelf.nix that builds a release for

PatchELF. Release pages are produced by the function patchelfRelease 149 that accepts

a single argument input that points to the component’s source code as obtained by the

script build+upload.sh by performing a checkout from PatchELF’s Subversion repository.

213

8. Continuous Integration and Release Management

inputs: distBaseURL:

145

with (import ../..) inputs.nixpkgs.path;

146

rec {

patchelfTarball = input: svnToSourceTarball "patchelf" input {

147

inherit (pkgs) stdenv fetchsvn;

buildInputs = [ pkgs.autoconf pkgs.automake ];

};

patchelfNixBuild = input: pkgs: nixBuild (patchelfTarball input) {

148

inherit (pkgs) stdenv;

};

patchelfRelease = input: makeReleasePage {

149

fullName = "PatchELF";

contactEmail = "eelco@cs.uu.nl";

sourceTarball = patchelfTarball input;

nixBuilds = [

(patchelfNixBuild input pkgsLinux)

];

inherit distBaseURL;

};

patchelfHeadRelease = patchelfRelease (inputs.patchelfHead);

150

}

Figure 8.3.: Build farm Nix expression for PatchELF

A release page produced by patchelfRelease is shown in Figure 8.4.

The actual production of the release page is done by the generic release page builder

function makeReleasePage (brought into scope in the with-expression at 146 ). It accepts

many arguments, only some of which are shown in the PatchELF example:

• The name of the component (e.g., "PatchELF").

• A contact e-mail address placed on the generated release pages.

• The target URL (e.g., distBaseURL) of the release page. The release page builder

does not perform uploads itself (since that is impure) but it needs the target URL for

self-references in the release page.

• A derivation that builds a source distribution (sourceTarball), e.g., the ﬁle patchelf-

0.1pre3663.tar.gz that can be downloaded, compiled, and installed by users. The

source distribution is produced from the Subversion sources (i.e., input) by the func-

tion patchelfTarball 147 . (A tarball is a Unix colloquialism for a source distribution.)

Here too the actual work is done by an external generic function svnToSourceTarball.

214

8.2. The Nix build farm

Figure 8.4.: Release page for PatchELF

• A list of derivations that perform normal builds of the component from the source

distribution (nixBuilds). These are used to automatically populate a channel to which

users can subscribe, and to generate packages for one-click installation (see Sec-

tion 7.4). In this case the builds are produced by the function patchelfNixBuild 148 ,

which in turn uses the (poorly named) generic function nixBuild.

• Similarly, makeReleasePage accepts attributes for the component’s manual, cover-

age analysis builds, and RPM packages.

The top-level derivation is produced by evaluation of the value patchelfHeadRelease

150 . The name of this attribute was speciﬁed in the XML job description at 142 . Building

of this derivation will produce the release page and all the distributions included on the

release page (in the example, a source distribution and a Nix channel distribution).

The build+upload.sh script, as its name implies, not only builds the derivation but also

uploads the release page to the server. Each release is stored under its own URL, e.g.,

http://nix.cs.uu.nl/dist/nix/patchelf-0.1pre3663/. It also performs a nix-push to build and

upload the Nix expressions in the channels provided by the release. The uploading of

the release is assisted by a server-side CGI script that stores the uploaded ﬁles and, when

the upload is done, updates various index pages listing the release. Figure 8.5 shows the

215

8. Continuous Integration and Release Management

Figure 8.5.: Release overview

automatically generated index of the most recent releases, concisely showing the extent to

which each release succeeded.

Reproducing releases An important conﬁguration management property is the ability

to reproduce releases in the future. E.g., when we need to ﬁx a bug in some old release

of a component, we need to be able to reproduce the entire build environment, including

compilers, libraries, and so on. So it is important that we have a record that describes

exactly what inputs went into a release.

Therefore the build farm stores a ﬁle job.xml as part of every release, e.g., under http:

//nix.cs.uu.nl/dist/nix/patchelf-0.1pre3663/job.xml. This ﬁle is the same as the XML job

description that went into the supervisor (e.g., the one in Figure 8.2), except that each

input element that referred to a non-constant input such as a HEAD revision has been

“absolutised”. For instance, the patchelfHead input element has been changed into:

<input id='patchelfHead' type='svn'

url='https://.../repos/trace/patchelf/trunk

rev='3663' hash='b252b5740a0d...' />

That is, it no longer refers to the HEAD revision of the Subversion repository of PatchELF,

but to a speciﬁc revision. Since this job.xml is a perfectly valid build job, we can feed it

into the supervisor to reproduce the build.

Release process As can be seen in Figure 8.5, each release has a symbolic name, such

as patchelf-0.1pre3663. The current build farm implements the policy that names including

216

8.3. Distributed builds

the string pre in the version string are “unstable” releases (i.e., only intended for developers

or bleeding edge users), and are “stable” otherwise. Stable releases are intended to be kept

indeﬁnitely, while unstable releases can be deleted eventually. More importantly, stable

and unstable releases appear in separate channels. For instance, the URL of the channel

for stable PatchELF releases is

http://nix.cs.uu.nl/dist/nix/channels/patchelf-stable

while the channel for unstable releases is

http://nix.cs.uu.nl/dist/nix/channels/patchelf-unstable

The release name is computed by the build jobs themselves. The component name (e.g.,

patchelf) is generally hard-coded into the build job (e.g., at 147 ). The version name is

usually computed by taking the version number hard-coded into the source (e.g., 0.1) and

appending preN, where N is the revision number of the Subversion repository (e.g., 3663),

if the release is unstable. Whether the release is stable or unstable is also hard-coded in

the sources.

For instance, for Autoconf-based components, the release name is usually computed by

the conﬁgure script, which contains a line

STABLE=0

in the sources in the main development branch of the project (trunk in Subversion termi-

nology [137]). Thus all releases built from the main branch will be unstable releases. To

build a stable release, it sufﬁces to change the line to

STABLE=1

and rebuild.

In practice, a more controlled process is used to build stable releases. To build a stable

release, e.g., patchelf-0.1, the development branch is copied to a special release branch,

e.g., branches/patchelf-0.1-release. In this branch, the stable ﬂag is set. A one-time

job for this branch is then added to jobs.conf. After the release succeeds, the release

branch is tagged and removed. That is, branches/X-release is moved to tags/X; e.g.,

branches/patchelf-0.1-release is moved to tags/patchelf-0.1.

8.3. Distributed builds

Nix expressions allow multi-platform builds to be described and built in a centralised man-

ner. In the Nix expression for the PatchELF job, we have one channel build for Linux:

nixBuilds = [

(patchelfNixBuild input pkgsLinux)

];

Here, pkgsLinux is an attribute set consisting of the derivations in the Nix Packages collec-

tion built for i686-linux. That is, the function patchelfNixBuild accepts the dependencies to

use as an argument 148 . Note that this includes the standard environment stdenv, which

217

8. Continuous Integration and Release Management

nix@mcflurry.labs.cs.uu.nl powerpc-darwin 1

nix@losser.labs.cs.uu.nl i686-freebsd 1

nix@itchy.labs.cs.uu.nl i686-linux 1

nix@scratchy.labs.cs.uu.nl i686-linux 2

Figure 8.6.: remote-systems.conf: deﬁnition of build farm machines

determines the platform on which to build (since the function mkDerivation in stdenv sets

the system attribute for the derivation).

So if we also have sets of dependencies built for other platforms, we can trivially perform

channel builds for these platforms, e.g.,:

nixBuilds = [

(patchelfNixBuild input pkgsLinux)

(patchelfNixBuild input pkgsFreeBSD)

(patchelfNixBuild input pkgsDarwin)

];

where pkgsFreeBSD is Nixpkgs built for i686-freebsd, and pkgsDarwin is for powerpc-

darwin (i.e., Apple’s Mac OS X). Thus this Nix expression abstracts over platforms and

machines; it is up to Nix to somehow build each derivation on a machine of the appropriate

type.

Of course, the supervisor and build+upload.sh run on some particular platform, e.g.,

i686-linux. Nix cannot build derivations for other platforms, and will fail if it has to do

so (

76 in Figure 5.11). This is where build hooks come in (Section 5.5.2). The script

build+upload.sh runs Nix with the environment variable NIX_BUILD_HOOK pointing at a

concrete build hook script build-remote.pl.

This hook uses a table of machines to perform builds, remote-systems.conf, shown in

Figure 8.6. Each entry in the table speciﬁes a user account and remote host on which

a build can be performed (e.g., nix@mcﬂurry.labs.cs.uu.nl), the machine’s platform type

(e.g., powerpc-darwin), and its maximum load, which is the maximum number of jobs that

the hook will concurrently execute on the machine (e.g., 1). Generally, the maximum load

is equal to the number of CPUs in the machine. There can be multiple machines with the

same platform type, allowing more derivations to be built in parallel. The hook maintains

the load on each machine in a persistent state ﬁle.

The hook uses the Rsync protocol [163] over Secure Shell (SSH) [61] to copy the input

closures to the Nix store of the selected remote machine. The build is then performed by

running Nix on the remote machine (also executed through SSH). Finally, the output is

copied back to the local Nix store using Rsync.

8.4. Discussion and related work

This section describes some of the advantages and disadvantages of the Nix build farm

relative to other continuous integration tools.

The main advantage is the use of Nix expressions to describe and build jobs. It makes the

management of the build environment (i.e., dependencies) quite easy and scalable. This

218

8.4. Discussion and related work

aspect is completely ignored by tools such as CruiseControl [158] and Tinderbox [69],

which expect the environment to be managed by the machine’s administrator. Anthill [165]

has the notion of “dependency groups” that allows an ordering between build jobs.

Most of these systems are targeted at testing, not producing releases. Sisyphus [171]

on the other hand is a continuous integration system that is explicitly intended to support

deployment of upgrades to clients. It uses a destructive update model, which makes it easy

to use with existing deployment tools, but bars side-by-side versioning and rollbacks.

The centralised view of the build job for a release is also a big plus. Systems such as

Tinderbox have a more “anarchistic” approach: build farm machines perform jobs essen-

tially independently, and send the results to a central machine that presents them on a web

page. This is ﬁne for continuous integration per se, but is not a good model if we want

integration with release management. Since each build independently selects which revi-

sion to build, there is no guarantee that any particular revision will always be built by all

machines. Thus there is no guarantee that a complete release page will ever be made.

A fundamental downside to the Nix build farm is that by building in Nix, by deﬁnition

we are building in a way that differs from the “native” method for the platform. If a com-

ponent builds and passes the tests on powerpc-darwin, we can conclude that the component

can work on that platform; but we cannot conclude that it will work if a user were to down-

load the source and build using the platform’s native tools (e.g., the C compiler provided

in /usr/bin). That is, while the build farm builds the component in a Nix environment,

most users will use it in a non-Nix environment. This limits the level of portability testing

attained by the Nix build farm.

On the other hand, this situation can be improved by simulating the native environment

as closely as possible, e.g., by providing the same tool versions. Nevertheless, there is no

getting around the fact that the paths of those tools differ; they are in the store, not in their

native locations in the ﬁle system.

However, we can still build “native” binary distributions in some cases. For example,

we use User-Mode Linux (UML) [45] to build RPM packages for various platforms. User-

Mode Linux is a normal user space program that runs a complete Linux operating system

inside a virtual machine. Thus, the builder of the derivation that builds RPMs is entirely

pure: it simply runs UML to build the RPM.

Of course, we can also do “native” builds directly in Nix builders if we are willing to

accept a level of impurity (e.g., by adding /usr/bin to PATH). We can even test whether the

component installs properly to an impure location (such as /usr/local/my-package) if the

builder has sufﬁcient permissions. In fact, the latter need not even be impure as long as the

following conditions hold:

• Subsequent derivations do not depend on output in impure locations. Thus, the

builder should remove the impure output at the end of the build script.

• Locking should be used to prevent multiple derivations from installing into the same

impure location. E.g., derivations that want to install into /usr/local/my-package can

acquire an exclusive lock on a lock /usr/local/my-package.lock.

A more transient limitation of the prototype build farm is its poor scheduling. (A better

supervisor is currently being implemented.) It simply runs jobs after each other in a con-

tinuous loop. It supports none of the more advanced scheduling methods discussed earlier.

219

8. Continuous Integration and Release Management

It also does not run jobs in parallel, which leads to poor utilisation of the build farm. For

instance, builds on Macs generally take (much) longer than on other machines. A multi-

platform job can therefore cause many machines to be idle, as the job waits for the Mac

build to ﬁnish. It is then useful to start a new job to put the idle machines to good use.

220

9. Service Deployment

This chapter shows that Nix extends naturally into the domain of service deployment. A

software service is a set of processes running on one or more machines that cooperate to

provide a useful facility to an end-user or to another software system. An example might

be a bug tracking service, implemented through a web server running certain web applica-

tions and a back-end database storing persistent data. A service is generally implemented

through a set of software components, static data ﬁles, dynamic state (such as databases

and log ﬁles), and conﬁguration ﬁles that tie all these together.

The deployment of such software services is very often a time-consuming and error-

prone activity for developers and system administrators. In order to produce a working

service, one must typically install a large set of components, put them in the right locations,

write conﬁguration ﬁles, create state directories or initialise databases, make sure that all

sorts of processes are started in the right order on the right machines, and so on. These

activities are quite often performed manually, or scripted in an ad hoc way.

A particularly troubling aspect of common service deployment practice is the lack of

good conﬁguration management. For instance, the software environment of a machine

may be under the control of a package manager, and the conﬁguration ﬁles and static data

of the service under the control of a version management system. That is, these two parts

of the service are both under CM control. However, the combination of the two isn’t:

we don’t have a way to express that (say) the conﬁguration of a web server consists of a

composition of a speciﬁc instance of Apache, speciﬁc versions of conﬁguration ﬁles and

data ﬁles, and so on.

This means that we lack important features of good CM practice. There is no identi-

ﬁcation: we do not have a way to name what the running conﬁguration on a system is.

We may have a way to identify the conﬁgurations of the code and data bits (e.g., through

package management and version management tools), but we have no handle on the com-

position. Likewise, there is no derivation management: a software service is ideally an

automatically constructed derivate of code and data artifacts, meaning that we can always

automatically rebuild it, e.g., to move it to another machine. However, if administrators

ever manually edit a ﬁle of a running service, we lose this property.

In practice, we see that important service deployment operations are quite hard. Moving

a service to another machine can be time-consuming if we have to ﬁgure out exactly what

software components to install in order to establish the proper environment for the service.

Having multiple instances of a service (e.g., a test and production instance) running side-

by-side on the same machine is difﬁcult since we must arrange for the instances not to

overwrite each other, which often entails manually copying ﬁles and tweaking paths in

conﬁguration ﬁles. Performing a rollback of a conﬁguration after an upgrade might be

very difﬁcult, in particular if software components were replaced.

Of course, these properties—automatic derivations, side-by-side existence, rollbacks,

identiﬁability and reproducibility—are all provided by Nix. Up till now, we have shown

221

9. Service Deployment

software

data

configuration

subversion

openssl db4perl expatperlBerkeleyDB

apache

/data/subversion/http.conf

control

Figure 9.1.: Dependencies between code, conﬁguration, and data components in a Subver-

sion service

how software components can be built and deployed through Nix. This chapter shows that

simply by treating services as components, we can build and deploy those as well.

9.1. Overview

9.1.1. Service components

From the perspective of a user, a service is a collection of data in some format with a

coherent set of operations (use cases) on those data. Typical examples are a Subversion

version management service [137] in which the data are repositories and the operations

are version management actions such as committing, adding, and renaming ﬁles; a TWiki

service [157] in which the data are web pages and operations are viewing, editing, and

adding pages; and a JIRA issue-tracking service in which the data consists of entries in an

issue data-base and the operations are issue management actions such as opening, updating,

and closing issues. In these examples, the service has persistent data that is stored on the

server and that can be modiﬁed by the operations of the server. However, a service can

also be stateless and just provide some computation on data provided by the client at each

request, e.g., a translation service that translates a document uploaded by the client.

From the perspective of a system administrator, a service consists of data directories

containing the persistent state, software components that implement the operations on the

data, and a conﬁguration of those components binding the software to the data and to the

local environment. Typically, the conﬁguration includes deﬁnitions of paths to data di-

rectories and software components, and items such as the URLs and ports at which the

service is provided. Furthermore, the conﬁguration provides meta-operations for initialis-

ing, starting, and stopping the service. Figure 9.1 illustrates this with a diagram sketching

the composition of a version management service based on Apache and Subversion com-

ponents. The control component is a script that implements the meta-operations, while the

222

9.1. Overview

ServerRoot "/var/httpd"

ServerAdmin eelco@cs.uu.nl

ServerName svn.cs.uu.nl:8080

DocumentRoot "/var/httpd/root"

LoadModule dav_svn_module /usr/lib/modules/mod_dav_svn.so

AuthType Basic

AuthDBMUserFile /data/subversion/db/svn-users

...

DAV svn

SVNParentPath /data/subversion/repos

</Location>

Figure 9.2.: Parts of httpd.conf showing hosting of Subversion by an Apache server

ﬁle httpd.conf deﬁnes the conﬁguration. The software components underlying a service are

generally not self-contained, but rather composed from a (large) number of auxiliary com-

ponents. For example, Figure 9.1 shows that Subversion and Apache depend on a number

of other software components such as OpenSSL and Berkeley DB.

9.1.2. Service conﬁguration and deployment

Setting up a service requires installing the software components and all of their dependen-

cies, ensuring that the versions of the installed components are compatible with each other.

The data directories should be initialised to the required format and possibly ﬁlled with

an initial data set. Finally, a conﬁguration ﬁle should be created to point to the software

and data components. For example, Figure 9.2 shows an excerpt of an Apache httpd.conf

conﬁguration ﬁle for a Subversion service. The ﬁle uses absolute pathnames to software

components such as a WebDAV module for Subversion, and data directories such as the

place where repositories are stored.

Installation of software components using traditional package management systems is

fraught with problems. Package managers do not enforce the completeness of dependency

declarations of a component. The fact that a component works in a test environment, does

not guarantee that it will work on a client site, since the test environment may provide

components that are not explicitly declared as dependencies. As a consequence, com-

ponent compositions are not reproducible. The installation of components in a common

directory makes it hard to install multiple versions of a component or to roll back to a

previous conﬁguration if it turns out that upgrading produces a faulty conﬁguration.

These problems are compounded in the case of service management. Typically, man-

agement of conﬁguration ﬁles is not coupled to management of software components.

Conﬁguration ﬁles are maintained in some speciﬁc directory in the ﬁle system and chang-

ing a conﬁguration is a destructive operation. Even if the conﬁguration ﬁles are under

version management, there is no coupling to the versions of the software components that

they conﬁgure. Thus, even if it is possible to do a rollback of the conﬁguration ﬁles to a

previous time, the installation of the software components may have changed in the mean-

time and become out of synch with the old conﬁguration ﬁles. Finally, having a global

223

9. Service Deployment

{ stdenv, apacheHttpd, subversion }:

stdenv.mkDerivation {

name = "svn-service";

builder = ./builder.sh;

# Build script.

control = ./control.in;

# Control script template.

conf = ./httpd.conf.in;

# Apache configuration template.

inherit apacheHttpd subversion;

}

Figure 9.3.: services/svn.nix: Function for creating an Apache-based Subversion service

pkgs = import ../pkgs/system/all-packages.nix;

subversion = import ../subversion/ {

# Get dependencies from all-packages.nix.

inherit (pkgs) stdenv fetchurl openssl httpd ...;

};

webServer = import ../services/svn.nix {

inherit (pkgs) stdenv apacheHttpd;

};

Figure 9.4.: conﬁgurations/svn.cs.uu.nl.nix: Composition of a Subversion service for a spe-

ciﬁc machine

conﬁguration makes it hard to have multiple versions of a service running side by side, for

instance a production version and a version for testing an upgrade.

9.1.3. Capturing component compositions with Nix expressions

A service can be constructed just like a software component by composing the required

software, conﬁguration, and control components. For example, Figure 9.3 deﬁnes a func-

tion for producing a Subversion service, given Apache (httpd) and Subversion components.

The build script of the service creates the Apache conﬁguration ﬁle httpd.conf and the con-

trol script bin/control from templates by ﬁlling in the paths to the appropriate software

components. That is, the template httpd.conf.in contains placeholders such as

LoadModule dav_svn_module @subversion@/modules/mod_dav_svn.so

rather than absolute paths. The funcion in Figure 9.3 can be instantiated to create a concrete

instance of the service. For example, Figure 9.4 deﬁnes the composition of a concrete

Subversion service using an ./httpd.conf ﬁle deﬁning the particulars of the service. Just

like installing a software component composition, service composition can be reproducibly

installed using the nix-env command:

$ nix-env -p /nix/var/nix/profiles/svn \

-f configuration/svn.nix -i

I.e., the service component is added to the speciﬁed proﬁle. The execution of the service,

initialisation, activation, and deactivation, can be controlled using the control script with

commands such as

224

9.2. Composition of services

# Recall the old server

oldServer=$(readlink -f $profiles/$serverName || true)

# Build and install the new server.

nix-env -p $profiles/$serverName -f "$nixExpr" -i

# Stop the old server.

if test -n "$oldServer"; then

$oldServer/bin/control stop || true

# Start the new server.

$profiles/$serverName/bin/control start

Figure 9.5.: upgrade-server: Upgrading coupled to activation and deactivation

$ /nix/var/nix/profiles/svn/bin/control start

Similarly, the stop action stops the server.

9.1.4. Maintaining the service

A service evolves over time. New versions of components become available with new

functionality or security ﬁxes; the conﬁguration needs to be adapted to reﬂect changing

requirements or changes in the environment; the machine that hosts the service needs to

be rebooted; or the machine is retired and the service needs to be migrated to another

machine. Such changes can be expressed by adapting the Nix expressions appropriately

and rebuilding the service. The upgrade-server script in Figure 9.5 implements a common

sequence of actions to upgrade a service: build the new service, stop the old service, and

start the new one. Since changes are non-destructive, it is possible (through a similar com-

mand sequence) to roll back to a previous installation if the new one is not satisfactory.

By keeping the Nix expressions deﬁning a service under version management, the com-

plete history of all aspects of the service is managed, and any version can be reproduced at

any time. An even better approach is to have upgrade-server build directly from a version

management repository, rather than from a working copy. In this way we can always trace

a running service conﬁguration back to its sources in the version management system.

9.2. Composition of services

The previous section showed a sketch of an Apache-based Subversion service deployed us-

ing Nix. The major advantage is that such a service is a closure: all code and conﬁguration

elements of the service are described in the Nix expression, and can thus be reproduced at

any time. Clearly, we can deploy other services along similar lines. For instance, we might

want to deploy an Apache-based TWiki service by providing Nix expressions to build the

TWiki components and to compose them with Apache (which entails writing a parame-

terised httpd.conf that enables the TWiki CGI scripts in Apache). However, suppose we

225

9. Service Deployment

software

data configuration

subversion

openssl db4perl expat

apachetwiki

mail

/data/subversion/

/data/twiki subversion.conftwiki.conf

http.conf

control

Figure 9.6.: An Apache-based composition of the Subversion and TWiki services

now want a single Apache server that provides both the Subversion and TWiki services.

How can we easily compose such services?

We can solve this problem by factoring out the service-speciﬁc parts of httpd.conf into

separate components called subservices. That is, we create conﬁguration components

that contain httpd.conf fragments such as twiki.conf and subversion.conf. The top-level

httpd.conf merely contains the global server conﬁguration such as host names, and includes

the service-speciﬁc fragments. A sketch of this composition is shown in Figure 9.6.

Figure 9.7 shows a concrete elaboration. Here, the functions imported from

../subversion-service and ../jira-service build the Subversion and JIRA conﬁguration frag-

ments and store them under a subdirectory types/apache-httpd/conf/ in their preﬁxes. The

function imported from ../apache-httpd then builds a top-level httpd.conf that includes

those fragments. That is, there is a contract between the Apache service builder and the

subservices that allows them to be composed.

The subServices argument of the function in ../apache-httpd speciﬁes the list of service

components that are to be composed. By modifying this list and running upgrade-server,

we can easily enable or disable subservices.

9.3. Variability and crosscutting conﬁguration choices

A major factor in the difﬁculty of deploying services is that many conﬁguration choices are

crosscutting: the realisation of such choices affects multiple conﬁguration ﬁles or multiple

points in the same ﬁle. For instance, the Apache server in Figure 9.7 requires a TCP port

number on which the server listens for requests, so we pass that information to the top-

level webServer component. However, the user management interface of the Subversion

service also needs the port number to produce URLs pointing to itself. Hence, we see that

226

9.3. Variability and crosscutting conﬁguration choices

subversionService = import ../subversion-service {

httpPort = 80;

# see Section 9.3.

reposDir = "/data/subversion"; ...

};

jiraService = import ../jira-service {

twikisDir = "/data/twiki"; ...

};

webServer = import ../apache-httpd {

inherit (pkgs) stdenv apacheHttpd;

hostName = "svn.cs.uu.nl";

httpPort = 80;

subServices = [subversionService jiraService];

};

Figure 9.7.: Nix expression for the service in Figure 9.6

the port number is speciﬁed in two different places. In fact, Apache’s conﬁguration itself

already needs the port in several places, e.g.,

ServerName example.org:8080

Listen 8080

are some conﬁguration statements that can occur concurrently in a typical httpd.conf. This

leads to obvious dangers: if we update one, we can easily forget to update the other.

A related problem is that we often want to build conﬁgurations in several variants. For

instance, we might want to build a server in test and production variants, with the former

listening on a different port. We could make the appropriate edits to the Nix expression

whenever we build either a test or production variant, or maintain two copies of the Nix

expression, but both are inconvenient and unsafe.

Using the Nix expression language we have the abstraction facilities to easily support

possibly crosscutting variation points. Figure 9.8 shows a reﬁnement of the Apache com-

position in Figure 9.7. This Nix expression is now a function accepting a single boolean

argument productionServer that determines whether the instance is a test or production

conﬁguration. This argument drives the value selected for the port number, which is prop-

agated to the two components that require this information. Thus, this crosscutting param-

eter is speciﬁed in only one place (though implemented in two). This is a major advantage

over most conﬁguration ﬁle formats, which typically lack variables or other means of ab-

straction. For example, Enterprise JavaBeans deployment descriptors frequently become

unwieldy due to crosscutting variation points impacting many descriptors.

Of course, due to Nix’s cryptographic hashing scheme, building the server with different

values for productionServer (or manually editing in the Nix expression any aspect such as

the port attribute) yields different hashes and thus different paths. Therefore, multiple

conﬁgurations automatically can exist side by side on the same machine.

227

9. Service Deployment

{productionServer}:

let {

port = if productionServer then 80 else 8080;

webServer = import ./apache-httpd {

inherit (pkgs) stdenv apacheHttpd;

hostName = "svn.cs.uu.nl";

httpPort = port;

subServices = [subversionService jiraService];

};

subversionService = import ./subversion {

httpPort = port;

reposDir = "/data/subversion"; ...

};

jiraService = import ./jira {

twikisDir = "/data/twiki"; ...

};

}

Figure 9.8.: Dealing with crosscutting conﬁguration choices

9.4. Distributed deployment

Complex services are often composed of several subservices running on different ma-

chines. For instance, consider a simple scenario of a JIRA bug tracking system. This

service consists of two separately running subservices, possibly on different machines: a

Jetty servlet container, and a PostgreSQL database server. These communicate with each

other through TCP connections.

Such conﬁgurations are often labourious to deploy and maintain, since we now have

two machines to administer and conﬁgure. This means that administrators have to log in

to multiple machines, make sure that services are started and running, and so on. That is,

without sufﬁcient automation, the deployment effort rises linearly.

This section shows how one can implement distributed services by writing a single Nix

expression that describes each subservice and the machine on which it is to run. A special

service runner component will then take care of distributing the closure of each subservice

to the appropriate machines, and remotely running their control scripts.

An interesting complication is that the various machines may be of different machine

types, or may be running different operating systems. For instance, the Jetty container

might be running on a Linux machine, and the PostgreSQL database on a FreeBSD ma-

chine. As we saw in Section 8.3, we can build multi-platform Nix expressions using the

build hook mechanism.

Figure 9.9 shows the Nix expression for the JIRA example. We have two generic ser-

vices, namely PostgreSQL and Jetty. There is one concrete subservice, namely the JIRA

web application. This component is plugged into both generic services as a subservice,

228

9.4. Distributed deployment

# Build a Postgres server on FreeBSD.

postgresService = import ./postgresql {

inherit (pkgsFreeBSD) stdenv postgresql;

host = "losser.labs.cs.uu.nl";

# Machine to run on.

dataDir = "/var/postgres/jira-data";

subServices = [jiraService];

allowedHosts = [jettyService.host];

# Access control.

};

# Build a Jetty container on Linux.

jettyService = import ./jetty {

inherit (pkgsLinux) stdenv jetty j2re;

host = "itchy.labs.cs.uu.nl";

# Machine to run on.

# Include the JIRA web application at URI

path

subServices = [ { path = "/jira"; war = jiraService; } ];

};

# Build a JIRA service.

jiraService = import ./jira/server-pkgs/jira/jira-war.nix {

inherit (pkgsLinux) stdenv fetchurl ant postgresql_jdbc;

databaseHost = postgresService.host;

# Database to use.

};

# Compose the two services.

serviceRunner = import ./runner {

inherit (pkgsLinux) stdenv substituter;

services = [postgresService jettyService];

};

Figure 9.9.: 2-machine distributed service

though it provides a different interface to each (i.e., implementing different contracts). To

PostgreSQL, it provides an initialisation script that creates the database and tables. To

Jetty, it provides a WAR that can be loaded at a certain URI path.

The PostgreSQL service is built for FreeBSD; the other components are all built for

Linux. This is accomplished by passing input packages to the builders either for FreeBSD

or for Linux (e.g., inherit (pkgsFreeBSD) stdenv ...), which include the standard environ-

ment and therefore specify the system on which to build.

The two generic servers are combined into a single logical service by building a service

runner component, which is a simple wrapper component that at build time takes a list of

services, and generates a control script that starts or stops each in sequence. It also takes

care of distribution by deploying the closure of each service to the machine identiﬁed by its

host attribute, e.g., itchy.labs.cs.uu.nl for the Jetty service. For instance, when we run the

service runner’s start action, it copies each service and executes its start action remotely.

An interesting point is that the Nix expression nicely deals with a crosscutting aspect of

229

9. Service Deployment

the conﬁguration: the host names of the machines on which the services are to run. These

are crosscutting because the services need to know each other’s host names. In order for

JIRA to access the database, the JIRA web application needs to know the host name of the

database server. Conversely, the database server must allow access to the machine running

the Jetty container. Here, host names are speciﬁed only once, and are propagated using

expressions such as allowedHosts = [jettyService.host].

9.5. Experience

We have used the Nix-based approach described in this chapter to deploy a number of

services, some in production use and some in education or development. The production

services are:

• An Apache-based Subversion server (svn.cs.uu.nl) with various extensions, such as

a user and repository management system. This is essentially the service described

in Section 9.2.

• An Apache-based TWiki server (http://www.cs.uu.nl/wiki/Center), also using the

composable infrastructure of Section 9.2. Thus, it is easy to create an Apache server

providing both the Subversion and TWiki services.

• A Jetty-based JIRA bug tracking system with a PostgreSQL database back-end, as

described in Section 9.4.

Also, Nix services were used in a Software Engineering course to allow teams of stu-

dents working on the implementation of a Jetty-based web service (a Wiki) to easily build

and run the service.

In all these cases, we have found that the greatest advantage of Nix service deployment

is the ease with which conﬁgurations can be reproduced: if a developer wants to create a

running instance of a service on his own machine, it is just a matter of a checkout of the

Nix expressions and associated ﬁles, and a call to upgrade-server. Without Nix, setting up

the required software environment for the service is much more work: installing software,

tweaking conﬁguration ﬁles to the local machine, creating state locations, and so on. The

Nix services described above are essentially “plug-and-play”. Also, developer machines

can be quite heterogeneous. For instance, since Nix closures are self-contained, there are

no dependencies on the particulars of various Linux distributions that might be used by the

developers.

The ability to easily set up a test conﬁguration is invaluable, as it makes it fairly trivial

to experiment with new conﬁgurations. The ability to reliably perform a rollback, even

in the face of software upgrades, is an important safety net if testing has failed to show a

problem with the new conﬁguration.

9.6. Related work

It is not a new insight that the deployment of software should be treated as part of software

conﬁguration. In particular the SWERL group at the University of Colorado at Boulder

230

9.6. Related work

has argued for the application of software conﬁguration management to software deploy-

ment [168], developed a framework for characterizing software deployment processes and

tools [24], and implemented the experimental system Software Dock integrating software

deployment processes [85, 84]. However, it appears that our work is unique in unifying the

deployment of software and conﬁguration components.

Make is sometimes used to build various conﬁguration ﬁles. However, Make doesn’t

allow side-by-side deployment, as running make overwrites the previously built conﬁgu-

rations. Thus, rollbacks are not possible. Also, the Make language is rather primitive. In

particular, since the only abstraction mechanisms are global variables and patterns, it is

hard to instantiate a subservice multiple times. As I discuss in Section 10.3, there are other

build managers that have more functional speciﬁcation languages.

Cfengine is a popular system administration tool [22]. A declarative description of sets

of machines and the functionality they should provide is given, along with imperative ac-

tions that can realise a conﬁguration, e.g., by rewriting conﬁguration ﬁles in /etc. The

principal downside of such a model is that it is destructive: it realises a conﬁguration by

overwriting the current one, which therefore disappears. Also, it is hard to predict what

the result of a Cfengine run will be, since actions are typically speciﬁed as edit actions on

system ﬁles, i.e., the initial state is not always speciﬁed. This is in contrast to the fully

generational approach advocated here, i.e., Nix builders generate conﬁgurations fully in-

depently from any previous conﬁgurations. Finally, since actions are speciﬁed with respect

to ﬁxed conﬁguration ﬁle locations (e.g., /etc/sendmail.mc), it is not easy for multiple con-

ﬁgurations to coexist. In the present work, ﬁxed paths are only used for truly mutable state

such as databases and log ﬁles.

231

9. Service Deployment

232

10. Build Management

This chapter shows the ﬁnal application of Nix: build management. It should be clear by

now that the low-level Nix system is essentially a build manager: it manages the automatic

construction of software artifacts based on a formal description of the build actions to be

performed (a Nix expression). This is exactly what tools such as Make [56] do.

However, the derivations that we have seen up till now have been large-grained: they

build complete components such as Mozilla Firefox. Such component build actions typi-

cally consist of many smaller build steps that are performed using conventional build tools

like Make. These are called by the builders of the derivations. There is no reason why

these smaller build steps (such as the compilation of a single source ﬁle, or linking object

ﬁles together into an executable) cannot be expressed in a Nix expression directly, obviat-

ing the need for separate build managers. Indeed, there are many advantages to using Nix

as a build manager over conventional (Make-like) approaches. Nix expressions constitute

a simple but powerful language to express the building of software systems, and Nix’s

isolation properties prevent the dependency problems that plague Make-like tools.

10.1. Low-level build management using Nix

Figure 10.1 shows an example of a Nix expression that builds the ATerm library [166] from

a set of C source ﬁles. The library (in this example) has a single variation point: whether it

is build as a shared library, or as a static library. Hence it is a function that takes a boolean

parameter sharedLib 151 . The constant sources deﬁnes the source ﬁles, which reside in

the same directory as the Nix expression 153 .

The library is produced by the attribute libATerm 155 . The result of this attribute is a call

to the function makeLibrary, deﬁned externally (and imported in 152 ). This function takes

a number of attributes: the name of the library, the set of object ﬁles to link, and a ﬂag

specifying whether to create a dynamic or static library. The object ﬁles are produced by

compiling each source ﬁle in sources:

objects = map compile sources;

That it, the function compile is applied to each source ﬁle. It is merely a wrapper around

the external function compileC to specify whether the object ﬁle should support dynamic

linking 154 . (On Unix, dynamic linking requires that source ﬁles are compiled as “position

independent code” [160].)

There are a few things worth noting here. First, we did not specify any header ﬁle

dependencies for the source ﬁles in sources, even though they include a number of system

and local header ﬁles (e.g., #include "aterm1.h"). However, these dependencies are found

automatically by compileC. How it does this is discussed below.

233

10. Build Management

{sharedLib ? true}:

151

with (import ../../../lib);

152

rec {

sources = [

153

./afun.c ./aterm.c ./bafio.c ./byteio.c ./gc.c ./hash.c

./list.c ./make.c ./md5c.c ./memory.c ./tafio.c ./version.c

];

compile = main: compileC {inherit main sharedLib;};

154

libATerm = makeLibrary {

155

libraryName = "ATerm";

objects = map compile sources;

inherit sharedLib;

};

}

Figure 10.1.: Nix expression for building the ATerm library

Second, every build step only uses sources in the Nix store. For instance, each compile

action (e.g., compile ./afun.c) compiles a source that resides in the Nix store; recall that the

Nix expression translation process (Section 5.4) copies each local ﬁle referenced in a Nix

expression to the Nix store. Header ﬁles found by compileC are also copied to the store.

Third, since the builder for each derivation (i.e., the compile and link steps) is pure, a

change to any input of a derivation will cause a rebuild. This includes changes to the main

C sources, header ﬁles, C compiler ﬂags, the C compiler and other tools, and so on.

Figure 10.2 shows another Nix expression that imports the one in Figure 10.1 to build

a few small programs that link against the ATerm library; these are actually from the tests

directory of the ATerm package. The function compileTest 156 compiles and links a single

test program. This function is called for each test program 158 . Of course, body could also

have been written as

body = map compileTest [./fib.c ./primes.c];

Note that λ-abstraction (i.e., functions) and map take the role of pattern rules in Make.

But contrary to Make’s pattern rules, they are not global: we can deﬁne any number of

functions to build things in different ways in different parts of the Nix expression.

The test programs are linked against the ATerm library built by the function in Fig-

ure 10.1, which is called here to obtain a speciﬁc instance 157 . It is absolutely trivial to

use multiple variants of the library in a single Nix expression. If we want two programs

that link against the dynamic and static library, respectively, that’s easy:

foo = link {...;

libraries = [(import ../aterm {sharedLib = true;}).libATerm];

234

10.1. Low-level build management using Nix

with (import ../../../lib);

let {

compileTest = main: link {

156

objects = [(compileC {inherit main; localIncludePath = [../aterm];})];

libraries = [(import ../aterm {sharedLib = true;}).libATerm];

157

};

body = [

(compileTest ./primes.c)

158

(compileTest ./fib.c)

];

}

Figure 10.2.: Nix expression for building clients of the ATerm library

}

bar = link {...;

libraries = [(import ../aterm {sharedLib = false;}).libATerm];

}

Of course, the hashing scheme ensures that the two variants of the library end up in differ-

ent paths in the store.

Finding dependencies In general, build formalisms require that the parts that constitute

a system are declared. That is, we have to specify the dependencies of every build action.

In Figure 10.1 we could have speciﬁed the header ﬁles referenced by each C source ﬁle

explicitly, e.g.,

compileC {

main = ./afun.c

localIncludes = [./aterm2.h ./memory.h ./util.h ...];

}

An important advantage of Nix over build managers such as Make is that if we forget to

specify a dependency, the build will fail since the compiler will not be able to locate the

missing ﬁle (after all, the build takes place in a temporary directory, using only inputs in

the store, and has no reference to the location from which the derivation was instantiated).

Thus, if a build succeeds, we know that we have speciﬁed all the dependencies. In Make

on the other hand, it is very common for dependency speciﬁcations to be incomplete. This

leads to ﬁles not being rebuilt even though they should be, thus causing an inconsistency

between sources and build results.

But specifying all dependencies is a lot of work, so it scales badly in terms of developer

effort. So we wish to discover dependencies automatically. This has to be done in advance,

since if the dependency is not present at build time, it is too late. (Systems like Vesta [92]

on the other hand can discover dependencies during the build.) So we wish to “generate”

part of the Nix expression, e.g., the value of the localIncludes attribute in the example

above.

235

10. Build Management

compileC = {main, localIncludePath ? [], cFlags ? "", sharedLib}:

stdenv.mkDerivation {

name = "compile-c";

builder = ./compile-c.sh;

localIncludes =

159

dependencyClosure {

scanner = file: import (findIncludes file);

searchPath = localIncludePath;

startSet = [main];

};

cFlags = [ ... ];

inherit main;

};

findIncludes = file: stdenv.mkDerivation {

160

name = "find-includes";

builder = ./find-includes.pl;

inherit file;

};

Figure 10.3.: Nix expression for compiling C sources, with automatic header ﬁle determi-

nation

This is possible in the Nix expression language, since the import primitive can import

Nix expressions from derivation outputs (page 81). That is, if we have a source ﬁle X and

a derivation function that can compute a Nix list expression containing its dependencies,

we can import it at translation time. The function compileC is implemented in this way,

as shown in Figure 10.3. The set of include ﬁles 159 is computed by the primop depen-

dencyClosure, which “grows” a set of ﬁles startSet using the dependencies discovered by

calling the function scanner for each ﬁle. In this case, scanner is a function that returns

the header ﬁles included by a ﬁle:

import (findIncludes file)

The function ﬁndIncludes 160 yields a derivation that builds in its output path a Nix ex-

pression containing the header ﬁles included by ﬁle, using a builder ﬁnd-includes.pl that

scans for preprocessor include statements. E.g., if ﬁle contains

#include "foo/bar.h"

#include "../xyzzy.h"

then the output of the derivation will be

[ "foo/bar.h" "../xyzzy.h" ]

Since these are relative paths, dependencyClosure will absolutise them relative to the path

of the including ﬁle. Of course, these might not be all the dependencies of ﬁle, since they

236

10.1. Low-level build management using Nix

might in turn include other header ﬁles. But dependencyClosure will apply scanner to

those as well, thus eventually obtaining all dependencies

. The result of dependencyClo-

sure is a list of path references, e.g.,

localIncludes = [./main.c ./foo/bar.h ../xyzzy.h];

These ﬁles will be copied to the Nix store during the translation to store derivation. The

resulting build will therefore only make use of sources in the store.

Thus parts of Nix expressions can be generated dynamically, during the translation to

store derivations, by leveraging the translation and build machinery. This also allows the

Nix expression language to be “extended” in a way, since if the language does not provide

a way to perform certain computations (e.g., string or list operations), then a derivation and

builder can be written to implement them. Odin [28] has a similar approach to extensibility.

A nice aspect of automatically ﬁnding dependencies in Nix is that the dependency scan-

ner does not need to be entirely perfect. For instance, keeping the dependencies of T

documents (such as those caused by \input{ﬁle} macros) up to date in Make is hard, since

X does not provide a tool analogous to GCC’s -MM operation to print out all depen-

dencies of a ﬁle. Since T

X is a programming language, writing such a tool is quite dif-

ﬁcult. However, we can also write a simple script that searches for commands such as

\input{ﬁle} and a few others using regular expressions. Such a script is not guaranteed to

ﬁnd all dependencies, but that’s okay. If it does not ﬁnd all dependencies, the build will

fail deterministically, and the user can add the missing dependencies to the Nix expression

manually.

This thesis is in fact built using Nix. The top-level Nix expression is essentially just

default = runLaTeX {

rootFile = ./doc.ltx;

}

where runLaTeX uses the techniques described above to guess all L

X source ﬁles and

images included by doc.ltx. (The Nix expression also builds derived artifacts such as pretty-

printed code, Graphviz-generated [54] pictures, and so on.)

Relative paths Often, source ﬁles expect that their dependencies are in certain relative

paths. If we compile a C source that contains the following line:

#include "../foo.h"

then we can only compile this ﬁle if the header ﬁle exists in location ../foo.h relative to

the C source ﬁle. To compile modules in such languages, the builder must reproduce the

relative source tree as it existed before translation to store derivations.

For this reason, dependencyClosure actually produces a list of dependencies that also

includes their paths relative to the start ﬁle. For the example above, it returns

The closure thus obtained should not be confused with the closures in the Nix store used in the rest of this

thesis. They are closures of source ﬁles relative to the Nix expression (e.g., in the user’s working copy), prior

to translation to store derivations.

The encoding used here (alternating between paths and strings) is rather poorly typed—if we had a type system.

Clearly, it would be better to use tuples (if the language had them) or attribute sets. However, there is currently

no way to pass attribute sets to builders.

237

10. Build Management

localIncludes = [

./main.c "./main.c"

./foo/bar.h "./foo/bar.h"

../xyzzy.h "../xyzzy.h"

];

This extra information may seem redundant, but recall that path constants such as ../xyzzy.h

are actually absolutised internally, e.g., to /home/eelco/project/xyzzy.h. And the path is

certainly lost during translation to store derivations (see the processBinding case for path

constants in Figure 5.7). Thus the relative paths, encoded as strings, are actually necessary

to pass the required information to the builder.

The builder of compilerC, compile-c.sh, will use this information to create in its tempo-

rary build directory a hierarchy of symlinks such as

./xyzzy.h

./dotdot/main.c

./dotdot/foo/bar.h

to the actual sources in the Nix store. (A number of dotdot directories equal to the max-

imum number of .. references in relative paths is inserted to make those relative paths

resolve properly.) Compiling ./dotdot/main.c will then yield the desired result.

Building So how do we actually build the derivations deﬁned in, e.g., Figure 10.2? A

tool such as nix-env is inappropriate, since we do not necessarily want to “install” the build

result. Hence there is a small tool nix-build (brieﬂy mentioned in Section 5.6.1), which

is just a wrapper around nix-instantiate and nix-store --realise that builds a Nix expression

and places a symlink to the derivation output (named result) in the current directory. If the

top-level Nix expression yields multiple derivations, the links are numbered. By default,

nix-build builds the expression default.nix in the current directory. E.g., when applied to the

expression in Figure 10.2, we get

$ nix-build

...

$ ./result-2/program

fib(32) == 3524578

since the symlink result-2 refers to the second derivation (the ﬁb.c program, which com-

putes the Fibonacci sequence).

10.2. Discussion

Low-level build management using Nix has numerous advantages over most build man-

agers (discussed below). Of course, most of these are exactly the advantages that it also

has for deployment, showing that build management and deployment are very much over-

lapping problems.

• It ensures that dependency speciﬁcations are complete (though most dependencies

can be found automatically).

238

10.2. Discussion

• All inputs, not just source ﬁles but also ﬂags, scripts, tools, etc., are dependencies.

Running nix-build is guaranteed to produce a result consistent with the sources and

the Nix expression. In Make, by contrast, changes to the build actions do not trigger

recompilation. This is why Make users have a habit of running make clean often,

just to be on the safe side.

• Since dependencies are complete and builds produce no output other than in out,

parallel and distributed builds are safe.

• There is no pollution of the working copy in which the source resides. Thus there is

no need for an operation like make clean. Garbage collection takes care of freeing

disk space.

• Due to locking (page 98), it is safe to run multiple Nix builds in parallel, even on the

same Nix expression.

• It is safe to edit sources while a build is in progress. In contrast, with Make we can

run into race conditions, e.g., if we start a big L

X job or C++ compile, then edit

the source in Emacs and save while the build is in progress. The build will use the

old sources, but subsequent Make invocations will believe that the output is up to

date.

• The Nix expression language makes it very easy to specify variants. Since it is

a functional language, almost any repetitiveness can be abstracted away. So the

functionality provided by tools such as Automake and Libtool can be implemented

through Nix expression libraries.

• If a build succeeds, we know all the source dependencies in the working copy, so we

know which ﬁles at the least must be placed under version management.

• As mentioned, the automatic determination of header ﬁle dependencies is also safer

than what we can get in Make derivates. It is common to use tricks like gcc -MM

to ﬁnd header ﬁle dependencies automatically in Makeﬁle rules, but this has subtle

failure conditions. For instance, gcc -MM causes the found header ﬁles to be declared

as dependencies, but it neglects to add the ﬁles that were not found. If the C compiler

search path contains directories A and B, and a header X is found in B/X, then the

non-existent ﬁle A/X should also be a dependency. After all, if A/X were to be

added by the user, a rebuild would be necessary. Since we recompute the closure

of the dependencies on every invocation, the new ﬁle A/X will be found and will

trigger a rebuild.

However, there is an unresolved problem with Nix as a build manager: how can we use it

recursively, that is, from within a Nix builder? It is quite common for a Nix builder to call

conventional build tools such as Make. That is what almost all components in Chapter 7

do. But can we call nix-build in a builder? The problem is that the Nix store is global, and

nix-build would compute and create all sorts of paths that are not known inputs or outputs

of the current Nix build. A solution might be for the recursive Nix invocations to use a

private store (e.g., a subdirectory of the builder’s temporary directory).

239

10. Build Management

Instead of recursive Nix builds, the alternative is to have one gigantic build graph. For

instance, if we are building a component that needs a C compiler, the Nix expression for

that component simply imports the Nix expression that builds the compiler. The problem

with this approach is scalability: the resulting build graphs would become huge. The graph

for a simple component such as GNU Hello would include the build graphs for dozens of

large components, such as Glibc, GCC, etc. The resulting graph could easily have hundreds

of thousands of nodes, far exceeding the graphs typically occurring in deployment (e.g., the

one in Figure 1.5). However, apart from its efﬁciency, this is possibly the most desirable

solution because of its conceptual simplicity. Thus it is interesting to develop efﬁcient

ways of dealing with very large build graphs.

10.3. Related work

The archetypical build manager is Make [56], introduced in Seventh Edition Unix (1979).

According to Stuart Feldman’s 2003 ACM Software System Award citation, “there is prob-

ably no large software system in the world today that has not been processed by a version

or offspring of MAKE.” It builds systems from a Makeﬁle that describes a dependency

graph using rules of the form

targets : dependencies

actions

with the semantics that if the ﬁles targets do not exist or are older than the ﬁles

dependencies, then the shell commands listed in actions should be run to (re-)build targets.

Makeﬁles have some abstraction facilities to reduce repetitive rules, such as pattern rules

(generic rules that are applied automatically on the basis of ﬁle extensions) and variables.

Make has spawned dozens of derivates. Some are backwards-compatible extensions,

such as BSD Make and GNU Make [68]. Others are not, e.g., Plan 9’s mk [96], AT&T

and Lucent’s NMake [70], Microsoft’s NMake, Jam [148], and so on. Some of these add

support for parallel builds [7], have syntactic sugar to allow more concise Makeﬁles, or

have built-in support for languages such as C and C++ (e.g., to ﬁnd dependencies). There

are also tools that generate Makeﬁles from higher-level speciﬁcations, e.g., Automake [65,

172]. The generators unfortunately do not shield the user from the lower layers; it is the

user’s job to map problems that occur in a Makeﬁle back to the high-level speciﬁcation

from which it was generated.

Ant [63] is a very popular build manager, but its popularity stems mainly from the fact

that it comes with a large set of useful tasks for Java development, contrary to Make.

(Make’s built-in rules essentially reﬂect the Unix programming language landscape of

1979.) However, Ant in many ways is a step backwards compared to Make

. The (XML-

based) build formalism is actually quite crude, and describes a simple partial ordering of

build tasks. Ant’s core provides no incremental build facilities; these must be implemented

by tasks themselves. Ant however shows that it is important for a build tool to have good

libraries for building common languages.

Ant’s homepage [63] lists two reasons for its development: Make uses non-portable shell commands, and

Makeﬁles use TAB characters. This indicates that Ant’s development was not inspired by any fundamental

reﬂection on Make’s limitations.

240

10.3. Related work

An interesting performance improvement is provided by optimistic Make [21], which

starts building targets before the user has issued the make command, putting unused CPU

cycles to good use. A median reduction in response time as high as a factor of 5.1 is

reported.

However, none of these tools solve Make’s fundamental problems, which are:

• A lack of abstraction facilities. The abstraction mechanisms—variables and pattern

rules—are all global. Hence, if we need to specify different ways of building tar-

gets, we cannot use them, unless we split the system into multiple Makeﬁles. This,

however, creates the much greater problem of incomplete dependency graphs [121].

[71] shows some of the rather clumsy techniques used for stretching Make’s expres-

siveness.

• A lack of support for variant builds. Makeﬁle variables allow variability to be ex-

pressed, but variants will overwrite each other unless tortuous hacks are used to

arrange otherwise.

• The inability to ensure complete dependency speciﬁcations. If builds are not per-

formed in isolation but rather in the workspace that holds all the sources, there is no

way to prevent undeclared dependencies (unless ﬁle system calls are intercepted, as

discussed below).

However, there are a number of build managers that do fundamentally improve on Make.

Odin [28, 29] has a functional language to describe build actions (queries). (Indeed, a reim-

plementation of Odin’s semantics as an embedded domain-speciﬁc language in Haskell is

presented in [150].) For example, the expression hello.c denotes a source, while hello.c

:exe denotes the executable obtained by compiling and linking hello.c. Odin also supports

variants automatically. Variant builds can be expressed easily, e.g., hello.c +debug :exe.

Vesta (already discussed in Section 7.6) has a build manager, the Vesta evaluator, that

builds software from speciﬁcations written in the System Description Language (SDL),

which is a functional language [94]. Since Vesta uses a virtual (NFS-based) ﬁle system, it

can detect all dependencies used by a build action. The results of builds are cached and can

be shared between users. Since the language is functional, it is easy to specify and build

variants. Thus Vesta solves all three of the Make limitations listed above. The price is the

use of a virtual ﬁle system and the integration with a speciﬁc version management system.

Amake [9, 8, 164] is the build tool for the Amoeba distributed operating system. Like

Odin, it separates the speciﬁcation of build tools and system speciﬁcations. Given a set

of sources Amake automatically completes the build graph, that is, it ﬁnds instances of

tool deﬁnitions that build the desired targets from the given sources. This is contrary to

the functional model of explicit tool application in Odin, Vesta, and Nix. The obvious

advantage is that speciﬁcations become shorter; the downside is that it becomes harder to

specify alternative ways of building, and to see what’s going on.

Maak [46], by the author of this thesis, was in many ways a precursor to Nix. It has a

functional language similar to the Nix expression language, and supported build variability

automatically; in fact, the language has some features that the Nix expression language

does not, such as automatic propagation of attributes. The Nix language backs away from

this level of expressivity since a) the Nix language is intended foremost for deployment

241

10. Build Management

instead of build management, and so involves less complicated build descriptions; and b)

concise syntax is not always a blessing; it may make the language harder to understand.

Also, Maak allows circular dependencies, a classical “hard” problem for build man-

agers. A notable example is L

X, which requires any number of runs until certain out-

puts stop changing. Circular dependencies are fundamentally incompatible with Nix, since

store derivations describe a static set of build actions from which all variability has been

removed (and in the extensional model, the output paths must be known in advance). I now

feel that circular dependencies are an unnecessary complication for build managers: they

occur quite seldom—in fact, L

X is the only common example—and so are better ﬁxed

by wrapping the offending tool in a script that executes it an appropriate number of times.

The Maak paper [46] also introduced the notion of transparent source/binary deploy-

ment, but in the absence of a cryptographic hashing scheme, it used nominal version

equality, which is of course less safe. But Maak’s fundamental problem was that it could

not ensure complete dependency speciﬁcations, since (like Make) it performed builds “lo-

cally”, in the workspace holding the sources. (On Linux, it did have a feature to use the

strace command to intercept ﬁle system accesses, allowing dependency detection.) This

prompted the development of a component store based on cryptographic hashing.

242

Part IV.

Conclusion

243

11. Conclusion

The main objective of the research described in this thesis was to develop a system for

correct software deployment that ensures that the deployment is complete and does not

cause interference. This objective was successfully met in the Nix deployment system, as

the experience with Nixpkgs described in Section 7.1.5 has shown.

The objective of improving deployment correctness is reached through the two main

ideas described in this thesis. The ﬁrst is the use of cryptographic hashes in Nix store

paths. It gives us isolation, automatic support for variability, and the ability to determine

runtime dependencies. This however can be considered an (important) implementation

detail—maybe even a “trick”. However, it addresses the deployment problem at the most

fundamental level: the storage of components in the ﬁle system.

The second and more fundamental idea is the purely functional deployment model,

which means that components never change after they have been built and that their build

processes only depend on their declared inputs. In conjunction with the hashing scheme,

the purely functional model prevents interference between deployment actions, provides

easy component and composition identiﬁcation, and enables reproducibility of conﬁgura-

tions both in source and binary form—in other words, it gives predictable, deterministic

semantics to deployment actions.

Nix is a work in progress. The implementation of the extensional model (Chapter 5) is

in use. The experience of the Nix Packages collection shows that it works very well for its

primary goal of software deployment (Chapter 7), though more experience is necessary to

make conclusive statements about the scalability of the purely functional model. The inten-

sional model (Chapter 6) exists in prototype form and must be fully implemented. Services

deployed using Nix (Chapter 9) are in production use, as is a build farm (Chapter 8). Build

management (Chapter 10) works, as its properties are really a direct application of Nix,

but more work is necessary to improve its usability (e.g., by providing good libraries for

common languages and tasks) and scalability.

Our experience with Nix over the last two years allows the following conclusions to be

drawn:

• The purely functional deployment model implemented in Nix and the cryptographic

hashing scheme of the Nix store in particular give us important features that are

lacking in most deployment systems, such as complete dependencies, complete

deployment, side-by-side deployment, atomic upgrades and rollbacks, transparent

source/binary deployment and reproducibility (see Section 1.5).

• The cryptographic hashing scheme is effective in preventing undeclared dependen-

cies, assuming a fully Nixiﬁed environment. In other environments, the techniques

of Section 7.1.3 are successful in preventing most impurities.

• Nix places a number of assumptions on components (Section 6.8), but the large and

245

11. Conclusion

varied set of components deployed through Nix shows that these assumptions are

quite reasonable (Section 7.1.5).

• Hash rewriting is a technique that intuitively might appear unlikely to work, but in

fact does work. It enables the intensional model, which is a breakthrough in that

it enables secure sharing, multiple output paths, and some future beneﬁts discussed

below.

However, there are also a number of things that we cannot conclude on the basis of our

current experience:

• That a functional language is a good way to describe components and compositions.

Certainly such a language makes it possible and easy to describe variability, and

function arguments seem a natural way to describe dependencies. But the usabil-

ity of functional languages (and the Nix expression language in particular) has not

been demonstrated; all users of the Nix expression language had prior exposure to

functional languages such as Haskell.

• Nix opens up the perspective of unifying the various points on the software build

and deployment timeline into a single formalism; e.g., Nix is not just a deployment

tool, but as we have seen in Chapter 10, it can replace “low-level” build management

tools such as Make. This is good because it prevents “impedance mismatch” between

build management and deployment. For instance, it is not necessary to specify the

same dependency both in the build formalism and the deployment formalism. But it

remains to be seen whether low-level build management can be made scalable. Also,

it is necessary to subsume the source conﬁguration stage, currently implemented by

tools such as Autoconf; this has not been done yet.

Finally, there are also a number of reasonable criticisms that can be levelled at the purely

functional model:

• Efﬁcient upgrading remains a problem. Using patch deployment, upgrades can be

done efﬁciently in terms of network bandwidth. But a change to a fundamental de-

pendency can still cause disk space requirements to double. This is a real problem

compared to destructive updates. However, it can be argued that disk space is abun-

dant, and that software components no longer dominate disk space consumption.

• Preventing undeclared dependencies is good, but the lack of scoped composition

mechanisms lessens its usefulness (as discussed on page 173).

• Most of our experience with Nix has been on Linux, which allows a self-contained,

pure Nixpkgs (page 169). That is, every dependency of the components in Nixp-

kgs can be built and deployed through Nix. This is an ideal situation that cannot

be achieved in most other operating systems. A GUI application on Mac OS X

will typically have a dependency on the Cocoa or Carbon GUI libraries. Since

these are closed source and cannot be copied legally, we can neither build them

nor use the techniques of Section 7.1.4 to deploy them in binary form. Thus we

are forced to resort to impurity (e.g., by linking against the non-store path /Sys-

tem/Library/Frameworks/Cocoa.framework/...), reducing Nix’s effectiveness.

246

11.1. Future work

There are a number of possibilities for further research, as well as a several remaining

practical issues.

A fully Nixiﬁed system Since Nix works best in a “pure” environment where all com-

ponents are built and deployed by Nix, it is natural to consider an operating system dis-

tribution on that basis; that is, a Unix system (probably based on Linux or FreeBSD) that

has no /bin, /lib, /usr, etc.—all components are in /nix/store. In addition, the conﬁguration

of the system can be managed along the lines of the services in Chapter 9. For instance,

the conﬁguration of which services (such as daemons) should be enabled or disabled can

be expressed in a Nix expression that builds a component that starts and stops other com-

ponents passed as build-time inputs. To change the conﬁguration, the Nix expression is

changed and rebuilt.

A pure

Nix-based Linux system (tentatively called NixOS) is currently in active devel-

opment and in a bootable state. Building components in this pure environment has already

revealed a few impurities in Nixpkgs. On the other hand, the number of such impurities

has turned out to be surprisingly low: once NixOS was up and running, it took only two

one-line modiﬁcations (to stdenv, no less!) to get Firefox and all its dependencies to build.

The intensional model There is a prototype implementation of the intensional model

that shows that hash rewriting works in practice (i.e., does not break components), but the

prototype is not usable yet. A full implementation is necessary to answer a number of open

questions. In particular, what is a good policy for path selection (page 149)?

The intensional model has the potential to enable better build management. A common

efﬁciency problem in incremental building is that a small change to some ﬁle causes many

rebuilds. For instance, a change to a C header ﬁle (say, conﬁg.h) forces us to rebuild all

source ﬁles that include it. This is the case even for source ﬁles that include the header but

are unaffected by the change.

Redundant rebuilds can be prevented in the intensional model by making the C prepro-

cessor invocation an explicit build step. For source ﬁles that are unaffected by the header

change, the preprocessed output will remain unchanged. We can optimise the build algo-

rithm as follows. If a derivation d

differs only from a derivation d

in subderivations that

yielded the same output, we can just copy the output of d

to d

A language for builders We need to ﬁnd a better way to write builders. Shell scripts

are simply not a good language. The Unix shell is quite good as component glue—which

is what we use it for—but it is not safe enough. It is in fact virtually impossible to write

“correct” shell scripts that do not fail, unexpectedly, for banal reasons on certain inputs or

in certain situations. Here are some of the main problems in the language:

• Variable uses (e.g., $foo) are “interpolated” by default. This is the reason why most

shell scripts fail on ﬁle names that contain spaces or newlines. The same applies for

automatic “globbing” (expansion of special characters such as "*").

With the exception of /bin/sh, which is required by Glibc and several standards.

247

11. Conclusion

• Undeﬁned variables do not signal an error.

• Non-zero exit codes from commands are ignored by default. Worse, it is extremely

hard to catch errors from the left-hand sides of pipes.

In fact, the whole Unix tool chain is not conducive to writing correct code:

• Command-line arguments in Unix are usually overloaded, e.g., there is an ambiguity

in most Unix tools between options and ﬁle names starting with a dash.

• The pipe model is that all tools read and write ﬂat text ﬁles without a well-deﬁned

structure. This makes it hard to extract relevant data in a structured way. Typically,

regular expressions in conjunction with grep, sed, awk and perl are used to extract

data, but this tends to be brittle—the regular expressions hardly ever cover all possi-

bilities.

• Similarly, the fact that options are ﬂat strings makes it hard to pass structure, neces-

sitating escaping. E.g., the command grep -q "$ﬁlename" to see whether a certain ﬁle

name occurs in a list fails to take into account that grep interprets some characters

in $ﬁlename as regular expression meta-characters.

These problems are far from speciﬁc to Nix builders. They have been the cause of many

serious bugs, including security problems. So what we need is a shell with a clean seman-

tics founded on the principle of least surprise [140], and a tool set based on the exchange of

structured data (e.g., XML or ATerms). MSH (the Microsoft Shell) may be a good model.

It is also necessary to improve the interaction between Nix expressions and builders.

The mechanism by which information is communicated to builders is rather crude, namely,

through environment variables that only allow ﬂat strings to be passed. It is much better to

pass structured information, such as attribute sets and lists. An obvious way is to convert

such expressions to XML (which is especially attractive in conjunction with an XML-based

tool set).

A type system A type system for the Nix expression language is desirable. Of course,

to the greatest extent possible types should be inferred. We probably need a basic Hindley-

Milner type system [122] extended with extensible records [111] to support attribute sets.

Efﬁcient storage Binary patch deployment greatly reduces the bandwidth needed to de-

ploy binary upgrades, but the upgraded components still need to be stored in full, and an

amount of CPU time linear in the size of the upgraded components is also required. These

problems may be solved by using a ﬁle system that supports delta storage [115], i.e., that

can store ﬁle revisions as deltas to previous revisions. Actually, for the Nix store, we need

a ﬁle system where revisions can refer to revisions of other ﬁles. Concretely, we need a

kernel API copy(p

dst

, p

src

, patch) that creates a path p

dst

with contents equal to those of

src

with patch applied. Then FSO creation using patches takes disk space and CPU time

linear in the size of the patch, not the resulting FSO.

248

11.1. Future work

</within>

</condition>

Zlib 1.2.1 is vulnerable to a denial-of-service condition. See

http://www.kb.cert.org/vuls/id/238678. Upgrade to 1.2.2.

</reason>

</item>

Figure 11.1.: A blacklist entry

Blacklisting The purely functional model creates a serious problem in some situations:

how can we be sure that all uses of a certain “bad” component have been eliminated? For

example, suppose that we ﬁnd that a certain fundamental component (say, Zlib) contains

a security bug [1]. This component is used by many other components. Thus we want to

ensure that we upgrade all installed applications in all user environments to new instances

that do not use the bad component anymore. Of course, we can just run nix-env -u "*"

and hope that this gets rid of all bad components, but there is no guarantee of that, espe-

cially if we have installed components from arbitrary third parties (i.e., components not

in Nixpkgs). The purely functional model’s non-interference property now bites us. Thus

there should be a generic mechanism that detects uses of bad components on the basis of a

blacklist that describes these components.

A simple prototype of this idea has been implemented. It evaluates an XML speciﬁcation

that contains declarations such as the one shown in Figure 11.1. This example speciﬁes

that any output path built from a derivation graph that at any point used the Zlib 1.2.1

source distribution (as identiﬁed by its cryptographic hash, ef1cb003448b...), is tainted.

The condition is applied to every element of every user environment, and prints out a

warning if a match is found. Note that since the condition inspects the derivation graph

(found through the deriver link described in Section 5.5.1), it also matches statically linked

uses of Zlib. Thus the blacklisting approach ﬁnds dangerous uses of libraries that cannot

be ﬁxed by destructive upgrading in conventional deployment models.

A blacklisting approach should also specify what should be done if a match is found,

which may depend on the type of the machine, user environment, or application: for a

server the Zlib bug is critical, for a client less so.

Upgrading in the intensional model The technique of hash rewriting in the intensional

model (Section 6.3.2) allows something very similar to destructive upgrading in conven-

tional systems, but in safe way. We can generically upgrade all components that reference

some “bad” path p to a “good” path p

, by rewriting (using a variation of algorithm resolve

in Section 6.4.1) every use of p to p

. The difference with the normal mode of upgrading

249

11. Conclusion

in Nix (deploying replacement Nix expressions) is that no Nix expressions are necessary.

That’s also the problem of this technique: the resulting components do not have a deriver,

that is, they do not necessarily correspond to the output of any known derivation. But it

is not destructive: since resolve works by rewriting copies, the old components are still

available. Thus, atomicity and the ability to roll back still hold.

Feature selection and automatic instantiation Nix supports component variability

both in the expression language and at the store level, but current deployment policies

are not very good at making it available to users. For instance, nix-env does not provide

a way to pass arguments to functions; the top-level Nix expression should evaluate to a

set of derivations. Similarly, deployment policies such as channels do not support local

customisation.

At the very least, tools such as nix-env should be extended to allow variation points

to be bound (i.e., to allow the user to set parameters). One can imagine this being done

through a nice graphical user interface, e.g., as in the Linux kernel conﬁguration tool, Con-

sul@GUI [13] or the Software Dock [85]. Such interfaces lead to an interesting problem: if

the user binds some but not all of the variation points, how can we ﬁnd component instances

or compositions that satisfy all constraints (as expressed by assertions in Nix expressions)?

This is of course an NP-complete problem (since it is essentially just the boolean satisﬁa-

bility problem [35]), but that may not be a problem in practice. For example, CML2 [139]

ﬁnds Linux kernel instances satisfying user-speciﬁed constraints automatically, and Binary

Decision Diagrams (BDD) also appear to be a useful technique for this purpose [170].

250

Bibliography

[1] Mark Adler and Jean-loup Gailly. Zlib advisory 2002-03-11. http://www.zlib.net/advisory-2002-03-11.txt,

2002.

[2] Eric Anderson and Dave Patterson. A retrospective on twelve years of LISA proceedings. In Proceedings

of the 13th Systems Administration Conference (LISA ’99), pages 95–107. USENIX Association, Novem-

ber 1999.

[3] Gregory R. Andrews. Concurrent Programming: Principles and Practice. Addison-Wesley, 1991.

[4] Andrew W. Appel. Modern Compiler Implementation in ML. Cambridge University Press, 1998.

[5] Franz Baader and Tobias Nipkow. Term rewriting and all that. Cambridge University Press, 1998.

[6] Erik H. Baalbergen. Parallel and distributed compilations in loosely-coupled systems: A case study. In

Workshop on Large Grain Parallelism, Providence, RI, October 1986. Also in [123].

[7] Erik H. Baalbergen. Design and implementation of parallel make. Computing Systems, 1(2):135–158,

1988.

[8] Erik H. Baalbergen. The declarative operating system model. PhD thesis, Vrije Universiteit, Amsterdam,

The Netherlands, 1992.

[9] Erik H. Baalbergen, Kees Verstoep, and Andrew S. Tanenbaum. On the design of the Amoeba conﬁgura-

tion manager. In 2nd International Workshop on Software Conﬁguration Management (SCM-2), volume

17/7 of ACM SIGSOFT Software Engineering Notes, pages 15–22, November 1989.

[10] Henk Barendregt. The Lambda Calculus: Its Syntax and Semantics, volume II of Studies in Logic and

the Foundations of Mathematics. Elsevier Science Publishers B.V. (North-Holland), Amsterdam, second

edition, 1984.

[11] Bryan Kendall Beatty. Compact text encoding of latitude/longitude coordinates. United States Patent

Application 20050023524, February 2005.

[12] Tim Berners-Lee, Roy T. Fielding, and Larry Masinter. Uniform resource identiﬁers (URI): Generic

syntax. RFC 2396, August 1998.

[13] Danilo Beuche, Holger Papajewski, and Wolfgang Schröder-Preikschat. Variability management with

feature models. In Jilles van Gurp and Jan Bosch, editors, Proceedings of the Software Variability Man-

agement Workshop (SVM ’2003), February 2003.

[14] Hans-Juergen Boehm. Space efﬁcient conservative garbage collection. In Proceedings of the ACM SIG-

PLAN ’93 Conference on Programming Language Design and Implementation, number 28/6 in SIGPLAN

Notices, pages 197–206, June 1993.

[15] Hans-Juergen Boehm and Mark Weiser. Garbage collection in an uncooperative environment. Software—

Practice and Experience, 18(9):807–820, September 1988.

[16] Don Box. Essential COM. Addison-Wesley, 1998.

[17] Don Box. Essential .NET, Volume I: The Common Language Runtime. Addison-Wesley, November 2002.

[18] Richard S. Boyer and J Strother Moore. A fast string searching algorithm. Communications of the ACM,

20(10):762–772, 1977.

[19] Martin Bravenboer and Eelco Visser. Concrete syntax for objects. Domain-speciﬁc language embedding

and assimilation without restrictions. In Douglas C. Schmidt, editor, Proceedings of the 19th ACM SIG-

PLAN Conference on Object-Oriented Programing, Systems, Languages, and Applications (OOPSLA’04),

pages 365–383, Vancouver, Canada, October 2004. ACM Press.

[20] Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, Eve Maler, and François Yergeau, editors. Extensible

Markup Language (XML) 1.0. W3C, third edition, February 2004.

251

Bibliography

[21] Rick Bubenik and Willy Zwaenepoel. Optimistic Make. IEEE Transactions on Software, 41(2), February

1992.

[22] Mark Burgess. Cfengine: a site conﬁguration engine. Computing Systems, 8(3), 1995.

[23] Brent Callaghan, Brian Pawlowski, and Peter Staubach. Nfs version 3 protocol speciﬁcation. RFC 1813,

June 1995.

[24] Antonio Carzaniga, Alfonso Fuggetta, Richard S. Hall, André van der Hoek, Dennis Heimbigner, and

Alexander L. Wolf. A characterization framework for software deployment technologies. Technical Report

CU-CS-857-98, University of Colorado, April 1998.

[25] Antonio Carzaniga, David S. Rosenblum, and Alexander L. Wolf. Design and evaluation of a wide-area

event notiﬁcation service. ACM Transactions on Computer Systems, 19(3):332–383, 2001.

[26] Anders Christensen and Tor Egge. Store — a system for handling third-party applications in a heteroge-

neous computer environment. In Selected papers from the ICSE SCM-4 and SCM-5 Workshops on Soft-

ware Conﬁguration Management, number 1005 in Lecture Notes in Computer Science. Springer-Verlag,

1995.

[27] James Clark, editor. XSL Transformations (XSLT) Version 1.0. W3C, November 1999.

[28] Geoffrey Clemm. The Odin System — An Object Manager for Extensible Software Environments. PhD

thesis, University of Colorado at Boulder, February 1986.

[29] Geoffrey Clemm. The Odin system. In Selected papers from the ICSE SCM-4 and SCM-5 Workshops on

Software Conﬁguration Management, number 1005 in Lecture Notes in Computer Science, pages 241–

262. Springer-Verlag, 1995.

[30] Geoffrey Clemm, Jim Amsden, Tim Ellison, Christopher Kaler, and Jim Whitehead. Versioning extensions

to WebDAV (web distributed authoring and versioning). RFC 3253, March 2002.

[31] CollabNet. Subversion. http://subversion.tigris.org/, 2005.

[32] Wallace Colyer and Walter Wong. Depot: A tool for managing software environments. In Proceedings

of the 6th Systems Administration Conference (LISA VI), pages 153–162. USENIX Association, October

1992.

[33] International Electrotechnical Commission. Letter symbols to be used in electrical technology — Part 2:

Telecommunications and electronics. IEC, second edition, November 2000. IEC 60027-2.

[34] The Unicode Consortium. The Unicode Standard, Version 4.0. Addison-Wesley, 2003.

[35] Stephen A. Cook. The complexity of theorem-proving procedures. In STOC ’71: Proceedings of the

third annual ACM symposium on Theory of computing, pages 151–158, New York, NY, USA, 1971. ACM

Press.

[36] Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest. Introduction to Algorithms. MIT Press,

1990.

[37] Intel Corporation. IA-32 Intel Architecture Software Developer’s Manual, volume 2B—Instruction Set

Reference, N–Z. Intel Corporation, June 2005. Order number 253667-016.

[38] Microsoft Corporation. Joliet speciﬁcation — version 1, May 1995.

[39] Microsoft Corporation. Binary delta compression. Whitepaper, March 2002.

[40] Krzysztof Czarnecki and Ulrich Eisenecker. Generative Programming: Methods, Tools, and Applications.

Addison-Wesley, 2000.

[41] Homayoun Dayani-Fard, Yijun Yu, John Mylopoulos, and Periklis Andritsos. Improving the build archi-

tecture of legacy C/C++ software systems. In Maura Cerioli, editor, Fundamental Approaches to Software

Engineering: 8th International Conference (FASE 2005), volume 3443 of Lecture Notes in Computer

Science, pages 96–110, Edinburgh, Scotland, April 2005. Springer-Verlag.

[42] Merijn de Jonge. To Reuse or To Be Reused. PhD thesis, University of Amsterdam, March 2003.

[43] Merijn de Jonge. Build-level components. IEEE Transactions on Software Engineering, 31(7):588–600,

July 2005.

[44] Edsger W. Dijkstra. Notes on structured programming. Technical Report 70-WSK-03, Technische

Hogeschool Eindhoven, March 1970.

252

Bibliography

[45] Jeff Dike. A user-mode port of the Linux kernel. In 4th Annual Linux Showcase & Conference (ALS

2000), October 2000.

[46] Eelco Dolstra. Integrating software construction and software deployment. In Bernhard Westfechtel,

editor, 11th International Workshop on Software Conﬁguration Management (SCM-11), volume 2649 of

Lecture Notes in Computer Science, pages 102–117, Portland, Oregon, USA, May 2003. Springer-Verlag.

[47] Eelco Dolstra. Efﬁcient upgrading in a purely functional component deployment model. In George Heine-

man et al., editor, Eighth International SIGSOFT Symposium on Component-based Software Engineering

(CBSE 2005), volume 3489 of Lecture Notes in Computer Science, pages 219–234, St. Louis, Missouri,

USA, May 2005. Springer-Verlag.

[48] Eelco Dolstra. Secure sharing between untrusted users in a transparent source/binary deployment model.

In 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005), pages 154–

163, Long Beach, California, USA, November 2005.

[49] Eelco Dolstra, Martin Bravenboer, and Eelco Visser. Service conﬁguration management. In 12th Interna-

tional Workshop on Software Conﬁguration Management (SCM-12), September 2005.

[50] Eelco Dolstra, Merijn de Jonge, and Eelco Visser. Nix: A safe and policy-free system for software

deployment. In Lee Damon, editor, 18th Large Installation System Administration Conference (LISA

’04), pages 79–92, Atlanta, Georgia, USA, November 2004. USENIX.

[51] Eelco Dolstra and Eelco Visser. Building interpreters with rewriting strategies. In Mark van den Brand and

Ralf Lämmel, editors, Workshop on Language Descriptions, Tools and Applications (LDTA’02), volume

65/3 of Electronic Notes in Theoretical Computer Science, Grenoble, France, April 2002. Elsevier Science

Publishers.

[52] Eelco Dolstra, Eelco Visser, and Merijn de Jonge. Imposing a memory management discipline on software

deployment. In Proceedings of the 26th International Conference on Software Engineering (ICSE 2004),

pages 583–592. IEEE Computer Society, May 2004.

[53] ECMA. ECMA-48: Control Functions for Coded Character Sets. ECMA, ﬁfth edition, June 1991. Also

ISO/IEC 6429:1992.

[54] John Ellson, Emden R. Gansner, Eleftherios Koutsoﬁos, Stephen C. North, and Gordon Woodhull.

Graphviz — open source graph drawing tools. In Graph Drawing 2001, number 2265 in Lecture Notes in

Computer Science, pages 483–485. Springer-Verlag, 2001.

[55] Robert Ennals and Simon Peyton Jones. HsDebug: debugging lazy programs by not being lazy. In Johan

Jeuring, editor, Proceedings of the ACM SIGPLAN workshop on Haskell, pages 84–87, Uppsala, Sweden,

August 2003. ACM Press.

[56] Stuart I. Feldman. Make—a program for maintaining computer programs. Software—Practice and Expe-

rience, 9(4):255–65, 1979.

[57] Roy T. Fielding, James Gettys, Jeffrey C. Mogul, Henrik Frystyk Nielsen, Larry Masinter, Paul J. Leach,

and Tim Berners-Lee. Hypertext transfer protocol — HTTP/1.1. RFC 2616, June 1999.

[58] Gert Florijn, Ernst Lippe, Atze Dijkstra, Norbert van Oosterom, and Doaitse Swierstra. Camera: Co-

operation in open distributed environments. In Proceedings of the EurOpen and USENIX Spring 1992

Workshop/Conference, pages 123–136. EurOpen and USENIX, April 1992.

[59] Gert Florijn, Leo Soepenberg, and Atze Dijkstra. Mapping objects to ﬁles: a UNIX ﬁle system interface

to an object management system. In H. Wijshoff, editor, Proceedings Computing Science in the Nether-

lands (CSN’93), 1993. Also published as Utrecht University, Department of Information and Computing

Sciences technical report RUU-CS-93-08.

[60] Karl Fogel and Moshe Bar. Open Source Development with CVS. Paraglyph Press, July 2003.

[61] Internet Engineering Task Force. Secure shell (secsh) working group. http://www.ietf.org/html.charters/

secsh-charter.html, 2005. Accessed 21 August 2005.

[62] Eric Foster-Johnson. Red Hat RPM Guide. John Wiley & Sons, 2003. Also at http://fedora.redhat.com/

docs/drafts/rpm-guide-en/.

[63] Apache Software Foundation. Apache Ant. http://ant.apache.org/, 2005. Accessed 15 August 2005.

[64] Free Software Foundation. Autoconf. http://www.gnu.org/software/autoconf/, 2005.

253

Bibliography

[65] Free Software Foundation. Automake. http://www.gnu.org/software/automake/, 2005.

[66] Free Software Foundation. Bison. http://www.gnu.org/software/bison/, 2005.

[67] Free Software Foundation. GNU Hello. http://www.gnu.org/software/hello/, 2005.

[68] Free Software Foundation. GNU Make. http://www.gnu.org/software/make/, 2005.

[69] Mozilla Foundation. Tinderbox. http://www.mozilla.org/tinderbox.html, 2005.

[70] Glenn Fowler. The fourth generation Make. In USENIX 1985 Summer Conference, June 1985.

[71] Glenn Fowler. A case for make. Software—Practice and Experience, 20(S1):S1/35–S1/46, 1990.

[72] Martin Fowler and Matthew Foemmel. Continuous integration. http://www.martinfowler.com/articles/

continuousIntegration.html. Accessed 11 August 2005.

[73] FreeBSD Project. FreeBSD Ports Collection. http://www.freebsd.org/ports/.

[74] Ned Freed and Nathaniel S. Borenstein. Multipurpose internet mail extensions (MIME) part two: Media

types. RFC 2046, November 1996.

[75] freedesktop.org. pkg-conﬁg. http://pkgconﬁg.freedesktop.org/wiki/, 2005. Accessed 21 August 2005.

[76] Alan O. Freier, Philip L. Karlton, and Paul C. Kocher. The SSL protocol — version 3.0. IETF Internet

Draft draft-freier-ssl-version3-02.txt, November 1996.

[77] Gentoo Foundation. Gentoo Linux. http://www.gentoo.org/.

[78] Bob Glickstein. GNU Stow. http://www.gnu.org/software/stow/, 2005.

[79] James Gosling, Bill Joy, Guy Steele, and Gilad Bracha. The Java Language Speciﬁcation. Addison-

Wesley, third edition, June 2005.

[80] Mark Grechanik and Dewayne Perry. Secure deployment of components. In Wolfgang Emmerich and

Alexander L. Wolf, editors, 2nd International Working Conference on Component Deployment (CD 2004),

volume 3083 of Lecture Notes in Computer Science. Springer-Verlag, May 2004.

[81] Filesystem Hierarchy Standard Group. Filesystem hierarchy standard, version 2.3. http://www.pathname.

com/fhs/pub/fhs-2.3.html, 2004.

[82] Free Standards Group. Linux Standard Base Core Speciﬁcation 3.0. http://www.linuxbase.org/, 2005.

[83] Nicholas Gulrajani and Karen Wade. Bridging the chasm between development and operations. Prod-

uct white paper, ftp://ftp.software.ibm.com/software/rational/web/whitepapers/2003/Bridging_the_chasm_

CC-TCM.pdf, June 2004.

[84] Richard S. Hall, Dennis Heimbigner, André van der Hoek, and Alexander L. Wolf. An architecture for

post-development conﬁguration management in a wide area network. In Proceedings of the 17th Interna-

tional Conference on Distributed Computing Systems, Baltimore, Maryland, USA, May 1997.

[85] Richard S. Hall, Dennis Heimbigner, and Alexander L. Wolf. A cooperative approach to support software

deployment using the Software Dock. In ICSE ’99: Proceedings of the 21st International Conference on

Software Engineering, pages 174–183, Los Alamitos, CA, USA, 1999. IEEE Computer Society Press.

[86] Richard S. Hall, Dennis M. Heimbigner, and Alexander L. Wolf. Requirements for software deployment

languages and schema. In 8th International Symposium on System Conﬁguration Management (SCM-8),

volume 1439 of Lecture Notes in Computer Science, pages 198–203. Springer-Verlag, 1998.

[87] John Hart and Jeffrey D’Amelia. An analysis of RPM validation drift. In Proceedings of the 16th Systems

Administration Conference (LISA ’02), pages 155–166. USENIX Association, November 2002.

[88] Red Hat. Cygwin. http://www.cygwin.com/, 2005. Accessed 21 August 2005.

[89] Les Hatton. Reexamining the fault density-component size connection. IEEE Software, 14(2):89–97,

1997.

[90] Jan Heering, Paul R. H. Hendriks, Paul Klint, and Jan Rekers. The syntax deﬁnition formalism SDF —

reference manual. SIGPLAN Notices, 24(11):43–75, 1989.

[91] Armijn Hemel. Using buildfarms to improve code. In UKUUG Linux 2003 Conference, August 2003.

[92] Allan Heydon, Roy Levin, Timothy Mann, and Yuan Yu. The Vesta approach to software conﬁguration

management. Technical Report Research Report 168, Compaq Systems Research Center, March 2001.

254

Bibliography

[93] Allan Heydon, Roy Levin, Timothy Mann, and Yuan Yu. The Vesta software conﬁguration management

system. Technical Report Research Report 177, Compaq Systems Research Center, January 2002.

[94] Allan Heydon, Roy Levin, and Yuan Yu. Caching function calls using precise dependencies. In ACM

SIGPLAN ’00 Conference on Programming Language Design and Implementation, pages 311–320. ACM

Press, 2000.

[95] Graydon Hoare. Monotone. http://www.venge.net/monotone/, 2005.

[96] Andrew Hume. Mk: a successor to make. In USENIX 1987 Summer Conference, 1987.

[97] James J. Hunt, Kiem-Phong Vo, and Walter F. Tichy. Delta algorithms: An empirical analysis. ACM

Transactions on Software Engineering and Methodology, 7(2):192–214, 1998.

[98] IEEE. IEEE Std 1003.1-1988 Standard Portable Operating System Interface for Computer Environments

(POSIX). IEEE, 1988.

[99] IEEE and The Open Group. The Open Group Base Speciﬁcations Issue 6 — IEEE Std 1003.1, 2004

Edition. http://www.opengroup.org/onlinepubs/009695399/, 2004.

[100] British Standards Institute. The C Standard: Incorporating Technical Corrigendum 1. Wiley, July 2003.

[101] British Standards Institute. The C++ Standard: Incorporating Technical Corrigendum No. 1. Wiley,

October 2003.

[102] Slinger Jansen, Gerco Ballintijn, and Sjaak Brinkkemper. A process model and typology for software prod-

uct updaters. In 9th European Conference on Software Maintenance and Reengineering (CSMR 2005),

March 2005.

[103] Steven C. Johnson. Yacc: Yet Another Compiler Compiler. In UNIX Programmer’s Manual, volume 2,

pages 353–387. Holt, Rinehart, and Winston, New York, NY, USA, 1979.

[104] Richard Jones and Rafael Lins. Garbage Collection: Algorithms for Automatic Dynamic Memory Man-

agement. John Wiley & Sons, August 1996.

[105] Dan Kaminsky. MD5 to be considered harmful someday. http://www.doxpara.com/md5_someday.pdf,

December 2004.

[106] Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language. Prentice Hall, second edition,

March 1988.

[107] David G. Korn and Kiem-Phong Vo. vdelta: Differencing and compression. In Balachander Krishna-

murthy, editor, Practical Reusable UNIX Software. John Wiley & Sons, 1995.

[108] N. Jesper Larsson and Kunihiko Sadakane. Faster sufﬁx sorting. Technical Report LU-CS-TR-99-214,

Lund University, 1999.

[109] David B. Leblang. The CM challenge: conﬁguration management that works. In W. F. Tichy, editor,

Conﬁguration management, pages 1–37. John Wiley & Sons, New York, NY, USA, 1995.

[110] Daan Leijen. The λ Abroad: A Functional Approach to Software Components. PhD thesis, Universiteit

Utrecht, November 2003.

[111] Daan Leijen. Extensible records with scoped labels. In Proceedings of the 2005 Symposium on Trends in

Functional Programming (TFP’05), September 2005.

[112] Thomas Leonard. The Zero Install system. http://zero-install.sourceforge.net/. Accessed 22 July 2005.

[113] Michael E. Lesk and Eric Schmidt. lex: A lexical analyzer generator. In UNIX Programmer’s Manual,

volume 2, pages 388–400. Holt, Rinehart, and Winston, New York, NY, USA, 1979.

[114] Ernst Lippe. Camera — Support for Distributed Cooperative Work. PhD thesis, Rijksuniversiteit Utrecht,

October 1992.

[115] Joshua P. MacDonald. File system support for delta compression. Master’s thesis, Department of Electrical

Engineering and Computer Science, University of California at Berkeley, 2000.

[116] Kenneth Manheimer, Barry A. Warsaw, Stephen N. Clark, and Walter Rowe. The Depot: A framework for

sharing software installation across organizational and UNIX platform boundaries. In Proceedings of the

4th Systems Administration Conference (LISA ’90), pages 37–46. USENIX Association, October 1990.

[117] Mauro Marinilli. Java Deployment with JNLP and WebStart. Sams, September 2001.

255

Bibliography

[118] Steve McConnell. Software Project Survival Guide. Microsoft Press, 1998.

[119] Bertrand Meyer. The grand challenge of trusted components. In Proceedings of the 25th International

Conference on Software Engineering (ICSE 2003), pages 660–667, May 2003.

[120] Ondrej Mikle. Practical attacks on digital signatures using MD5 message digest. Cryptology ePrint

Archive, Report 2004/356, 2004. http://eprint.iacr.org/.

[121] Peter Miller. Recursive make considered harmful. http://aegis.sourceforge.net/auug97.pdf, 1997. Accessed

12 August 2005.

[122] Robin Milner. A theory of type polymorphism in programming. Journal of Computer and System Sciences,

17:348–375, August 1978.

[123] Sape J. Mullender, editor. The Amoeba distributed operating system: selected papers 1984–1987. Num-

ber 41 in CWI tracts. Centrum voor Wiskunde en Informatica, Amsterdam, 1987.

[124] NCSA. Common gateway interface, version 1.1. http://hoohoo.ncsa.uiuc.edu/cgi/. Accessed 4 August

2005.

[125] Tobias Oetiker. SEPP: Software installation and sharing system. In Proceedings of the 12th Systems

Administration Conference (LISA ’98), pages 253–259. USENIX Association, December 1998.

[126] National Institute of Standards and Technology. Advanced encryption standard. http://www.nist.gov/aes/,

2001. Accessed 21 August 2005.

[127] National Institute of Standards and Technology. Secure hash standard, February 2004.

[128] Michael A. Olson. The design and implementation of the Inversion ﬁle system. In Proceedings of the

USENIX Winter 1993 Technical Conference, pages 205–218, January 1993.

[129] Kyle Oppenheim and Patrick McCormick. Deployme: Tellme’s package management and deployment

system. In Proceedings of the 14th Systems Administration Conference (LISA 2000), pages 187–196.

USENIX Association, December 2000.

[130] Vern Paxson. Flex — a fast scanner generator. http://www.gnu.org/software/ﬂex/, 2005.

[131] Colin Percival. An automated binary security update system for FreeBSD. In Proceedings of BSDCON

’03. USENIX, September 2003.

[132] Colin Percival. Binary diff/patch utility. http://www.daemonology.net/bsdiff/, 2003.

[133] Simon Peyton Jones. The Implementation of Functional Programming Languages. Prentice Hall, 1987.

[134] Simon Peyton Jones. Implementing lazy functional languages on stock hardware: the Spineless Tagless

G-machine. Journal of Functional Programming, 2(2):127–202, April 1992.

[135] Simon Peyton Jones, editor. Haskell 98 Language and Libraries: The Revised Report. Cambridge Uni-

versity Press, April 2004.

[136] Rob Pike. Systems software research is irrelevant. Research talk, http://www.cs.bell-labs.com/who/rob/

utah2000.pdf, February 2000.

[137] C. Michael Pilato, Ben Collins-Sussman, and Brian W. Fitzpatrick. Version Control with Subversion.

O’Reilly, June 2004.

[138] The Fink Project. Fink. http://ﬁnk.sourceforge.net/.

[139] Eric S. Raymond. The CML2 language: Python implementation of a constraint-based interactive conﬁg-

urator. In 9th International Python Conference, March 2001.

[140] Eric S. Raymond. The Art of Unix Programming. Addison-Wesley, September 2003.

[141] Ron Rivest. The MD5 message-digest algorithm. RFC 1321, April 1992.

[142] Mendel Rosenblum and John K. Ousterhout. The design and implementation of a log-structured ﬁle

system. ACM Transactions on Computer Systems, 10(1):26–52, 1992.

[143] John P. Rouillard and Richard B. Martin. Depot-Lite: a mechanism for managing software. In Proceedings

of the 8th Systems Administration Conference (LISA ’94), pages 83–91. USENIX Association, 1994.

[144] Antony Rowstron and Peter Druschel. Pastry: Scalable, decentralized object location and routing for large-

scale peer-to-peer systems. In 18th IFIP/ACM International Conference on Distributed Systems Platforms

(Middleware 2001), November 2001.

256

Bibliography

[145] Bruce Schneier. Applied Cryptography. John Wiley & Sons, second edition, 1996.

[146] Bruce Schneier. New cryptanalytic results against SHA-1. Crypto-Gram newsletter, http://www.schneier.

com/blog/archives/2005/08/new_cryptanalyt.html, August 2005.

[147] Bruce Schneier. SHA-1 broken. Crypto-Gram newsletter, http://www.schneier.com/crypto-gram-0503.

html, March 2005.

[148] Christopher Seiwald. Jam — Make(1) redux. In USENIX Applications Development Symposium, April

1994.

[149] Gustavo Noronha Silva. APT HOWTO. http://www.debian.org/doc/manuals/apt-howto/index.en.html,

2004. Accessed 26 August 2005.

[150] Anthony M. Sloane. Post-design domain-speciﬁc language embedding: A case study in the software

engineering domain. In Proceedings of the 35th Annual Hawaii International Conference on System

Sciences (HICSS’02), Washington, DC, USA, 2002. IEEE Computer Society.

[151] Guy L. Steele. RABBIT: A compiler for SCHEME. Technical Report AI-TR-474, MIT AI Lab, May

1978.

[152] W. Richard Stevens and Stephen A. Rago. Advanced Programming in the UNIX Environment. Addison-

Wesley, second edition, June 2005.

[153] Bjarne Stroustrup. The C++ Programming Language. Addison-Wesley, third edition, 1997.

[154] Clemens Szyperski. Component Software: Beyond Object-Oriented Programming. Addison-Wesley,

second edition, 2002.

[155] Clemens Szyperski. Component technology—what, where, and how? In Proceedings of the 25th Inter-

national Conference on Software Engineering (ICSE 2003), pages 684–693, May 2003.

[156] Andrew S. Tanenbaum. Computer Networks. Prentice Hall, fourth edition, August 2002.

[157] Peter Thoeny. Twiki—an enterprise collaboration platform. http://twiki.org/, 2005.

[158] ThoughtWorks. Cruise Control. http://cruisecontrol.sourceforge.net/, 2005.

[159] Walter F. Tichy. Tools for software conﬁguration management. In Jürgen F. H. Winkler, editor, Proceed-

ings of the International Workshop on Software Version and Conﬁguration Control, volume 30 of Berichte

des German Chapter of the ACM, Grassau, January 1988. Teubner.

[160] TIS Committee. Tool Interface Speciﬁcation (TIS) Executable and Linking Format (ELF) Speciﬁcation,

Version 1.2. http://www.x86.org/ftp/manuals/tools/elf.pdf, May 1995.

[161] TraCE Project. Nix deployment system. http://www.cs.uu.nl/wiki/Trace/Nix, 2005.

[162] Dimitre Trendaﬁlov, Nasir Memon, and Torsten Suel. zdelta: An efﬁcient delta compression tool. Tech-

nical Report TR-CIS-2002-02, Polytechnic University, 2002.

[163] Andrew Tridgell. Efﬁcient Algorithms for Sorting and Synchronization. PhD thesis, Australian National

University, February 1999.

[164] Vrije Universiteit. The Amoeba Reference Manual — Programming Guide, chapter Chapter 6.3 (Amake

User Reference Manual), pages 83–104. Vrije Universiteit, Amsterdam, 1996.

[165] Urbancode. Anthill. http://www.urbancode.com/projects/anthill/default.jsp, 2005. Accessed 21 August

2005.

[166] Mark van den Brand, Hayco de Jong, Paul Klint, and Pieter Olivier. Efﬁcient annotated terms. Software—

Practice and Experience, 30:259–291, 2000.

[167] André van der Hoek. Integrating conﬁguration management and software deployment. In Working Con-

ference on Complex and Dynamic Systems Architecture (CDSA 2001), December 2001.

[168] André van der Hoek, Richard S. Hall, Antonio Carzaniga, Dennis Heimbigner, and Alexander L. Wolf.

Software deployment: Extending conﬁguration management support into the ﬁeld. Crosstalk, The Journal

of Defense Software Engineering, 11(2), February 1998.

[169] André van der Hoek, Richard S. Hall, Dennis Heimbigner, and Alexander L. Wolf. Software release man-

agement. In M. Jazayeri and H. Schauer, editors, Proceedings of the Sixth European Software Engineering

Conference (ESEC/FSE 97), pages 159–175. Springer-Verlag, 1997.

257

Bibliography

[170] Tijs van der Storm. Variability and component composition. In Jan Bosch and Charles Krueger, editors,

Software Reuse: Methods, Techniques and Tools: 8th International Conference (ICSR-8), volume 3107 of

Lecture Notes in Computer Science, pages 86–100. Springer-Verlag, June 2004.

[171] Tijs van der Storm. Continuous release and upgrade of component-based software. In 12th International

Workshop on Software Conﬁguration Management (SCM-12), September 2005.

[172] Gary V. Vaughan, Ben Elliston, Tom Tromey, and Ian Lance Taylor. GNU Autoconf, Automake, and

Libtool. New Riders, October 2000. Also at http://sources.redhat.com/autobook/, accessed 21 august

2005.

[173] V. N. Venkatakrishnan, R. Sekar, Tapan Kamat, Soﬁa Tsipa, and Zhenkai Liang. An approach for secure

software installation. In Proceedings of the 16th Systems Administration Conference (LISA ’02), pages

219–226. USENIX Association, November 2002.

[174] Seth Vidal. Yellow dog Updater, Modiﬁed (yum). http://linux.duke.edu/projects/yum/.

[175] Eelco Visser. Syntax Deﬁnition for Language Prototyping. PhD thesis, University of Amsterdam, Septem-

ber 1997.

[176] Eelco Visser. Program transformation with Stratego/XT: Strategies, tools, and systems in StrategoXT-0.9.

In C. Lengauer et al., editor, Domain-Speciﬁc Program Generation, volume 3016 of Lecture Notes in

Computer Science, pages 216–238. Spinger-Verlag, June 2004.

[177] Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu. Finding collisions in the full SHA-1. In Victor Shoup,

editor, Advances in Cryptology — CRYPTO 2005: 25th Annual International Cryptology Conference,

volume 3621 of Lecture Notes in Computer Science. Springer-Verlag, August 2005.

[178] Xiaoyun Wang and Hongbo Yu. How to break MD5 and other hash functions. In Ronald Cramer, editor,

Advances in Cryptology — EUROCRYPT 2005: 24th Annual International Conference on the Theory and

Applications of Cryptographic Techniques, volume 3494 of Lecture Notes in Computer Science. Springer-

Verlag, May 2005.

[179] Paul R. Wilson. Uniprocessor garbage collection techniques. In International Workshop on Memory

Management, volume 637 of Lecture Notes in Computer Science. Springer-Verlag, September 1992.

[180] Stephen P. Berczuk with Brad Appleton. Software conﬁguration management patterns: effective team-

work, practical integration. Addison-Wesley, 2003.

[181] Walter Wong. Local disk depot: customizing the software environment. In Proceedings of the 7th Systems

Administration Conference (LISA ’93), pages 49–53. USENIX Association, November 1993.

[182] X.Org Foundation. X.Org homepage. http://wiki.x.org/wiki/, 2005. Accessed 27 November 2005.

258

Index

β -REDUCE semantic rule, 77, 83, 85

β -REDUCE’ semantic rule, 77, 85

acquirePathLock function, 99

addTempRoot function, 132

addToStore function

in the extensional model, 98

in the intensional model, 154

Amake, 62, 241

Ant, 240

application bundle, 12

APT, 201

ASSERT semantic rule, 78

assert keyword, 33

assertion, 33, 206

ATerm, 40, 105

ATerm library, 81

atom, 97

atomicity, 10, 37

attribute, 27

attribute set, 27, 73

recursive, 73, 77

selection, 77

Autoconf, 29, 246

Berkeley DB, 95

bias, 183

binary deployment, 11, 42, 45, 185, 187

binary patching, 192

binding time, 52

blacklist, 249

body attribute, 73

bootstrapping, 44, 177, 197

Boyer-Moore algorithm, 114

bsdiff, 193

build function

in the extensional model, 110, 117

in the intensional model, 156

build farm, 16, 210

build hook, 115, 218, 228

builder, 25

builder attribute, 27, 102

Camera, 202

canonical path, 73, 93

canonical serialisation, 91

canonicalise function, 73

canonicalisePathContents function, 112

channel, 43, 46, 188

circular dependency, 242

CLASSPATH, 4

ClearCase, 202

CLOSED semantic rule, 86

closure, 55, 96, 147

closure function, 96

closure invariant, 96, 109

closure’ function, 147

comment, 66

complete deployment, 8, 24

component, 19, 49

deﬁnition of, 50

component store, 19

composition, 25

concurrency, 98

conﬁguration management, 51, 114,

200, 216, 221

conﬂuence, 76

content-addressability, 97, 135, 140

continuous integration, 16, 209

correct deployment, 3

cryptographic hash, 8, 88

default arguments, 32

default.nix, 81

deleteAndSignal function, 99, 100

deletion token, 100

259

Index

dependency, 4, 21

exact, 24

nominal, 8, 24

dependency hell, 10

deployment policy, 42

DERIVATION semantic rule, 80, 100

derivation, 27

abstract syntax, 100

ﬁxed-output, 106

multiple outputs, 160

translation to store derivations, 100

deriver, 114, 249

deriver table, 114

deserialise function, 92

destructive upgrading, 9, 21, 36

determinism, 21

DLL hell, 5, 12

drvPath attribute, 101

dynamic composition, 170

ELF executable, 23, 171, 179, 180

eqClassMembers table, 146

equivalence class, 145

equivalence class unicity invariant, 148

extensional model, 87, 134

fetchsvn Nix expression, 107

fetchurl Nix expression, 27, 67, 106

ﬁle system object, 90

serialisation, 91

ﬁnd function, 113, 144

ﬁndRoots function, 127

ﬁxed-output derivation, 106

followRef function, 147

free variable, 75

FreeBSD, 169

FreeBSD Ports Collection, 11, 201

FSO, see ﬁle system object

garbage collection, 38, 124

conservative, 24, 56

liveness, 128

roots, 38, 125

stop-the-world approach, 129

generic builder, 33, 175

Gentoo Linux, 6, 201

Glibc, 21, 106, 169, 181, 190

Global Assembly Cache, 13, 203

hash function, 89

hash invariant, 142

hash rewriting, 135, 143, 246, 249

hashDrv function, 108

hashModulo function, 144

hashPart function, 94, 144

head normal form, 76

IFELSE semantic rule, 78

IFTHEN semantic rule, 78

IMPORT semantic rule, 80

import keyword, 29

IMPORT’ semantic rule, 81

impurity, 177, 179

incomplete deployment, see complete

deployment, 21

inherit keyword, 28, 74

instantiate function, 102

integrity invariant, 97

intensional model, 87, 135

interference, 9, 21

isolation, 14, 19, 182

Java, 4, 204

λ -calculus, 77

late binding, 53, 170

laziness, 62, 83

LET semantic rule, 77

let keyword, 73, 77

let-expression, 73, 77

Linux, 6, 169

livePaths function, 129

locking, 99

logging, 176

Maak, 241

Mac OS X, 12, 169

Make, 233, 240

makePath function, 94, 142

manifest, 45, 186

maximal laziness, 83

maximal sharing, 59

maybeRewrite function, 151

260

Index

MD5, 88

Microsoft Windows, 12, 170

multiple outputs, 105, 160

name attribute, 27, 108

namePart function, 94, 142

NAR, see Nix Archive

.NET, 12, 203

Nix, 3, 14

homepage, 14

Nix Archive, 92

Nix expression, 25

semantics, 71

syntax, 64

Nix Packages collection, 25, 42, 44, 167

nix-build command, 108, 125, 238

nix-env command, 35

--import (-I), 35

--install (-i), 35, 187

--query (-q), 35, 63

--remove-generations, 38

--rollback, 36

--switch-proﬁle, 37

--uninstall (-e), 38

--upgrade (-u), 43, 189

nix-instantiate command, 39, 100

nix-store command, 38

--gc, 38

--query, 41, 97

--realise, 41, 100, 108

--register-substitutes, 158, 187

--register-substitute, 119

--register-validity, 118

--verify, 96

NixOS, iii, 180, 247

Nixpkgs, see Nix Packages collection

non-strictness, 83

non-termination, 76

Odin, 62, 241

one-click installation, 43, 189

OPAND semantic rule, 79

OPEQ semantic rule, 78

OPHASATTR+ semantic rule, 79

OPHASATTR- semantic rule, 79

OPIMPL semantic rule, 79

OPNEG semantic rule, 79

OPOR semantic rule, 79

OPPLUSPATH semantic rule, 79

OPPLUSSTR semantic rule, 79

OPUPDATE semantic rule, 79

outPath attribute, 101

output path, 28

outputHash attribute, 107

outputHashAlgo attribute, 107

outputHashMode attribute, 107

package, 19, 49

parseDrv function, 108

partial parameterisation, 31

patch chaining, 195

PatchELF, 213

patchelf command, 179, 181

path literal, 67

Perl, 174

pointer discipline, 87

pointer graph, 56

pointer hiding, 58, 182

Portage, 201

primop, 80

printDrv function, 105

printHash

function, 89

printHash

function, 89

privacy, 158

processBinding function, 103

proﬁle, 37

purely functional deployment model, 14,

21, 167, 245

purity, 178, 183

Python, 174

queries, 35

readPath function, 91

REC semantic rule, 77, 83, 85

rec keyword, 29, 73, 77

recursion, 73

Red Hat Package Manager, 6, 201

refClasses table, 147

reference, 23, 53, 55, 96

reference invariant, 119

references table, 96

261

Index

referers table, 96

referrer closure, 97

release, 17

releasePathLock function, 99

replace function, 144

resolve function, 150

retained dependency, 23, 54

rollback, 5, 36

RPATH, 23, 171, 179

RPM, see Red Hat Package Manager

scanForReferences function, 113, 163

SDF, see Syntax Deﬁnition Formalism

security, 135

SELECT semantic rule, 77, 83, 85

serialise function, 93

service, 221

service deployment, 17, 221

setReferences function, 97

SHA-1, 88

SHA-256, 88

sharing, 59, 135

Software Dock, 204

source deployment, 11, 42, 44, 185, 187

spec ﬁle, 7

spyware, 137

standard environment, 26, 174

static composition, 170

store, 19, 90

store derivation, 39

abstract syntax, 101

instantiation, 100

realisation, 41

store object, 92

validity, 95

store path, 20, 92

stored hash invariant, 96

string literal, 67

subpath, 72

subst function, 75

substitute, 45, 108, 118, 185

substitute function, 109

in the extensional model, 120

in the intensional model, 158

substitutes table, 118, 157

Subversion, 31, 107

Syntax Deﬁnition Formalism, 64

system attribute, 102

tarball, 214

TraCE, iii, 14

traceability, 114

traceability invariant, 114

transparent source/binary deployment,

15, 45, 118, 185, 188

Trojan horse, 88, 135

trust, 137

type attribute, 101

Unix, 6, 200

update operator, 79

usable path, 119, 146

user environment, 35, 36, 184

valid table, 95

valid path, 95

validation, 181

validity invariant, 95

variability, 4, 31, 245

Vesta, 202, 241

weak head normal form, 76

Web Start, 204

WITH semantic rule, 78, 85

working copy, 43

writePath function, 91

XML, 177, 248, 249

XSLT, 177

Yum, 11, 201

Zero Install, 13, 136, 170

262

Samenvatting

De installatie van software en de daarmee samenhangende activiteiten lijken altijd te mis-

lukken. Zo komt het regelmatig voor dat het installeren of upgraden van een software-

toepassing leidt tot het falen van eerder geïnstalleerde toepassingen. Of het blijkt dat een

applicatie afhankelijk is van componenten die niet aanwezig zijn op het systeem. Het is

meestal ook niet mogelijk om dit soort acties terug te draaien als ze naderhand niet blij-

ken te bevallen. Deze activiteiten worden aangeduid als software deployment, d.w.z. het

in gebruik nemen of “uitrollen” van software. Het onderwerp van dit proefschrift is de

ontwikkeling van betere technieken om deployment te ondersteunen.

De onderliggende problemen die de deploymentmalaise veroorzaken zijn een gebrek

aan isolatie tussen componenten, en het feit dat het lastig is om afhankelijkheden (depen-

dencies) tussen componenten correct te identiﬁceren. Voorts schaalt deployment moeilijk

op. Zo toont Figuur 1.5 op pagina 10 de afhankelijkheden van de browser Mozilla Firefox.

Elke component kan in potentie de deployment van Firefox doen mislukken. Zo kan de

component afwezig zijn, aanwezig zijn in een incompatibele versie, of aanwezig zijn in

een compatibele versie maar met verkeerde compilerinstellingen gebouwd zijn.

Er zijn talloze deploymenttechnologieën, in de Unix-wereld ook wel package managers

genaamd. Ik noem met name RPM, Debian APT, Gentoo Portage, .NET assemblies en

Mac OS application bundles. Het belang van deployment—en het verlangen naar verbete-

ringen op dit vlak—wordt onderstreept door het feit dat Linux-distributies zich in de eerste

plaats onderscheiden in hun package-managementsystemen. Deze hebben echter allemaal

ernstige tekortkomingen. Ze zijn niet in staat om volledige afhankelijkheidsspeciﬁcaties af

te dwingen, ze ondersteunen niet de aanwezigheid van meerdere versies van een compo-

nent, enzovoorts.

Er is tot op heden betrekkelijk weinig onderzoek geweest naar deployment, en vrijwel

helemaal niet naar de low-level aspecten zoals de opslag van componenten in het bestands-

systeem (waar isolatie immers gerealiseerd moet worden). In plaats daarvan is vooruitgang

op dit gebied voornamelijk geboekt door systeembeheerders en Unix-hackers. Dit is niet

verbazingwekkend: deployment is een onderdeel van het vakgebied van systeembeheer,

een onderwerp dat volgens Pike [136] vooralsnog “deeply difﬁcult” is.

Het Nix deploymentsysteem

De Nix store Dit proefschrift beschrijft een betere aanpak, een zogenaamd puur functi-

oneel deploymentmodel. Zo’n model is geïmplementeerd in een deploymentsysteem ge-

naamd Nix. Nix slaat componenten in isolatie van elkaar op in een Nix store, een speciale

directory in het bestandssysteem.

In Nix wordt elke component opgeslagen onder een speciaal pad zoals /nix/store/-

2zbay49r2ihrznny6vbrcjvils4nqm1w-ﬁrefox-1.0.7. Het deel 2zbay49r2ihr... is een crypto-

graﬁsche hash van alle invoer die heeft bijgedragen aan de berekening van de component.

263

Samenvatting

Daarom heet het model puur functioneel: net als in puur functionele programmeertalen

wordt het resultaat van een bouwactie alleen bepaald door de opgegeven invoer. Crypto-

graﬁsche hashes hebben de plezierige eigenschap dat het erg moeilijk is om twee verschil-

lende sets van invoeren te vinden die dezelfde hash opleveren. Hierdoor kunnen we er in

de praktijk vanuit gaan dat als twee componenten op enige wijze verschillen, ze een ver-

schillende hash opleveren en dus op verschillende locaties in het bestandssysteem worden

opgeslagen. Dit zorgt voor de gewenste isolatie tussen componenten. De installatie van een

applicatie kan niet langer leiden tot de beschadiging van reeds geïnstalleerde applicaties.

Het gebruik van hashes lost ook het andere grote deploymentprobleem op—onvolledige

afhankelijkheidsinformatie. Ten eerste voorkomt het ongedeclareerde afhankelijkheden

tijdens bouwtijd. Immers, bouwtools zoals compilers en linkers zoeken niet standaard in

paden zoals /nix/store/2zbay49r2ihr...-ﬁrefox-1.0.7.

Ten tweede maakt het het ons mogelijk om te scannen naar de afhankelijkheden die

kunnen optreden gedurende de executie van een component. Dit houdt in dat we de binaire

inhoud van een component doorzoeken op de aanwezigheid van de hashonderdelen van

de bestandsnamen van andere componenten. Stel dat de Nix store een GUI-component

/nix/store/4kd0ma2pxf6w...-gtk+-2.8.6 bevat (een GUI-bibliotheek). Als we vervolgens de

tekenreeks 4kd0ma2pxf6w... tegenkomen in de inhoud van /nix/store/2zbay49r2ihr...-ﬁrefox-

1.0.7, kunnen we concluderen dat de Firefox-component een afhankelijkheid heeft op

/nix/store/4kd0ma2pxf6w...-gtk+-2.8.6. De geldigheid van deze aanpak motiveer ik door

middel van een analogie met technieken die worden gebruikt in de implementatie van pro-

grammeertalen, in het bijzonder conservatieve garbage collection (hoofdstuk 3).

De scanaanpak zorgt voor volledige kennis van de afhankelijkheden tussen componen-

ten. Deployment van een component verloopt correct indien we de afsluiting (closure)

van de component onder de afhankelijkheidsrelatie kopiëren. Voorts kunnen componenten

veilig verwijderd worden: een component mag alleen verwijderd worden indien er geen

componenten in de store meer naar verwijzen. Het is zelfs mogelijk om ongebruikte com-

ponenten volledig automatisch weg te laten gooien (garbage collection).

Abstractie over hashes Uiteraard dienen gebruikers en ontwikkelaars niet geconfron-

teerd te worden met bestandsnamen met eerdergenoemde hashes. Dat is gelukkig ook

niet het geval. Ze worden verborgen voor gebruikers door zogeheten user environments

die een verzameling “geactiveerde” componenten beschikbaar maken voor de gebruiker.

User environments zijn zelf ook (automatisch gegenereerde) componenten. Ze kunnen dus

gebroederlijk naast elkaar bestaan. Dit stelt de gebruikers van een systeem in staat om ver-

schillende user environments te gebruiken. Ook maakt het een rollback mogelijk waarbij

installatie- of upgradeacties teruggedraaid worden. Dit is een belangrijke eigenschap in

bijvoorbeeld serveromgevingen.

Evenmin hoeven ontwikkelaars rechtstreeks met hashes te werken. Nix componenten

worden namelijk gebouwd uit Nix-expressies (hoofdstukken 2 en 4). Dit is een eenvoudige

functionele taal die beschrijft hoe componenten gebouwd en samengesteld moeten worden.

Uit deze beschrijvingen berekent Nix de paden waar de componenten opgeslagen worden.

Transparant source/binary deploymentmodel Het uitrollen van Nix-expressies naar

doelmachines levert een source-deploymentmodel op, omdat ze beschrijven hoe compo-

264

Samenvatting

nenten uit broncode gebouwd kunnen worden. Zo’n model—ook toegepast in deploy-

mentsystemen zoals Gentoo Linux—is ﬂexibel en prettig voor ontwikkelaars, maar ook

erg resource-intensief: het kost dikwijls veel tijd en schijfruimte om een component uit

broncode te bouwen. Daarentegen is een binair deploymentmodel, waarin componenten

in binaire vorm worden gedistribueerd naar doelmachines, efﬁciënter maar ook minder

ﬂexibel.

Nix combineert het beste van beide werelden door middel van een transparant sour-

ce/binary deploymentmodel. Softwaredistributeurs of systeembeheerders bouwen een Nix-

expressie en plaatsen de resulterende binaire componenten op een centrale server, ge-

indexeerd onder hun hashes. Doelmachines kunnen vervolgens het bestaan van deze voor-

gebouwde componenten registreren. Deze registraties heten substituten. De bouw van

een Nix-expressie kan dan worden “kortgesloten” indien voor het te bouwen pad (zoals

/nix/store/2zbay49r2ihr...-ﬁrefox-1.0.7) een substituut bekend is die opgehaald en uitgepakt

kan worden. Op die manier verandert source-deployment op een voor de gebruiker trans-

parante wijze automatisch in binaire deployment.

Extensioneel en intensioneel model Het proefschrift beschrijft twee subtiel verschil-

lende modellen voor de implementatie van Nix. Het oorspronkelijke en momenteel ge-

bruikte model is het extensionele model (hoofdstuk 5), waarin store-paden vooraf worden

berekend uit Nix-expressies. Dit is nodig omdat het pad van een component bekend moet

zijn vóórdat de component gebouwd wordt. Het maakt het echter lastig om een Nix store te

delen tussen meerdere, elkaar wellicht wederzijds niet vertrouwende gebruikers. Immers,

een kwaadwillende gebruiker kan zomaar een onzinnig substituut registreren voor een pad

dat daarna geïnstalleerd wordt door een andere gebruiker.

Het intensionele model (hoofdstuk 6) heft deze beperking op door alle componenten

op een inhoudsgeadresseerde wijze op te slaan. Hierbij wordt de naam van een compo-

nent bepaald door de cryptograﬁsche hash over de inhoud van de component in plaats van

over de Nix-expressie. Twee componenten kunnen dus alleen op dezelfde plaats in het be-

standssysteem worden opgeslagen als ze inhoudelijk (intensioneel) gelijk zijn. Gebruikers

kunnen hierdoor verschillende substituten registreren voor dezelfde Nix-expressie. Het

resultaat is een Nix store die op veilige wijze door gebruikers gedeeld kan worden.

Toepassingen

De voornaamste toepassing van Nix is deployment (hoofdstuk 7). We hebben Nix ge-

bruikt om meer dan 278 bestaande Unix-componenten uit te rollen. Deze pakketten zijn

onderdeel van de Nix Packages collection (Nixpkgs, sectie 7.1). Nixpkgs dient zowel ter

validatie van onze aanpak, als om gebruikers van Nix een bruikbare basis van pakketten

aan te bieden waarop verder ontwikkeld kan worden.

Nix is echter ook toepasbaar op een aantal aan deployment grenzende probleemgebie-

den. Het streven is om zoveel mogelijk aspecten van het bouw- en deploymentproces onder

één enkel formalisme te brengen, namelijk Nix-expressies.

Continue integratie en releasemanagement (hoofdstuk 8) Het is een goede gewoonte

om tijdens het ontwikkelen van grote softwaresystemen het samenvoegen van de verschil-

265

lende onderdelen niet te lang uit te stellen, daar dit kan leiden tot zogenaamde ‘Big Bang

integratie” waar in een laat stadium geconstateerd wordt dat de onderdelen niet samen-

werken. Bij continue integratie worden componenten voortdurend gebouwd vanuit het

versiebeheersysteem, dus iedere keer dat een ontwikkelaar een verandering toepast. Een

build farm kan deze praktijk ondersteunen. Dit is een verzameling machines die de softwa-

re bouwt, typisch onder verschillende besturingssystemen en architecturen. Een build farm

is niet alleen nuttig om software te testen, maar kan ook meteen releases van de software

maken—kant en klare pakketten die door gebruikers geïnstalleerd kunnen worden.

Het beheer van zo’n build farm is vaak nogal tijdrovend, omdat elke gewenste verande-

ring in de softwareomgeving (zoals een nieuwe versie van een compiler) op elke machine

doorgevoerd moet worden. Dit kan echter geautomatiseerd worden met Nix, omdat het

op een transparante wijze Nix-expressies voor meerdere platformtypes kan bouwen. Nix

voorkomt ook dat componenten die niet gewijzigd zijn steeds opnieuw gebouwd worden.

Tenslotte kunnen de in een op Nix gebaseerde build farm geproduceerde releases op be-

trouwbare wijze via Nix uitgerold worden.

Service deployment (hoofdstuk 9) Het uitrollen van draaiende netwerkdiensten (servi-

ces) zoals webservers is een erg tijdrovend en meestal slecht beheerd proces. Veel acties—

het installeren van software, aanpassen van conﬁguratiebestanden, enz.—worden met de

hand uitgevoerd en zijn daardoor niet automatisch reproduceerbaar. Hierdoor is het lastig

om meerdere versies van een dienst (bijvoorbeeld test- en productieversies) naast elkaar te

draaien, of om een rollback uit te voeren naar een eerdere toestand.

We kunnen dit probleem met Nix voor een groot gedeelte aanpakken door de relatief

statische delen van een dienst (nl. de software en de conﬁguratiebestanden) als een Nix-

component te behandelen. Dit wil zeggen dat ze vanuit een Nix-expressie gebouwd worden

en in de Nix store terecht komen. Alle voordelen van Nix voor deployment—zoals het

gebruik van meerdere versies naast elkaar, reproduceerbaarheid, rollbacks, enz.—zijn op

die manier onmiddelijk van toepassing op de uitrol van diensten. De dynamische toestand

van een dienst (zoals databases) is echter niet onder Nix-beheer.

Build management (hoofdstuk 10) Het bouwen van software is een klassiek probleem

dat typisch door tools zoals Make (1979) wordt opgelost. Zulke bouwtools hebben ech-

ter allerlei beperkingen die veel lijken op die van deploymentsystemen: ze kunnen niet

garanderen dat afhankelijkheidsspeciﬁcaties volledig zijn, ze hebben slechte ondersteu-

ning voor meerdere varianten van bouwacties, enz. Dit is niet verrassend, omdat bouwen

en deployment goeddeels hetzelfde probleem zijn, met dien verstande dat bouwtools ty-

pisch betrekking hebben op “kleine” componenten (zoals individuele broncodebestanden),

terwijl deploymentsystemen te maken hebben met “grote” componenten (zoals complete

programma’s). Het is dus vrij eenvoudig om Nix te gebruiken als een Make-vervanger,

simpelweg door Nix-expressies te schrijven die voornoemde “kleine” componenten bou-

wen.

266

Curriculum Vitae

Eelco Dolstra

18 augustus 1978

Geboren te Wageningen

1990–1996

Voorbereidend Wetenschappelijk Onderwijs aan de Christelijke Scholengemeenschap “Het

Streek” te Ede

Diploma (gymnasium) behaald op 12 juni 1996

1996–2001

Studie Informatica aan de Universiteit Utrecht

Propedeutisch diploma behaald op 27 augustus 1997 (cum laude)

Doctoraal diploma behaald op 10 augustus 2001 (cum laude)

2001–2005

Assistent in Opleiding, Department of Information and Computing Sciences, Universiteit

Utrecht

267

268

Titles in the IPA Dissertation Series

J.O. Blanco. The State Operator in Process Alge-

bra. Faculty of Mathematics and Computing Science,

TUE. 1996-01

A.M. Geerling. Transformational Development of

Data-Parallel Algorithms. Faculty of Mathematics

and Computer Science, KUN. 1996-02

P.M. Achten. Interactive Functional Programs:

Models, Methods, and Implementation. Faculty of

Mathematics and Computer Science, KUN. 1996-03

M.G.A. Verhoeven. Parallel Local Search. Faculty

of Mathematics and Computing Science, TUE. 1996-

M.H.G.K. Kesseler. The Implementation of Func-

tional Languages on Parallel Machines with Distrib.

Memory. Faculty of Mathematics and Computer Sci-

ence, KUN. 1996-05

D. Alstein. Distributed Algorithms for Hard Real-

Time Systems. Faculty of Mathematics and Comput-

ing Science, TUE. 1996-06

J.H. Hoepman. Communication, Synchronization,

and Fault-Tolerance. Faculty of Mathematics and

Computer Science, UvA. 1996-07

H. Doornbos. Reductivity Arguments and Program

Construction. Faculty of Mathematics and Comput-

ing Science, TUE. 1996-08

D. Turi. Functorial Operational Semantics and its

Denotational Dual. Faculty of Mathematics and Com-

puter Science, VUA. 1996-09

A.M.G. Peeters. Single-Rail Handshake Circuits.

Faculty of Mathematics and Computing Science,

TUE. 1996-10

N.W.A. Arends. A Systems Engineering Speciﬁcation

Formalism. Faculty of Mechanical Engineering, TUE.

1996-11

P. Severi de Santiago. Normalisation in Lambda Cal-

culus and its Relation to Type Inference. Faculty of

Mathematics and Computing Science, TUE. 1996-12

D.R. Dams. Abstract Interpretation and Partition Re-

ﬁnement for Model Checking. Faculty of Mathematics

and Computing Science, TUE. 1996-13

M.M. Bonsangue. Topological Dualities in Seman-

tics. Faculty of Mathematics and Computer Science,

VUA. 1996-14

B.L.E. de Fluiter. Algorithms for Graphs of Small

Treewidth. Faculty of Mathematics and Computer

Science, UU. 1997-01

W.T.M. Kars. Process-algebraic Transformations in

Context. Faculty of Computer Science, UT. 1997-02

P.F. Hoogendijk. A Generic Theory of Data Types.

Faculty of Mathematics and Computing Science,

TUE. 1997-03

T.D.L. Laan. The Evolution of Type Theory in Logic

and Mathematics. Faculty of Mathematics and Com-

puting Science, TUE. 1997-04

C.J. Bloo. Preservation of Termination for Explicit

Substitution. Faculty of Mathematics and Computing

Science, TUE. 1997-05

J.J. Vereijken. Discrete-Time Process Algebra. Fac-

ulty of Mathematics and Computing Science, TUE.

1997-06

F.A.M. van den Beuken. A Functional Approach to

Syntax and Typing. Faculty of Mathematics and Infor-

matics, KUN. 1997-07

A.W. Heerink. Ins and Outs in Refusal Testing. Fac-

ulty of Computer Science, UT. 1998-01

G. Naumoski and W. Alberts. A Discrete-Event Sim-

ulator for Systems Engineering. Faculty of Mechani-

cal Engineering, TUE. 1998-02

J. Verriet. Scheduling with Communication for Mul-

tiprocessor Computation. Faculty of Mathematics and

Computer Science, UU. 1998-03

J.S.H. van Gageldonk. An Asynchronous Low-Power

80C51 Microcontroller. Faculty of Mathematics and

Computing Science, TUE. 1998-04

A.A. Basten. In Terms of Nets: System Design with

Petri Nets and Process Algebra. Faculty of Mathemat-

ics and Computing Science, TUE. 1998-05

E. Voermans. Inductive Datatypes with Laws and

Subtyping — A Relational Model. Faculty of Math-

ematics and Computing Science, TUE. 1999-01

H. ter Doest. Towards Probabilistic Uniﬁcation-

based Parsing. Faculty of Computer Science, UT.

1999-02

J.P.L. Segers. Algorithms for the Simulation of Sur-

face Processes. Faculty of Mathematics and Comput-

ing Science, TUE. 1999-03

C.H.M. van Kemenade. Recombinative Evolution-

ary Search. Faculty of Mathematics and Natural Sci-

ences, UL. 1999-04

E.I. Barakova. Learning Reliability: a Study on In-

decisiveness in Sample Selection. Faculty of Mathe-

matics and Natural Sciences, RUG. 1999-05

M.P. Bodlaender. Scheduler Optimization in Real-

Time Distributed Databases. Faculty of Mathematics

and Computing Science, TUE. 1999-06

M.A. Reniers. Message Sequence Chart: Syntax and

Semantics. Faculty of Mathematics and Computing

Science, TUE. 1999-07

J.P. Warners. Nonlinear approaches to satisﬁabil-

ity problems. Faculty of Mathematics and Computing

Science, TUE. 1999-08

J.M.T. Romijn. Analysing Industrial Protocols with

Formal Methods. Faculty of Computer Science, UT.

1999-09

P.R. D’Argenio. Algebras and Automata for Timed

and Stochastic Systems. Faculty of Computer Science,

UT. 1999-10

G. Fábián. A Language and Simulator for Hybrid

Systems. Faculty of Mechanical Engineering, TUE.

1999-11

J. Zwanenburg. Object-Oriented Concepts and

Proof Rules. Faculty of Mathematics and Computing

Science, TUE. 1999-12

R.S. Venema. Aspects of an Integrated Neural Pre-

diction System. Faculty of Mathematics and Natural

Sciences, RUG. 1999-13

J. Saraiva. A Purely Functional Implementation of

Attribute Grammars. Faculty of Mathematics and

Computer Science, UU. 1999-14

R. Schiefer. Viper, A Visualisation Tool for Parallel

Program Construction. Faculty of Mathematics and

Computing Science, TUE. 1999-15

K.M.M. de Leeuw. Cryptology and Statecraft in the

Dutch Republic. Faculty of Mathematics and Com-

puter Science, UvA. 2000-01

T.E.J. Vos. UNITY in Diversity. A stratiﬁed approach

to the veriﬁcation of distributed algorithms. Faculty

of Mathematics and Computer Science, UU. 2000-02

W. Mallon. Theories and Tools for the Design of

Delay-Insensitive Communicating Processes. Faculty

of Mathematics and Natural Sciences, RUG. 2000-03

W.O.D. Grifﬁoen. Studies in Computer Aided Veriﬁ-

cation of Protocols. Faculty of Science, KUN. 2000-

P.H.F.M. Verhoeven. The Design of the MathSpad

Editor. Faculty of Mathematics and Computing Sci-

ence, TUE. 2000-05

J. Fey. Design of a Fruit Juice Blending and Packag-

ing Plant. Faculty of Mechanical Engineering, TUE.

2000-06

M. Franssen. Cocktail: A Tool for Deriving Correct

Programs. Faculty of Mathematics and Computing

Science, TUE. 2000-07

P.A. Olivier. A Framework for Debugging Hetero-

geneous Applications. Faculty of Natural Sciences,

Mathematics and Computer Science, UvA. 2000-08

E. Saaman. Another Formal Speciﬁcation Language.

Faculty of Mathematics and Natural Sciences, RUG.

2000-10

M. Jelasity. The Shape of Evolutionary Search Dis-

covering and Representing Search Space Structure.

Faculty of Mathematics and Natural Sciences, UL.

2001-01

R. Ahn. Agents, Objects and Events a computational

approach to knowledge, observation and communica-

tion. Faculty of Mathematics and Computing Science,

TU/e. 2001-02

M. Huisman. Reasoning about Java programs in

higher order logic using PVS and Isabelle. Faculty

of Science, KUN. 2001-03

I.M.M.J. Reymen. Improving Design Processes

through Structured Reﬂection. Faculty of Mathemat-

ics and Computing Science, TU/e. 2001-04

S.C.C. Blom. Term Graph Rewriting: syntax and se-

mantics. Faculty of Sciences, Division of Mathemat-

ics and Computer Science, VUA. 2001-05

R. van Liere. Studies in Interactive Visualization.

Faculty of Natural Sciences, Mathematics and Com-

puter Science, UvA. 2001-06

A.G. Engels. Languages for Analysis and Testing of

Event Sequences. Faculty of Mathematics and Com-

puting Science, TU/e. 2001-07

J. Hage. Structural Aspects of Switching Classes.

Faculty of Mathematics and Natural Sciences, UL.

2001-08

M.H. Lamers. Neural Networks for Analysis of Data

in Environmental Epidemiology: A Case-study into

Acute Effects of Air Pollution Episodes. Faculty of

Mathematics and Natural Sciences, UL. 2001-09

T.C. Ruys. Towards Effective Model Checking. Fac-

ulty of Computer Science, UT. 2001-10

D. Chkliaev. Mechanical veriﬁcation of concurrency

control and recovery protocols. Faculty of Mathemat-

ics and Computing Science, TU/e. 2001-11

M.D. Oostdijk. Generation and presentation of for-

mal mathematical documents. Faculty of Mathemat-

ics and Computing Science, TU/e. 2001-12

A.T. Hofkamp. Reactive machine control: A simula-

tion approach using χ. Faculty of Mechanical Engi-

neering, TU/e. 2001-13

D. Bošna

cki. Enhancing state space reduction tech-

niques for model checking. Faculty of Mathematics

and Computing Science, TU/e. 2001-14

M.C. van Wezel. Neural Networks for Intelligent

Data Analysis: theoretical and experimental aspects.

Faculty of Mathematics and Natural Sciences, UL.

2002-01

V. Bos and J.J.T. Kleijn. Formal Speciﬁcation and

Analysis of Industrial Systems. Faculty of Mathemat-

ics and Computer Science and Faculty of Mechanical

Engineering, TU/e. 2002-02

T. Kuipers. Techniques for Understanding Legacy

Software Systems. Faculty of Natural Sciences, Math-

ematics and Computer Science, UvA. 2002-03

S.P. Luttik. Choice Quantiﬁcation in Process Alge-

bra. Faculty of Natural Sciences, Mathematics, and

Computer Science, UvA. 2002-04

R.J. Willemen. School Timetable Construction: Al-

gorithms and Complexity. Faculty of Mathematics

and Computer Science, TU/e. 2002-05

M.I.A. Stoelinga. Alea Jacta Est: Veriﬁcation

of Probabilistic, Real-time and Parametric Systems.

Faculty of Science, Mathematics and Computer Sci-

ence, KUN. 2002-06

N. van Vugt. Models of Molecular Computing. Fac-

ulty of Mathematics and Natural Sciences, UL. 2002-

A. Fehnker. Citius, Vilius, Melius: Guiding and

Cost-Optimality in Model Checking of Timed and Hy-

brid Systems. Faculty of Science, Mathematics and

Computer Science, KUN. 2002-08

R. van Stee. On-line Scheduling and Bin Packing.

Faculty of Mathematics and Natural Sciences, UL.

2002-09

D. Tauritz. Adaptive Information Filtering: Concepts

and Algorithms. Faculty of Mathematics and Natural

Sciences, UL. 2002-10

M.B. van der Zwaag. Models and Logics for Process

Algebra. Faculty of Natural Sciences, Mathematics,

and Computer Science, UvA. 2002-11

J.I. den Hartog. Probabilistic Extensions of Seman-

tical Models. Faculty of Sciences, Division of Mathe-

matics and Computer Science, VUA. 2002-12

L. Moonen. Exploring Software Systems. Faculty

of Natural Sciences, Mathematics, and Computer Sci-

ence, UvA. 2002-13

J.I. van Hemert. Applying Evolutionary Computa-

tion to Constraint Satisfaction and Data Mining. Fac-

ulty of Mathematics and Natural Sciences, UL. 2002-

S. Andova. Probabilistic Process Algebra. Faculty of

Mathematics and Computer Science, TU/e. 2002-15

Y.S. Usenko. Linearization in µCRL. Faculty of

Mathematics and Computer Science, TU/e. 2002-16

J.J.D. Aerts. Random Redundant Storage for Video

on Demand. Faculty of Mathematics and Computer

Science, TU/e. 2003-01

M. de Jonge. To Reuse or To Be Reused: Techniques

for component composition and construction. Faculty

of Natural Sciences, Mathematics, and Computer Sci-

ence, UvA. 2003-02

J.M.W. Visser. Generic Traversal over Typed Source

Code Representations. Faculty of Natural Sciences,

Mathematics, and Computer Science, UvA. 2003-03

S.M. Bohte. Spiking Neural Networks. Faculty of

Mathematics and Natural Sciences, UL. 2003-04

T.A.C. Willemse. Semantics and Veriﬁcation in Pro-

cess Algebras with Data and Timing. Faculty of Math-

ematics and Computer Science, TU/e. 2003-05

S.V. Nedea. Analysis and Simulations of Catalytic Re-

actions. Faculty of Mathematics and Computer Sci-

ence, TU/e. 2003-06

M.E.M. Lijding. Real-time Scheduling of Tertiary

Storage. Faculty of Electrical Engineering, Mathe-

matics & Computer Science, UT. 2003-07

H.P. Benz. Casual Multimedia Process Annotation —

CoMPAs. Faculty of Electrical Engineering, Mathe-

matics & Computer Science, UT. 2003-08

D. Distefano. On Modelchecking the Dynamics of

Object-based Software: a Foundational Approach.

Faculty of Electrical Engineering, Mathematics &

Computer Science, UT. 2003-09

M.H. ter Beek. Team Automata — A Formal Ap-

proach to the Modeling of Collaboration Between Sys-

tem Components. Faculty of Mathematics and Natural

Sciences, UL. 2003-10

D.J.P. Leijen. The λ Abroad — A Functional Ap-

proach to Software Components. Faculty of Mathe-

matics and Computer Science, UU. 2003-11

W.P.A.J. Michiels. Performance Ratios for the Dif-

ferencing Method. Faculty of Mathematics and Com-

puter Science, TU/e. 2004-01

G.I. Jojgov. Incomplete Proofs and Terms and Their

Use in Interactive Theorem Proving. Faculty of Math-

ematics and Computer Science, TU/e. 2004-02

P. Frisco. Theory of Molecular Computing — Splic-

ing and Membrane systems. Faculty of Mathematics

and Natural Sciences, UL. 2004-03

S. Maneth. Models of Tree Translation. Faculty of

Mathematics and Natural Sciences, UL. 2004-04

Y. Qian. Data Synchronization and Browsing for

Home Environments. Faculty of Mathematics and

Computer Science and Faculty of Industrial Design,

TU/e. 2004-05

F. Bartels. On Generalised Coinduction and Prob-

abilistic Speciﬁcation Formats. Faculty of Sciences,

Division of Mathematics and Computer Science,

VUA. 2004-06

L. Cruz-Filipe. Constructive Real Analysis: a Type-

Theoretical Formalization and Applications. Fac-

ulty of Science, Mathematics and Computer Science,

KUN. 2004-07

E.H. Gerding. Autonomous Agents in Bargaining

Games: An Evolutionary Investigation of Fundamen-

tals, Strategies, and Business Applications. Faculty of

Technology Management, TU/e. 2004-08

N. Goga. Control and Selection Techniques for the

Automated Testing of Reactive Systems. Faculty of

Mathematics and Computer Science, TU/e. 2004-09

M. Niqui. Formalising Exact Arithmetic: Represen-

tations, Algorithms and Proofs. Faculty of Science,

Mathematics and Computer Science, RU. 2004-10

A. Löh. Exploring Generic Haskell. Faculty of Math-

ematics and Computer Science, UU. 2004-11

I.C.M. Flinsenberg. Route Planning Algorithms for

Car Navigation. Faculty of Mathematics and Com-

puter Science, TU/e. 2004-12

R.J. Bril. Real-time Scheduling for Media Processing

Using Conditionally Guaranteed Budgets. Faculty of

Mathematics and Computer Science, TU/e. 2004-13

J. Pang. Formal Veriﬁcation of Distributed Systems.

Faculty of Sciences, Division of Mathematics and

Computer Science, VUA. 2004-14

F. Alkemade. Evolutionary Agent-Based Economics.

Faculty of Technology Management, TU/e. 2004-15

E.O. Dijk. Indoor Ultrasonic Position Estimation Us-

ing a Single Base Station. Faculty of Mathematics and

Computer Science, TU/e. 2004-16

S.M. Orzan. On Distributed Veriﬁcation and Veriﬁed

Distribution. Faculty of Sciences, Division of Mathe-

matics and Computer Science, VUA. 2004-17

M.M. Schrage. Proxima — A Presentation-oriented

Editor for Structured Documents. Faculty of Mathe-

matics and Computer Science, UU. 2004-18

E. Eskenazi and A. Fyukov. Quantitative Prediction

of Quality Attributes for Component-Based Software

Architectures. Faculty of Mathematics and Computer

Science, TU/e. 2004-19

P.J.L. Cuijpers. Hybrid Process Algebra. Faculty of

Mathematics and Computer Science, TU/e. 2004-20

N.J.M. van den Nieuwelaar. Supervisory Machine

Control by Predictive-Reactive Scheduling. Faculty

of Mechanical Engineering, TU/e. 2004-21

E. Ábrahám. An Assertional Proof System for Multi-

threaded Java — Theory and Tool Support. Faculty of

Mathematics and Natural Sciences, UL. 2005-01

R. Ruimerman. Modeling and Remodeling in Bone

Tissue. Faculty of Biomedical Engineering, TU/e.

2005-02

C.N. Chong. Experiments in Rights Control — Ex-

pression and Enforcement. Faculty of Electrical En-

gineering, Mathematics & Computer Science, UT.

2005-03

H. Gao. Design and Veriﬁcation of Lock-free Parallel

Algorithms. Faculty of Mathematics and Computing

Sciences, RUG. 2005-04

H.M.A. van Beek. Speciﬁcation and Analysis of

Internet Applications. Faculty of Mathematics and

Computer Science, TU/e. 2005-05

M.T. Ionita. Scenario-Based System Architecting —

A Systematic Approach to Developing Future-Proof

System Architectures. Faculty of Mathematics and

Computing Sciences, TU/e. 2005-06

G. Lenzini. Integration of Analysis Techniques in

Security and Fault-Tolerance. Faculty of Electrical

Engineering, Mathematics & Computer Science, UT.

2005-07

I. Kurtev. Adaptability of Model Transformations.

Faculty of Electrical Engineering, Mathematics &

Computer Science, UT. 2005-08

T. Wolle. Computational Aspects of Treewidth —

Lower Bounds and Network Reliability. Faculty of

Science, UU. 2005-09

O. Tveretina. Decision Procedures for Equality

Logic with Uninterpreted Functions. Faculty of Math-

ematics and Computer Science, TU/e. 2005-10

A.M.L. Liekens. Evolution of Finite Populations in

Dynamic Environments. Faculty of Biomedical Engi-

neering, TU/e. 2005-11

J. Eggermont. Data Mining using Genetic Program-

ming: Classiﬁcation and Symbolic Regression. Fac-

ulty of Mathematics and Natural Sciences, UL. 2005-

B.J. Heeren. Top Quality Type Error Messages. Fac-

ulty of Science, UU. 2005-13

G.F. Frehse. Compositional Veriﬁcation of Hybrid

Systems using Simulation Relations. Faculty of Sci-

ence, Mathematics and Computer Science, RU. 2005-

M.R. Mousavi. Structuring Structural Operational

Semantics. Faculty of Mathematics and Computer

Science, TU/e. 2005-15

A. Sokolova. Coalgebraic Analysis of Probabilistic

Systems. Faculty of Mathematics and Computer Sci-

ence, TU/e. 2005-16

T. Gelsema. Effective Models for the Structure of

pi-Calculus Processes with Replication. Faculty of

Mathematics and Natural Sciences, UL. 2005-17

P. Zoeteweij. Composing Constraint Solvers. Faculty

of Natural Sciences, Mathematics, and Computer Sci-

ence, UvA. 2005-18

J.J. Vinju. Analysis and Transformation of Source

Code by Parsing and Rewriting. Faculty of Natural

Sciences, Mathematics, and Computer Science, UvA.

2005-19

M. Valero Espada. Modal Abstraction and Replica-

tion of Processes with Data. Faculty of Sciences, Di-

vision of Mathematics and Computer Science, VUA.

2005-20

A. Dijkstra. Stepping through Haskell. Faculty of

Science, UU. 2005-21

Y.W. Law. Key management and link-layer security

of wireless sensor networks: energy-efﬁcient attack

and defense. Faculty of Electrical Engineering, Math-

ematics & Computer Science, UT. 2005-22

E. Dolstra. The Purely Functional Software Deploy-

ment Model. Faculty of Science, UU. 2006-01