toraritte.github.io

The original builtins.fetchGit docs; see also comment in Nix issue #5128.

Quick notes beforehand:

improvement ideas:

…and questions:


builtins.fetchGit referenceToGitRepo -> storeResult

Fetch a Git repo.

0. Pseudo Types

referenceToGitRepo = URL | path | gitArgs

URL :: string = httpURL | httpsURL | ftpURL | fileURL
  Supported code hosting services: GitHub, GitLab, SourceHut.

httpURL = string
  Needs to conform to the http:// URI scheme (see RFC 9110, section 4.2).

httpsURL = string
  Needs to conform to the https:// URI scheme (see RFC 9110, section 4.2).

ftpURL = string
  Needs to conform to the ftp:// URI scheme (see RFC 1738, section 3.2).

webLikeURL :: string = httpURL httpsURL ftpURL

fileURL :: string = "file://" + fileURLPathPart
fileURLPathPart = string
  fileURLPathPart is a shorthand for fileURL (i.e., it will be prefixed with "file://" during evaluation) therefore both need to conform to the file:// URI scheme (see the path syntax of RFC 8089).

path = Nix path | fileURLPathPart

gitArgs :: attribute set =
  { url :: (URL | path);
    [ name :: string ? "source" ];
    [ ref :: gitReference ? "HEAD" ];
    [ rev :: gitFullCommitHash ? <ref dereferenced> ];
    [ submodules :: boolean ? false ];
    [ shallow :: boolean ? false ];
    [ allRefs :: boolean ? false ];
  }

webLikeGitArgs :: attribute set =   (Erlang-y)  gitArgs#{ url :: webLikeURL; }   (Haskell-y) gitArgs { url :: webLikeURL; }

pathLikeGitArgs :: attribute set =   (Erlang-y)  gitArgs#{ url :: (path | fileURL); }   (Haskell-y) gitArgs#{ url :: (path | fileURL); }

webLike = webLikeURL | webLikeGitArgs   Argument that is a URL or has a URL member conforming to the http://, https://, and ftp:// URI schemes.

pathLike = path | fileURL | pathLikeGitArgs   Argument that resolves or has a member that resolves to a file system path.

gitReference = string
  Needs to be valid Git reference.

gitFullCommitHash = string
  Has to be full SHA-1 (for now object name (40-byte hexadecimal string) that refers to an existing commit in the repo.

storeResult :: attribute set =
  { lastModified :: ?;
    lastModifiedDate :: ?;
    narHash :: ?;
    outPath :: nixStorePath;
    rev :: gitFullCommitHash;
    revCount :: ?;
    shortRev :: ?;
    submodules :: boolean;
  }

1. Behaviour

builtins.fetchGit behaves differently when called with pathLike or webLike arguments.

1.1 “Web-like” semantics

These sections describe the behaviour of builtins.fetchGit when called with webLike arguments:

1.1.1 webLikeURL type argument

NOTE

The file:// URI scheme is omitted on purpose, and is discussed in section 1.2 “Path-like” semantics.

Table 1.1.1-1 builtins.fetchGit string
String format Outcome
webLikeURL httpURL "http://..." The latest commit (or HEAD)
of the repo's default branch
(typically called main or
master) will be fetched.
httpsURL "https://..."
ftpURL "ftp://..."

HTTPS examples with the supported code hosting sites:

  builtins.fetchGit "https://github.com/NixOS/nix"
  builtins.fetchGit "https://git.sr.ht/~rycee/configurations"
  builtins.fetchGit "https://gitlab.com/rycee/home-manager"

1.1.2 webLikeGitArgs type argument

NOTE

gitArgs attributes rev and ref will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.

Table 1.1.2-1 builtins.fetchGit attribute set
gitArgs
attributes
Outcome Example argument Example resolved to full webLikeGitArgs attribute set
url
attribute

(mandatory)
rev
attribute

   (optional)   
ref
attribute

   (optional)   
webLikeURL omitted1.1.2-1 omitted
(or default value of HEAD used)
Same as
builtins.fetchGit webLikeURL
(see Table 1.1.1-1 above)
{ url = "https://github.com/nixos/nix"; }
{
  url = "https://github.com/NixOS/nix";
  name = "source";
  ref = "HEAD";
  rev = "<SHA-1 commit hash of HEAD>"
  submodules = false;
  shallow = false;
  allRefs = false;
}
present ignored1.1.2-2 Fetch repo at rev commit
{ url = "https://github.com/nixos/nix";
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
}
{ url = "https://github.com/nixos/nix";
  name = "source";
  ref = ""
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
  submodules = false;
  shallow = false;
  allRefs = false;
}
omitted1.1.2-1 present Fetch repo at ref branch / tag
{ url = "https://github.com/nixos/nix";
  ref = "refs/tags/2.10.3";
}
{ url = "https://github.com/nixos/nix";
  name = "source";
  ref = "refs/tags/2.10.3";
  rev = "309c2e27542818b74219d6825e322b8965c7ad69";
  submodules = false;
  shallow = false;
  allRefs = false;
}

[1.1.2-1]: See section 3.3 rev

[1.1.2-2]: See section 3.4 ref

1.2 “Path-like” semantics

Calls with pathLike arguments attempt to fetch a repo in a directory on a local or remote file system. The target repo may be a project under active development so their status and state may need to be determined before trying to copy the repo to the Nix store.

1.2.1 Git repository characteristics

That is, characteristics that builtins.fetchGit cares about.

1.2.1.1 Status

The status of a Git repo is

1.2.1.2 State

The state of a Git repo is the specific commit where the HEAD reference points to (directly or indirectly) at the moment when the repo is fetched.

Directly, if the repo is in a “detached HEAD” state, and indirectly when the commit is also the target of other references as shown on the figure below.

Visualized Git repo showing two branches and HEAD points to a commit that is tagged and is also the head of a branch.
1.2.1.2-1. State of a Git repo

LEGEND: orange label = branch, blue label = tag

1.2.2. Argument of type Nix path, fileURL, or fileURLPathPart

</tbody> </table> In fact, the 3 "STATE" rows could easily be collapsed into one as Git branches and tags are only labels to a Git object and what matters to `fetchGit` is the specific commit at the end of the de-reference process. Example calls: + via [Nix path]():\ `builtins.fetchGit ~/clones/nix` + via **fileURL**:\ `builtins.fetchGit "file:///home/nix_user/clones/nix"` + via **fileURLPathPart**:\ `builtins.fetchGit "/home/nix_user/clones/nix"` #### 1.2.3 `pathLikeGitArgs` type argument This means one of the following: + via { `url` :: [Nix path]() } `builtins.fetchGit { url = ~/clones/nix; ... }` + via { `url` :: **fileURLPathPart** } `builtins.fetchGit { url = "/home/nix_user/clones/nix"; ... }` + via { `url` :: **fileURL** } `builtins.fetchGit { url = "file:///home/nix_user/clones/nix"; ... }` The following table takes advantage of the fact that [state]() is simply determined by the current value of the [HEAD reference][HEAD]: > NOTE > > [`gitArgs`]() attributes [`rev`]() and [`ref`]() will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.
Table 1.2.2-1
STATUS De-reference process
dirty clean
STATE on BRANCH Copy directory contents verbatim Fetch repo at HEAD of BRANCH HEAD -> refs/heads/BRANCH -> <SHA-1 commit hash>
at TAG Copy directory contents verbatim Fetch repo at TAG HEAD -> refs/tags/TAG -> <SHA-1 commit hash>
detached HEAD Copy directory contents verbatim Fetch repo at HEAD HEAD -> <SHA-1 commit hash>
</tr> </thead> </table> \[1.2.3-1]: See section [3.3 `rev`]() \[1.2.3-2]: When `ref` or `rev` is present, the intention is probably to fetch a known past state from the repo's history, thus the most recent changes are not relevant (neither the status of the repo). \[1.2.3-3]: See section [3.4 `ref`]() As a corollary, here are some tips: + If you need to fetch a local repo, calling `builtins.fetchGit` with `ref` (branch or tag) or `rev` (commit hash) will make sure that a repo is fetched with a predictable content, ignoring any changes that may have been made since you last touched it. + If you are packaging a project under active development and want to test changes without commiting, you'll probably want to call `builtins.fetchGit` with `{ url = ...; }` or the specified in [1.2.2. Argument of type `Nix path`, `fileURL`, or `fileURLPathPart`](). ## 2. **gitArgs** attributes Reminder: **gitArgs** :: [attribute set]() =\   { `url` :: (**URL** | **path**);\     [ `name` :: [string]() ? `"source"` ];\     [ `ref` :: **gitReference** ? `"HEAD"` ];\     [ `rev` :: **gitFullCommitHash** ? <`ref` dereferenced> ];\     [ `submodules` :: [boolean]() ? `false` ];\     [ `shallow` :: [boolean]() ? `false` ];\     [ `allRefs` :: [boolean]() ? `false` ];\   } ### 3.1 `url` (mandatory)
Table 1.2.3-1.
    STATUS</a>    </th> gitArgs
attributes
        Outcome         Example argument
url
attribute

         (mandatory)         
rev
attribute

   (optional)   
ref
attribute

   (optional)   
dirty   Nix path
| fileURLPathPart
| fileURL

(See examples at the top of this section.)
omitted1.2.3-1 omitted
(or default value of HEAD used)
Copy directory contents verbatim
{ url = "https://github.com/nixos/nix"; }
clean omitted1.2.3-1 omitted
(or default value of HEAD used)
Fetch repo at HEAD
ignored1.2.3-2 present ignored1.2.3-3 Ignore changes (if any) and fetch repo at rev commit
{ url = "https://github.com/nixos/nix";
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
}
ignored1.2.3-2 omitted1.2.3-1 present Ignore changes (if any) and fetch repo at ref tag / branch
{ url = "https://github.com/nixos/nix";
  ref = "refs/tags/2.10.3";
}
</table> ### 3.2 `name` (optional)
Description This attribute is covered extensively in section 1. Behaviour (specifically, in sections 1.1.2 webLikeGitArgs type argument and 1.2.3 pathLikeGitArgs type argument).
Type string
Default value none
</table> Examples: ```text nix-repl> builtins.fetchGit { url = ./.; } { ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-source"; ...; } nix-repl> builtins.fetchGit { url = ./.; name = "miez"; } { ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-miez"; ...; } ``` ### 3.3 `rev` (optional)
Description The name part of the Nix store path where the Git repo's content will be copied to.
Type string
Default value "source"
</table> Sections [1.1.2 **webLikeGitArgs** type argument]() and [1.2.3 **pathLikeGitArgs** type argument]()) in [1. Behaviour]() describe the prevailing behaviour `builtins.fetchgit` when the `rev` attribute is used. > NOTE > > Specifying the `rev` attribute will render the `ref` attribute irrelevant no matter if it is included in the input attribute set or not. See next section for more. ### 3.4 `ref` (optional)
Description The rev attribute is used to refer to a specific commit by the full SHA-1 Git object name (40-byte hexadecimal string) - or as it is more colloquially called, the commit hash.
Type string
Additional
constraints
40-byte hexadecimal SHA-1 string
Default value The dereferenced value of the Git reference held by the ref attribute. (See next section.)
</table> > WARNING > > By default, the `ref` value is prefixed with `refs/heads/`. After Nix 2.3.0, it will not be prefixed with `refs/heads/` if `ref` starts with `refs/`. #### 3.3.1 [`ref` attribute]() ignored when the `rev` attribute is provided The `rev` attribute (i.e., the commit hash) has higher specificity; a `ref` reference will need to be resolved and its value may change with time, but a commit hash will always point to the same exact commit object and thus to the same state of the the repo during the lifetime of a Git repo. (TODO: right?) ## 4. Examples > TODO: Re-work original examples --- TODO/NOTE: Stopping here for now to wait for the resolution of [comment on Nix issue #5128](https://github.com/NixOS/nix/issues/5128#issuecomment-1198254451) Here are some examples of how to use `builtins.fetchGit`. - To fetch a private repository over SSH: ```nix builtins.fetchGit { url = "git@github.com:my-secret/repository.git"; ref = "master"; rev = "adab8b916a45068c044658c4158d81878f9ed1c3"; } ``` - To fetch an arbitrary reference: ```nix builtins.fetchGit { url = "https://github.com/NixOS/nix.git"; ref = "refs/heads/0.5-release"; } ``` - If the revision you're looking for is in the default branch of the git repository you don't strictly need to specify the branch name in the `ref` attribute. However, if the revision you're looking for is in a future branch for the non-default branch you will need to specify the the `ref` attribute as well. ```nix builtins.fetchGit { url = "https://github.com/nixos/nix.git"; rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452"; ref = "1.11-maintenance"; } ``` > **Note** > > It is nice to always specify the branch which a revision > belongs to. Without the branch being specified, the fetcher > might fail if the default branch changes. Additionally, it can > be confusing to try a commit from a non-default branch and see > the fetch fail. If the branch is specified the fault is much > more obvious. - If the revision you're looking for is in the default branch of the git repository you may omit the `ref` attribute. ```nix builtins.fetchGit { url = "https://github.com/nixos/nix.git"; rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452"; } ``` - To fetch a specific tag: ```nix builtins.fetchGit { url = "https://github.com/nixos/nix.git"; ref = "refs/tags/1.9"; } ``` - To fetch the latest version of a remote branch: ```nix builtins.fetchGit { url = "ssh://git@github.com/nixos/nix.git"; ref = "master"; } ``` > **Note** > > Nix will refetch the branch in accordance with > the option `tarball-ttl`. > **Note** > > This behavior is disabled in *Pure evaluation mode*. TODO: move this to the end \[2.2-1]: See [`src/libfetchers/git.cc`#`445`](https://github.com/NixOS/nix/blob/86fcd4f6923b3a8ccca261596b9db0d8c0a873ec/src/libfetchers/git.cc#L445-L452). \[2.2-2]: See [`src/libfetchers/git.cc`#`fetch()`](https://github.com/NixOS/nix/blob/86fcd4f6923b3a8ccca261596b9db0d8c0a873ec/src/libfetchers/git.cc#L393), line [556](https://github.com/NixOS/nix/blob/86fcd4f6923b3a8ccca261596b9db0d8c0a873ec/src/libfetchers/git.cc#L556). + `submodules` (optional) + `shallow` (optional) + `allRef` (optional) - `ref` (optional)\ The git ref to look for the requested revision under. This is often a branch or tag name. Defaults to `HEAD`. - `rev` (optional)\ The [Git revision](https://git-scm.com/docs/git-rev-parse#_specifying_revisions) to fetch.\ *Default value*: if the `ref` attribute (see above) is specified, - `submodules` (optional)\ A Boolean parameter that specifies whether submodules should be checked out. Defaults to `false`. - `shallow` (optional)\ A Boolean parameter that specifies whether fetching a shallow clone is allowed. Defaults to `false`. NOTE [`git clone --depth=1 ` creates a shallow clone](https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/#:~:text=git%20clone%20%2D%2Ddepth%3D1%20%3Curl%3E%C2%A0creates%20a%C2%A0shallow%20clone) QUESTION How does this affect rev and ref? this is what i think + "no ref, no rev": ref=HEAD rev=resolve(HEAD) +++++++++clean repo: only diff is revCount The store hash stayed the same! Probably because .git is never copied - but then what is the point of this switch because then it will always be shallow. https://stackoverflow.com/questions/11497457/git-clone-without-git-directory QUESTION also, if .git is never copied, what is the point of revcount? NOTE should be called commitCount nix-repl> builtins.fetchGit { url = ~/shed/my-project; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 5; shortRev = "5f45e9c"; submodules = false; } nix-repl> builtins.fetchGit { url = ~/shed/my-project; shallow = true; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 0; shortRev = "5f45e9c"; submodules = false; } --------- +++++++++dirty repo: completely identical outputs nix-repl> builtins.fetchGit { url = ./.; } warning: Git tree '/home/toraritte/clones/nix' is dirty { lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; } nix-repl> builtins.fetchGit { url = ./.; shallow = true; } warning: Git tree '/home/toraritte/clones/nix' is dirty { lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; } +++++++++stopping this experiment here because adding rev and/or ref doesn't make a difference: only revCount will differ - `allRefs` (optional)\ Whether to fetch all refs of the repository. With this argument being true, it's possible to load a `rev` from *any* `ref` (by default only `rev`s from the specified `ref` are supported). NOTE allRefs also seems kind of pointless nix-repl> builtins.fetchGit { url = ~/shed/my-project; allRefs = true; ref = "main";} { lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source"; rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules = false; } nix-repl> builtins.fetchGit { url = ~/shed/my-project; ref = "main";} { lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source"; rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules = false; } nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; } nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRef = true; } error: unsupported Git input attribute 'allRef' nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRefs = true; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; } ##################### Only 2 uses of `allRefs` in the entirety of Nixpkgs: TODO i don't think it matters if used or not; except if `builtins.fetchGit` used "improperly" -> see issue below 0 [07:38:22] ag 'allRefs' . pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js 54: allRefs = true; pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix 179: allRefs = true; ##################### https://github.com/NixOS/nix/issues/5128 but the error makes sense as an SHA-1 hash **is not** a valid reference nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; } fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128 nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; allRefs = true; } warning: could not update mtime for file '/home/toraritte/.cache/nix/gitv3/0240dfgnkwmgqs7sma8rns 8wlwxiv40b1lddl2sg2i0hnw7ym5c0/refs/heads/db1442a0556c2b133627ffebf455a78a1ced64b9': No such file or directory { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; } nix-repl> nix-repl> nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128 nix-repl> builtins.fetchGit { rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; }
Description The ref attribute accepts a Git reference that is present in the target repo.
Type string
Additional
constraints
See Git reference syntax
Default value "HEAD"