OPAM 1.2: Repository Pinning

---

Most package managers support some pin functionality to ensure that a given package remains at a particular version without being upgraded. The stable OPAM 1.1 already supported this by allowing any existing package to be pinned to a target, which could be a specific released version, a local filesystem path, or a remote version-controlled repository.

However, the OPAM 1.1 pinning workflow only lets you pin packages that already exist in your OPAM repositories. To declare a new package, you had to go through creating a local repository, registering it in OPAM, and adding your package definition there. That workflow, while reasonably clear, required the user to know about the repository format and the configuration of an internal repository in OPAM before actually getting to writing a package. Besides, you were on your own for writing the package definition, and the edit-test loop wasn't as friendly as it could have been.

A natural, simpler workflow emerged from allowing users to pin new package names that don't yet exist in an OPAM repository:

1. choose a name for your new package
2. opam pin add in the development source tree
3. the package is created on-the-fly and registered locally.

To make it even easier, OPAM can now interactively help you write the package definition, and you can test your updates with a single command. This blog post explains this new OPAM 1.2 functionality in more detail; you may also want to check out the new Packaging tutorial relying on this workflow.

From source to package

For illustration purposes in this post I'll use a tiny tool that I wrote some time ago and never released: ocp-reloc. It's a simple binary that fixes up the headers of OCaml bytecode files to make them relocatable, which I'd like to release into the public OPAM repository.

"opam pin add"

The command opam pin add <name> <target> pins package <name> to <target>. We're interested in pinning the ocp-reloc package name to the project's source directory.

cd ocp-reloc
opam pin add ocp-reloc .

If ocp-reloc were an existing package, the metadata would be fetched from the package description in the OPAM repositories. Since the package doesn't yet exist, OPAM 1.2 will instead prompt for on-the-fly creation:

Package ocp-reloc does not exist, create as a NEW package ? [Y/n] y
ocp-reloc is now path-pinned to ~/src/ocp-reloc

NOTE: if you are using beta4, you may get a version-control-pin instead, because we added auto-detection of version-controlled repos. This turned out to be confusing (issue #1582), because your changes wouldn't be reflected until you commit, so this has been reverted in favor of a warning. Add the --kind path option to make sure that you get a path-pin.

OPAM Package Template

Now your package still needs some kind of definition for OPAM to acknowledge it; that's where templates kick in, the above triggering an editor with a pre-filled opam file that you just have to complete. This not only saves time in looking up the documentation, it also helps getting consistent package definitions, reduces errors, and promotes filling in optional but recommended fields (homepage, etc.).

opam-version: "1.2"
name: "ocp-reloc"
version: "0.1"
maintainer: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
authors: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
homepage: ""
bug-reports: ""
license: ""
build: [
  ["./configure" "--prefix=%{prefix}%"]
  [make]
]
install: [make "install"]
remove: ["ocamlfind" "remove" "ocp-reloc"]
depends: "ocamlfind" {build}

After adding some details (most importantly the dependencies and build instructions), I can just save and exit. Much like other system tools such as visudo, it checks for syntax errors immediately:

[ERROR] File "/home/lg/.opam/4.01.0/overlay/ocp-reloc/opam", line 13, character 35-36: '.' is not a valid token.
Errors in /home/lg/.opam/4.01.0/overlay/ocp-reloc/opam, retry editing ? [Y/n]

Installation

You probably want to try your brand new package right away, so OPAM's default action is to try and install it (unless you specified -n):

ocp-reloc needs to be installed.
The following actions will be performed:
 - install   cmdliner.0.9.5                        [required by ocp-reloc]
 - install   ocp-reloc.0.1*
=== 1 to install ===
Do you want to continue ? [Y/n]

I usually don't get it working the first time around, but opam pin edit ocp-reloc and opam install ocp-reloc -v can be used to edit and retry until it does.

Package Updates

How do you keep working on your project as you edit the source code, now that you are installing through OPAM? This is as simple as:

opam upgrade ocp-reloc

This will pick up changes from your source repository and reinstall any packages that are dependent on ocp-reloc as well, if any.

So far, we've been dealing with the metadata locally used by your OPAM installation, but you'll probably want to share this among developers of your project even if you're not releasing anything yet. OPAM takes care of this by prompting you to save the opam file back to your source tree, where you can commit it directly into your code repository.

cd ocp-reloc
git add opam
git commit -m 'Add OPAM metadata'
git push

Publishing your New Package

The above information is sufficient to use OPAM locally to integrate new code into an OPAM installation. Let's look at how other developers can share this metadata.

Picking up your development package

If another developer wants to pick up ocp-reloc, they can directly use your existing metadata by cloning a copy of your repository and issuing their own pin.

git clone git://github.com/OCamlPro/ocp-reloc.git
opam pin add ocp-reloc/

Even specifying the package name is optional since this is documented in ocp-reloc/opam. They can start hacking, and if needed use opam pin edit to amend the opam file too. No need for a repository, no need to share anything more than a versioned opam file within your project.

Cloning already existing packages

We have been focusing on an unreleased package, but the same functionality is also of great help in handling existing packages, whether you need to quickly hack into them or are just curious. Let's consider how to modify the omd Markdown library.

opam source omd --pin
cd omd.0.9.7
...patch...
opam upgrade omd

The new opam source command will clone the source code of the library you specify, and the --pin option will also pin it locally to ensure it is used in preference to all other versions. This will also take care of recompiling any installed packages that are dependent on omd using your patched version so that you notice any issues right away.

There's a new OPAM field available in 1.2 called dev-repo. If you specify this in your metadata, you can directly pin to the upstream repository via opam source --dev-repo --pin.

If the upstream repository for the package contains an opam file, that file will be picked up in preference to the one from the OPAM repository as soon as you pin the package. The idea is to have:

• a development opam file that is versioned along with your source code (and thus accurately tracks the latest dependencies for your package).
• a release opam file that is published on the OPAM repository and can be updated independently without making a new release of the source code.

How to get from the former to the latter will be the subject of another post! In the meantime, all users of the beta are welcome to share their experience and thoughts on the new workflow on the bug tracker.

---

Au sujet d'OCamlPro :

OCamlPro développe des applications à haute valeur ajoutée depuis plus de 10 ans, en utilisant les langages les plus avancés, tels que OCaml et Rust, visant aussi bien rapidité de développement que robustesse, et en ciblant les domaines les plus exigeants (méthodes formelles, cybersécurité, systèmes distribués/blockchain, conception de DSLs). Fort de plus de 20 ingénieurs R&D, avec une expertise unique sur les langages de programmation, aussi bien théorique (plus de 80% de nos ingénieurs ont une thèse en informatique) que pratique (participation active au développement de plusieurs compilateurs open-source, prototypage de la blockchain Tezos, etc.), diversifiée (OCaml, Rust, Cobol, Python, Scilab, C/C++, etc.) et appliquée à de multiples domaines. Nous dispensons également des [formations sur mesure certifiées Qualiopi sur OCaml, Rust, et les méthodes formelles] (https://training.ocamlpro.com/) Pour nous contacter : contact@ocamlpro.com.