Opam 101: The First Steps

Authors: Dario Pinto, Raja Boujbel

Date: 2024-01-23

Category: Trainings

Tags: tutorial, on-boarding, opam, beginner, opam10x

Welcome, dear reader, to a new series of blog posts!

This series will be about everything opam. Each article will cover a specific aspect of the package manager, and make sure to dissipate any confusion or misunderstandings on this keystone of the OCaml distribution!

Each technical article will be tailored for specific levels of engineering -- everyone, be they beginners, intermediate or advanced in the OCaml Arts will find answers to some questions about opam right here.

Checkout each article's tags to get an idea of the entry level required for the smoothest read possible!

New to the expansive OCaml sphere? As said on the official opam website, opam has been a game changer for the OCaml distribution, since it first saw the light of day here, almost a decade ago.

Walking the path of opam, treading on solid ground

We are aware that it can be quite a daunting task to get on-board with the OCaml distribution. Be it because of its decentralised characteristics: plethora of different tools, a variety of sometimes clashing modi operandi and practices, usually poorly documented edge use-cases, the variety of ways to go about having a working environment or many a different reason...

We have been thinking about making it easier for everyone, even the more confirmed Cameleers, by releasing a set of blogposts progressively detailing the depths at which opam can go.

Be sure to read these articles from the start if you are new to the beautiful world of OCaml and, if you are already familiar, use it as a trust-worthy documentation on speed-dial... You never know when you will have to setup an opam installation while off-the-grid, do you ?

Are you ready to dive in ?

First step: installing opam

First, let's talk about installing opam.

DISCLAIMER: In this tutorial, we will only be addressing a fresh install of opam on Linux and Mac. For more information about a Windows installation, stay tuned with this blog!

One would expect to have to interact with the package manager of one's favourite distribution in order to install opam, and, to some extent, one would be correct. However, we cannot guarantee that the version of opam you have at your disposal through these means is indeed the one expected by this tutorial, and every subsequent one for that matter.

You can check that here, make sure the version available to you is 2.1.5 or above.

Thus, in order for us to guarantee that we are on the same version, we will use the installation method found here and add an option to specify the version of opam we will be working with from now on.

Note that if you don't add the --version 2.1.5 option to the following command line, the script will download and install the latest opam release. The cli of opam is made to remain consistent between versions so, unless you have a very old version, or if you read this article in the very distant future, you should not have problems by not using the exact same version as we do. For the sake of consistency though, I will use this specific version.

$ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.5"

This script will download the necessary binaries for a proper installation of opam. Once done, you can move on to the nitty gritty of having a working opam environment with opam init.

Second step: initialisation

The first command to launch, after the initial opam binaries have been downloaded and opam has been installed on your system, is opam init.

This is when you step into the OCaml distribution for the first time.

opam init does several crucial things for you when you launch it, and the rest of this article will detail what exactly these crucial things are and what they mean:

• it checks some required and recommended tools;
• it syncs with the official OCaml opam-repository, which you can find here;
• it sets up the opam environment in your *rc files;
• it creates a switch and installs an ocaml-compiler for you;

Lets take a step-by-step look at the output of that command:

$ opam init
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs.
Perfect!

<><> Fetching repository information ><><><><><><><><><><><><><><><><>
[default] Initialised

<><> Required setup - please read <><><><><><><><><><><><><><><><><><>

  In normal operation, opam only alters files within ~/.opam.

  However, to best integrate with your system, some environment
  variables should be set. If you allow it to, this initialisation
  step will update your bash configuration by adding the following
  line to ~/.profile:

    test -r ~/.opam/opam-init/init.sh && . ~/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

  Otherwise, every time you want to access your opam installation,
  you will need to run:

    eval $(opam env)

  You can always re-run this setup with 'opam init' later.

Do you want opam to modify ~/.profile? [N/y/f]
(default is 'no', use 'f' to choose a different file) y

User configuration:
  Updating ~/.profile.
[NOTE] Make sure that ~/.profile is well sourced in your ~/.bashrc.


<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

The main result for an opam init call is to setup what is called your opam root. It does so by creating a ~/.opam directory to operate inside of. opam modifies and writes in this location only as a default.

First, opam checks that there is at least one required tool for syncing to the opam-repository. Then it checks what backends are available in your system. Here all are available: rsync, git, mercurial, and darcs. They will be used to sync repositories or packages.

$ opam init
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs.
Perfect!

Then, opam fetches the default opam repository: opam.ocaml.org.

<><> Fetching repository information ><><><><><><><><><><><><><><><><>
[default] Initialised

Secondly, opam requires your input in order to configure your shell for the smoothest possible experience. For more details about the opam environment, refer to the next section.

Something interesting to remember for later is, in the excerpt below, we grant opam with the permission to edit the ~/.profile file. This part of the Quality of Life features for an everyday use an opam environment and we will detail how so below.

<><> Required setup - please read <><><><><><><><><><><><><><><><><><>

  In normal operation, opam only alters files within ~/.opam.

  However, to best integrate with your system, some environment
  variables should be set. If you allow it to, this initialisation
  step will update your bash configuration by adding the following
  line to ~/.profile:

    test -r ~/.opam/opam-init/init.sh && . ~/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

  Otherwise, every time you want to access your opam installation,
  you will need to run:

    eval $(opam env)

  You can always re-run this setup with 'opam init' later.

Do you want opam to modify ~/.profile? [N/y/f]
(default is 'no', use 'f' to choose a different file) y

User configuration:
  Updating ~/.profile.
[NOTE] Make sure that ~/.profile is well sourced in your ~/.bashrc.

The next action is the installation of your very first switch alongside a version of the OCaml compiler, by default a compiler >= 4.05.0 to be exact.

For more information about what is a switch be sure to read the rest of the article.

<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

Great! So let's focus on the actions performed by the opam init call!

Acclimating to the environment

Well, as said previously, the first action was to setup an opam root in your $HOME directory, (i.e: ~/.opam). This is where opam will operate. opam will never modify other locations in your filesystem without notifying you first.

An opam root is made to resemble a linux-like architecture. You will find inside it directories such as /usr, /etc, /bin and so on. This is by default where opam will store everything relative to your system-wide installation. Config files, packages and their configurations, and also binaries.

This leads us to the need for an eval $(opam env) call.

Indeed, in order to make your binaries and such accessible as system-wide tools, you need to update all the relevant environment variables (PATH, MANPATH, etc.) with all the locations for all of your everyday OCaml tools.

To see what variables are exported when evaluating the opam env command, you can check the following codeblock:

$ opam env
OPAM_SWITCH_PREFIX='~/.opam/default'; export OPAM_SWITCH_PREFIX;
CAML_LD_LIBRARY_PATH='~/.opam/default/lib/stublibs:~/.opam/default/lib/ocaml/stublibs:~/.opam/default/lib/ocaml'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='~/.opam/default/lib/toplevel'; export OCAML_TOPLEVEL_PATH;
MANPATH=':~/.opam/default/man'; export MANPATH;
PATH='~/.opam/default/bin:$PATH'; export PATH;

Remember when we granted opam init with the permission to edit the ~/.profile file, earlier in this tutorial ? That comes in handy now: it keeps us from having to use the eval $(opam env) more than necessary.

Indeed, you would otherwise have to call it every time you launch a new shell among other things. What it does instead is adding hook at prompt level that keeps opam environment synced, updating it every time the user presses Enter. Very handy indeed.

Switches, tailoring your workspace to your vision

The second task accomplished by opam init was installing the first switch inside your fresh installation.

A switch is one of opam's core operational concepts, it's definition can vary depending on your exact use-case but in the case of OCaml, a switch is a named pair:

• an arbitrary version of the OCaml compiler
• a list of packages available for that specific version of the compiler.

In our example, we see that the only packages installed in the process were the dependencies for the OCaml compiler version 5.1.0 inside the switch named default.

<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

You can create an arbitrary amount of parallel switches in opam. This allows users to manage parallel, independent OCaml environments for their developments.

There are two types of switches:

• global switches have their packages, binaries and tools available anywhere on your computer. They are useful when you consider a given switch to be your default and most adequate environment for your everyday use of opam and OCaml.
• local switches on the other hand are only available in a given directory. Their packages and binaries are local to that specific directory. This allows users to make specific projects have their own self-contained working environments. The local switch is automatically selected by opam as the current one when you are located inside the appropriate directory. More details on local switches below.

The default behaviour for opam when creating a switch at init-time is to make it global and name it default.

$ opam switch show
default
$ opam switch
#  switch   compiler     description
→  default  ocaml.5.1.0  default

Now that you have a general understanding of what exactly is a switch and how it is used, let's get into how you can go about manually creating your first switch.

Creating a global switch

NB: Remember that opam's command-line interface is beginner friendly. You can, at any point of your exploration, use the --help option to have every command and subcommand explained. You may also checkout the opam cheat-sheet that was released a while ago and might still hold some precious insights on opam's cli.

So how does one create a switch ? The short answer is bafflingly straightforward:

# Installs a switch named "my-switch" based OCaml compiler version > 4.05.0
# Here 4.05 is the default lower compiler version opam selects when unspecified
$ opam switch create my-switch

Easy, right? Now let's imagine that you would like to specify a later version of the OCaml compiler. The first thing you would want to know is which version are available for you to specify, and you can use opam list for that.

Other commands can be used to the same effect but we prefer introducing you to this specific one as it may also be used for any other package available via opam.

So, as for any other package than ocaml itself, opam list will give you all available versions of that package for your currently active switch. Since we don't yet have an OCaml compiler installed, it will list all of them so that we may pick and choose our favourite to use for the switch we are making.

$ opam list ocaml
# Packages matching: name-match(ocaml) & (installed | available)
# Package    # Installed # Synopsis
ocaml.3.07   --          The OCaml compiler (virtual package)
ocaml.3.07+1 --          The OCaml compiler (virtual package)
ocaml.3.07+2 --          The OCaml compiler (virtual package)
ocaml.3.08.0 --          The OCaml compiler (virtual package)
(...)
ocaml.4.13.1 --          The OCaml compiler (virtual package)
ocaml.4.13.2 --          The OCaml compiler (virtual package)
(...)
ocaml.5.2.0  --          The OCaml compiler (virtual package)

Let's use it for a switch:

# Installs a switch named "my-switch" based OCaml compiler version = 4.13.1
$ opam switch create my-switch ocaml.4.13.1

That's it, for the first time, you have manually created your own global switch tailored to your specific needs, congratulations!

NB: Creating a switch can be a fairly time-consuming task depending on whether or not the compiler version you have queried from opam is already installed on your machine, typically in a previously created switch. Every time you ask opam to install a version of the compiler, it will first scour your installation for a locally available version of that compiler to save you the time necessary for downloading, compiling and installing a brand new one.

Now, onto local switches.

Creating a local switch

As said previously, the use of a local switch is to constrain a specific OCaml environment to a specific location on your workstation.

Let's imagine you are about to start a new development called my-project.

While preparing all necessary pre-requisites for it, you notice something problematic: your global default switch is drastically incompatible with the dependencies of your project. In this imaginary situation, you have a default global switch that is useful for most of your other tasks but now have only one project that differs from your usual usage of OCaml.

To remedy this situation, you could go about creating another global switch for your upcoming dev requirements on my-project and proceed to install all relevant packages and remake a full switch from scratch for that specific project. However this would require you to always keep track of which one is your currently active switch, while possibly having to regularly oscillate between your global default switch and your alternative global my-project switch which you could understandably find to be suboptimal and tedious to incorporate to your workflow on the long run.

That's when local switches come in handy because they allow you to leave the rest of your OCaml dev environment unaffected by whatever out-of-bounds or specific workload you're undertaking. Additionally, the fact that opam automatically selects your local switch as your current active one as soon as you step inside the relevant directory makes the developers's context switch seemless.

Let's examine how you can create such a switch:

# Hop inside the directory of your project
$ cd my-project
# We consider your project already has an opam file describing only
# its main dependency: ocaml.4.14.1
$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.
$ opam switch
#  switch                   compiler      description
→  /home/ocp/my-project     ocaml.4.14.1  /home/ocp/my-project
   default                  ocaml.5.1.0   default
   my-switch                ocaml.4.13.1  my-switch

[NOTE] Current switch has been selected based on the current directory.
       The current global system switch is default.

Here it is, you can now hop into your local switch /home/ocp/my-project whenever you have time to deviate from your global environment.

The official opam-repository, the safe for all your packages

Among all the things that opam init did when it was executed, there is still one detail we have yet to explain and that's the first action of the process: retrieving packages specification from the official OCaml opam-repository.

Explaining what exactly an opam-repository is requires the recipient to have a slightly deeper understanding of how opam works than the average reader this article was written for might have; so you will have to wait for us to go deeper into that subject in another blogpost when the time is ripe.

What we will do now though is explain what the official OCaml opam-repository is and how it relates to our use of opam in this blog post.

The Official OCaml opam-repository is an open-source project where all released software of the OCaml distributions are referenced. It holds different compilers, basic tools, thousands of libraries, approximatively 4500 packages in total as of today and is configured to be the default repository for opam to sync to. You may add your own repositories for your own use of opam, but again, that's a subject for another time.

In case the repository itself is not what you are looking for, know that all packages available throughout the entire OCaml distribution may be browsed directly on ocaml.org.

It is essentially a collection of opam packages described in opam file format. Checkout the manual for more information about the opam file format.

A short explanation for it is that an opam package file holds every information necessary for opam to operate and provide. The file lists all of the packages direct dependencies, where to find its source code, the names and emails of maintainers and authors, different checksums for each archive release and the list goes on.

Here's a quick example for you to have an idea of what it looks like:

opam-version: "2.0"
synopsis: "OCaml bindings to Zulip API"
maintainer: ["Dario Pinto <dario.pinto@ocamlpro.com>"]
authors: ["Mohamed Hernouf <mohamed.hernouf@ocamlpro.com>"]
license: "LGPL-2.1-only WITH OCaml-LGPL-linking-exception"
homepage: "https://github.com/OCamlPro/ozulip"
doc: "https://ocamlpro.github.io/ozulip"
bug-reports: "https://github.com/OCamlPro/ozulip/issues"
dev-repo: "git+https://github.com/OCamlPro/ozulip.git"
tags: ["zulip" "bindings" "api"]
depends: [
  "ocaml" {>= "4.10"}
  "dune" {>= "2.0"}
  "ez_api" {>= "2.0.0"}
  "re"
  "base64"
  "json-data-encoding" {>= "1.0.0"}
  "logs"
  "lwt" {>= "5.4.0"}
  "ez_file" {>= "0.3.0"}
  "cohttp-lwt-unix"
  "yojson"
  "logs"
]
build: [ "dune" "build" "-p" name "-j" jobs "@install" ]
url {
  src: "https://github.com/OCamlPro/ozulip/archive/refs/tags/0.1.tar.gz"
  checksum: [
    "md5=4173fefee440773dd0f8d7db5a2e01e5"
    "sha512=cb53870eb8d41f53cf6de636d060fe1eee6c39f7c812eacb803b33f9998242bfb12798d4922e7633aa3035cf2ab98018987b380fb3f380f80d7270e56359c5d8"
  ]
}

Okay so now, how do we go about populating a switch with packages and really get started?

Installing packages in your current switch

It's elementary. This simple command will do the trick of trying to install a package, and its dependencies, in your currently active switch.

$ opam install my-package

I say trying because opam will notify you if the current package version and its dependencies you are querying are or not compatible with the current state of your switch. It will also offer you solutions for the compatibility constraints between packages to be satisfiable: it may suggest to upgrade some of your packages, or even to remove them entirely.

The key thing about this process is that opam is designed to solve compatibility constraints in the global graph of dependencies that the OCaml packages form. This design is what makes opam the average Cameleer's best friend. It will highlight inconsistencies within dependencies, it will figure out a way for your specific query to be satisfiable somehow and save you a lot of headscratching, that is, if you are willing to accommodate a bit of getting-used to.

The next command allows you to uninstall a package from your currently active switch as well as the packages that depend on it:

$ opam remove my-package

And the two following will update the state of the repositories opam is synchronised with and upgrade the packages installed while always keeping package compatibility in mind.

$ opam update
$ opam upgrade

Conclusion

Here it is, you should now be knowledgeable enough about opam to jump right in the OCaml discovery!

Today we learned everything elementary about opam.

From installation, to initialisation and explanations about the core concepts of the opam environment, switches, packages and the Official OCaml opam-repository.

Be sure to stay tuned with our blog, the journey into the rabbit hole has only started and opam is a deep one indeed!

Thank you for reading,

From 2011, with love,

The OCamlPro Team

About OCamlPro:

OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.

• We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
• We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
• We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
• We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.

Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.