opam 2.1.0 is released!
Feedback on this post is welcomed on Discuss!
We are happy to announce the release of opam 2.1.0.
Many new features made it in (see the pre-release changelogs or release notes for the details), but here are a few highlights.
What's new in opam 2.1?
- Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
- Creation of lock files for reproducible installations (formerly the opam-lock plugin)
- Switch invariants, replacing the "base packages" in opam 2.0 and allowing for easier compiler upgrades
- Improved options configuration (see the new
option
and expandedvar
sub-commands) - CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
- opam root readability by newer and older versions, even if the format changed
- Performance improvements to opam-update, conflict messages, and many other areas
Seamless integration of System dependencies handling (a.k.a. "depexts")
opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:
- You no longer have to remember to run
opam depext
, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy asopam install .
, without having to do the dance ofopam pin add -n/depext/install
. Just one command now for the common case! - The solver is only called once, which both saves time and also stabilises the
behaviour of opam in cases where the solver result is not stable. It was
possible to get one package solution for the
opam depext
stage and a different solution for theopam install
stage, resulting in some depexts missing. - opam now has full knowledge of depexts, which means that packages can be
automatically selected based on whether a system package is already installed.
For example, if you have neither MariaDB nor MySQL dev libraries installed,
opam install mysql
will offer to installconf-mysql
andmysql
, but if you have the MariaDB dev libraries installed, opam will offer to installconf-mariadb
andmysql
.
Hint: You can set OPAMCONFIRMLEVEL=unsafe-yes
or
--confirm-level=unsafe-yes
to launch non interactive system package commands.
opam lock files and reproducibility
When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 19500 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.
While this works well for libraries, we need a different strategy for projects
that need to test and ship using a fixed set of dependencies. To satisfy this
use-case, opam 2.0.0 shipped with support for using project.opam.locked
files. These are normal opam files but with exact versions of dependencies. The
lock file can be used as simply as opam install . --locked
to have a
reproducible package installation.
With opam 2.1.0, the creation of lock files is also now integrated into the client:
opam lock
will create a.locked
file for your current switch and project, that you can check into the repository.opam switch create . --locked
can be used by users to reproduce your dependencies in a fresh switch.
This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).
Hint: You can export the full configuration of a switch with opam switch export
new options, --full
to have all packages metadata included, and
--freeze
to freeze all VCS to their current commit.
Switch invariants
In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam's solver. This was a much bigger concern early on in opam 2.0's development, but is less of a problem with the default mccs solver.
However, it's a problem for system compilers. opam would detect that your
system compiler version had changed, but be unable to upgrade the ocaml-system
package unless you went through a slightly convoluted process with
--unlock-base
.
In opam 2.1, base packages have been replaced by switch invariants. The switch
invariant is a package formula which must be satisfied on every upgrade and
install. All existing switches' base packages could just be expressed as
package1 & package2 & package3
etc. but opam 2.1 recognises many existing
patterns and simplifies them, so in most cases the invariant will be
"ocaml-base-compiler" {= "4.11.1"}
, etc. This means that opam switch create my_switch ocaml-system
now creates a switch invariant of "ocaml-system"
rather than a specific version of the ocaml-system
package. If your system
OCaml package is updated, opam upgrade
will seamlessly switch to the new
package.
This also allows you to have switches which automatically install new point releases of OCaml. For example:
opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop
Creates a switch with OCaml 4.11.0 (the --repos=
was just to select a version
of opam-repository from before 4.11.1 was released). Now issue:
opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade
and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There's not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.
Hint: You can set up a default invariant that will apply for all new switches,
via a specific opamrc
. The default one is ocaml >= 4.05.0
Configuring opam from the command-line
Configuring opam is not a simple task: you need to use an opamrc
at init
stage, or hack global/switch config file, or use opam config var
for
additional variables. To ease that step, and permit a more consistent opam
config tweaking, a new command was added : opam option
.
For example:
opam option download-jobs
gives the globaldownload-jobs
value (as it exists only in global configuration)opam option jobs=6 --global
will set the number of parallel build jobs opam is allowed to run (along with the associatedjobs
variable)opam option depext-run-commands=false
disables the use ofsudo
for handling system dependencies; it will be replaced by a prompt to run the installation commandsopam option depext-bypass=m4 --global
bypassm4
system package check globally, whileopam option depext-bypass=m4 --switch myswitch
will only bypass it in the selected switch
The command opam var
is extended with the same format, acting on switch and
global variables.
Hint: to revert your changes use opam option <field>=
, it will take its
default value.
CLI Versioning
A new --cli
switch was added to the first beta release, but it's only now
that it's being widely used. opam is a complex enough system that sometimes bug
fixes need to change the semantics of some commands. For example:
opam show --file
needed to change behaviour- The addition of new controls for setting global variables means that the
opam config
was becoming cluttered and some things want to move toopam var
opam switch install 4.11.1
still works in opam 2.0, but it's really an OPAM 1.2.2 syntax.
Changing the CLI is exceptionally painful since it can break scripts and tools
which themselves need to drive opam
. CLI versioning is our attempt to solve
this. The feature is inspired by the (lang dune ...)
stanza in dune-project
files which has allowed the Dune project to rename variables and alter
semantics without requiring every single package using Dune to upgrade their
dune
files on each release.
Now you can specify which version of opam you expected the command to be run
against. In day-to-day use of opam at the terminal, you wouldn't specify it,
and you'll get the latest version of the CLI. For example: opam var --global
is the same as opam var --cli=2.1 --global
. However, if you issue opam var --cli=2.0 --global
, you will told that --global
was added in 2.1 and so is
not available to you. You can see similar things with the renaming of opam upgrade --unlock-base
to opam upgrade --update-invariant
.
The intention is that --cli
should be used in scripts, user guides (e.g. blog
posts), and in software which calls opam. The only decision you have to take is
the oldest version of opam which you need to support. If your script is using
a new opam 2.1 feature (for example opam switch create --formula=
) then you
simply don't support opam 2.0. If you need to support opam 2.0, then you can't
use --formula
and should use --packages
instead. opam 2.0 does not have the
--cli
option, so for opam 2.0 instead of --cli=2.0
you should set the
environment variable OPAMCLI
to 2.0
. As with all opam command line
switches, OPAMCLI
is simply the equivalent of --cli
which opam 2.1 will
pick-up but opam 2.0 will quietly ignore (and, as with other options, the
command line takes precedence over the environment).
Note that opam 2.1 sets OPAMCLI=2.0
when building packages, so on the rare
instances where you need to use the opam
command in a package build:
command (or in your build system), you must specify --cli=2.1
if you're
using new features.
Since 2.1.0~rc2, CLI versioning applies to opam environment variables. The previous behavior was to ignore unknown or wrongly set environment variable, while now you will have a warning to let you know that the environment variable won't be handled by this version of opam.
To ensure not breaking compatibility of some widely used deprecated options,
a default CLI is introduced: when no CLI is specified, those deprecated
options are accepted. It concerns opam exec
and opam var
subcommands.
There's even more detail on this feature in our wiki. We're hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.
Note: For opam libraries users, since 2.1 environment variable are no more loaded by the libraries, only by opam client. You need to load them explicitly.
opam root portability
opam root format changes during opam life-cycle, new field are added or removed, new files are added ; an older opam version sometimes can no longer read an upgraded or newly created opam root. opam root format has been updated to allow new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots). It is a read-only best effort access, any attempt to modify the opam root fails.
Hint: for opam libraries users, you can safely load states with
OpamStateConfig
load functions.
Tremendous thanks to all involved people, who've developed, tested & retested, helped with issue reports, comments, feedback...
Try it!
In case you plan a possible rollback, you may want to first backup your
~/.opam
directory.
The upgrade instructions are unchanged:
- Either from binaries: run
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0"
or download manually from the Github "Releases" page to your PATH.
- Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
About OCamlPro:
OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.
- We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
- We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
- We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
- We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.
Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.
Most Recent Articles
2024
- Flambda2 Ep. 3: Speculative Inlining
- opam 2.2.0 release!
- Flambda2 Ep. 2: Loopifying Tail-Recursive Functions
- Fixing and Optimizing the GnuCOBOL Preprocessor
- OCaml Backtraces on Uncaught Exceptions
- Opam 102: Pinning Packages
- Flambda2 Ep. 1: Foundational Design Decisions
- Behind the Scenes of the OCaml Optimising Compiler Flambda2: Introduction and Roadmap
- Lean 4: When Sound Programs become a Choice
- Opam 101: The First Steps
2023
- Maturing Learn-OCaml to version 1.0: Gateway to the OCaml World
- The latest release of Alt-Ergo version 2.5.1 is out, with improved SMT-LIB and bitvector support!
- 2022 at OCamlPro
- Autofonce, GNU Autotests Revisited
- Sub-single-instruction Peano to machine integer conversion
- Statically guaranteeing security properties on Java bytecode: Paper presentation at VMCAI 23
- Release of ocplib-simplex, version 0.5
- The Growth of the OCaml Distribution