new opam features: "opam build"

---

UPDATE: after discussions following this post, this feature was abandoned with the interface presented below. See this post for the details and the new interface!

The new opam 2.0 release, currently in beta, introduces several new features. This post gets into some detail on the new opam build command, its purpose, its use, and some implementation aspects.

opam build is run from the source tree of a project, and does not rely on a pre-existing opam installation. As such, it adds a new option besides the existing workflows based on managing shared OCaml installations in the form of switches.

What does it do ?

Typically, this is used in a fresh git clone of some OCaml project. Like when pinning the package, opam will find and leverage package definitions found in the source, in the form of opam files.

• if opam hasn't been initialised (no ~/.opam), this is taken care of.
• if no switch is otherwise explicitely selected, a local switch is used, and created if necessary (i.e. in ./_opam/)
• the metadata for the current project is registered, and the package installed after its dependencies, as opam usually does

This is particularly useful for distributing projects to people not used to opam and the OCaml ecosystem: the setup steps are automatically taken care of, and a single opam build invocation can take care of resolving the dependency chains for your package.

If building the project directly is preferred, adding --deps-only is a good way to get the dependencies ready for the project:

opam build --deps-only
eval $(opam config env)
./configure; make; etc.

Note that if you just want to handle project-local opam files, opam build can also be used in your existing switches: just specify --no-autoinit, --switch or make sure the OPAMSWITCH variable is set. E.g. opam build --no-autoinit --deps-only is a convenient way to get the dependencies for the local project ready in your current switch.

Additional functions

Installation

The installation of the packages happens as usual to the prefix corresponding to the switch used (<project-root>/_opam/ for a local switch). But it is possible, with --install-prefix, to further install the package to the system:

opam build --install-prefix ~/local

will install the results of the package found in the current directory below ~/local.

The dependencies of the package won't be installed, so this is intended for programs, assuming they are relocatable, and not for libraries.

Choosing custom repositories

The user can pre-select the repositories to use on the creation of the local switch with:

opam build --repositories <repos>

where <repos> is a comma-separated list of repositories, specified either as name=URL, or name if already configured on the system.

Multiple packages

Multiple packages are commonly found to share a single repository. In this case, opam build registers and builds all of them, respecting cross-dependencies. The opam files to use can also be explicitely selected on the command-line.

In this case, specific opam files must be named <package-name>.opam.

Implementation details

The choice of the compiler, on automatic initialisation, is either explicit, using the --compiler option, or automatic. In the latter case, the default selection is used (see opam init --help, section "CONFIGURATION FILE" for details), but a compiler compatible with the local packages found is searched from that. This allows, for example, to choose a system compiler when available and compatible, avoiding a recompilation of OCaml.

When using --install-prefix, the normal installation is done, then the tracking of package-installed files, introduced in opam 2.0, is used to extract the installed files from the switch and copy them to the prefix.

The packages installed through opam build are not registered in any repository, and this is not an implicit use of opam pin: the rationale is that packages installed this way will also be updated by repeating opam build. This means that when using other commands, e.g. opam upgrade, opam won't try to keep the packages to their local, source version, and will either revert them to their repository definition, or remove them, if they need recompilation.

Planned extensions

This is still in beta: there are still rough edges, please experiment and give feedback! It is still possible that the command syntax and semantics change significantly before release.

Another use-case that we are striving to improve is sharing of development setups (share sets of pinned packages, depend on specific remotes or git hashes, etc.). We have many ideas to improve on this, but opam build is not, as of today, a direct solution to this. In particular, installing this way still relies on the default opam repository; a way to define specific options for the switch that is implicitely created on opam build is in the works.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

Comments

Louis Gesbert (16 March 2017 at 14 h 31 min):

Some discussion on a better naming and making some parts of this more widely available in the opam CLI is ongoing at https://github.com/ocaml/opam/issues/2882

Hez Carty (16 March 2017 at 17 h 23 min):

Is it possible/planned to support sharing of compilers across local (or global) switches? It would be very useful to have a global 4.04.0+flambda switch including only the compiler itself or the compiler + basic tools like ocp-indent and merlin. Then a number of projects could share this base installation but have their own locally installed dependencies without duplicating the entire build time per-project.

Louis Gesbert (17 March 2017 at 10 h 10 min):

Sharing compilers, or other packages across switches is not supported at the moment. However:

You can still use the global system compiler on any switch, local or not, to avoid its recompilation What is planned, as a first step, for after the 2.0 release, is to add a cache of compiled packages. Hooks are already in place to allow this, and opam is able to track the files installed by each package already, so the most difficult part is probably going to be the relocation issues with OCaml itself.

A cache is an easier solution to warrant consistency: with shared switches, the problem of reinstallations and keeping everything consistent gets much more complex — what happens when you change the compiler of your “master” switch ?

Hez Carty (20 March 2017 at 16 h 46 min):

That sounds great, thank you. Should make this kind of local switch more useful when working with large numbers of projects.

---

Au sujet d'OCamlPro :

OCamlPro développe des applications à haute valeur ajoutée depuis plus de 10 ans, en utilisant les langages les plus avancés, tels que OCaml et Rust, visant aussi bien rapidité de développement que robustesse, et en ciblant les domaines les plus exigeants (méthodes formelles, cybersécurité, systèmes distribués/blockchain, conception de DSLs). Fort de plus de 20 ingénieurs R&D, avec une expertise unique sur les langages de programmation, aussi bien théorique (plus de 80% de nos ingénieurs ont une thèse en informatique) que pratique (participation active au développement de plusieurs compilateurs open-source, prototypage de la blockchain Tezos, etc.), diversifiée (OCaml, Rust, Cobol, Python, Scilab, C/C++, etc.) et appliquée à de multiples domaines. Nous dispensons également des [formations sur mesure certifiées Qualiopi sur OCaml, Rust, et les méthodes formelles] (https://training.ocamlpro.com/) Pour nous contacter : contact@ocamlpro.com.