OCamlPro Feed

opam 2.3.0 release!

2024-11-13T13:56:36Z

Feedback on this post is welcomed on Discuss!

As mentioned in our talk at the OCaml Workshop 2024, we decided to switch to a time-based release cycle (every 6 months), starting with opam 2.3.

As promised, we are very pleased to announce the release of opam 2.3.0, and encourage all users to upgrade. Please read on for installation and upgrade instructions.

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam or $env:LOCALAPPDATAopam directory.

The upgrade instructions are unchanged:

Either from binaries: run

For Unix systems

$ bash -c "sh <(curl -fsSL https://opam.ocaml.org/install.sh)"

or from PowerShell for Windows systems

Invoke-Expression "& { $(Invoke-RestMethod https://opam.ocaml.org/install.ps1) }"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

Major breaking change: extra-files

When loading a repository, opam now ignores files in packages' files/ directories which aren't listed in the extra-files field of the opam file. This was done to simplify the opam specification where we hope the opam file to be the only thing that you have to look at when reading a package specification. It being optional to list all files in the extra-files: field went against that principle. This change also reduces the surface area for potential file corruption as all extra-files must have checksums.

This is a breaking change and means that if you are using the files/ directory without listing them in the extra-files: field, you need to make sure that all files in that directory are included in the extra-files field. The resulting opam file remains compatible with all previous opam 2.x releases.

If you have an opam repository, you should make sure all files are listed so every packages continues to work without any issue, which can be done automatically using the opam admin update-extrafiles command.

Major changes

Packages requiring an unsupported version of opam are now marked unavailable, instead of causing a repository error. This means an opam repository can now allow smoother upgrade in the future where some packages can require a newer version of opam without having to fork the repository to upgrade every package to that version as was done for the upgrade from opam 1.2 to 2.0
Add a new opam list --latests-only option to list only the latest versions of packages. Note that this option respects the order options were given on the command line. For example: --available --latests-only will first list all the available packages, then choose only the latest packages in that set; while --latests-only --available will first list all the latest packages, then only show the ones that are available in that set
Fix and improve opam install --check, which now checks if the whole dependency tree of the package is installed instead of only the root dependencies
Add a new --verbose-on option to enable verbose output for specified package names. Thanks to @desumn for this contribution
Add a new opam switch import --deps-only option to install only the dependencies of the root packages listed in the opam switch export file
opam switch list-available no longer displays compilers flagged with avoid-version/deprecated unless --all is given, meaning that pre-release or unreleased OCaml packages no longer appear to be the latest version
opam switch create --repositories now correctly infers --kind=git for URLs ending with .git rather than requiring the git+https:// protocol. This is consistant with other commands such as opam repository add. Thanks to @Keryan-dev for this contribution
opam switch set-invariant now displays the switch invariant using the same syntax as the --formula flag
The builtin-0install solver was improved and should now be capable of being your default solver instead of builtin-mccs+glpk. It was previously mostly only suited for automated tasks such as Continuous Integration. If you wish to give it a try, simply calling opam option solver=builtin-0install (call opam option solver= restores the default)
Most of the unhelpful conflict messages were fixed. (#4373)
Fix an opam 2.1 regression where the initial pin of a local VCS directory would store untracked and ignored files. Those files would usually be cleaned before building the package, however git submodules would not be cleaned and would cause issues when paired with the new behaviour added in 2.3.0~alpha1 which makes opam error when git submodules fail to update (it was previously a warning). (#5809)
Fix the value of the arch variable when the current OS is 32bit on a 64bit machine (e.g. Raspberry Pi OS). (#5949)
opam now fails when git submodules fail to update instead of ignoring the error and just showing a warning
opam's libraries now compile with OCaml >= 5.0 on Windows
Fix the installed packages internal cache, which was storing the wrong version of the opam file after a build failure. This could be triggered easily for users with custom repositories with non-populated extra-files. (#6213)
Several improvements to the pre-built release binaries were made:
- The Linux binaries are now built on Alpine 3.20
- The FreeBSD binary is now built on FreeBSD 14.1
- The OpenBSD binary is now built on OpenBSD 7.6 and loses support for OpenBSD 7.5 and earlier
- Linux/riscv64 and NetBSD/x86_64 binaries are now available

And many other general, performance and UI improvements were made and bugs were fixed. You can take a look to previous blog posts. API changes and a more detailed description of the changes are listed in:

This release also includes PRs improving the documentation and improving and extending the tests.

Please report any issues to the bug-tracker.

We hope you will enjoy the new features of opam 2.3!

Optimisation de Geneweb, 1er logiciel français de Généalogie depuis près de 30 ans

2024-11-06T13:56:36Z

Un bonsaï sous sa cloche de verre. De nos jours, l'accès à la généalogie grand public est préservé surtout grâce à la maintenance de codes patrimoniaux.

L’équipe d’OCamlPro a récemment été sollicitée par l’association Roglo, une association française de généalogie qui gère une base de plus de 10 millions de personnes connectées dans un même arbre généalogique, et dont la base s'accroît d’environ 500 000 nouvelles contributions tous les ans. L’association s’appuie sur le logiciel libre Geneweb, l’un des plus puissants logiciels du domaine, créé en 1997 à l’Inria, permettant de partager sur le web des arbres généalogiques, et utilisé aussi bien par des particuliers que par des leaders du secteur, comme la société française Geneanet, acquise en 2021 par l’Américain Ancestry.

Notre mission s’est d’abord concentrée sur l'optimisation des performances, pour ramener le traitement de certaines requêtes sur la base gargantuesque de Roglo à des temps raisonnables. Après avoir rapidement survolé le code de plus de 80 000 lignes et profilé les requêtes les plus coûteuses, nous avons pu proposer une solution, l’implanter et l’intégrer dans la branche principale. Pour l’une des requêtes, le temps passe ainsi de 77s à 4s, soit 18 fois plus rapide ! Nous travaillons maintenant à enrichir Geneweb de nouvelles fonctionnalités pour ces utilisateurs, mais aussi pour ses contributeurs et les mainteneurs de la plateforme !

Cette mission, fragmentée en sprints de développement, s'inscrit dans une démarche continue visant à faire évoluer Geneweb pour qu'il puisse gérer des volumes de données encore plus importants. Nous sommes ravis de contribuer à cette évolution, en apportant notre expertise en optimisation et en développement logiciel pour faire grandir cette plateforme de référence.

Alt-Ergo 2.6 is Out!

2024-09-30T13:56:36Z

The Alt-Ergo 2.6 release comes with many enhancements!

We are excited to announce the release of Alt-Ergo 2.6!

Alt-Ergo is an open-source automated prover used for formal verification in software development. It is part of the arsenal behind static analysis frameworks such as TrustInSoft Analyzer and Frama-C, and is one of the solvers behind Why3, a platform for deductive program verification. The newly released version 2.6 brings new features and performance improvements.

Development on Alt-Ergo has accelerated significantly this past year, thanks to the launch of the DéCySif joint research project (i-Démo) with AdaCore, Inria, OCamlPro and TrustInSoft. The improvements to bit-vectors and algebraic data types in this release are sponsored by the Décysif project.

The highlights of Alt-Ergo 2.6 are:

Support for reasoning and model generation with bit-vectors
Model generation for algebraic data types
Optimization with (maximize) and (minimize)
FPA support is enabled by default and available in SMT-LIB format
Binary releases now on GitHub

Alt-Ergo 2.6 also includes other improvements to the user interface (notably the set-option SMT-LIB command), use of Dolmen as the default frontend for SMT-LIB and native input, and many bug fixes.

Bit-vectors

In Alt-Ergo 2.5, we introduced built-in functions for the bit-vector primitives from the SMT-LIB standard, but only provided limited reasoning support. For Alt-Ergo 2.6, we set out to improve this reasoning support, and have developed a new and improved relational theory for bit-vectors. This new theory is based on an also new constraint propagation core that draws heavily on the architecture of the Colibri solver (as in Sharpening Constraint Programming approaches for Bit-Vector Theory), integrated into Alt-Ergo's existing normalizing Shostak solver.

Bit-vectors are commonly used in verification of low-level code and in cryptography, so improved support significantly enhances Alt-Ergo’s applicability in these domains.

There are still areas of improvements, so please share any issue you encounter with the bit-vector theory (or Alt-Ergo in general) via our issue tracker.

To showcase improvements in Alt-Ergo 2.6, we compared it against the version 2.5 and industry-leading solvers Z3 and CVC5 on a dataset of bit-vector problems collected from our partners in the DéCySif project. The (no BV) variants for Alt-Ergo do not use the new bit-vector theory but instead an axiomatization of bit-vector primitives provided by Why3. The percentages represent the proportion of bit-vector problems solved successfully in each configuration.

	AE 2.5		AE 2.6		Z3 (4.12.5)	CVC5 (1.1.2)	Total
	(BV)	(no BV)	(BV)	(no BV)
#	4128	4870	6265	4940	5482	7415	9038
%	46%	54%	69%	54%	61%	82%	100%

As the table shows, Alt-Ergo 2.6 significantly outperforms version 2.5, and the new built-in bit-vector theory outperforms Why3's axiomatization. We even surpass Z3 on this benchmark, a testament to the new bit-vector theory in Alt-Ergo 2.6.

Model Generation

Bit-vector is not the only theory Alt-Ergo 2.6 improves upon. Model generation was introduced in Alt-Ergo 2.5 with support for booleans, integers, reals, arrays, enumerated types, and records. Alt-Ergo 2.6 extends this support to bit-vector and arbitrary algebraic data types, which means that model generation is now enabled for all the theories supported by Alt-Ergo.

Model generation allows users to extract concrete examples or counterexamples, aiding in debugging and verification of their systems.

Model generation is also more robust in Alt-Ergo 2.6, with numerous bug fixes and improvements for edge cases.

Optimization

Alt-Ergo 2.6 introduces optimization capabilities, available via SMT-LIB input using OptiSMT primitives such as (minimize) and (maximize) and compatible with Z3 and OptiMathSat. Optimization allows guiding the solver towards simpler and smaller counterexamples, helping users find more concrete and realistic scenarios to trigger a bug.

See some examples in the documentation.

SMT-LIB command support

Alt-Ergo 2.6 supports more SMT-LIB syntax and commands, such as:

The (get-info :all-statistics) command to obtain information about the solver's statistics
The (reset), (exit) and (echo) commands
The (get-assignment) command, as well as the :named attribute and :produce-assignments option

See the SMT-LIB standard for more details about these commands.

Floating-point theory

In this release, we have made Alt-Ergo's floating-point theory enabled by default: there is no need to provide the --enable-theories fpa flag anymore. The theory can be disabled with --disable-theories fpa,nra,ria (the nra and ria theories were automatically enabled along with the fpa theory in Alt-Ergo 2.5).

We have also made the floating-point primitives available in the SMT-LIB format as the indexed constant ae.round and the convenience ae.float16, ae.float32, ae.float64 and ae.float128 functions; see the documentation.

Dolmen is the new default frontend

Introduced in Alt-Ergo 2.5, the Dolmen frontend has been rigorously tested for regressions and is now the default for both .smt2 and .ae files; the --frontend dolmen flag that was introduced in Alt-Ergo 2.5 is no longer necessary.

The Dolmen frontend is based on the Dolmen library developed by Guillaume Bury at OCamlPro. It provides excellent support for the SMT-LIB standard and is used to check validity of all new problems in the SMT-LIB benchmark collection, as well as the results of the annual SMT-LIB affiliated solver competition SMT-COMP.

The preferred input format for Alt-Ergo is now the SMT-LIB format. The legacy .ae format is still supported, but is now deprecated and users are encouraged to migrate to the SMT-LIB format if possible. Please reach out if you find any issue while migrating to the SMT-LIB format.

As we announced when releasing Alt-Ergo 2.5, the legacy frontend (supports .ae files only) is deprecated in Alt-Ergo 2.6, but it can still be enabled with the --frontend legacy option. It will be removed entirely from Alt-Ergo 2.7.

Parser extensions, such as the built-in AB-Why3 plugin, only work with the legacy frontend, and will no longer work with Alt-Ergo 2.7. We are not aware of any current users of either parser extensions or the AB-Why3 plugin: if you need these features, please reach out to us on GitHub or by email so that we can figure out a path forward.

Use of `dune-site` for plugins

Starting with Alt-Ergo 2.6, we are using the plugin mechanism from dune-site to replace the custom plugin loading Dynlink. Plugins now need to be registered in the (alt-ergo plugins) site with the plugin stanza.

This does not impact users, but only impacts developers of Alt-Ergo plugins. See the dune file for Alt-Ergo's built-in FM-Simplex plugin for reference.

Binary releases on GitHub

Starting with Alt-Ergo 2.6, we will be providing binary releases on the GitHub Releases page for Linux (x86_64) and macOS (x86_64 and arm). These are released under the same licensing conditions as the Alt-Ergo source code.

The binary releases are statically linked and have no dependencies, except for system dependencies on macOS. They do not support dynamically loading plugins.

Performance

For Alt-Ergo 2.6, our main focus of improvement in term of reasoning was on bit-vectors and algebraic data types. Other theories also benefit from broader performance improvements we made. On our internal problem dataset, Alt-Ergo 2.6 is about 5% faster than Alt-Ergo 2.5 on the goals they both prove.

And more!

This release also includes significant internal refactoring, notably a rewrite from scratch of the interval domain. This improves the accuracy of Alt-Ergo in handling interval arithmetic and facilitates mixed operations involving integers and bit-vectors, resulting in shorter and more reliable proofs.

See the complete changelog here.

We encourage you to try out Alt-Ergo 2.6 and share your experience or any feedback on our GitHub or by email at alt-ergo@ocamlpro.com. Your input will help share future releases!

Acknowledgements

We thank the Alt-Ergo Users' Club members: AdaCore, the CEA, Thales, Mitsubishi Electric R&D Center Europe (MERCE) and TrustInSoft.

Special thanks to David Mentré and Denis Cousineau at MERCE for funding the initial optimization work. MERCE has been a Member of the Alt-Ergo Users' Club for four years. This partnership allowed Alt-Ergo to evolve and we hope that more users will join the Club on our journey to make Alt-Ergo a must-have tool.

The dedicated members of our Alt-Ergo Club!

Flambda2 Ep. 3: Speculative Inlining

2024-08-09T13:56:36Z

Credit: The Weighing of the Heart Ceremony, Ammit. Angus McBride (British, 1931-2007)" src="/blog/assets/img/picture_egyptian_weighing_of_heart.jpg"/>

A representation of Speculative Inlining through the famous Weighing Of The Heart of Egyptian Mythology. Egyptian God Anubis weighs his OCaml function, to see if it is worth inlining.
Credit: The Weighing of the Heart Ceremony, Ammit. Angus McBride (British, 1931-2007)

Welcome to a new episode of The Flambda2 Snippets!

The F2S blog posts aim at gradually introducing the world to the inner-workings of a complex piece of software engineering: The Flambda2 Optimising Compiler for OCaml, a technical marvel born from a 10 year-long effort in Research & Development and Compilation; with many more years of expertise in all aspects of Computer Science and Formal Methods.

Today's article will serve as an introduction to one of the key design decisions structuring Flambda2 that we will cover in the next episode in the series: Upward and Downward Traversals.

See, there are interesting things to be said about how inlining is conducted inside of our compiler. Inlining in itself is rather ubiquitous in compilers. The goal here is to show how we approach inlining, and present what we call Speculative Inlining.

Table of contents

Inlining in general
When inlining is detrimental
How to decide when inlining is beneficial
Speculative inlining
Speculative inlining in practice
Summary
Conclusion

Inlining in general

Given the way people write functional programs, inlining is an important part of the optimisation pipeline of such functional langages.

What we call inlining in this series is the process of duplicating some code to specialise it to a specific context.

Usually, this can be thought as copy-pasting the body of a function at its call site. A common misunderstanding is to think that the main benefit of this optimisation is to remove the cost of the function call. However, with modern computer architectures, this has become less and less relevant in the last decades. The actual benefit is to use the specific context to trigger further optimisations.

Suppose we have the following option_map and double functions:

let option_map f x =
  match x with
  | None -> None
  | Some x -> Some (f x)

let double i =
  i + i

Additionally, suppose we are currently considering the following function:

let stuff () =
  option_map double (Some 21)

In this short example, inlining the option_map function would perform the following transformation:

let stuff () =
  let f = double in
  let x = Some 21 in
  match x with
  | None -> None
  | Some x -> Some (f x)

Now we can inline the double function.

let stuff () =
  let x = Some 21 in
  match x with
  | None -> None
  | Some x ->
    Some (let i = x in i + i)

As you can see, inlining alone isn't that useful of an optimisation per se. In this context, appliquing Constant Propagation will optimise and simplify it to the following:

let stuff () = Some 42

Although this is a toy example, combining small functions is a common pattern in functional programs. It's very convenient that using combinators is not significantly worse than writing this function by hand.

When inlining is detrimental

We cannot just go around and inline everything, everywhere... all at once.

As we said, inlining is mainly code duplication and that would be detrimental and blow the size of the compiled code drastically. However, there is a sweet spot to be found, between both absolute inlining and no inlining at all, but it is hard to find.

Here's an example of exploding code at inlining time:

(* val h : int -> int *)
let h n = (* Some non constant expression *)

(* val f : (int -> int) -> int -> int *)
let f g x = g (g x)

(* 4 calls to f -> 2^4 calls to h *)
let n = f (f (f (f h))) 42

Following through with the inlining process will produce a very large binary relative to its source code. This contrived example highlights potential problems that might arise in ordinary codebases in the wild, even if this one is tailored to be quite nasty for inlining: notice the exponential blowup in the number of nested calls, every additional call to f doubles the number of calls to h after inlining.

How to decide when inlining is beneficial

Most compilers use a collection of heuristics to guide them in the decision making. A good collection of heuristics is hard to both design, and fine-tune. They also can be quite specific to a programming style and unfit for other compilers to integrate. The take away is: there is no best way.

Side Note:

This topic would make for an interesting blog post but, unfortunately, rather remote from the point of this article. If you are interested in going deeper into that subject right now, we have found references for you to explore until we get around to writing a comprehensive, and more digestable, explanation about the heuristic nature of inlining:

Secrets of the Glasgow Haskell Compiler inliner, by SIMON PEYTON JONES and SIMON MARLOW, 2002.

Extending the Scope of Syntactic Abstraction, by OSCAR WADDELL, 1999. Section 4.4 (PDF Download link), for the case of Scheme.

Towards Better Inlining Decisions Using Inlining Trials, by JEFFREY DEAN and CRAIG CHAMBERS, 1994.

Understanding and Exploiting Optimal Function Inlining, by THEODOROS THEODORIDIS, TOBIAS GROSSER, ZHENDONG SU, 2022.

Before we get to a concrete example, and break down Speculative Inlining for you, we would like to discuss the trade-offs of duplicating code.

CPUs execute instructions one by one, or at least they pretend that they do. In order to execute an instruction, they need to load up into memory both code and data. In modern CPUs, most instructions take only a few cycles to execute and in practice, the CPUs often execute several at the same time. To put into perspective, loading memory, however, in the worst case, can take hundreds of CPU cycles... Most of the time it's not the case because CPUs have complex memory cache hierarchies such that loading from instruction cache can take just a few cycles, loading from level 2 caches may take dozens of them, and the worst case is loading from main memory which can take hundreds of cycles.

The take away is, when executing a program, the cost of one instruction that has to be loaded from main memory can be larger than the cost of executing a hundred instructions in caches.

There is a way to avoid the worst case scenario. Since caches are rather small in size, the main component to keeping from loading from main memory is to keep your program rather small, or at least the parts of it that are regularly executed.

Keep these orders of magnitude in mind when we address the trade-offs between improving the number of instructions that we run and keeping the program to a reasonably small size.

Before explaining Speculative Inlining let's consider a piece of code.

The following pattern is quite common in OCaml and other functional languages, let's see how one would go about inlining this code snippet.

Example 1: Notice the higher-order function f:

(*
  val f :
    (condition:bool -> int -> unit) 
    -> condition:bool
    -> int
    -> unit
 *)
let f g ~condition n =
  for i = 0 to n do
    g ~condition i
  done

let g_real ~condition i =
  if condition then
    (* small operation *)
  else
    (* big piece of code *)

let condition = true

let foo n =
  f g_real ~condition n

Even for such a small example we will see that the heuristics involved to finding the right solution can become quite complex.

Keeping in mind the fact that condition is always true, the best set of inlining decisions would yield the following code:

(* All the code before [foo] is kept as is, from the previous codeblock *)
let foo x = 
  for i = 0 to x do
    (* small operation *)
  done

But if condition had been always false, instead of small operation, we would have had a big chunk of g_real duplicated in foo (i.e: (* big piece of code *)). Moreover it would have only spared us the running time of a few call instructions. Therefore, we would have probably preferred to have kept ourselves from inlining anything.

Specifically, we would have liked to have stopped from inlining g, as well as to have avoided inlining f because it would have needlessly increased the size of the code with no substantial benefit.

However, if we want to be able to take an educated decision based on the value of condition, we will have to consider the entirety of the code relevant to that choice. Indeed, if we just look at the code for f, or its call site in foo, nothing would guide us to the right decision. In order to take the right decision, we need to understand that if the ~condition parameter to the g_real function is true, then we can remove a large piece of code, namely: the else branch and the condition check as well.

But to understand that the ~condition in g_real is always true, we need to see it in the context of f in foo. This implies again that, that choice of inlining is not based on a property of g_real but rather a property of the context of its call.

There exists a very large number of combinations of such difficult situations that would each require different heuristics which would be incredibly tedious to design, implement, and maintain.

Speculative inlining

We manage to circumvent the hurdle that this decision problem represents thanks to what we call Speculative Inlining. This strategy requires two properties from the compiler: the ability to inline and optimise at the same time, as well as being able to backtrack inlining decisions.

Lets look at Example 1 again and look into the Speculative Inlining strategy.

let f g ~condition n =
  for i = 0 to n do
    g ~condition i
  done

let g_real ~condition x =
  if condition then
    (* small operation *)
  else
    (* big piece of code *)

let condition = true

let foo x =
  f g_real ~condition x

We will focus only on the traversal of the foo function.

Before we try and inline anything, there are a couple things we have to keep in mind about values and functions in OCaml:

Application arity may not match function arity

To give you an idea, the function foo could also been written in the following way:

let foo x =
  let f1 = f in
  let f2 = f1 g_real in 
  let f3 = f2 ~condition in
  f3 x

We expect the compiler to translate it as well as the original, but we cannot inline a function unless all its arguments are provided. To solve this, we need to handle partial applications precisely. Over-applications also present similar challenges.

Functions are values in OCaml

We have to understand that the call to f in foo is not trivially a direct call to f in this context. Indeed, at this point functions could instead be stored in pairs, or lists, or even hashtables, to be later retrieved and applied at will, and we call such functions general functions.

Since our goal is to inline it, we need to know the body of the function. We call a function concrete when we have knowledge of its body. This entails Constant Propagation in order to associate a concrete function to general function values and, consequently, be able to simplify it while inlining.

Here's the simplest case to demonstrate the importance of Constant Propagation.

let foo_bar y =
  let pair = foo, y in
  (fst pair) (snd pair)

In this case, we have to look inside the pair in order to find the function, this demonstrates that we sometimes have to do some amount of value analysis in order to proceed. It's quite common to come across such cases in OCaml programs due to the module system and other functional languages present similar characteristics.

There are many scenarios which also require a decent amount of context in order to identify which function should be called. For example, when a function passed as parameter is called, we need to know the context of the caller functions, sometimes up to an arbitrarily large context. Analysing the relevant context will tell us which function is being called and thus help us make educated inlining decisions. This problem is specific to functional languages, functions in good old imperative languages are seldom ambiguous; even though such considerations would be relevant when function pointers are involved.

This small code snippet shows us that we have to inline some functions in order to know whether we should have inlined them.

Speculative inlining in practice

In practice, Speculative Inlining is being able to quantify the benefits brought by a set of optimisations, which have to be applied after a given inlining decision, and use these results to determine if said inlining decision is in fact worth to carry out all things considered.

The criteria for accepting an inlining decision is that the resulting code should be faster that the original one. We use "should be" because program speed cannot be fully understood with absolutes.

That's why we use a heuristic algorithm in order to compare the original and the optimised versions of the code. It roughly consists in counting the number of retired (executed) instructions and comparing it to the increase in code size introduced by inlining the body of that function. The value of that cut-off ratio is by definition heuristic and different compilation options given to ocamlopt change it.

As said previously, we cannot go around and evaluate each inlining decision independently because there are cases where inlining a function allows for more of them to happen, and sometimes a given inlining choice validates another one. We can see this in Example 1, where deciding not to inline function g_real would make the inlining of function f useless.

Naturally, every combination of inlining decision cannot be explored exhaustively. We can only explore a small subset of them, and for that we have another heuristic that was already used in Flambda1, although Flambda2 does not yet implement it in full.

It's quite simple: we choose to consider inlining decision relationships only when there are nested calls. As for any other heuristic, it does not cover every useful case, but not only is it the easiest to implement, we are also fairly confident that it covers the most important cases.

Here's a small rundown of that heuristic:

A is a function which calls B
- Case 1: we evaluate the body of A at its definition, possibly inlining B in the process
- Case 2: at a specific callsite of A, we evaluate A in the inlining context.
  - Case 2.a: inlining A is beneficial no matter the decision on B, so we do it.
  - Case 2.b: inlining A is potentially detrimental, so we go and evaluate B before deciding to inline A for good.

Keep in mind that case 2.b is recursive and can go arbitrarily deep. This amounts to looking for the best leaf in the decision tree. Since we can't explore the whole tree, we do have a some limit to the depth of the exploration.

Reminder for our fellow Cameleers: Flambda1 and Flambda2 have a flag you can pass through the CLI which will generate a .org file which will detail all the inlining decisions taken by the compiler. That flag is: -inlining-report. Note that .org files allow to easily visualise a decision tree inside of the Emacs editor.

Summary

By now, you should have a better understanding of the intricacies inherent to Speculative Inlining. Prior to its initial inception, it was fair to question how feasible (and eligible, considering the many requirements for developping a compiler), such an algorithm would be in practice. Since then, it has demonstrated its usefulness in Flambda1 and, consequently, its porting to Flambda2 was called for.

So before we move on to the next stop in the F2S series, lets summarize what we know of Speculative Inlining.

We learned that inlining is the process of copying the body of a function at its callsite. We also learned that it is not a very interesting transformation by itself, especially nowadays with how efficient modern CPUs are, but that its usefulness is found in how it facilitates other optimisations to take place later.

We also learned about the heuristic nature of inlining and how it would be difficult to maintain finely-tailored heuristics in the long run as many others have tried before us. Actually, it is because there is no best way that we have come up with the need for an algorithm that is capable of simultaneously performing inlining and optimising as well as backtracking when needed which we called Speculative Inlining. In a nutshell, Speculative Inlining is one of the algorithms of the optimisation framework of Flambda2 which facilitates other optimisations to take place.

We have covered the constraints that the algorithm has to respect for it to hold ground in practice, like performance. We value a fast compiler and aim to keep both its execution but also the code it generates to be so. Take an optimisation such as Constant Propagation as an example. It would be a naïve approach to try and perform this transformation everywhere because the resulting complexity of the compiler would amount to something like size_of_the_code * number_of_inlinings_performed which is unacceptable to say the least. We aim at making the complexity of our compiler linear to the code size, which in turn entails plenty of logarithms anytime it is possible. Instead, we choose to apply any transformation only in the inlined parts of the code.

With all these parameters in mind, can we imagine ways to tackle these multi-layered challenges all at the same time ? There are solutions out there that do so in an imperative manner. In fact, the most intuitive way to implement such an algorithm may be fairly easily done with imperative code. You may want to read about Equality Saturation for instance, or even download Manuel Serrano's Paper inside the Scheme Bigloo compiler to learn more about it. However, we require backtracking, and the nested nature of these transformations (inlining, followed by different optimising transformations) would make backtracking bug-prone and tedious to maintain if it was to be written imperatively.

It soon became evident for us that we were going to leverage one of the key characteristics of functional languages in order to make this whole ordeal easier to design, implement and maintain: purity of terms. Indeed, not only is it easier to support backtracking when manipulating pure code, but it also becomes impossible for us to introduce cascades of hard to detect nested bugs by avoiding transforming code in place. From this point on, we knew we had to perform all transformations at the same time, making our inlining function one that would return an optimised inlined function. This does introduce complexities that we have chosen over the hurdles of maintaining an imperative version of that same algorithm, which can be seen as pertaining to graph traversal and tree rewriting for all intents and purposes.

Despite the density of this article, keep in mind that we aim at explaining Flambda2 in the most comprehensive manner possible and that there are voluntary shortcuts taken throughout these snippets for all of this to make sense for the broader audience. In time, these articles will go deep into the guts of the compiler and by then, hopefully, we will have done a good job at providing our readers with all necessary information for all of you to continue enjoying this rabbit-hole with us!

Here's a pseudo-code snippet representing Speculative Inlining.

(* Pseudo-code to rpz the actual speculation *)
let try_inlining f env args =
  let inlined_version_of_f = inline f env args in
  let benefit = compare inlined_version_of_f f in
  if benefit > 0 then
    inlined_version_of_f
  else
    f

Conclusion

As we said at the start of this article, this one is but an introduction to a major topic we will cover next, namely: Upwards and Downwards Traversals.

We had to cover Speculative Inlining first. It is a reasonably approachable solution to a complex problem, and having an idea of all the requirements for its good implementation is half of the work done for understanding key design decisions such as how code traversal was designed for algorithms such as Speculative Inlining to hold out.

Thank you all for reading! We hope that these articles will keep the community hungry for more!

Until next time, keep calm and OCaml! ⚱️🐫🏺📜

opam 2.2.0 release!

2024-07-01T13:56:36Z

Feedback on this post is welcomed on Discuss!

We are very pleased to announce the release of opam 2.2.0, and encourage all users to upgrade. Please read on for installation and upgrade instructions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com, and published in discuss.ocaml.org.

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam or $env:LOCALAPPDATAopam directory.

The upgrade instructions are unchanged:

Either from binaries: run

For Unix systems

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0"

or from PowerShell for Windows systems

Invoke-Expression "& { $(Invoke-RestMethod https://raw.githubusercontent.com/ocaml/opam/master/shell/install.ps1) }"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

Changes

Major change: Windows support

After 8 years' effort, opam and opam-repository now have official native Windows support! A big thank you is due to Andreas Hauptmann (@fdopen), whose WODI and OCaml for Windows projects were for many years the principal downstream way to obtain OCaml on Windows, Jun Furuse (@camlspotter) whose initial experimentation with OPAM from Cygwin formed the basis of opam-repository-mingw, and, most recently, Jonah Beckford (@jonahbeckford) whose DkML distribution kept - and keeps - a full development experience for OCaml available on Windows.

OCaml when used on native Windows requires certain tools from the Unix world which are provided by either Cygwin or MSYS2. We have engineered opam init so that it is possible for a user not to need to worry about this, with opam managing the Unix world, and the user being able to use OCaml from either the Command Prompt or PowerShell. However, for the Unix user coming over to Windows to test their software, it is also possible to have your own Cygwin/MSYS2 installation and use native Windows opam from that. Please see the previous blog post for more information.

There are two "ports" of OCaml on native Windows, referred to by the name of provider of the C compiler. The mingw-w64 port is GCC-based. opam's external dependency (depext) system works for this port (including providing GCC itself), and many packages are already well-supported in opam-repository, thanks to the previous efforts in opam-repository-mingw. The MSVC port is Visual Studio-based. At present, there is less support in this ecosystem for external dependencies, though this is something we expect to work on both in opam-repository and in subsequent opam releases. In particular, it is necessary to install Visual Studio or Visual Studio BuildTools separately, but opam will then automatically find and use the C compiler from Visual Studio.

Major change: opam tree / opam why

opam tree is a new command showing packages and their dependencies with a tree view. It is very helpful to determine which packages bring which dependencies in your installed switch.

$ opam tree cppo
cppo.1.6.9
├── base-unix.base
├── dune.3.8.2 (>= 1.10)
│   ├── base-threads.base
│   ├── base-unix.base [*]
│   └── ocaml.4.14.1 (>= 4.08)
│       ├── ocaml-base-compiler.4.14.1 (>= 4.14.1~ & < 4.14.2~)
│       └── ocaml-config.2 (>= 2)
│           └── ocaml-base-compiler.4.14.1 (>= 4.12.0~) [*]
└── ocaml.4.14.1 (>= 4.02.3) [*]

Reverse-dependencies can also be displayed using the new opam why command. This is useful to examine how dependency versions get constrained.

$ opam why cmdliner
cmdliner.1.2.0
├── (>= 1.1.0) b0.0.0.5
│   └── (= 0.0.5) odig.0.0.9
├── (>= 1.1.0) ocp-browser.1.3.4
├── (>= 1.0.0) ocp-indent.1.8.1
│   └── (>= 1.4.2) ocp-index.1.3.4
│       └── (= version) ocp-browser.1.3.4 [*]
├── (>= 1.1.0) ocp-index.1.3.4 [*]
├── (>= 1.1.0) odig.0.0.9 [*]
├── (>= 1.0.0) odoc.2.2.0
│   └── (>= 2.0.0) odig.0.0.9 [*]
├── (>= 1.1.0) opam-client.2.2.0~alpha
│   ├── (= version) opam.2.2.0~alpha
│   └── (= version) opam-devel.2.2.0~alpha
├── (>= 1.1.0) opam-devel.2.2.0~alpha [*]
├── (>= 0.9.8) opam-installer.2.2.0~alpha
└── user-setup.0.7

Special thanks to @cannorin for contributing this feature.

Major change: with-dev-setup

There is now a way for a project maintainer to share their project development tools: the with-dev-setup dependency flag. It is used in the same way as with-doc and with-test: by adding a {with-dev-setup} filter after a dependency. It will be ignored when installing normally, but it's pulled in when the package is explicitly installed with the --with-dev-setup flag specified on the command line.

For example

opam-version: "2.0"
depends: [
  "ocaml"
  "ocp-indent" {with-dev-setup}
]
build: [make]
install: [make "install"]
post-messages:
[ "Thanks for installing the package"
  "as well as its development setup. It will help with your future contributions" {with-dev-setup} ]

Major change: opam pin --recursive

When pinning a package using opam pin, opam looks for opam files in the root directory only. With recursive pinning, you can now instruct opam to look for .opam files in subdirectories as well, while maintaining the correct relationship between the .opam files and the package root for versioning and build purposes.

Recursive pinning is enabled by the following options to opam pin and opam install:

With --recursive, opam will look for .opam files recursively in all subdirectories.
With --subpath <path>, opam will only look for .opam files in the subdirectory <path>.

The two options can be combined: for instance, if your opam packages are stored as a deep hierarchy in the mylib subdirectory of your project you can try opam pin . --recursive --subpath mylib.

These options are useful when dealing with a large monorepo-type repository with many opam libraries spread about.

New Options

opam switch -, inspired by git switch -, makes opam switch back to the previously selected global switch.
opam pin --current fixes a package to its current state (disabling pending reinstallations or removals from the repository). The installed package will be pinned to its current installed state, i.e. the pinned opam file is the one installed.
opam pin remove --all removes all the pinned packages from a switch.
opam exec --no-switch removes the opam environment when running a command. It is useful when you want to launch a command without opam environment changes.
opam clean --untracked removes untracked files interactively remaining from previous packages removal.
opam admin add-constraint <cst> --packages pkg1,pkg2,pkg3 applies the given constraint to a given set of packages
opam list --base has been renamed into --invariant, reflecting the fact that since opam 2.1 the "base" packages of a switch are instead expressed using a switch invariant.
opam install --formula <formula> installs a formula instead of a list of packages. This can be useful if you would like to install one package or another one. For example opam install --formula '"extlib" |"extlib-compat"' will install either extlib or extlib-compat depending on what's best for the current switch.

Miscellaneous changes

The UI now displays a status when extracting an archive or reloading a repository
Overhauled the implementation of opam env, fixing many corner cases for environment updates and making the reverting of package environment variables precise. As a result, using setenv in an opam file no longer triggers a lint warning.
Fix parsing pre-opam 2.1.4 switch import files containing extra-files
Add a new sys-ocaml-system default global eval variable
Hijack the "%{var?string-if-true:string-if-false-or-undefined}%" syntax to support extending the variables of packages with + in their name (conf-c++ and conf-g++ already exist) using "%{?pgkname:var:}%"
Fix issues when using fish as shell
Sandbox: Mark the user temporary directory (as returned by getconf DARWIN_USER_TEMP_DIR) as writable when TMPDIR is not defined on macOS
Add Warning 69: Warn for new syntax when package name in variable in string interpolation contains several '+' (this is related to the "hijack" item above)
Add support for Wolfi OS, treating it like Alpine family as it also uses apk
Sandbox: /tmp is now writable again, restoring POSIX compliance
Add a new opam admin: new add-extrafiles command to add/check/update the extra-files: field according to the files present in the files/ directory
Add a new opam lint -W @1..9 syntax to allow marking a set of warnings as errors
Fix bugs in the handling of the OPAMCURL, OPAMFETCH and OPAMVERBOSE environment variables
Fix bugs in the handling of the --assume-built argument
Software Heritage fallbacks is now supported, but is disabled-by-default for now. For more information you can read one of our previous blog post

And many other general and performance improvements were made and bugs were fixed. You can take a look to previous blog posts. API changes and a more detailed description of the changes are listed in:

This release also includes PRs improving the documentation and improving and extending the tests.

Please report any issues to the bug-tracker.

We hope you will enjoy the new features of opam 2.2! 📯

Flambda2 Ep. 2: Loopifying Tail-Recursive Functions

2024-05-07T13:56:36Z

Two camels are taking a break from crossing the desert, they know their path could not have been more optimised.

Welcome to a new episode of The Flambda2 Snippets!

Today's topic is Loopify, one of Flambda2's many optimisation algorithms which specifically deals with optimising both purely tail-recursive and/or functions annotated with the [@@loop] attribute in OCaml.

A lazy explanation for its utility would be to say that it simply aims at reducing the number of memory allocations in the context of recursive and tail-recursive function calls in OCaml. However, we will see that is just part of the point and thus we will tend to address the broader context: what are tail-calls, how they are optimised and how they fit in the functional programming world, what dilemma does Loopify nullify exactly and, in time, many details on how it's all implemented!

If you happen to be stumbling upon this article and wish to get a bird's-eye view of the entire F2S series, be sure to refer to Episode 0 which does a good amount of contextualising as well as summarising of, and pointing to, all subsequent episodes.

All feedback is welcome, thank you for staying tuned and happy reading!

The F2S blog posts aim at gradually introducing the world to the inner-workings of a complex piece of software engineering: The Flambda2 Optimising Compiler, a technical marvel born from a 10 year-long effort in Research & Development and Compilation; with many more years of expertise in all aspects of Computer Science and Formal Methods.

Table of contents

Tail-Call Optimisation
Tail-Calls in OCaml
The Conundrum of Reducing allocations Versus Writing Clean Code
Loopify
Conclusion

Tail-Call Optimisation

As far as we know, Tail-Call optimisation (TCO) has been a reality since at least the 70s. Some LISP implementations used it and Scheme specified it into its language around 1975.

The debate to support TCO happens regularly today still. Nowadays, it's a given that most functional languages support it (Scala, OCaml, Haskell, Scheme and so on...). Other languages and compilers have supported it for some time too. Either optionally, with some C compilers (gcc and clang) that support TCO in some specific compilation scenarios; or systematically, like Lua, which, despite not usually being considered a functional language, specifies that TCO occurs whenever possible (you may want to read section 3.4.10 of the Lua manual here).

So what exactly is Tail-Call Optimisation ?

A place to start would be the Wikipedia page. You may also find some precious insight about the link between the semantics of GOTO and tail calls here, a course from Xavier Leroy at the College de France, which is in French.

Additionally to these resources, here are images to help you visualise how TCO improves stack memory consumption. Assume that g is a recursive function called from f:

A representation of the textbook behaviour for recursive functions stackframe allocations. You can see here that the stackframes of non-tail-recursive functions are allocated sequentially on decreasing memory addresses which may eventually lead to a stack overflow.

Now, let's consider a tail-recursive implementation of the g function in a context where TCO is not supported. Tail-recursion means that the last thing t_rec_g does before returning is calling itself. The key is that we still have a frame for the caller version of t_rec_g but we know that it will only be used to return to the parent. The frame itself no longer holds any relevant information besides the return address and thus the corresponding memory space is therefore mostly wasted.

A representation of the textbook behaviour for tail-recursive functions stackframe allocations without Tail Call Optimisation (TCO). When TCO is not implemented the behaviour for these allocations and the potential for a stack overflow are the same as with non-tail-recursive functions.

And finally, let us look at the same function in a context where TCO is supported. It is now apparent that memory consumption is much improved by the fact that we reuse the space from the previous stackframe to allocate the next one all the while preserving its return address:

A representation of the textbook behaviour for tail-recursive functions stackframe allocations with TCO. Since TCO is implemented, we can see that the stack memory consumption is now constant, and that the potential that this specific tail-recursive function will lead to a stack overflow is diminished.

Tail-Calls in OCaml

The List data structure is fundamental to and ubiquitous in functional programming. Therefore, it's important to not have an arbitrary limit on the size of lists that one can manipulate. Indeed, most List manipulation functions are naturally expressed as recursive functions, and can most of the time be implemented as tail-recursive functions. Without guaranteed TCO, a programmer could not have the assurance that their program would not stack overflow at some point. That reasoning also applies to a lot of other recursive data structures that commonly occur in programs or libraries.

In OCaml, TCO is guaranteed. Ever since its inception, Cameleers have unanimously agreed to guarantee the optimisation of tail-calls. While the compiler's support for TCO has been a thing from the beginning, an attribute, [@tailcall] was later added to help users ensure that their calls are in tail position.

Recently, TCO was also extended with the Tail Mod Cons optimisation which allows to generate tail-calls in more cases.

The Conundrum of Reducing Allocations Versus Writing Clean Code

One would find one of the main purposes for the existence of Loopify in the following conversation: a Discuss Post about the unboxing of floating-point values in OCaml and performance.

This specific comment sparks a secondary conversation that you may want to read yourself but will find a quick breakdown of below and that will be a nice starting point to understand today's subject.

Consider the following code:

let sum l =
  let rec loop s l =
    match l with
    | [] -> s
    | hd :: tl ->
      (* This allocates a boxed float *)
      let s = s +. hd in
      loop s tl 
  in
  loop 0. l

This is a simple tail-recursive implementation of a sum function for a list of floating-point numbers. However this is not as efficient as we would like it to be.

Indeed, OCaml needs an uniform representation of its values in order to implement polymorphic functions. In the case of floating-point numbers this means that the numbers are boxed whenever they need to be used as generic values.

Besides, everytime we call a function all parameters have to be considered as generic values. We thus cannot avoid their allocation at each recursive call in this function.

If we were to optimise it in order to get every last bit of performance out of it, we could try something like:

Warning: The following was coded by trained professionnals, do NOT try this at home.

let sum l = 
  (* Local references *)
  let s = ref 0. in
  let cur = ref l in
  try
    while true do
      match !cur with
      | [] -> raise Exit
      | hd :: tl ->
        (* Unboxed floats -> No allocation *)
        s := !s +. hd;
        cur := tl
    done; assert false
  with Exit -> !s (* The only allocation *)

While in general references introduce one allocation and a layer of indirection, when the compiler can prove that a reference is strictly local to a given function it will use mutable variables instead of reference cells.

In our case s and cur do not escape the function and are therefore eligible to this optimisation.

After this optimisation, s is now a mutable variable of type float and so it can also trigger another optimisation: float unboxing.

You can see more details here but note that, in this specific example, all occurrences of boxing operations disappear except a single one at the end of the function.

We like to think that not forcing the user to write such code is a benefit, to say the least.

Loopify

Concept

There is a general concept of transforming function-level control-flow into direct IR continuations to benefit from "basic block-level" optimisations. One such pattern is present in the local-function optimisation triggered by the [@local] attribute. Here's the link to the PR that implements it. Loopify is an attempt to extend the range of this kind of optimisation to proper (meaning self) tail-recursive calls.

As you saw previously, in some cases (e.g.: numerical calculus), recursive functions sometimes hurt performances because they introduce some allocations.

That lost performance can be recovered by hand-writing loops using local references however it's unfortunate to encourage non-functional code in a language such as OCaml.

One of Flambda and Flambda2's goals is to avoid situations such as those and allow for good-looking, functional code, to be as performant as code which is written and optimised by hand at the user-level.

Therefore, we introduce a solution to the specific problem described above with Loopify, which, in a nutshell, transforms tail-recursive functions into non-recursive functions containing a loop, hence the name.

Deciding to Loopify or not

The decision to loopify a given function is made during the conversion from the Lambda IR to the Flambda2 IR. The conversion is triggered in two cases:

when a function is purely tail-recursive -- meaning all its uses within its body are self-tail calls, they are called proper calls;
when an annotation is given by the user in the source code using the [@loop] attribute;

Let's see two examples for them:

(* Not a tail-rec function: is not loopified *)
let rec map f = function
  | [] -> []
  | x :: r -> f x :: map f r

(* Is tail-rec: is loopified *)
let rec fold_left f acc = function
  | [] -> acc
  | x :: r -> fold_left f (f acc x) r

Here, the decision to loopify is automatic and requires no input from the user. Quite straightforward.

Onto the second case now:

(* Helper function, not recursive, nothing to do. *)
let log dbg f arg =
  if dbg then
    print_endline "Logging...";
  f arg
[@@inline]

(* 
  Not tail-rec in the source, but may become
  tail-rec after inlining of the [log] function.
  At this point we can loopify, provided that the
  user specified a [@@loop] attribute.
*)
let rec iter_with_log dbg f = function
  | [] -> ()
  | x :: r ->
    f x;
    log dbg (iter_with_log dbg f) r
[@@loop]

The recursive function iter_with_log, is not initially purely tail-recursive.

However after the inlining of the log function and then simplification, the new code for iter_with_log becomes purely tail-recursive.

At that point we have the ability to loopify the function, but we keep from doing so unless the user specifies the [@@loop] attribute on the function definition.

The nature of the transformation

Onto the details of the transformation.

First, we introduce a recursive continuation at the start of the function. Lets call it self.

Then, at each tail-recursive call, we replace the function call with a continuation call to self with the same arguments as the original call.

let rec iter_with_log dbg f l =
  let_cont rec k_self dbg f l =
    match l with
    | [] -> ()
    | x :: r ->
      f x;
      log dbg (iter_with_log dbg f) r
  in
  apply_cont k_self (dbg, f, l)

Then, we inline the log function:

let rec iter_with_log dbg f l =
  let_cont k_self dbg f l =
    match l with
    | [] -> ()
    | x :: r ->
      f x;
      (* Here the inlined code starts *)
      (*
        We first start by binding the arguments of the
        original call to the parameters of the function's code
       *)
      let dbg = dbg in
      let f = iter_with_log dbg f in
      let arg = r in
      if dbg then
        print_endline "Logging...";
      f arg
  in
  apply_cont k_self (dbg, f, l)

Then, we discover a proper tail-recursive call subsequently to these transformations that we replace with the adequate continuation call.

let rec iter_with_log dbg f l =
  let_cont k_self dbg f l =
    match l with
    | [] -> ()
    | x :: r ->
      f x;
      (* Here the inlined code starts *)
      (*
        Here, the let bindings have been substituted
        by the simplification.
       *)
      if dbg then
        print_endline "Logging...";
      apply_cont k_self (dbg, f, r)
  in
  apply_cont k_self (dbg, f, l)

In this context, the benefit of transforming a function call to a continuation call is mainly about allowing other optimisations to take place. As shown in the previous section, one of these optimisations is unboxing which can be important in some cases like numerical calculus. Such optimisations can take place because continuations are local to a function while OCaml ABI-abiding function calls require a prior global analysis.

One could think that a continuation call is intrinsically cheaper than a function call. However, the OCaml compiler already optimises self-tail-calls such that they are already as cheap as continuation calls (i.e, a single jump instruction).

An astute reader could realise that this transformation can apply to any function and will result in one of three outcomes:

if the function is not tail-recursive, or even not recursive at all, nothing will happen, the transformation does nothing.
if a function is purely tail-recursive then all recursive calls will be replaced to a continuation call and the function after optimisation will no longer be recursive. This allows us to later inline it and even specialise some of its arguments. This happens precisely when we automatically decide to loopify a function;
if a function is not purely tail-recursive, but contains some tail-recursive calls then the transformation will rewrite those calls but not the other ones. This may result in better code but it's hard to be sure in advance. In such cases (and cases where functions become purely tail-recursive only after inlining), users can force the transformation by using the [@@loop] attribute

Conclusion

Here it is, the concept behind the Loopify optimisation pass as well as the general context and philosophy which led to its inception!

It should be clear enough now that having to choose between writing clean or efficient code was always unsatisfactory to us. With Loopify, as well as with the rest of the Flambda and Flambda2 compiler backends, we aim at making sure that users should not have to write imperative code for it to be as efficient as functional code. Thus ideally making any which way of writing a piece of code as efficient as the next.

This article describes one of the very first user-facing optimisations of this series of snippets on Flambda2. We have not gotten into any of the neat implementation details yet. This is a topic for another time. The functioning of Loopify will be much clearer next time we talk about it.

Loopify is only applied automatically when the tail-recursive nature of a function call is visible in the source from the get-go. However, the optimisations applied by Loopify can still very much be useful in other situations as seen in this section. That is why we have the [@loop] attribute in order to enforce loopification. Good canonical examples for applying Loopify with the [@loop] attribute would be either of the following: loopifying a partially tail-recursive function (i.e, a function with only some tail-recursive paths), or for functions which are not obviously tail-recursive in the source code, but could become so after some optimisation steps.

This transformation illustrates a core principle behind the Flambda2 design: applying a somewhat naïve optimisation that is not transformative by itself, but changes the way the compiler can look at the code and trigger a whole lot of other useful ones. Conversely, it being triggered in the middle of the inlining phase can allow some non-obvious cases to become radically better. Coding a single optimisation that would discover the cases demonstrated in the examples above would be quite complex, while this one is rather simple thanks to these principles.

Throughout the entire series of snippets, we will continue seeing these principles in action, starting with the next blog post that will introduce Downward and Upward Traversals.

Stay tuned, and thank you for reading, until next time, see you Space Cowboy. 🤠

Fixing and Optimizing the GnuCOBOL Preprocessor

2024-04-30T13:56:36Z

In this post, I will present some work that we did on the GnuCOBOL compiler, the only fully-mature open-source compiler for COBOL. It all started with a bug issued by one of our customers that we fixed by improving the preprocessing pass of the compiler. We later went on and optimised it to get better performances than the initial version.

Supporting the GnuCOBOL compiler has become one of our commercial activities. If you are interested in this project, we have a dedicated website on our SuperBOL offer, a set of tools and services to ease deploying GnuCOBOL in a company to replace proprietary COBOL environments.

At OCamlPro, we often favor correctness over performance. But at the end, our software is correct AND often faster than its competitors! Optimizing software is an art, that often contradicts popular beliefs.

Table of contents

Preprocessing and Replacements in COBOL
Preprocessing in the GnuCOBOL Compiler
Conformance to the ISO Standard
Preprocessing with Automata on Streams
Some Performance Issues
Optimising Allocations
What about Fast Paths ?
Conclusion

Preprocessing and Replacements in COBOL

COBOL was born in 1959, at a time where the science of programming languages was just starting. If you had to design a new language for the same purpose today, the result would be very different, you would do different mistakes, but maybe not fewer. Actually, COBOL has shown to be particularly resilient to time, as it is still used, 70 years later! Though it has evolved over the years (the last ISO standard for COBOL was released in January 2023), the kernel of the language is still the same, showing that most of the initial design choices were not perfect, but still got the job done.

One of these choices, which would sure scare off young developers, is how COBOL favors code reusability and sharing, through replacements done in its preprocessor.

Let's consider this COBOL code, this will be our example for the rest of this article:

DATA DIVISION.
WORKING-STORAGE SECTION.
  01 VAL1.
    COPY MY-RECORD REPLACING ==:XXX:== BY ==VAL1==.
  01 VAL2.
    COPY MY-RECORD REPLACING ==:XXX:== BY ==VAL2==.
  01 COUNTERS.
     05 COUNTER-NAMES  PIC 999 VALUE 0.
     05 COUNTER-VALUES PIC 999 VALUE 0.

We are using the free format, a modern way of formatting code, the older fixed format would require to leave a margin of 7 characters on the left. We are in the DATA division, the part of the program that defines the format of data, and specifically, in the WORKING-STORAGE section, where global variables are defined. In standard COBOL, there are no local variables, so the WORKING-STORAGE section usually contains all the variables of the program, even temporary ones.

In COBOL, there are variables of basic types (integers and strings with specific lengths), and composite types (arrays and records). Records are defined using levels: global variables are at level 01 (such as VAL1, VAL2 and COUNTERS in our example), whereas most other levels indicate inner fields: here, COUNTER-NAMES and COUNTER-VALUES are two direct fields of COUNTERS, as shown by their lower level 05 (both are actually integers of 3 digits as specified by PIC 999). Moreover, COBOL programmers like to be able to access fields directly, by making them unique in the program: it is thus possible to use COUNTER-NAMES everywhere in the program, without refering to COUNTERS itself (note that if the field wasn't assigned a unique name, it would be possible to use COUNTER-NAMES OF COUNTERS to disambiguate them).

On the other hand, in older versions of COBOL, there were no type definitions.

So how would one create two record variables with the same content?

One would use the preprocessor to include the same file several times, describing the structure of the record into your program. One would also use that same file to describe the format of some data files storing such records. Actually, COBOL developers use external tools that are used to manage data files and generate the descriptions, that are then included into COBOL programs in order to manipulate the files (pacbase for example is one such tool).

In our example, there would be a file MY-RECORD.CPY (usually called a copybook), containing something like the following somewhere in the filesystem:

05 :XXX:-USERNAME PIC X(30).
05 :XXX:-BIRTHDATE.
  10 :XXX:-BIRTHDATE-YEAR PIC 9999.
  10 :XXX:-BIRTHDATE-MONTH PIC 99.
  10 :XXX:-BIRTHDATE-MDAY PIC 99.
05 :XXX:-ADDRESS PIC X(100).

This code except is actually not really correct COBOL code because identifiers cannot contain a :XXX: part:. It was written instead for it to be included and modified in other COBOL programs.

Indeed, the following line will include the file and perform a replacement of a :XXX: partial token by VAL1:

COPY MY-RECORD REPLACING ==:XXX:== BY ==VAL1==.

So, in our main example, we now have two global record variables VAL1 and VAL2, of the same format, but containing fields with unique names such as VAL1-USERNAME and VAL2-USERNAME.

Allow me to repeat that, despite pecular nature, these features have stood the test of the time.

The journey continues. Suppose now that you are in a specific part of your program, and that wish to manipulate longer names, say, you would like the :XXX:-USERNAME variable to be of size 60 instead of 30.

Here is how you could do it:

  [...]
REPLACE ==PIC X(30)== BY ==PIC X(60)==.
  01 VAL1.
    COPY [...]
REPLACE OFF.
  01 COUNTERS.
  [...]

Here, we can replace a list of consecutive tokens PIC X(30) by another list of tokens PIC X(60). The result is that the fields VAL1-USERNAME and VAL2-USERNAME are now 60 bytes long.

REPLACE and COPY REPLACING can both perform the same kind of replacements on both parts of tokens (using LEADING or TRAILING keywords) and lists of tokens. COBOL programmers combine them to perform their daily job of building consistent software, by sharing formats using shared copybooks.

Let's see now how GnuCOBOL can deal with that.

Preprocessing in the GnuCOBOL Compiler

The GnuCOBOL compiler is a transpiler: it translates COBOL source code into C89 source code, that can then be compiled to executable code by a C compiler. It has two main benefits: high portability, as GnuCOBOL will work on any platform with any C compiler, including very old hardware and mainframes, and simplicity, as code generation is reduced to its minimum, most of the code of the compiler is its parser... Which is actually still huge as COBOL is a particularly rich language.

GnuCOBOL implements many dialects, (i.e.: extensions of COBOL available on proprietary compilers such as IBM, MicroFocus, etc.), in order to provide a solution to the migration issues posed by proprietary platforms.

The support of dialects is one of the most interesting features of GnuCOBOL: by supporting natively many extensions of proprietary compilers, it is possible to migrate applications from these compilers to GnuCOBOL without modifying the sources, allowing to run the same code on the old platform and the new one during all the migration.

One of OCamlPro's main contributions to GnuCOBOL has been to create such a dialect for GCOS7, a former Bull mainframe still in use in some places.

This is a Bull DPS-7 mainframe around 1980, running the GCOS7 operating system. Such systems are still used to run COBOL critical applications in some companies, though running on software emulators on PCs. GnuCOBOL is a mature solution to migrate such applications to modern Linux computers.

To perform its duty, GnuCOBOL processes COBOL source files in two passes: it preprocesses them during the first phase, generating a new temporary COBOL file with all inclusions and replacement done, and then parses this file and generates the corresponding C code.

To do that, GnuCOBOL includes two pairs of lexers and parsers, one for each phase. The first pair only recognises a very limited set of constructions, such as COPY... REPLACING, REPLACE, but also some other ones like compiler directives.

The lexer/parser for preprocessing directly works on the input file, and performed all these operations in a single pass before version 3.2.

The output can be seen using the -E argument:

$ cobc -E --free foo.cob
#line 1 "foo.cob"
DATA DIVISION.
WORKING-STORAGE SECTION.
 
 01 VAL1.
 
#line 1 "MY-RECORD.CPY"
05 VAL1-USERNAME PIC X(60).
05 VAL1-BIRTHDATE.
 10 VAL1-BIRTHDATE-YEAR PIC 9999.
 10 VAL1-BIRTHDATE-MONTH PIC 99.
 10 VAL1-BIRTHDATE-MDAY PIC 99.
05 VAL1-ADDRESS PIC X(100).
#line 5 "foo.cob"

 01 VAL2.
 
#line 1 "MY-RECORD.CPY"
05 VAL2-USERNAME PIC X(60).
05 VAL2-BIRTHDATE.
 10 VAL2-BIRTHDATE-YEAR PIC 9999.
 10 VAL2-BIRTHDATE-MONTH PIC 99.
 10 VAL2-BIRTHDATE-MDAY PIC 99.
05 VAL2-ADDRESS PIC X(100).
#line 7 "foo.cob"

 
 01 COUNTERS.
 05 COUNTER-NAMES PIC 999 VALUE 0.
 05 COUNTER-VALUES PIC 999 VALUE 0.

The -E option is particularly useful if you want to understand the final code that GnuCOBOL will compile. You can also get access to this information using the option --save-temps (save intermediate files), in which case cobc will generate a file with extension .i (foo.i in our case) containing the preprocessed COBOL code.

You can see that cobc successfully performed both the REPLACE and COPY REPLACING instructions.

The corresponding code in version 3.1.2 is in file cobc/pplex.l, function ppecho. Fully understanding it is left as an exercice for the motivated reader.

The general idea is that replacements defined by COPY REPLACING and REPLACE are added to the same list of active replacements.

We show in the next section that such an implementation does not conform to the ISO standard.

Conformance to the ISO Standard

You may wonder if it is possible for REPLACE statements to perform replacements that would change a COPY statement, such as :

REPLACE ==COPY MY-RECORD== BY == COPY OTHER-RECORD==.
COPY MY-RECORD.

You may also wonder what happens if we try to combine replacements by COPY and REPLACE on the same tokens, for example:

REPLACE ==VAL1-USERNAME PIC X(30)== BY ==VAL1-USERNAME PIC X(60)==

Such a statement only makes sense if we assume the COPY replacements have been performed before the REPLACE replacements are performed.

Such ambiguities have been resolved in the ISO Standard for COBOL: in section 7.2.1. Text Manipulation >> General, it is specified that preprocessing is executed in 4 phases on the streams of tokens:

1. `COPY` statements are performed, and the corresponding `REPLACING`
   replacements too;
2. Conditional compiler directives are then performed;
3. `REPLACE` statements are performed;
4. `COBOL-WORDS` statements are performed (allowing to enabled/disable
   some keywords)

So, a REPLACE cannot modify a COPY statement (and the opposite is also impossible, as REPLACE are not allowed in copybooks), but it can modify the same set of tokens that are being modified by the REPLACING part of a COPY.

The ISO standard specifies the different steps to preprocess COBOL files and perform replacements in a specific order.

As described in the previous section, GnuCOBOL implements all phases 1, 2 and 3 in a single one, even mixing replacements defined by COPY and by REPLACE statements. Fortunately, this behavior is good enough for most programs. Unfortunately, there are still programs that combine COPY and REPLACE on the same tokens, leading to hard to debug errors, as the compiler does not conform to the specification.

A difficult situation which happened to one of our customers and that we prompty addressed by patching a part of the compiler.

Preprocessing with Automata on Streams

Correctly implementing the specification written in the standard would make the preprocessing phase quite complicated. Indeed, we would have to implement a small parser for every one of the four steps of preprocessing. That's actually what we did for our COBOL parser in OCaml used by the LSP (Language Server Protocol) of our SuperBOL Studio COBOL plugin for VSCode.

However, doing the same in GnuCOBOL is much harder: GnuCOBOL is written in C, and such a change would require a complete rewriting of the preprocessor, something that would take more time than we had on our hands. Instead, we opted for rewriting the replacement function, to split COPY REPLACING and REPLACE into two different replacement phases.

The corresponding C code has been moved into a file cobc/replace.c. It implements an automaton that applies a list of replacements on a stream of tokens, returning another stream of tokens. The preprocessor is thus composed of two instances of this automaton, one for COPY REPLACING statements and another one for REPLACE statements.

The second instance takes the stream of tokens produced by the first one as input. The automaton is implemented using recursive functions, which is particularly suitable to allow reasoning about its correctness. Actually, several bugs were found in the former C implementation while designing this automaton. Each automaton has an internal state, composed of a set of tokens which are queued (and waiting for a potential match) and a list of possible replacements of these tokens.

Thanks to this design, it was possible to provide a working implementation in a very short delay, considering the complexity of that part of the compiler.

We added several tests to the testsuite of the compiler for all the bugs that had been detected in the process to prevent regressions in the future, and the corresponding pull request was reviewed by Simon Sobisch, the GnuCOBOL project leader, and later upstreamed.

Some Performance Issues

Unfortunately, it was not the end of the work: Simon performed some performance evaluations on this new implementation, and although it had improved the conformance of GnuCOBOL to the standard, it did affect the performance negatively.

Compiler performance is not always critical for most applications, as long as you compile only individual COBOL source files. However, some source files can become very big, especially when part of the code is auto-generated. In COBOL, a typical case of that is the use of a pre-compiler, typically for SQL. Such programs contain EXEC SQL statements, that are translated by the SQL pre-compiler into much longer COBOL code, consisting mostly of CALL statements calling C functions into the SQL library to build and execute SQL requests.

For such a generated program, of a whopping 700 kLines, Simon noticed an important degradation in compilation time, and profiling tools concluded that the new preprocessor implementation was responsible for it, as shown in the flamegraph below:

perf stats visualised on hotspot: the horizontal axis is the total duration. We can see that ppecho, the function for replacements, takes most of the preprocessing time, with the two-automata replacement phases. Credit: Simon Sobisch" src="/blog/assets/img/cobc-callgraph-pplex1.png"/>

A flamegraph generated by perf stats visualised on hotspot: the horizontal axis is the total duration. We can see that ppecho, the function for replacements, takes most of the preprocessing time, with the two-automata replacement phases. Credit: Simon Sobisch

So we started investigating to fix the problem in a new pull-request.

Optimizing Allocations

Our first intuition was that the main difference with the previous implementation came from allocating too many lists in the temporary state of the two automatons. This intuition was only partially right, as we will see.

Mutable lists were used in the automaton (and also in the former implementation) to store a small part of the stream of tokens, while they were being matched with a replacement source. On a partial match, the list had to wait for additionnal tokens to check for a full match. Actually, these lists were used as queues, as tokens were always added at the end, while matched or un-matched tokens were removed from the top. Also, the size of these lists was bounded by the maximal replacement that was defined in the code, that would unlikely be more than a few dozen tokens.

Our first idea was to replace these lists by real queues, that can be efficiently implemented using circular buffers and arrays. Each and every allocation of a new list element would then be replaced by the single allocation of a circular buffer, granted with a few possible reallocations further down the road if the list of replacements was to grow bigger.

The results were a bit disappointing: on the flamegraph, there was some improvement, but the replacement phase still took a lot of time:

token_list_add. But our work is not yet finished! Credit: Simon Sobisch" src="/blog/assets/img/cobc-callgraph-pplex2.png"/>

The flamegraph is better, as shown by the disappearance of calls to token_list_add. But our work is not yet finished! Credit: Simon Sobisch

Another intuition we had was that we had been a bit naive about allocating tokens: in the initial implementation of version 3.1.2, tokens were allocated when copied from the lexer into the single queue for replacement; in our implementation, that job was also done, but twice, as they were allocated in both automata. So, we modified our implementation to only allocate tokens when they are first entered in the COPY REPLACING stream, and not anymore when entering the REPLACE stream. A simple idea, that reduced again the remaining allocations by a factor of 2.

Yet, the new optimised implementation still didn't match the performance of the former 3.1.2 version, and we were running out of ideas on how the allocations performed by the automata could again be improved:

Using circular buffers instead of mutable lists for queues decreased allocations by a factor of 3. Removing the re-allocations between the two streams would also improve it by a factor of 2. A nice improvement, but not yet the performances of version 3.1.2

What about Fast Paths ?

So we decided to study some of the code from 3.1.2 to understand what could cause such a difference, and it became immediately obvious: the former version had two fast paths, that we had left out of our own implementation!

The two fast paths that completely shortcut the replacement mechanisms are the following:

The first one is when there are no replacements defined in the source. In COBOL, most replacements are only performed in the DATA DIVISION, and moreover, COPY REPLACING ones are only performed during copies. This means that a large part of the code that did not need to go through our two automata still did!

The second fast path is for spaces: replacements always start and finish by a non-space token in COBOL, so, if we check that we are not in the middle of partial match (i.e. both internal token queues are empty), we can safely make the space token skip the automata. Again, given the frequency of space tokens (about half, as there are very few other separators), this fast path is likely to be used very, very frequently.

Implementing them was straigthforward, and the results were the one expected:

After implementing the same fast paths as in 3.1.2, the flamegraph is back to normal, with the time spent in the replacement function being almost not noticeable. Credit: Simon Sobisch

Conclusion

As often with optimisations, intuitions do not always lead to the expected improvements: in our case, the real improvement came not with improving the algorithm, but from shortcutting it!

Yet, we are still very pleased by the results: the new optimised implementation of replacements in GnuCOBOL makes it more conformant to the standard, and also more efficient than the former 3.1.2 version, as shown by the final results sent to us by Simon:

These results show that the new implementation is now a little better than 3.1.2. It comes from using the circular buffers instead of the mutable lists for queues, but the optimisation only happens when replacements are defined, which is a very small part of the code source.

OCaml Backtraces on Uncaught Exceptions

2024-04-25T13:56:36Z

A mystical Camel using its net to catch all uncaught... Butterflies.

Uncaught exception: Not_found

This blog post probably won't teach anything new to OCaml veterans; but for the others, you might be glad to learn that this very basic, yet surprisingly little-known feature of OCaml will give you backtraces with source file positions on any uncaught exception.

Since it can save hours of frustrating debugging, my intent is to give some publicity to this accidentally hidden feature.

PSA: define OCAMLRUNPARAM=b in your environment.

For those wanting to go further, I'll then go on with hints and guidelines for good exception management in OCaml.

For the details, everything here is documented in the Printexc module.

Table of contents

Uncaught exception: Not_found
Get your stacktraces!
Improve your traces
- Properly Re-raising exceptions, and finalisers
- There are holes in my backtrace!
Guidelines for exception handling, and Control-C
- Controlling the backtraces from OCaml

Get your stacktraces!

Compile-time errors are good, but sometimes you just have to cope with run-time failures.

Here is a simple (and buggy) program:

let dict = [
    "foo", "bar";
    "foo2", "bar2";
]

let rec replace = function
  | [] -> []
  | w :: words -> List.assoc w dict :: words

let () =
  let words = Array.to_list Sys.argv in
  List.iter print_endline (replace words)

Side note

For purposes of the example, we use List.assoc here; this relies on OCaml's structural equality, which is often a bad idea in projects, as it can break in surprising ways when the matched type gets more complex. A more serious implementation would use e.g. Map.Make with an explicit comparison function.

Here is the result of executing this program with no options:

$ ./foo
Fatal error: exception Not_found

This isn't very helpful, but no need for a debugger, lots of printf or tedious debugging, just do the following:

$ export OCAMLRUNPARAM=b
$ ./foo
Fatal error: exception Not_found
Raised at Stdlib__List.assoc in file "list.ml", line 191, characters 10-25
Called from Foo.replace in file "foo.ml", line 8, characters 18-35
Called from Foo in file "foo.ml", line 12, characters 26-41

Much more helpful! In most cases, this will be enough to find and fix the bug.

If you still don't get the backtrace, you may need to recompile with -g (with dune, ensure your default profile is dev or specify --profile=dev)

So, now we know where the failure occured... But not on what input. This is not a matter of backtraces: if that's an issue, define your own exceptions, with arguments, and raise that rather than the basic Not_found.

Hint

If you run the program directly from your editor, with a properly configured OCaml mode, the file positions in the backtrace should be parsed and become clickable, making navigation very quick and easy.

Improve your traces

The above works well in general, but depending on the complexity of the programs, there are some more advanced tricks that may be helpful, to preserve or improve the backtraces.

Properly Re-raising exceptions, and finalisers

It's pretty common to want a finaliser after some processing, here to remove a temporary file:

let with_temp_file basename (f: unit -> 'a) : 'a =
  let filename = Filename.temp_file basename in
  match f filename with
  | result ->
    Sys.remove filename;
    result
  | exception e ->
    Sys.remove filename;
    raise e

In simple cases this will work, but if e.g. you are using the Printf module before re-raising, it will break the printed backtrace.

Solution 1: use Fun.protect ~finally f that handles the backtrace properly.

Solution 2: manually, use raw backtrace access from the Printexc module:

| exception e ->
  let bt = Printexc.get_raw_backtrace () in
  Sys.remove filename;
  Printexc.raise_with_backtrace e bt

Re-raising exceptions after catching them should always be done in this way.

There are holes in my backtrace!

Indeed, it may appear that not all function calls show up in the backtrace.

There are two main reasons for that:

functions can get inlined by the compiler, so they don't actually appear in the concrete backtrace at runtime;
tail-call optimisation also affects the stack, which can be visible here;

Don't run and disable all optimisations though! Some effort has been put in recording useful debugging information even in these cases. The Flambda pass of the compiler, which does more inlining, also actually makes it more traceable.

As a consequence, switching to Flambda will often give you more helpful backtraces with recursive functions and tail-calls. It can be done with opam install ocaml-option-flambda (this will recompile the whole opam switch).

Well, what if my program uses lwt?

Backtraces in this context are a complex matter -- but they can be simulated: a good practice is to use ppx_lwt and the let%lwt syntax rather than let* or Lwt.bind directly, because the ppx will insert calls that reconstruct "fake" backtrace information.

Guidelines for exception handling, and Control-C

Exceptions in OCaml can happen anywhere in the program: besides uses of raise, system errors can trigger them. In particular, if you want to implement clean termination on the user pressing Control-C without manually handling signals, you should call Sys.catch_break true ; you will then get a Sys.Break exception raised when the user interrupts the program.

Anyway, this is one reason why you must never use try .. with _ ->

let find_opt x m =
  try Some (Map.find x m)
  with _ -> None

The programmer was too lazy to write with Not_found. They may think this is OK since Map.find won't raise anything else. But if Control-C is pressed at the wrong time, this will catch it, and return None instead of stopping the program !

let find_debug x m =
  try Map.find x m
  with e ->
    let bt = Printexc.get_raw_backtrace () in
    Printf.eprintf "Error on %s!" (to_string x);
    Printexc.raise_with_backtrace e bt

This version is OK since it re-raises the exception. If you absolutely need to catch all exceptions, a last resort is to explicitely re-raise "uncatchable" exceptions:

let this_is_a_last_resort =
  try .. with
  | (Sys.Break | Assert_failure _ | Match_failure _) as e -> raise e
  | _ -> ..

In practice, you'll finally want to catch exceptions from your main function (cmdliner already offers to do this, for example); catching Sys.Break at that point will offer a better message than Uncaught exception, give you control over finalisation and the final exit code (the convention is to use 130 for Sys.Break).

Controlling the backtraces from OCaml

Setting OCAMLRUNPARAM=b in the environment works from the outside, but the module Printexc can also be used to enable or disable them from the OCaml program itself.

Printexc.record_backtrace: bool -> unit toggles the recording of backtraces. Forcing it off when running tests, or on when a debug flag is specified, can be good ideas;
Printexc.backtrace_status: unit -> bool checks if recording is enabled. This can be used when finalising the program to print the backtraces when enabled;

Nota Bene

The base library turns on backtraces recording by default. While I salute an attempt to remedy the issue that this post aims to address, this can lead to surprises when just linking the library can change the output of a program (e.g. this might require specific code for cram tests not to display backtraces)

The Printexc module also allows to register custom exception printers: if, following the advice above, you defined your own exceptions with parameters, use Printexc.register_printer to have that information available when they are uncaught.

Opam 102: Pinning Packages

2024-03-25T13:56:36Z

Pins standout. They help us anchor interest points, thus helping us focus on what's important. They become the catalyst for experimentation and help us navigating the strong safety features that opam provides users with.

Welcome, dear reader, to a new opam blog post!

Today we take an additional step down the metaphorical rabbit hole with opam pin, the easiest way to catch a ride on the development version of a package in opam.

We are aware that our readers are eager to see these blog posts venture on the developer side of the opam experience, and so are we, but we need to spend just a bit little more time on the beginner and user-side of it for now so please, bear with us! 🐻

This tutorial is the second one in this on-going series about the OCaml package manager opam. Be sure to read the first one to get up to speed. Also, check out each article's tags to get an idea of the entry level required for the smoothest read possible!

New to the expansive OCaml sphere? As said on the official opam website, opam has been a game changer for the OCaml distribution, since it first saw the day of light here, almost a decade ago.

Table of contents

Tutorial context
Use-case for opam pin
- Pinning a released package development version: opam pin add --dev-repo
- Pinning an unreleased package development version: opam pin add <url>
Dig into opam pin, find spicy features
Conclusion

Tutorial context and basis

As far as context goes for this article, we will consider that you already are familiar with the concepts introduced in our tutorial opam 101.

Your current environment should thus be somewhat similar to the one we had by the end of that tutorial. Meaning: your version of opam is a least 2.1.5 (all outputs were generated with this version), you have already launched opam init, created a global switch my-switch and, possibly, you have even populated it with a few packages with a few calls to the opam install command.

Furthermore, keep in mind that, in this blog post, we are approaching this subject from the perspective of a developer who is looking into integrating new packages to his current workload, not from the perspective of someone who is looking into sharing a project or publishing a new software.

opam pin is a feature that will quickly become necessary for you to use as you continue your exploration of opam. It allows for the user to pin a given package to a specific version, or even change the source from which said package is pulled, installed, and synchronised with from within your currently active switch.

This feature shines the most in contexts such as:

when doing ordinary switch management;
for incorporating external, still under-construction, libraries to your own current workload;
when designing a specific switch: pinning a specific package version will make it the main compatibility constraint for that switch, thus tailoring the environment around it in the process.

Reminder

Remember that opam's command-line interface is beginner friendly. You can, at any point of your exploration, use the --help option to have every command and subcommand explained. You may also check out the opam cheat-sheet that was released a while ago and still holds some precious insights on opam's CLI.

Use-case for `opam pin`

Now onto today's use-cases for opam pin, the premise is as follows:

The package on which your current development depends on has just had a major update on its development branch. This package is available on the opam repository and its name is hc.

That update introduced a new feature that you would very much like to experiment with for your own on-going project.

However, that feature is still very much a work-in-progress and the maintainers of hc are not about to release their package anytime soon...

That's when opam pin comes in. In this article, we will cover two similar use-cases for opam pin, namely the one dealing with pinning a version of a package that is already available on the opam repository, and that of pinning a version of an unreleased package, directly from its public URL.

After all the basics have been laid out, we will eventually cover some of the more underground ⛏ and dangerous 🔥 features available when pinning packages.

Important Notice

For the sake of convenience and brevity, we will breakdown the opam pin command, and some of its options, by only dealing with addresses that obey the classic definition of the word URL.

However do keep in mind that opam uses a broader definition for that word, going as far as to consider a filesystem path to be a valid string for a URL argument, thus allowing all opam pin calls and options to be valid when manipulating opam packages inside a local filesystem or local network instead of just on the web.

Pinning the dev version of a released package: `opam pin add --dev-repo`

Picking up from the base context: our project depends on hc, and hc has just received an update. The first option available for us to access this fresh update on the hc repository is to use opam pin add --dev-repo <pkg> command.

$ opam pin add --dev-repo hc
[hc.0.3] synchronised (git+https://git.zapashcanon.fr/zapashcanon/hc.git)
hc is now pinned to git+https://git.zapashcanon.fr/zapashcanon/hc.git (version 0.3)

The following actions will be performed:
  ∗ install dune 3.14.0 [required by hc]
  ∗ install hc   0.3*
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved hc.0.3  (no changes)
⬇ retrieved dune.3.14.0  (https://opam.ocaml.org/cache)
∗ installed dune.3.14.0
∗ installed hc.0.3
Done.

So what exactly did `opam pin` do here?

$ opam pin add --dev-repo hc
[hc.0.3] synchronised (git+https://git.zapashcanon.fr/zapashcanon/hc.git)

When you feed a package name to the opam pin add --dev-repo command, it will first retrieve the package definition found inside the opam file in the directory of the corresponding package on the the Official OCaml opam repository or any other opam repositories that your local opam installation happens to be synchronised with.

You can inspect said package definition directly yourself with the opam show <pkg> command.

Let's take a look at the package definition for hc:

$ opam show hc

<><> hc: information on all versions ><><><><><><><><><><><><><><><><>
name         hc
all-versions 0.0.1  0.2  0.3

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><>
version      0.3
repository   default
url.src      "https://git.zapashcanon.fr/zapashcanon/hc/archive/0.3.tar.gz"
url.checksum
          "sha256=61b443056adec3f71904c5775b8521b3ac8487df618a8dcea3f4b2c91bedc314"
          "sha512=a1d213971230e9c7362749d20d1bec6f5e23af191522a65577db7c0f9123ea4c0fc678e5f768418d6dd88c1f3689a49cf564b5c744995a9db9a304f4b6d2c68a"
homepage     "https://git.zapashcanon.fr/zapashcanon/hc"
doc          "https://doc.zapashcanon.fr/hc/"
bug-reports  "https://git.zapashcanon.fr/zapashcanon/hc/issues"
dev-repo     "git+https://git.zapashcanon.fr/zapashcanon/hc.git"
authors      "Léo Andrès <contact@ndrs.fr>"
maintainer   "Léo Andrès <contact@ndrs.fr>"
license      "ISC"
depends      "dune" {>= "3.0"} "ocaml" {>= "4.14"} "odoc" {with-doc}
synopsis     Hashconsing library
description  hc is an OCaml library for hashconsing. It provides
             easy ways to use hashconsing, in a type-safe and
             modular way and the ability to get forgetful
             memoïzation.

Here, you can see the dev-repo field which contains the URL of the development repository of that package. Opam will use that information to retrieve package sources for you.

hc is now pinned to git+https://git.zapashcanon.fr/zapashcanon/hc.git (version 0.3)

Once it has retrieved hc sources, opam will then store the status of the pin internally, which is that hc is git pinned to url git.zapashcanon.fr/zapashcanon/hc at version 0.3.

$ opam pin list
hc.0.3    git  git+https://git.zapashcanon.fr/zapashcanon/hc.git

Did you know? The default behaviour for opam pin is the list option. The option to see all pinned packages in the current active switch.

On the other hand, the default behaviour for opam pin <target> command is the add option. Keep it in mind if you happen to grow tired of typing opam pin add <target> every time.

Opam will then analyse hc dependencies and compute a solution that respects the dependencies constraints and state of your current switch (i.e. the compatibility constraints between the packages currently installed in your switch).

If it manages to do so, it will come forth with a prompt to install the pinned package and its dependencies.

The following actions will be performed:
  ∗ install dune 3.14.0 [required by hc]
  ∗ install hc   0.3*
===== ∗ 2 =====
Do you want to continue? [Y/n] y

Pressing Enter or y + Enter will perform the installation.

Notice that sometimes a * character is found next to some package actions? It's the shorthand signal that the package is pinned, you can get that information at a quick glance when opam outputs the actions to perform for you if you know what to look for.

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved hc.0.3  (no changes)
⬇ retrieved dune.3.14.0  (https://opam.ocaml.org/cache)
∗ installed dune.3.14.0
∗ installed hc.0.3
Done.

Congratulations, you now have a pinned development version of the hc package. You can now start exploring the neat feature you have been looking forward to!

Pinning the dev version of an unreleased package: `opam pin add <url>`

Every once in a while on your OCaml journey, you will come across unreleased software.

These OCaml programs and libraries can still very much have active repositories but their maintainers have not yet gone as far as to release them in order to distribute their work through opam to the rest of the OCaml ecosystem.

Yet, you might still want to have seamless access to these software solutions on your local opam installation for your own personal enjoyment and developments. That's when opam pin add <url> comes in handy.

Modern OCaml projects will most often have one or several opam files in their tree which opam can operate with.

$ opam pin git+https://github.com/rjbou/opam-otopop
Package opam-otopop does not exist, create as a NEW package? [Y/n] y
opam-otopop is now pinned to git+https://github.com/rjbou/opam-otopop (version 0.1)

The following actions will be performed:
  ∗ install opam-client 2.0.10 [required by opam-otopop]
  ∗ install opam-otopop 0.1*
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved opam-client.2.0.10  (https://opam.ocaml.org/cache)
∗ installed opam-client.2.0.10
∗ installed opam-otopop.0.1
Done.

As you can see, the course of an opam pin add <url> call is very close to that of an opam pin add --dev-repo <pkg>, the only exception being the following line:

Package opam-otopop does not exist, create as a NEW package? [Y/n] y

Since the package is unavailable on the opam repositories that your opam installation is synchronised with, opam doesn't know about it.

That's why it will ask you if you want to create it as a NEW package.

Once pinned, that package is available in your switch as any other ordinarily available repository package.

You can see here that opam has pinned the opam-otopop package to a specific 0.1 version.

opam-otopop is now pinned to git+https://github.com/rjbou/opam-otopop (version 0.1)

The reason for that is found inside the opam file at the root of the source repository for that package:

version: "0.1"

In any instance where this specific field is not found in the opam file, the version name would then be pinned to the verbatim ~dev version.

Dig into opam pin, find spicy features

Add a pin without installing with `--no-action`

Here are the two main use-cases for a call to opam pin with the --no-action option:

You don't want to install a package immediately, but do want to inform opam of its existence to allow opam to keep the compatibility constraints of that specific package in the equation whenever you are undertaking operations that would require such calculations;
You just want to be assured that your package will be synchronised with the right sources;

--no-action will only perform the first actions of an opam pin call and will quit before installing the package, it can be used with all pin subcommands.

$ opam pin add hc --dev-repo --no-action
[hc.0.3] synchronised (git+https://git.zapashcanon.fr/zapashcanon/hc.git)
hc is now pinned to git+https://git.zapashcanon.fr/zapashcanon/hc.git (version 0.3)
$

Update your pinned packages

There are two ways to go about updating and upgrading your pinned packages. They are the same no matter if you used the --dev-repo option, or <url> argument, or any other method for pinning them.

The first one you may consider is to either install, or reinstall the specific package(s). The reason is that opam will always first synchronise with the linked source, and then proceed to recompiling.

$ opam install opam-otopop

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
[opam-otopop.0.1] synchronised (git+https://github.com/rjbou/opam-otopop#master)

The following actions will be performed:
  ↻ recompile opam-otopop 0.1*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   opam-otopop.0.1
∗ installed opam-otopop.0.1
Done.

In the above code block, opam-otopop has been upgraded by that opam install call.

The second method is to use the specific opam update and opam upgrade mechanisms. These commands are very common in an opam abiding workflow. Their general usage was briefly mentioned in our article opam 101.

By default, opam update updates the state of your opam repositories, for you to have access to the most recent version of your packages. If you add the --development flag to it, it will also update the source code of your pinned packages internally.

$ opam update --development

<><> Synchronising development packages <><><><><><><><><><><><><><><><><><><><>
[opam-otopop.0.1] synchronised (git+https://github.com/rjbou/opam-otopop#master)
Now run 'opam upgrade' to apply any package updates.

Then you run upgrade as you would in any other package upgrade scenario.

$ opam upgrade
The following actions will be performed:
  ↻ recompile opam-otopop 0.1* [upstream or system changes]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   opam-otopop.0.1
∗ installed opam-otopop.0.1
Done.

Unpin packages

When you are done with your experimentation and wish to remove a pinned package, you can simply call the remove subcommand.

Keep in mind that opam unpin is an alias for opam pin remove.

The behaviour of opam unpin is slightly different between released and unreleased packages.

Released packages

If the pinned package is released, by default, opam will retrieve and install the released version of the package instead of removing that package altogether.

$ opam pin list
hc.0.3    git  git+https://git.zapashcanon.fr/zapashcanon/hc.git

$ opam list hc
# Packages matching: name-match(hc) & (installed | available)
# Package # Installed # Synopsis
hc.0.3    0.3         pinned to version 0.3 at git+https://git.zapashcanon.fr/zapashcanon/hc.git

$ opam pin remove hc
Ok, hc is no longer pinned to git+https://git.zapashcanon.fr/zapashcanon/hc.git (version 0.3)
The following actions will be performed:
  ↻ recompile hc 0.3
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved hc.0.3  (https://opam.ocaml.org/cache)
⊘ removed   hc.0.3
∗ installed hc.0.3
Done.

$ opam list hc
# Packages matching: name-match(hc) & (installed | available)
# Package # Installed # Synopsis
hc.0.3    0.3         Hashconsing library

As we can see in the details:

⬇ retrieved hc.0.3  (https://opam.ocaml.org/cache)

opam has retrieved the sources from the archive that is specified in the opam file of the relevant opam repository, thus pulling hc back down to its latest available, current-switch compatible, release.

Notice the absence of the * character next to the package action? It means the package is no longer pinned.

Unreleased packages

On the other hand, an unreleased package, since its only definition source—meaning both the location of its source code as well as all information required for opam to operate, found in the corresponding opam file—is the pin itself, opam will have no other choice than to offer to remove it for you.

$ opam pin list
opam-otopop.0.1    git  git+https://github.com/rjbou/opam-otopop#master

In this case, opam unpin <package-name> (or idempotently: opam pin remove <package-name>) launches an opam remove action:

$ opam pin remove opam-otopop
Ok, opam-otopop is no longer pinned to git+https://github.com/rjbou/opam-otopop#master (version 0.1)
The following actions will be performed:
  ⊘ remove opam-otopop 0.1
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   opam-otopop.0.1
Done.

Unpin but do no action

Just like with the opam pin add command, the --no-action option is available when removing pins. It will only unpin the package, without removing it, or recompiling it.

$ opam pin remove opam-otopop --no-action
Ok, opam-otopop is no longer pinned to git+https://github.com/rjbou/opam-otopop#master (version 0.1)

$ opam list opam-otopop
# Packages matching: name-match(opam-otopop) & (installed | available)
# Package      # Installed # Synopsis
opam-otopop.0.1 0.1         An opam-otopop package

You may use it for removing the pin from a package while still keeping it installed in your switch, or replacing it by its opam repository definition version.

The resulting package remains linked to its URL, but it is not considered as pinned, so there will be no update or automatic syncing to follow the changes of the upstream branch.

You may also consider this feature to prepare a specific action, say, as a temporary state. For example, you could unpin several packages in a row, and then proceed to recompiling the whole batch in one go.

One URL to pin them all: handling a multi-package repository

Every example seen so far had but one opam file at the root of their respective work tree (sometimes in a specific opam/ directory).

Yet it is possible for some projects to have several packages distributed by a single repository. An example of this would be the opam project source repository itself. If that is the case, and you pin that URL, the default behaviour is that all the packages defined at that address will be pinned.

Let's take this project.

You can see that several packages are defined: ocp-index and ocp-browser.

Here's how a pin action behaves when given that URL:

$ opam pin add git+https://github.com/OCamlPro/ocp-index
This will pin the following packages: ocp-browser, ocp-index.
Continue? [Y/n] y
ocp-browser is now pinned to git+https://github.com/OCamlPro/ocp-index (version 1.3.6)
ocp-index is now pinned to git+https://github.com/OCamlPro/ocp-index (version 1.3.6)

The following actions will be performed:
  ∗ install ocp-indent  1.8.1  [required by ocp-index]
  ∗ install ocp-index   1.3.6*
  ∗ install ocp-browser 1.3.6*
===== ∗ 3 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved ocp-indent.1.8.1  (https://opam.ocaml.org/cache)
∗ installed ocp-indent.1.8.1
∗ installed ocp-index.1.3.6
∗ installed ocp-browser.1.3.6
Done.

As you can see, this process is exactly the same as before, but with 3 packages in one go.

What if I do not want to pin every package in that repository?

Easy: if you just need one of the packages found at that URL, you can just feed that package name to the opam pin add <package-name> <url> CLI call, just like we did at the beginning of this tutorial!

$ opam pin add ocp-index git+https://github.com/OCamlPro/ocp-index
[ocp-index.1.3.6] synchronised (git+https://github.com/OCamlPro/ocp-index)
ocp-index is now pinned to git+https://github.com/OCamlPro/ocp-index (version 1.3.6)

The following actions will be performed:
  ∗ install ocp-indent 1.8.1  [required by ocp-index]
  ∗ install ocp-index  1.3.6*
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved ocp-indent.1.8.1  (cached)
∗ installed ocp-indent.1.8.1
∗ installed ocp-index.1.3.6
Done.

If you do not know the exact names of these different packages, you may also consider using the very handy opam pin scan command which will lookup the contents repository at the URL and list its opam packages for you:

$ opam pin scan git+https://github.com/OCamlPro/ocp-index
# Name       # Version  # Url
ocp-index    1.3.6      git+https://github.com/OCamlPro/ocp-index
ocp-browser  1.3.6      git+https://github.com/OCamlPro/ocp-index

Setting arbitrary version numbers, toying with fire

As demonstrated earlier, opam will choose a version of the pinned package according to the contents of the opam file.

The important thing to take away from that is, in most usual scenarios, the contents of the opam file are paramount to how opam will calculate compatibility constraints in a given switch.

It is from the information that is hardcoded inside the opam file that opam will be able to take educated decisions whenever changes to the state of your current switch are to be made. There is a way, however, to circumvent that behaviour, that we want to inform you of, even if it entails a bit of precaution.

Naturally, directly tinkering with such a key stability feature like compatibility constraints solving does require you to tread carefully. We will see together some of the pitfalls and things to do that will keep you from finding yourself in confusing situations in regards to the state of your switch and the dependencies within it.

Ready? Lets get acquainted with our first slightly dangerous opam feature:

You are allowed to append an arbitrary version number to the name of the pinned package for opam to incorporate in its calculations, as seen in the following code block:

$ opam pin add directories.1.0 git+https://github.com/ocamlpro/directories --no-action
[directories.1.0] synchronised (git+https://github.com/ocamlpro/directories)
directories is now pinned to git+https://github.com/ocamlpro/directories (version 1.0)

In this specific example, package directories is available in the opam repository, that our opam installation is synchronised with. However, there is no such 1.0 version in that repository. Not a single reference to such a version number can be found at that address, neither in the tags, nor releases of the repository, and not even in the opam file.

$ opam show directories --field all-versions
0.1  0.2  0.3  0.4  0.5

What we have done here is effectively telling opam that directories is at a different version number than it actually is in the most purely technical aspect...

But why would we want to do such a thing?

Let's consider a reasonable use-case for opam pin add <package>.<my-version-number> <url>:

You have been working on a project called my-project for some time and you are using a package named fst-dep for your development.

Below, you will find an excerpt of the fst-dep.opam file, specifically its dependencies:

depends: [
  "dep-to-try" { <= "3.0.0" }
  "other-dep"
]

All three packages (fst-dep,dep-to-try and other-dep) are installed in your current switch and are available on your favourite opam repository.

One day you go about checking the repository for each dependency, and you find that dep-to-try has just had one of its main features reimplemented, improved and optimised, they are preparing to release a 4.0.0 version soon.

See, these changes would have been available for you to fetch directly from it's development repository had you been working with it directly, but you are not. It is up to the maintainers of fst-dep to do that work.

Since you have no ownership over any of these dependencies. You have no way of changing any of the version constraints in this tiny dependency tree that ranges from fst-dep and upwards.

Here are the three mainstream solutions to this problem:

Wait for both packages to publish new releases. A new official release from the dep-to-try team, which would ship said reimplementation, and another from the fst-dep team which would update its dependency tree to include dep-to-try's latest version. Needless to say that this could take an arbitrary amount of time which is unsatisfying at best.
Another suboptimal solution would be to copy the current state of the entire opam repository relevant to your package distribution, go to the corresponding directory for fst-dep inside that repository, relax the hard dependency "dep-to-try" { <= "3.0.0" } and reinstall all the packages that are directly or indirectly affected by that change. A very time consuming task for such a small edit to the global dependency tree.
Last option would be to pin fst-dep, then go about manually editing the dependencies of fst-dep with the opam pin --edit option to relax the dependency. The only pitfall with this solution is that, in a context where dep-to-try is a key package in the OCaml distribution, and many other packages depend on it as well, you might have to do a lot of editing to make your switch a stable environment with all dependency constraints met...

So neither of these solutions fit our needs. They are all unsatisfactory at best and even counter-productive at worst.

That's when arbitrary version pinning shines.

The main benefit of this feature is that it allows for added flexibility in navigating and tweaking the compatibility tree of any opam repository at the switch-level. It provides the user with ways to circumvent all tasks pertaining to a larger operation on the global graph of packages.

$ opam pin dep-to-try.3.0.0 git+https://github.com/OCamlPro/dep-to-try
[dep-to-try.3.0.0] synchronised (file:///home/rjbou/ocamlpro/opam_bps_examples/dep-to-try)
dep-to-try is now pinned to git+https://github.com/OCamlPro/dep-to-try#master (version 3.0.0)

opam will still think that dep-to-try's version is valid ({ <= "3.0.0"}), even if you are synchronised with the state of its development branch, thus giving you access to the latest changes with the minimal amount of manual editing required. Pretty neat, right?

Now, onto the pitfalls that you should keep in mind when tinkering with your dependencies like that.

What kind of predicament awaits you?

You could introduce unforeseen behaviours. This could be anything from errors at compile-time, if dep-to-try's interfaces have changed significantly, to runtime crashes if you're unlucky.
Another source of confusion could arise if you happen to use the opam unpin dep-to-try --no-action command on such a package. After unpinning it, there's a chance that you would later forget it used to be pinned to a development version. There would be little to no way for you to remember which package it was that you had experimented with at some point. You would either have to inspect all you installed packages or even remake a switch from scratch which would not be affected by your reckless arbitrary version pinning and would work just fine after that.

Our advice is rather simple: use this feature with discretion and try to avoid unpinning packages if it's not to reinstall or remove them altogether. If you follow these instructions, you should be safe...

Setting multiple arbitrary version numbers

One last bit of black magic for you to play around with.

Instead of pinning package-name.my-version-number, you may use the --with-version option to pin packages at that URL to an arbitrary version. A key detail is that it is compatible with multiple opam file pinning... Just keep in mind that all the pitfalls mentioned previously apply here too, only with multiple packages at once, which could make it more confusing.

Below, you can see that we are setting all the packages found in that repository to the same version:

$ opam pin add git+https://github.com/OCamlPro/ocp-index --with-version 2.0.0
This will pin the following packages: ocp-browser, ocp-index.
Continue? [Y/n] y
ocp-browser is now pinned to git+https://github.com/OCamlPro/ocp-index (version 2.0.0)
ocp-index is now pinned to git+https://github.com/OCamlPro/ocp-index (version 2.0.0)

The following actions will be performed:
  ∗ install ocp-indent  1.8.1  [required by ocp-index]
  ∗ install ocp-index   2.0.0*
  ∗ install ocp-browser 2.0.0*
===== ∗ 3 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved ocp-indent.1.8.1  (cached)
⬇ retrieved ocp-index.2.0.0  (no changes)
⬇ retrieved ocp-browser.2.0.0  (no changes)
∗ installed ocp-indent.1.8.1
∗ installed ocp-index.2.0.0
∗ installed ocp-browser.2.0.0
Done.

You can see that all these packages are pinned to 2.0.0 now.

$ opam pin list
ocp-browser.2.0.0    git  git+https://github.com/OCamlPro/ocp-index
ocp-index.2.0.0      git  git+https://github.com/OCamlPro/ocp-index

Conclusion

Here it is, the opam pin command in most of its glory.

If you have managed to stick this long to read this article, you should no longer feel confused about pinning projects and should now have another of opam's most commonly used feature in your arsenal when tackling your own development challenges!

So it is that we have learned about pinning both released and unreleased packages. Additionally, we showcased several features for orthogonal use-cases: from the more quality of life-oriented calls such as opam show and opam pin scan, to obscure features like arbitrary version pinning as well as ordinary options like --no-action, --dev-repo and subcommands like opam unpin.

We are steadily approaching a level of familiarity with opam that will allow us to get into some really neat features soon.

Be sure to stay tuned with our blog, the journey into the rabbit hole has only started and opam is a deep one indeed!

Thank you for reading,

From 2011, with love,

The OCamlPro Team

Flambda2 Ep. 1: Foundational Design Decisions

2024-03-19T13:56:36Z

Welcome to The Flambda2 Snippets!

In this first post of The Flambda2 Snippets, we dive into the powerful CPS-based internal representation used within the Flambda2 optimizer, which was one of the main motivation to move on from the former Flambda optimizer.

Credit goes to Andrew Kennedy's paper Compiling with Continuations, Continued for pointing us in this direction.

The F2S blog posts aim at gradually introducing the world to the inner-workings of a complex piece of software engineering: The Flambda2 Optimising Compiler, a technical marvel born from a 10 year-long effort in Research & Development and Compilation; with many more years of expertise in all aspects of Computer Science and Formal Methods.

Table of contents

CPS (Continuation Passing Style)
Double Barrelled CPS
The Flambda2 Term Language
Following up

CPS (Continuation Passing Style)

Terms in the Flambda2 IR are represented in CPS style, so let us briefly explain what that means.

Some readers may already be familiar with what we call First-Class CPS where continuations are represented using functions of the language:

(* Non-tail-recursive implementation of map *)
let rec map f = function
| [] -> []
| x :: r -> f x :: map f r

(* Tail-recursive CPS implementation of map *)
let rec map_cps f l k =
match l with
| [] -> k []
| x :: r ->  let fx = f x in map_cps f r (fun r -> k (fx :: r))

This kind of transformation is useful to make a recursive function tail-recursive and sometimes to avoid allocations for functions returning multiple values.

In Flambda2, we use Second-Class CPS instead, where continuations are control-flow constructs in the Intermediate Language. In practice, this is equivalent to an explicit representation of a control-flow graph.

Here's an example using some hopefully intuitive syntax for the Flambda2 IR.

let rec map f = function
| [] -> []
| x :: r -> f x :: map f r

(* WARNING: FLAMBDA2 PSEUDO-SYNTAX INBOUND *)
let rec map
  ((f : <whatever_type1> ),
  (param : <whatever_type2>))
  {k_return_continuation : <return_type>}
{
  let_cont k_empty () = k_return_continuation [] in 
  let_cont k_nonempty x r =
    let_cont k_after_f fx =
      let_cont k_after_map map_result =
        let result = fx :: map_result in
        k_return_continuation result 
      in
      Apply (map f r {k_after_map})
    in
    Apply (f x {k_after_f})
  in
  match param with
  | [] -> k_empty ()
  | x :: r -> k_nonempty x r
}

Every let_cont binding declares a new sequence of instructions in the control-flow graph, which can be terminated either by:

calling a continuation (for example, k_return_continuation) which takes a fixed number of parameters;
applying an OCaml function (Apply), this function takes as a special parameter the continuation which it must jump to at the end of its execution. Unlike continuations, OCaml functions can take a number of arguments that does not match the number of parameters at their definition;
branching constructions like match _ with and if _ then _ else _, in these cases each branch is a call to a (potentially different) continuation;

This image shows the previous code represented as a graph.

Notice that some boxes are nested to represent scoping relations: variables defined in the outer boxes are available in the inner ones.

To demonstrate the kinds of optimisations that such control-flow graphs allow us, see the following simple example:

Original Program:

let f cond =
  let v =
    if cond then
      raise Failure
    else 0
  in
  v, v

We then represent the same program using CPS in two steps, the first is the direct translation of the original program, the second is an equivalent program represented in a more compact form.

Minimal CPS transformation, using pseudo-syntax

(* showing only the body of f *)
(* STEP 1 - Before graph compaction *)
let_cont k_after_if v =
  let result = v, v in
  k_return_continuation result
in
let_cont k_then () = k_raise_exception Failure in 
let_cont k_else () = k_after_if 0 in
if cond then k_then () else k_else ()

which becomes after inlining k_after_if:

(* STEP 2 - After graph compaction *)
let_cont k_then () = k_raise_exception Failure in 
let_cont k_else () =
  let v = 0 in 
  let result = v, v in
  k_return_continuation result
in
if cond then k_then () else k_else ()

This allows us, by using the translation to CPS and back, to transform the original program into the following:

Optimized original program

let f cond =
  if cond then
    raise Failure
  else 0, 0

As you can see, the original program is simpler now. The nature of the changes operated on the code are in fact not tied to a particular optimisation but rather the nature of the CPS transformation itself. Moreover, we do want to actively perform optimisations and to that extent, having an intermediate representation that is equivalent to a control-flow graph allows us to benefit from the huge amount of literature on the subject of static analysis of imperative programs which often are represented as control-flow graphs.

To be fair, in the previous example, we have cheated in how we have translated the raise primitive. Indeed we used a simple continuation (k_raise_exception) but we haven't defined it anywhere prior. This is possible because our use of Double Barrelled CPS.

Double Barrelled CPS

In OCaml, all functions can not only return normally (Barrel 1) but also throw exceptions (Barrel 2), it corresponds to two different paths in the control-flow and we need the ability to represent it in our own control-flow graph.

Hence the name: Double Barrelled CPS, that we took from this paper, by Hayo Thielecke. In practice this only has consequences in four places:

the function definitions must have two special parameters instead of one: the exception continuation (k_raise_exception) in addition to the normal return continuation (k_return_continuation);
the function applications must have two special arguments, reciprocally;
try ... with terms are translated using regular continuations with the exception handler (the with path of the construct) compiled to a continuation handler (let_cont);
raise terms are translated into continuation calls, to either the current function exception continuation (e.g. in case of uncaught exception) or the enclosing try ... with handler continuation.

The Flambda2 Term Language

This CPS form has directed the concrete implementation of the FL2 language.

We can see that the previous IRs have very descriptive representations, with about 20 constructors for Clambda and 15 for Flambda while Flambda2 has regrouped all these features into only 6 categories which are sorted by how they affect the control-flow.

type expr =
  | Let of let_expr
  | Let_cont of let_cont_expr
  | Apply of apply
  | Apply_cont of apply_cont
  | Switch of switch
  | Invalid of { message : string }

The main benefits we reap from such a strong design choice are that:

Code organisation is better: dealing with control-flow is only done when matching on full expressions and dealing with specific features of the language is done at a lower level;
Reduce code duplication: features that behave in a similar way will have their common code shared by design;

Following up

The goal of this article was to show a fundamental design choice in Flambda2 which is using a CPS-based representation. This design is felt throughout the Flambda2 architecture and will be mentioned and strengthened again in later posts.

Flambda2 takes the Lambda IR as input, then performs CPS conversion, followed by Closure conversion, each of them worth their own blog post, and this produces the terms in the Flambda2 IR.

From there, we have our main optimisation pass that we call Simplify which first performs static analysis on the term during a single Downwards Traversal, and then rebuilds an optimised term during the Upwards Traversal.

Once we have an optimised term, we can convert it to the CMM IR and feed it to the rest of the backend. This part is mostly CPS elimination but with added original and interesting work we will detail in a specific snippet.

The single-pass design allows us to consider all the interactions between optimisations

Some examples of optimisations performed during Simplify:

Inlining of function calls;
Constant propagation;
Dead code elimination
Loopification, that is transforming tail-recursive functions into loops;
Unboxing;
Specialisation of polymorphic primitives;

Most of the following snippets will detail one or several parts of these optimisations.

Stay tuned, and thank you for reading!

Behind the Scenes of the OCaml Optimising Compiler Flambda2: Introduction and Roadmap

2024-03-18T13:56:36Z

Introducing our Flambda2 snippets

At OCamlPro, the main ongoing task on the OCaml Compiler is to improve the high-level optimisation. This is something that we have been doing for quite some time now. Indeed, we are the authors behind the Flambda optimisation pass and today we would like to introduce the series of blog snippets showcasing the direct successor to it, the creatively named Flambda2.

This series of blog posts will cover everything about Flambda2, a new optimising backend for the OCaml native compiler. This introductory episode will provide you with some context and history about Flambda2 but also about its predecessor Flambda and, of course, the OCaml compiler!

This work may be considered as a completement to an on-going documentation effort at OCamlPro as well as to the many different talks we have given last year on the subject, two of which you can watch online: OCaml Workshop ( slideshow ), ML Workshop ( slideshow ).

This work was developed in collaboration with, and funded by Jane Street. Warm thanks to Mark Shinwell for shepherding the Flambda project and to Ron Minsky for his support.

Table of contents

Introduction
Compiling OCaml
Snippets Roadmap
The F2S Series!

Compiling OCaml

The compiling of OCaml is done through a multitude of passes (see simplified representation below), and the bulk of high-level optimisations happens between the Lambda IR (Intermediate Representation) and CMM (which stands for C--). This set of optimisations will be the main focus of this series of snippets.

Flambda." src="/blog/assets/img/flambda2_snippets_ep0_figure3_1.png"/>

The different passes of the OCaml compilers, from sources to executable code, before the addition of Flambda.

Indeed, that part of the compiler is quite crowded. Originally, after the frontend has type-checked the sources, the Closure pass was in charge of transforming the Lambda IR (see source code) into the Clambda IR (see source code). This transformation handles Constant Propagation, some inlining, and some Constant Lifting (moving constant structures to static allocation). Then, a subsequent pass (called Cmmgen) transforms the Clambda IR into the CMM IR (see source code) and handles some peep-hole optimisations and unboxing. This final representation will be used by architecture-specific backends to produce assembler code.

Before we get any further into the hairy details of Flambda2 in the upcoming snippets, it is important that we address some context.

We introduced the Flambda framework which was released with OCaml 4.03. This was a success in improving inlining and related optimisations, and has been stable ever since, with very few bug reports.

We kept both Closure and Flambda alive together because some users cared a lot about the compilation speed of OCaml - Flambda is indeed a bit slower than Closure.

Flambda provides an alternative to the classic Closure transformation, with additionnal optimizations." src="/blog/assets/img/flambda2_snippets_ep0_figure3_2.png"/>

Flambda provides an alternative to the classic Closure transformation, with additionnal optimizations.

Now is time to introduce another choice to both Flambda and Closure: Flambda2, which is meant to eventually replace Flambda and potentially Closure as well. In fact, Janestreet has been gradually moving from Closure and Flambda to Flambda2 during the past year and has to this day no more systems relying on Closure or Flambda.

You can read more about the transition from staging to production-level workloads of Flambda2 right here.

Flambda is still maintained and will be for the forseeable future. However, we have noticed some limitations that prevented us from doing some kinds of optimisations and on which we will elaborate in the following episodes of The Flambda2 Snippets series.

Flambda2 provides a much extended alternative to Flambda, from Lambda IR to CMM." src="/blog/assets/img/flambda2_snippets_ep0_figure3.png"/>

Flambda2 provides a much extended alternative to Flambda, from Lambda IR to CMM.

One obvious difference to notice is that Flambda2 translates directly to CMM, circumventing the Clambda IR, allowing us to lift some limitations inherent to Clambda itself.

Furthermore, we experimented after releasing Flambda with the aim to incrementally improve and add new optimisations. We tried to improve its internal representation and noticed that we could gain a lot by doing so, but also that it required deeper changes and that is what led us to Flambda2.

Snippets Roadmap

This is but the zeroth snippet of the series. It aims at providing you with history and context for Flambda2.

You can expect the rest of the snippets to alternate between deep dives into the technical aspects of Flambda2, and user-facing descriptions of the new optimisations that we enable.

The F2S Series!

Episode 1: Foundational Design Decisions in Flambda2

The first snippet covers the characteristics and benefits of a CPS-based internal representation for the optimisation of the OCaml language. It was already covered in part at the OCaml Workshop in 2023 and we go deeper into the subject in these blog posts.
Episode 2: Loopifying Tail-Recursive Functions

Loopify is the first optimisation algorithm that we introduce in the F2S series. In this post, we breakdown the concept of transforming tail-recursive functions in the context of reducing memory allocations inside of the Flambda2 compiler. We start with giving broader context around tail-recursion and tail-recursion optimisation before diving into how this transformation is both simple and representative of the philosophy behind all the optimisations conducted by the Flambda2 compiler.
Episode 3: Speculative Inlining

This article introduces Speculative Inlining, which is the name of the algorithm responsible for computing and inlining optimised function code inside of Flambda2. We cover how quickly we are faced with complex questions with only heuristic answers when it comes down to an optimal inlining choice. Speculative Inlining is also the best demonstration of how we traverse code in our compilation pipeline.
Episode 4: Upward and Downward Traversals

Coming soon...

Stay tuned, and thank you for reading!

Lean 4: When Sound Programs become a Choice

2024-03-07T13:56:36Z

Monitoring Edge Technical Endeavours

As a company specialized in strongly-typed programming languages with strong static guarantees, OCamlPro closely monitors the ongoing trend of bringing more and more of these elements into mainstream programming languages. Rust is a relatively recent example of this trend; another one is the very recent Lean 4 language.

Table of contents

Monitoring Edge Technical Software
Lean 4, the Promise of Proven Software
OCamlPro for a Future of Trustworthy Software

Lean 4, the Promising Future of Proven Software

Lean 4 builds on the shoulders of giants like the Coq proof assistant, and languages such as OCaml and Haskell, to put programmers in a world where they can write elegant programs, express their specification with the full power of modern logics, and prove that these programs are correct with respect to their specification. Doing all this in the same language is crucial as it can streamline the certification process: once Lean 4 is trusted (audits, certification...), then programs, specifications, and proofs are also trusted. This contrasts with having a programming language, a specification language, and a separate verification/certification tool, and then having to argue about the trustworthiness of each of them, and that the glue linking all of them together makes sense. This is extremely interesting in the context of critical embedded systems in particular, and in qualified/certified "high-trust" development in general.

While admittedly not as mainstream as Rust, Lean 4 has recently seen an explosion in interest from the media, developers, mathematicians, and (some) industrials. Quanta now routinely publishes articles about/mentioning Lean 4; Fields medalist Terry Tao is increasingly vocal about (and productive with) its use of Lean 4, see here and here for (very technical) example(s). On the industrial side, Leonardo de Moura (Lean 4's lead designer) recently went from a position at Microsoft Research to Amazon Web Service, which was followed by a fast and still ongoing expansion of the infrastructure around Lean 4.

Pushing for a Future of Trustworthy Software

OCamlPro has been closely monitoring Lean 4's progress by regularly developing in-house prototypes in Lean 4. Getting involved in the community and Lean 4's development effort is also part of our culture. This is to give back to the community, but also to closely follow the evolution of Lean 4 and sharpen our skills.

There are a few notable and public examples of our involvement. As part of our in-house prototyping, we discovered a "major bug" in Lean 4's dependent pattern-matching; later, we contributed on improving aspects of the by notation (used to construct proofs), which then ricocheted into fixing problems into the calc tactic. More recently, we contributed on various fronts such as improving the ecosystem's ergonomics, adding useful lemmas to Lean 4's standard library, contributing to the documentation effort...

Lean 4 is not of industrial-strength yet, but it gets closer and closer. Quickly enough for us to think that now's a reasonable time to spend some time exploring it.

Opam 101: The First Steps

2024-01-23T13:56:36Z

Opam is like a magic box that allows people to be tidy when they share their work with the world, thus making the environment stable and predictable for everybody!

Welcome, dear reader, to a new series of blog posts!

This series will be about everything opam. Each article will cover a specific aspect of the package manager, and make sure to dissipate any confusion or misunderstandings on this keystone of the OCaml distribution!

Each technical article will be tailored for specific levels of engineering -- everyone, be they beginners, intermediate or advanced in the OCaml Arts will find answers to some questions about opam right here.

Checkout each article's tags to get an idea of the entry level required for the smoothest read possible!

Table of contents

Walking the path of opam, treading on solid ground
First step: installing opam
Second step: initialisation
Acclimating to the environment
Switches, tailoring your workspace to your vision
- Creating a global switch
- Creating a local switch
The official opam-repository, the safe for all your packages
Installing packages in your current switch
Conclusion

New to the expansive OCaml sphere? As said on the official opam website, opam has been a game changer for the OCaml distribution, since it first saw the light of day here, almost a decade ago.

Walking the path of opam, treading on solid ground

We are aware that it can be quite a daunting task to get on-board with the OCaml distribution. Be it because of its decentralised characteristics: plethora of different tools, a variety of sometimes clashing modi operandi and practices, usually poorly documented edge use-cases, the variety of ways to go about having a working environment or many a different reason...

We have been thinking about making it easier for everyone, even the more confirmed Cameleers, by releasing a set of blogposts progressively detailing the depths at which opam can go.

Be sure to read these articles from the start if you are new to the beautiful world of OCaml and, if you are already familiar, use it as a trust-worthy documentation on speed-dial... You never know when you will have to setup an opam installation while off-the-grid, do you ?

Are you ready to dive in ?

First step: installing opam

First, let's talk about installing opam.

DISCLAIMER: In this tutorial, we will only be addressing a fresh install of opam on Linux and Mac. For more information about a Windows installation, stay tuned with this blog!

One would expect to have to interact with the package manager of one's favourite distribution in order to install opam, and, to some extent, one would be correct. However, we cannot guarantee that the version of opam you have at your disposal through these means is indeed the one expected by this tutorial, and every subsequent one for that matter.

You can check that here, make sure the version available to you is 2.1.5 or above.

Thus, in order for us to guarantee that we are on the same version, we will use the installation method found here and add an option to specify the version of opam we will be working with from now on.

Note that if you don't add the --version 2.1.5 option to the following command line, the script will download and install the latest opam release. The cli of opam is made to remain consistent between versions so, unless you have a very old version, or if you read this article in the very distant future, you should not have problems by not using the exact same version as we do. For the sake of consistency though, I will use this specific version.

$ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.5"

This script will download the necessary binaries for a proper installation of opam. Once done, you can move on to the nitty gritty of having a working opam environment with opam init.

Second step: initialisation

The first command to launch, after the initial opam binaries have been downloaded and opam has been installed on your system, is opam init.

This is when you step into the OCaml distribution for the first time.

opam init does several crucial things for you when you launch it, and the rest of this article will detail what exactly these crucial things are and what they mean:

it checks some required and recommended tools;
it syncs with the official OCaml opam-repository, which you can find here;
it sets up the opam environment in your *rc files;
it creates a switch and installs an ocaml-compiler for you;

Lets take a step-by-step look at the output of that command:

$ opam init
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs.
Perfect!

<><> Fetching repository information ><><><><><><><><><><><><><><><><>
[default] Initialised

<><> Required setup - please read <><><><><><><><><><><><><><><><><><>

  In normal operation, opam only alters files within ~/.opam.

  However, to best integrate with your system, some environment
  variables should be set. If you allow it to, this initialisation
  step will update your bash configuration by adding the following
  line to ~/.profile:

    test -r ~/.opam/opam-init/init.sh && . ~/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

  Otherwise, every time you want to access your opam installation,
  you will need to run:

    eval $(opam env)

  You can always re-run this setup with 'opam init' later.

Do you want opam to modify ~/.profile? [N/y/f]
(default is 'no', use 'f' to choose a different file) y

User configuration:
  Updating ~/.profile.
[NOTE] Make sure that ~/.profile is well sourced in your ~/.bashrc.


<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

The main result for an opam init call is to setup what is called your opam root. It does so by creating a ~/.opam directory to operate inside of. opam modifies and writes in this location only as a default.

First, opam checks that there is at least one required tool for syncing to the opam-repository. Then it checks what backends are available in your system. Here all are available: rsync, git, mercurial, and darcs. They will be used to sync repositories or packages.

$ opam init
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs.
Perfect!

Then, opam fetches the default opam repository: opam.ocaml.org.

<><> Fetching repository information ><><><><><><><><><><><><><><><><>
[default] Initialised

Secondly, opam requires your input in order to configure your shell for the smoothest possible experience. For more details about the opam environment, refer to the next section.

Something interesting to remember for later is, in the excerpt below, we grant opam with the permission to edit the ~/.profile file. This part of the Quality of Life features for an everyday use an opam environment and we will detail how so below.

<><> Required setup - please read <><><><><><><><><><><><><><><><><><>

  In normal operation, opam only alters files within ~/.opam.

  However, to best integrate with your system, some environment
  variables should be set. If you allow it to, this initialisation
  step will update your bash configuration by adding the following
  line to ~/.profile:

    test -r ~/.opam/opam-init/init.sh && . ~/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

  Otherwise, every time you want to access your opam installation,
  you will need to run:

    eval $(opam env)

  You can always re-run this setup with 'opam init' later.

Do you want opam to modify ~/.profile? [N/y/f]
(default is 'no', use 'f' to choose a different file) y

User configuration:
  Updating ~/.profile.
[NOTE] Make sure that ~/.profile is well sourced in your ~/.bashrc.

The next action is the installation of your very first switch alongside a version of the OCaml compiler, by default a compiler >= 4.05.0 to be exact.

For more information about what is a switch be sure to read the rest of the article.

<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

Great! So let's focus on the actions performed by the opam init call!

Acclimating to the environment

Well, as said previously, the first action was to setup an opam root in your $HOME directory, (i.e: ~/.opam). This is where opam will operate. opam will never modify other locations in your filesystem without notifying you first.

An opam root is made to resemble a linux-like architecture. You will find inside it directories such as /usr, /etc, /bin and so on. This is by default where opam will store everything relative to your system-wide installation. Config files, packages and their configurations, and also binaries.

This leads us to the need for an eval $(opam env) call.

Indeed, in order to make your binaries and such accessible as system-wide tools, you need to update all the relevant environment variables (PATH, MANPATH, etc.) with all the locations for all of your everyday OCaml tools.

To see what variables are exported when evaluating the opam env command, you can check the following codeblock:

$ opam env
OPAM_SWITCH_PREFIX='~/.opam/default'; export OPAM_SWITCH_PREFIX;
CAML_LD_LIBRARY_PATH='~/.opam/default/lib/stublibs:~/.opam/default/lib/ocaml/stublibs:~/.opam/default/lib/ocaml'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='~/.opam/default/lib/toplevel'; export OCAML_TOPLEVEL_PATH;
MANPATH=':~/.opam/default/man'; export MANPATH;
PATH='~/.opam/default/bin:$PATH'; export PATH;

Remember when we granted opam init with the permission to edit the ~/.profile file, earlier in this tutorial ? That comes in handy now: it keeps us from having to use the eval $(opam env) more than necessary.

Indeed, you would otherwise have to call it every time you launch a new shell among other things. What it does instead is adding hook at prompt level that keeps opam environment synced, updating it every time the user presses Enter. Very handy indeed.

Switches, tailoring your workspace to your vision

The second task accomplished by opam init was installing the first switch inside your fresh installation.

A switch is one of opam's core operational concepts, it's definition can vary depending on your exact use-case but in the case of OCaml, a switch is a named pair:

an arbitrary version of the OCaml compiler
a list of packages available for that specific version of the compiler.

In our example, we see that the only packages installed in the process were the dependencies for the OCaml compiler version 5.1.0 inside the switch named default.

<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler)

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.1.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.5.1.0
∗ installed ocaml-config.3
∗ installed ocaml.5.1.0
∗ installed base-domains.base
∗ installed base-nnp.base
Done.

You can create an arbitrary amount of parallel switches in opam. This allows users to manage parallel, independent OCaml environments for their developments.

There are two types of switches:

global switches have their packages, binaries and tools available anywhere on your computer. They are useful when you consider a given switch to be your default and most adequate environment for your everyday use of opam and OCaml.
local switches on the other hand are only available in a given directory. Their packages and binaries are local to that specific directory. This allows users to make specific projects have their own self-contained working environments. The local switch is automatically selected by opam as the current one when you are located inside the appropriate directory. More details on local switches below.

The default behaviour for opam when creating a switch at init-time is to make it global and name it default.

$ opam switch show
default
$ opam switch
#  switch   compiler     description
→  default  ocaml.5.1.0  default

Now that you have a general understanding of what exactly is a switch and how it is used, let's get into how you can go about manually creating your first switch.

Creating a global switch

NB: Remember that opam's command-line interface is beginner friendly. You can, at any point of your exploration, use the --help option to have every command and subcommand explained. You may also checkout the opam cheat-sheet that was released a while ago and might still hold some precious insights on opam's cli.

So how does one create a switch ? The short answer is bafflingly straightforward:

# Installs a switch named "my-switch" based OCaml compiler version > 4.05.0
# Here 4.05 is the default lower compiler version opam selects when unspecified
$ opam switch create my-switch

Easy, right? Now let's imagine that you would like to specify a later version of the OCaml compiler. The first thing you would want to know is which version are available for you to specify, and you can use opam list for that.

Other commands can be used to the same effect but we prefer introducing you to this specific one as it may also be used for any other package available via opam.

So, as for any other package than ocaml itself, opam list will give you all available versions of that package for your currently active switch. Since we don't yet have an OCaml compiler installed, it will list all of them so that we may pick and choose our favourite to use for the switch we are making.

$ opam list ocaml
# Packages matching: name-match(ocaml) & (installed | available)
# Package    # Installed # Synopsis
ocaml.3.07   --          The OCaml compiler (virtual package)
ocaml.3.07+1 --          The OCaml compiler (virtual package)
ocaml.3.07+2 --          The OCaml compiler (virtual package)
ocaml.3.08.0 --          The OCaml compiler (virtual package)
(...)
ocaml.4.13.1 --          The OCaml compiler (virtual package)
ocaml.4.13.2 --          The OCaml compiler (virtual package)
(...)
ocaml.5.2.0  --          The OCaml compiler (virtual package)

Let's use it for a switch:

# Installs a switch named "my-switch" based OCaml compiler version = 4.13.1
$ opam switch create my-switch ocaml.4.13.1

That's it, for the first time, you have manually created your own global switch tailored to your specific needs, congratulations!

NB: Creating a switch can be a fairly time-consuming task depending on whether or not the compiler version you have queried from opam is already installed on your machine, typically in a previously created switch. Every time you ask opam to install a version of the compiler, it will first scour your installation for a locally available version of that compiler to save you the time necessary for downloading, compiling and installing a brand new one.

Now, onto local switches.

Creating a local switch

As said previously, the use of a local switch is to constrain a specific OCaml environment to a specific location on your workstation.

Let's imagine you are about to start a new development called my-project.

While preparing all necessary pre-requisites for it, you notice something problematic: your global default switch is drastically incompatible with the dependencies of your project. In this imaginary situation, you have a default global switch that is useful for most of your other tasks but now have only one project that differs from your usual usage of OCaml.

To remedy this situation, you could go about creating another global switch for your upcoming dev requirements on my-project and proceed to install all relevant packages and remake a full switch from scratch for that specific project. However this would require you to always keep track of which one is your currently active switch, while possibly having to regularly oscillate between your global default switch and your alternative global my-project switch which you could understandably find to be suboptimal and tedious to incorporate to your workflow on the long run.

That's when local switches come in handy because they allow you to leave the rest of your OCaml dev environment unaffected by whatever out-of-bounds or specific workload you're undertaking. Additionally, the fact that opam automatically selects your local switch as your current active one as soon as you step inside the relevant directory makes the developers's context switch seemless.

Let's examine how you can create such a switch:

# Hop inside the directory of your project
$ cd my-project
# We consider your project already has an opam file describing only
# its main dependency: ocaml.4.14.1
$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.
$ opam switch
#  switch                   compiler      description
→  /home/ocp/my-project     ocaml.4.14.1  /home/ocp/my-project
   default                  ocaml.5.1.0   default
   my-switch                ocaml.4.13.1  my-switch

[NOTE] Current switch has been selected based on the current directory.
       The current global system switch is default.

Here it is, you can now hop into your local switch /home/ocp/my-project whenever you have time to deviate from your global environment.

The official opam-repository, the safe for all your packages

Among all the things that opam init did when it was executed, there is still one detail we have yet to explain and that's the first action of the process: retrieving packages specification from the official OCaml opam-repository.

Explaining what exactly an opam-repository is requires the recipient to have a slightly deeper understanding of how opam works than the average reader this article was written for might have; so you will have to wait for us to go deeper into that subject in another blogpost when the time is ripe.

What we will do now though is explain what the official OCaml opam-repository is and how it relates to our use of opam in this blog post.

The Official OCaml opam-repository is an open-source project where all released software of the OCaml distributions are referenced. It holds different compilers, basic tools, thousands of libraries, approximatively 4500 packages in total as of today and is configured to be the default repository for opam to sync to. You may add your own repositories for your own use of opam, but again, that's a subject for another time.

In case the repository itself is not what you are looking for, know that all packages available throughout the entire OCaml distribution may be browsed directly on ocaml.org.

It is essentially a collection of opam packages described in opam file format. Checkout the manual for more information about the opam file format.

A short explanation for it is that an opam package file holds every information necessary for opam to operate and provide. The file lists all of the packages direct dependencies, where to find its source code, the names and emails of maintainers and authors, different checksums for each archive release and the list goes on.

Here's a quick example for you to have an idea of what it looks like:

opam-version: "2.0"
synopsis: "OCaml bindings to Zulip API"
maintainer: ["Dario Pinto <dario.pinto@ocamlpro.com>"]
authors: ["Mohamed Hernouf <mohamed.hernouf@ocamlpro.com>"]
license: "LGPL-2.1-only WITH OCaml-LGPL-linking-exception"
homepage: "https://github.com/OCamlPro/ozulip"
doc: "https://ocamlpro.github.io/ozulip"
bug-reports: "https://github.com/OCamlPro/ozulip/issues"
dev-repo: "git+https://github.com/OCamlPro/ozulip.git"
tags: ["zulip" "bindings" "api"]
depends: [
  "ocaml" {>= "4.10"}
  "dune" {>= "2.0"}
  "ez_api" {>= "2.0.0"}
  "re"
  "base64"
  "json-data-encoding" {>= "1.0.0"}
  "logs"
  "lwt" {>= "5.4.0"}
  "ez_file" {>= "0.3.0"}
  "cohttp-lwt-unix"
  "yojson"
  "logs"
]
build: [ "dune" "build" "-p" name "-j" jobs "@install" ]
url {
  src: "https://github.com/OCamlPro/ozulip/archive/refs/tags/0.1.tar.gz"
  checksum: [
    "md5=4173fefee440773dd0f8d7db5a2e01e5"
    "sha512=cb53870eb8d41f53cf6de636d060fe1eee6c39f7c812eacb803b33f9998242bfb12798d4922e7633aa3035cf2ab98018987b380fb3f380f80d7270e56359c5d8"
  ]
}

Okay so now, how do we go about populating a switch with packages and really get started?

Installing packages in your current switch

It's elementary. This simple command will do the trick of trying to install a package, and its dependencies, in your currently active switch.

$ opam install my-package

I say trying because opam will notify you if the current package version and its dependencies you are querying are or not compatible with the current state of your switch. It will also offer you solutions for the compatibility constraints between packages to be satisfiable: it may suggest to upgrade some of your packages, or even to remove them entirely.

The key thing about this process is that opam is designed to solve compatibility constraints in the global graph of dependencies that the OCaml packages form. This design is what makes opam the average Cameleer's best friend. It will highlight inconsistencies within dependencies, it will figure out a way for your specific query to be satisfiable somehow and save you a lot of headscratching, that is, if you are willing to accommodate a bit of getting-used to.

The next command allows you to uninstall a package from your currently active switch as well as the packages that depend on it:

$ opam remove my-package

And the two following will update the state of the repositories opam is synchronised with and upgrade the packages installed while always keeping package compatibility in mind.

$ opam update
$ opam upgrade

Conclusion

Here it is, you should now be knowledgeable enough about opam to jump right in the OCaml discovery!

Today we learned everything elementary about opam.

From installation, to initialisation and explanations about the core concepts of the opam environment, switches, packages and the Official OCaml opam-repository.

Be sure to stay tuned with our blog, the journey into the rabbit hole has only started and opam is a deep one indeed!

Thank you for reading,

From 2011, with love,

The OCamlPro Team

Maturing Learn-OCaml to version 1.0: Gateway to the OCaml World

2023-12-13T13:56:36Z

Camels are known to be able to walk long distances. They have adapted to an inhospitable environment and help Humanity daily.

From the very start OCamlPro has been trying to help ease the learning of the OCaml language. OCaml has been used around the world to teach about a variety of Computer Science domains, from algorithmic to calculus, or functional programming and compilation.

The language had been long taught in Academia when arose initiatives to offer simple web tools to write and compile OCaml code in a simple web browser. We launched the TryOCaml web editor for OCaml all the way back in 2012. We were then appointed in 2015 by Roberto Di Cosmo from the French University Paris-Diderot, to create the OCaml FUN MOOC platform - and helped write the exercises used as pedagogical resources for the Introduction to Functional Programming.

That is how the Learn-OCaml open source learning platform was born, created then maintained at OCamlPro until 2018. Its steering was then transferred to the OCaml Software Foundation in 2019 and the project steadily grew into a fully fledged tool used by teachers and students around the world to this day.

Kudos to all OCaml teachers around the world, and to the LearnOCaml team, shepherded by Louis Gesbert

Learn-OCaml v1.0

What is Learn-OCaml today?

Learn-OCaml is a web platform for orchestrating exercises for OCaml programming, with automated grading. The interface features a code editor and client-side evaluation and grading; it can be served statically, but if running the bundled server there are also server-side saves, facilities for teachers to follow the progress of students, give assignments, get grades, etc.

We are thrilled to announce that the steady work that has been accomplished over the years on Learn-OCaml is finally bearing its fruits in the form of a long-awaited soon-to-be-released v1.0!

For all details relative to the upcoming 1.0 release, do refer to Louis' post on OCaml Discuss.

For all historical intents and purposes, do refer to the original 2016 OCaml Workshop paper on Learn-OCaml which kickstarted a long stream of updates and improvements to the platform and its public corpus exercices.

The maintenance and development work on the platform is now funded by the OCaml Software Foundation.

The latest release of Alt-Ergo version 2.5.1 is out, with improved SMT-LIB and bitvector support!

2023-09-18T13:56:36Z

Alt‑Ergo: An Automated SMT Solver for Program Verification

We are happy to announce a new release of Alt‑Ergo (version 2.5.1).

Alt-Ergo is a cutting-edge automated prover designed specifically for mathematical formulas, with a primary focus on advancing program verification.

This powerful tool is instrumental in the arsenal of static analysis solutions such as Trust-In-Soft Analyzer and Frama-C. It accompanies other major solvers like CVC5 and Z3, and is part of the solvers used behind Why3, a platform renowned for deductive program verification.

Find out more about Alt‑Ergo and how to join the Alt-Ergo Users' Club here!

This release includes the following new features and improvements:

support for bit-vectors in the SMT-LIB format;
new SMT-LIB parser and typechecker;
improved bit-vector reasoning;
partial support for SMT-LIB commands set-option and get-model;
simplified options to enable floating-point arithmetic theory;
various bug fixes.

Update for bug fixes

Since writing this blog post, we have released Alt-Ergo version 2.5.2 which fixes an incorrect implementation of the (distinct) SMT-LIB operator when applied to more than two arguments, and a (rare) crash in model generation. We strongly advise users interested in SMT-LIB or model generation support upgrade to version 2.5.2 on OPAM.

Better SMT-LIB Support

This release includes a better support of the SMT-LIB standard v2.6. More precisely, the release contains:

built-in primitives for the FixedSizeBitVectors;
Reals_Ints theories and the QF_BV logic;
new fully-featured parsers and type-checkers for SMT-LIB and native Alt-Ergo languages;
specific and meaningful messages for syntax and typing errors.

These features are powered by the Dolmen Library through a new frontend alongside the legacy one. Dolmen, developed by our own Guillaume Bury, is also used by the SMT community to check the conformity of the SMT-LIB benchmarks.

Important: In this release, the legacy frontend is still the default. If you want to enable the new Dolmen frontend, use the option --frontend dolmen. We encourage you to try it and report any bugs on our issue tracker.

Note: We plan to deprecate the legacy frontend and make Dolmen the default frontend in version 2.6.0, and to fully remove the legacy frontend in version 2.7.0.

Support For Bit-Vectors Primitives

Alt-Ergo has had support for bit-vectors in its native language for a long time, but bit-vectors were not supported by the old SMT-LIB parser, and hence not available in the SMT-LIB format. This has changed with the new Dolmen front-end, and support for bit-vectors in the SMT-LIB format is now available starting with Alt-Ergo 2.5.1!

The SMT-LIB theories for bit-vectors, BV and QF_BV, have more primitives than those previously available in Alt-Ergo. Alt-Ergo 2.5.1 supports all the primitives in the BV and QF_BV primitives when using the Dolmen frontend. Alt-Ergo's reasoning capabilities on the new primitives are limited, and will be gradually improved in future releases.

Built-in Primitives For Mixed Integer And Real Problems

In this release, we add the support for the primitives to_real, to_int and is_int of the SMT-LIB theory Reals_Ints. Notice that the support is only avalaible through the Dolmen frontend.

Example

For instance, the input file input.smt2:

(set-logic ALL)
(declare-const x Int)
(declare-const y Real)
(declare-fun f (Int Int) Int)
(assert (= (f x y) 0))
(check-sat)

with the command:

alt-ergo --frontend dolmen input.smt2

produces the limpid error message:

File "input.smt2", line 5, character 11-18:
5 | (assert (= (f x y) 0))
               ^^^^^^^
Error The term: `y` has type `real` but was expected to be of type `int`

Model Generation

Generating models (also known as counterexamples) is highly appreciated by users of SMT-solvers. Indeed, most builtin theories in common SMT-solvers are incomplete. As a consequence, solvers can fail to discharge goals and, without models, the SMT-solver behaves as a black box by outputting laconic answers: sat, unsat or unknown.

Providing best-effort counterexamples assists developers to understand why the solver failed to validate goals. If the goal isn't valid, the solver should, as much as it can, output a correct counter-example that helps users while fixing their specifications. If the goal is actually valid, the generated model is wrong but it can help SMT-solver's maintainers to understand why their solver didn't manage to discharge the goal.

Model generation for LIA theory and enum theory is available in Alt-Ergo. The feature for other theories is either in testing phase or being implemented. If you run into wrong models, please report them on our Github repository.

Usage

The present release provides convenient ways to invoking models. Notice that we change model invocation since the post Alt-Ergo: the SMT solver with model generation about model generation on the next development branch.

Check out the documentation for more details.

Floating Point Support

In version 2.5.1, the options to enable support for unbounded floating-point arithmetic have been simplified. The options --use-fpa and --prelude fpa-theory-2019-10-08-19h00.ae are gone: floating-point arithmetic is now treated as a built-in theory and can be enabled with --enable-theories fpa. We plan on enabling support for the FPA theory by default in a future release.

Usage

To turn on the fpa theory, use the new option --enable-theory fpa as follows:

alt-ergo --enable-theory fpa input.smt2

About Alt-Ergo 2.5.0

Version 2.5.0 should not be used, as it contains a soundness bug with the new bvnot primitive that slipped through the cracks. The bug was found immediately after the release, and version 2.5.1 released with a fix.

Acknowledgements

We thank members of the Alt-Ergo Users' Club: Thales, Trust-in-Soft, AdaCore, MERCE and the CEA.

We specially thank David Mentré and Denis Cousineau at Mitsubishi Electric R&D Center Europe for funding the initial work on model generation. Note that MERCE has been a Member of the Alt-Ergo Users' Club for three years. This partnership allowed Alt-Ergo to evolve and we hope that more users will join the Club on our journey to make Alt-Ergo a must-have tool.

The dedicated members of our Alt-Ergo Club!

2022 at OCamlPro

2023-06-30T13:56:36Z

Clear skies on OCamlPro's way of life.

For 12 years now, OCamlPro has been empowering a large range of customers, allowing them to harness state-of-the-art technologies and languages like OCaml and Rust. Our not-so-small-anymore company steadily grew into a team of highly-skilled and passionate engineers, experts in Computer Science, from Compilation and Software Analysis to Domain Specific Languages design and Formal Methods.

In this article, as every year (see last year's post) - albeit later than we do usually, we review some of the work we did during 2022, in many different worlds as shows the wide range of the missions we achieved.

Table of contents

Newcomers at OCamlPro

Modernizing Core Parts of Real Life Applications

MLANG, keystone of the French citizens' Income Tax Calculation
Contributing to GnuCOBOL, the Free Open-Source COBOL Alternative

Rust Expertise and Developments

Ecore, a heart of Rust for EMF
Open-Source Rust Contributions

The WebAssembly Garbage Collection Working-Group

Tooling for Formal Methods

The Alt-Ergo Theorem Prover
- The Alt-Ergo Users' Club
- Developing Alt-Ergo
Dolmen Library for Automated Deduction Languages

Contributions to OCaml

About opam, the OCaml Package Manager
The Flambda2 Optimizing Compiler

Organizing OCaml Meetups

OCaml Users in PariS (OUPS)
OCaml Meet-Up in Toulouse

Participation to External Events

The OCaml Workshop 2022 - ICFP Ljubljana
Journées Francophones Langages Applicatif 2022

Newcomers at OCamlPro

OCamlPro is not just a R&D software company, we like to think about it more as a team of people who like to work together. So, we are proud to introduce you the incredible human beings that joined us in 2022:

Pierre Villemot joined us in June. After three years of research at the Weizmann Institute on transcendental measures in Arithmetical Geometry, he was recruited and became the main maintainer of the Alt-Ergo Theorem Prover.
Milàn Martos joined us in July. He studied Chemistry and Computer Science at ENS, and he holds an MBA. He joined the Team as a Presales Engineer and as a Junior OCaml Web Developer.
Nathanaëlle Courant joined us in September. She holds a Master's degree from École Normale Supérieure in Paris, and is finishing her Ph.D. on efficient and verified reduction and convertibility tests for theorem provers. She joined OCamlPro in 2022 and works on the OCaml optimizer, in the Flambda team.
Arthur Carcano also joined us in September. Arthur is a Rust developer interested in performance optimization, software design, and crafting powerful and user-friendly tools. After completing his M.Sc. in Computer Science at ENS Ulm, he obtained a Ph.D. in Mathematics and Computer Science from Université de Paris.
Emilien Lemaire joined us in December 2022. After an internship on typechecking COBOL statements, he will be working with our COBOL team on creating a studio of modern tools for COBOL.

Modernizing Core Parts of Real Life Applications

We love to harness our IT expertise to give a competitive advantage to our clients by modernizing core chunks of key infrastructures. For example, we are working with the French Public Finances General Directorate on two of their modernization projects, to reimplement the language used for the computation of the Income Tax (MLang) and to provide support on the GnuCOBOL compiler used by the MedocDB application (COBOL).

MLANG, keystone of the French citizens' Income Tax Calculation

The M language, designed in the 80s to compute the French Income Tax, is still being rewritten in OCaml!

In 2022, our work on MLANG has passed a significant milestone: our work may no longer be considered a prototype! Code generation is now behaviourly compliant with the upstream compiler. David focused on rewriting the C architecture, which has been of great aid in iterating through each version of this new implementation of MLANG.

As far as testing goes, we were allowed to compare the results of our implementation against the ones of the upstream calculator, on real-life inputs too. We are talking about calculations of immense scale, which entails a highly performance-dependent project. Naturally, we managed to produce something of equivalent performance which was a very important matter for our contractors which have, since then, voiced their satisfaction. It is always great for us to feel appreciated for our work.

The next step is to make a production-level language by the end of 2023, so stay tuned if you are interested in this great project.

Wondering what MLANG is ? Be sure to read last year's post on the matter.

Contributing to GnuCOBOL, the Free Open-Source COBOL Alternative

Cobol is ran in gargantuan infrastructures of many an insurance companies and banks across the globe.

In 2022, we started contributing to the GnuCOBOL project: the GnuCOBOL compiler is, today, the only free, open-source, industrial-grade alternative to proprietary compilers by IBM and Micro-Focus. A cornerstone feature of GnuCOBOL is its ability to understand proprietary extensions to COBOL (dialects), to decrease the migration work of developers.

Last year's at OCamlPro presented our gradual introduction to the COBOL Universe as one of our latest technical endeavours. In the beginning, our main objective was to wrap our heads around the state of the environment for COBOL developers.

Our main contribution for now is to add support for the GCOS7 dialect, to ease migration from obsolete GCOS Bull mainframes to a cluster of PCs running GnuCOBOL for our first COBOL customer, the French DGFIP (Public Finances General Directorate). We also contributed a few fixes and small useful general features. Our contributions are gradually upstreamed in the official version of GnuCOBOL.

The other part of our COBOL strategy is to progressively develop our SuperBOL Studio, a set of modern tools for COBOL, and especially GnuCOBOL, based on an OCaml parser for COBOL that we have been improving during the year to support the full COBOL standard. More on this next year, hopefully !

Rust Expertise and Developments

Kind words sent our way by Florian Gilcher (skade), managing director at Ferrous Systems!

OCamlPro's culture is one of human values and appeal for everything scientific.

Programming languages of all nature have caught our attention at some point in time. As a consequence of years of expertise in all kinds of languages, we have grown fond of strongly-typed, memory-safe ones. Eventually gravitating towards Rust, we have since then invested significantly in adding this state-of-the-art language to our toolsets, one of which being the trainings we deliver to industrial actors of various backgrounds to help them grasp at such technological marvels.

Our trainers are qualified engineers, some of which have more than ten years of experience in the industry in R&D, Formal Methods and embedded systems alike, seven of which being solely in Rust.

Strong of our collective experiences, 2022 was indeed the stage for many contributions and missions, some of which we will share with you right now.

Ecore, a heart of Rust for EMF

Ecore is the code generator at the heart of the EMF Architecture.

In 2022, we have seized the opportunity to work at the threshold between Java and Rust for our clients and academic partners of the CEA (Commissariat aux Énergies Atomiques et aux Énergies Alternatives). The product was a Rust-written and Rust-encoded Java class hierarchy code generator.

Ecore is the core metamodel at the heart of the Eclipse Modeling Framework (EMF), which is used internally at the CEA. Ecore is a superset of UML and allows for the engineers of the CEA to express a Java class hierarchy through a graphical interface. In practice, this allows for the generation of basic Java models for the engineers to then build upon.

Our mission consisted in writing, in Rust, a new model generator for them to use in their workflows and to make it capable of generating Rust code instead of Java.

The cost for harnessing the objective qualities of a new implementation in Rust was to have us tackle the scientific challenges pertaining to the inherent structural differences between both languages. Our goal was to find a way to encode, in Rust, a way to express the semantics of the hierarchy of classes of Java, hence merging the worlds of Rust and Java on the way there.

Eventually, our partners were convinced the challenges were worth the improved speed at which models were generated. Furthermore, the now embedded-programming compliant platform, the runtime safety and even Rust's broader WebAssembly-ready toolchain have cleared a new path for the future iterations of their internal projects.

Open-Source Rust Contributions

Ferris the Crab is the mascot of the Rust Language. No wonder why we converged as well!

As we continue scouring the market for more and more Rust projects, and whenever the opportunity shows up, we still actively contribute to the open-source community, here are some of this year's OS work:

Lean4

Here's a project suited for all who, like us, are Formal Methods, functional programming and formal methods enthousiasts: Lean:

Lean is a functional programming language that makes it easy to write correct and maintainable code. You can also use Lean as an interactive theorem prover. Lean programming primarily involves defining types and functions. This allows your focus to remain on the problem domain and manipulating its data, rather than the details of programming.

The list of our contributions to the repository of lean4:

Detection of a major dependent pattern matching bug
Some QA with unintuitive calc indentation
And some more with strict indentation in nested by-s requirement

Matla, TLA+ Projects Manager

Last year, we shared a sneakpeek of Matla, introducing its use-case and the motivations for implementing such manager for TLA+ projects. As we tinkered with TLA+, sometimes finding a bug, we continued our development of Matla on the side.

The tool, although still a work-in-progress, has since then undergone a few changes and releases:

You are welcome to contribute if you happen to find yourself in the same situation we were in when we started the project.

Agnos, for Let's Encrypt Wildcard Certificates

Agnos is a single-binary program allowing you to easily obtain certificates (including wildcards) from Let's Encrypt using DNS-01 challenges. It answers Let's Encrypt DNS queries on its own, bypassing the need for API calls to your DNS provider, and strives to offer a user-friendly and easy configuration.

Often, the best contributions are of a practical nature, which is the case for Agnos.

If that sounds interesting to you, you can learn more about it by reading this article.

Make sure to give us some feedback when you end up using it!

The WebAssembly Garbage Collection Working-Group

WebAssembly is used to compile many languages to an efficient portable code for web-browsers.

Late 2022 was finally time for us to put into practice the knowledge we have acquired about WebAssembly over the years by writing and presenting the first compiler of a real-world functional language targeting the WasmGC proposal.

Although a relatively new technology, its great design, huge potential, and already very tangible and interesting use-cases have not escaped our watch and we are very happy to have kept a sharp eye on it.

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.

WasmGC is the name of the on-going working group and proposal towards eventually adding support for garbage collection to Wasm. December 2022 saw a significant amount of work accomplished by both Léo Andrès (whose thesis work is directed by Pierre and Jean-Christophe Filliâtre) and Pierre Chambart towards finding viable compilation strategies from OCaml to WasmGC. The goal was three-fold: make a prototype compiler to demonstrate the soundness of the proposal, show that our compilation strategies were viable and, finally, convince the commitee of the significance of the Wasm i31ref for OCaml.

Our success in these three distinct points was paramount for OCaml, and other languages that depend on the presence of i31ref, in order to one day benefit from having WebAssembly as a natively supported compilation target for Web-bound applications.

Here's a short listing of the work they have accomplished for that matter. Please rest assured, more detailed explanations are to be expected on this very blog in the future so stay tuned!

Introducing Wasocaml to the Wasm-GC Group and demonstrating the OCaml's dependency on Wasm keeping i31ref in their GC proposal.
Wasocaml, an OCaml compiler to Wasm. Wasocaml is also the first compiler for a real-world functional language to Wasm-GC.
owi, an OCaml toolchain to work with Wasm. It provides and interpreter as an executable and a library.

Tooling for Formal Methods

Programming languages theory is closely tied with the idea of proper mathematical formalisation. Hence the strong scientific background in Formal Methods that we draw from both for language design or formal verification for cybersecurity.

The Alt-Ergo Theorem Prover

OCamlPro develops and maintains Alt-Ergo, an automatic solver of mathematical formulas designed for program verification and based on Satisfiability Modulo Theories (SMT) technology. Alt-Ergo was initially created within the VALS team at University of Paris-Saclay.

Alt-Ergo proves mathematical formulas corresponding to software program properties.

The Alt-Ergo Users' Club

The Alt-Ergo Users' Club was launched in 2019. Its 4th annual meeting was held in late March 2022.

These meetings allow us to keep track of our members' needs and demands as well as keep them informed with the latest changes to the SMT Solver; they are the lifeline of our Club and help us guarantee that the project lives on, despite the enormous task it represents.

This is a good time to appreciate the scope of the project: Alt-Ergo is the fruit of more than 10 years' worth of Research & Development. Currently maintained by Pierre Villemot, whom we will introduce in the next section, as full-time R&D engineer.

The dedicated members of the Club!

This is the reason why we would like to thank our partners from the Alt-Ergo Users’ Club, for their trust: Thales, Trust-in-Soft, AdaCore, MERCE (Mitsubishi Electric R&D Centre Europe) and the CEA. Their support allows us to maintain our tool.

Congratulations and many thanks to our partners at Trust-In-Soft, who have upgraded their subscription to the Club to Gold-tier, allowing this great open-source tool to live on!

Developing Alt-Ergo

In 2022, the Alt-Ergo team welcomed Pierre Villemot as full-time maintainer of the project! His recruitement shows our commitment to the project's long term maintenance and evolution. We are looking forward to seeing him take it to new heights in future releases! Speaking of releases, 2022 has also been the stage for Alt-Ergo's v2.4.2 minor release which introduced an update of the labgltk library to version 3 and a set of bug fixes.

Now onto the more substantial changes to Alt-Ergo, the integration into next of all the following:

Integration of the SMT-LIB2 format parser Dolmen to Alt-Ergo's frontend;
Improvement and test of models generation;
Addition of mutually recursive functions for the legacy frontend and Dolmen alike;
Significant amounts of documentation and code-cleaning;
Implementation of systematical benchmarks of the SMT-LIB for regression prevention;
Prototypical Dockerisation;

These are significative improvements to the User Experience and overall ergonomy of the tool. You can already benefit from these changes by using Alt-Ergo's dev version.

Finally, let us inform you that our candidacy for the DECYSIF project was approved. Indeed, we and our partners at Adacore, Trust-In-Soft and the Laboratoire Méthodes Formelles have been selected to conduct this funded research project as consultant in Formal Methods. Now, we hope to be part of collaborative R&D projects to further fund core Alt-Ergo developments. This should allow us to deepen collaboration with old partners like the Why3 team at the Formal Methods Lab (LMF) and the ProofinUse consortium members. Stay tuned!

Dolmen Library for Automated Deduction Languages

Dolmens are Neolithic megalithic structures composed of menhirs and they can range from a few centimeters to several meters high!

Dolmen is an OCaml Library developed by Guillaume Bury as part of our Research and Development processes around Formal Methods and our development efforts for our automated SMT-Solver Alt-Ergo.

Dolmen is a testimony of our push towards standardised input languages for SMT-Solvers. Indeed, it provides flexible Menhir parsers and typecheckers for several languages used in automated deduction such as: smtlib, ae (Alt-Ergo), dimacs, iCNF, tptp and zf (zipperposition). And so, Dolmen aims at encompassing the largest amount of input languages for automated deduction as possible and provides the OCaml community with a centralised solution for their input parsing and typechecking, hence keeping them from having to reimplement them each time.

Furthermore, the Dolmen binary is used by the maintainers of the SMTLIB in order to assert that newly submitted benchmarks to the SMTLIB are compliant with the specification which makes Dolmen its de facto reference implementation. In time, Dolmen will become the default frontend for Alt-Ergo and, we hope, any other OCaml written SMT-Solver from now on.

Contributions to OCaml

Last but not least, OCamlPro’s DNA built itself on one of the most powerful and elegant programming languages ever, born from more than 30 years of French public Research in Computer Science, and widely used in safety critical industries. OCaml’s traits pervasively inspired many new languages (like F#). We are proud to be part of this great community of researchers and computer scientists.

About opam, the OCaml Package Manager

opam, the OCaml Package Manager, remains one of OCamlPro's greatest achievements!

2022 has been the theatre of a sustained and continuous effort from the opam team.

The fruits of their labor have been compiled into an alpha release of version 2.2.0 by June 28th 2023, so here is a taste of what should make the final 2.2.0 version of opam a treat for its users:

Windows support: opam 2.2 comes with native Windows compatibility. You can now use opam from your preferred Windows terminal!
Recursive pinning: allows to have opam lookup for opam files into subdirectories.
Software Heritage binding: opam now integrates a fallback to Software Heritage archive retrieval, based on SWHID. If an SWHID url is present in an opam file, the fallback can be activated.
Enhanced features for developers: as the development tools variable to share a development setup, the opam tree command to have better an overview of dependencies, new pinning subcommands, and so on.

That being said, 2022 was a very special year for opam. Indeed, 10 years prior, on the 26th of June 2012, OCamlPro birthed version 0.1 of what was to become the official OCaml Package Manager, the cornerstone of the OCaml environment.

It was no small feat to make opam what it is today. It took approximately 5 years to bring 1.0.0 up to 2.0.0 and another 3 to reach 2.1.0 all the while ensuring changes were compliant with the current ecosystem (opam repository, OCaml tooling) and the public's feedback and vision.

This work is allowed thanks to Jane Street's funding.

The Flambda2 Optimizing Compiler

Flambda2 is a powerful code optimizer for the OCaml compiler strong of many years of R&D.

OCamlPro is proud to be working on Flambda2, an ambitious OCaml optimizing compiler project, initiated with Mark Shinwell from Jane Street, our long-term partner and client. Flambda2 builds upon its predecessor: Flambda, which focused on reducing the runtime cost of abstractions and removing as many short-lived allocations as possible. Thus, Flambda2 not only shines with the maturity and experience its architects acquired through years worth of R&D and dev-time for Flambda, but it improves upon them.

In 2022, Flambda2 was for the first time used for production workloads and has been ever since! Indeed, we can officially say that Flambda2 left the realm of the prototype to enter one of real-life, production-tested software for which we continue to provide development and support as it has been for years now.

This achievement comes along having our engineers take more and more action in maintaining the OCaml compiler. Being part of the OCaml Core-Team is an honour.

Finally, in 2022, the Flambda Team welcomed a new member: Nathanaëlle Courant will be joining forces with Pierre Chambart, Damien Doligez, Vincent Laviron, Guillaume Bury to tackle the challenges inherent to maintaining Flambda2 and that of the Core-Team.

If you are interested in more things Flambda2, stay tuned in with our blog, there should be a series of very interesting articles coming up in the not-so distant future!

This work is allowed thanks to Jane Street's funding.

In other OCaml compiler news, 2022 was also the year of the official release of OCaml 5.0.0, also known as Multi-Core, on the 16th of December. This major release introduced a new runtime system with support for shared memory parallelism and effect handlers! This fabulous milestone is brought to us by the joined work of the amazing people of the OCaml Core-Team; among them some of our own.

Many thanks to all who take part in uncovering the yet untrodden paths of the OCaml distribution!

What a time to be an OCaml enthousiast!

Organizing Meetups for the OCaml Community

OCaml Users in PariS (OUPS)

Camels going to their pluri-annual OUPS Meet-up.

Just under 10 years ago, Fabrice Le Fessant initiated the very first OCaml Users in Paris.

This event allowed OCaml users in Paris, professionals and amateurs alike, to meet and exchange on OCaml novelties. This is still the case and the organising crew now includes several people of diverse affiliations, maintaining the purpose of this friendly event.

Every two months or so, the organisers reach out to the community, hail volunteers and select presentations of on-going works. When the time and place is settled, the ocaml-paris Meetup members are informed by various means. The OCaml Users in PariS meetup is the place to enthusiastically share knowledge and a few pizzas. It is supported by the OCaml Software Foundation who graciously pays for the pizzas.

You can register to the OCaml Users in PariS (OUPS) meetup group here.

Here are all the relevant links to the talks that happened in Paris in 2022:

OCaml Meet-Up in Toulouse

Toulouse also has its set of enthousiastic OCaml supporters.

Fortunately for OCaml Users that live in the French South-West, a new Meet-up is now available to them. On the 11th of October 2022, the first OCaml meet-up in Toulouse happened.

The first occurence of the OCaml Users in Toulouse Meetup kicked off with Erik Martin-Dorel (OCaml Software Foundation) presenting Learn-OCaml who was then followed by David Declerck (OCamlPro) presenting his OCaml-Canvas graphics library for OCaml.

You can register to the OCaml Meet-Up in Toulouse group here.

Here's to sharing a slice or two with you soon!

Participation to External Events

The OCaml Workshop 2022 - ICFP Ljubljana

ICFP 2022 took place in the beautiful town of Ljubjana, Slovenia.

The OCaml Workshop is an international conference that focuses on everything OCaml and is part of the ICFP (International Conference on Functional Programming).

We attended many of these and have presented numerous papers throughout the years.

In 2022, a paper co-authored by the maintainers of opam, the OCaml Package Manager, was submitted and approved for presentation: "Supporting a decade of opam".

You can find the textual references of the talk here and a replay of the presentation there.

You can expect more papers and interesting talks coming from us in upcoming editions of the conference!

Journées Francophones Langages Applicatifs 2022

the JFLA'2022 took place in the beautiful Domaine d'Essendiéras in Périgord, France.

Among the many scientific conferences we attend on an annual basis, the JFLA (Journée Francophones des Langages Applicatifs or French-Speaking annual gathering on Application Programming Languages, mainly Functional Languages) is the one we feel most at home since 2016.

Ever since have we remained among their faithful supporters and participants. This gathering of many of our fellow French computer scientists and industrial actors alike has been our go-to conference to catch-up with and present our work. The 2022 edition was no exception!

We submitted and presented the following papers:

Mikino, formal verification made accessible (link to dedicated blogpost);
Connecting Software Heritage with the OCaml ecosystem;
Alt-Ergo-Fuzz, hunting the bugs of the bug hunter;

You can find a more detailed recounting of our JFLA2022 submissions in this blog post as well as the links to the actual (french-written) submitted papers.

Conclusion

As always, we warmly thank all our clients, partners, and friends, for their support and collaboration throughout the year,

And to you, dear reader, thank you for tagging along,

Since 2011 with love,

The OCamlPro Team

Autofonce, GNU Autotests Revisited

2023-06-27T13:56:36Z

Since 2022, OCamlPro has been contributing to GnuCOBOL, the only fully open-source compiler for the COBOL language. To speed-up our contributions to the compiler, we developed a new tool, autofonce, to be able to easily run and modify the testsuite of the compiler, originally written as a GNU Autoconf testsuite. This article describes this tool, that could be useful for other project testsuites.

Table of contents

Introduction
The Gnu Autoconf Testsuite of GnuCOBOL
Main Features of Autofonce
Conclusion

Autofonce is a modern runner for GNU Autoconf Testsuite

Introduction

Since 2022, OCamlPro has been involved in a big modernization project for the French state: the goal is to move a large COBOL application, running on a former Bull mainframe (GCOS) to a cluster of Linux computers. The choice was made to use the most open-source compiler, GnuCOBOL, that had already been used in such projects.

One of the main problems in such migration projects is that most COBOL proprietary compilers provide extensions to the COBOL language standard, that are not supported by other compilers. Fortunately, GnuCOBOL has good support for several mainstream COBOL dialects, such as IBM or Micro-Focus ones. Unfortunately, GnuCOBOL had no support at the time for the GCOS COBOL dialect developed by Bull for its mainframes.

As a consequence, OCamlPro got involved in the project to extend GnuCOBOL with the support for the GCOS dialect needed for the application. This work implied a lot of (sometimes very deep) modifications of the compiler and its runtime library, both of them written in the C language. And of course, our modifications had first to pass the large existing testsuite of COBOL examples, and then extend it with new tests, so that the new dialect would continue to work in the future.

This work lead us to develop autofonce, a modern open-source runner for GNU Autoconf Testsuites, the framework used in GnuCOBOL to manage its testsuite. Our tool is available on Github, with Linux and Windows binaries on the release page.

The GNU Autoconf Testsuite of GnuCOBOL

GNU Autoconf is a set of powerful tools, developed to help developers of open-source projects to manage their projects, from configuration steps to testing and installation. As a very old technology, GNU Autoconf relies heavily on M4 macros both as its own development language, and as its extension language, typically for tests.

In GnuCOBOL, the testsuite is in a sub-directory tests/, containing a file testsuite.at, itself including other files from a sub-directory testsuite.src/.

As an example, a typical test from syn_copy.at looks like:

AT_SETUP([INITIALIZE constant])
AT_KEYWORDS([misc])
AT_DATA([prog.cob], [
       IDENTIFICATION   DIVISION.
       PROGRAM-ID.      prog.
       DATA             DIVISION.
       WORKING-STORAGE  SECTION.
       01  CON          CONSTANT 10.
       01  V            PIC 9.
       78  C78          VALUE 'A'.
       PROCEDURE DIVISION.
           INITIALIZE CON.
           INITIALIZE V.
           INITIALIZE V, 9.
           INITIALIZE C78, V.
])
AT_CHECK([$COMPILE_ONLY prog.cob], [1], [],
[prog.cob:10: error: invalid INITIALIZE statement
prog.cob:12: error: invalid INITIALIZE statement
prog.cob:13: error: invalid INITIALIZE statement
])
AT_CLEANUP

Actually, we were quite pleased by the syntax of tests, it is easy to generate test files (using AT_DATA macro) and to test the execution of commands (using AT_CHECK macro), checking its exit code, its standard output and error output separately. It is even possible to combine checks to run additional checks in case of error or success. In general, the testsuite is easy to read and complete.

However, there were still some issues:

At every update of the code or the tests, the testsuite runner has to be recompiled;
Running the testsuite requires to be in the correct sub-directory, typically within the _build/ sub-directory;
By default, tests are ran sequentially, even when many cores are available.

The output is pretty verbose, showing all tests that have been executed. Failed tests are often lost in the middle of other successful tests, and you have to wait for the end of the run to start investigating them;

## -------------------------------------------- ##
## GnuCOBOL 3.2-dev test suite: GnuCOBOL Tests. ##
## -------------------------------------------- ##
  
General tests of used binaries
  
  1: compiler help and information                   ok
  2: compiler warnings                               ok
  3: compiler outputs (general)                      ok
  4: compiler outputs (file specified)               ok
  5: compiler outputs (path specified)               ok
  6: compiler outputs (assembler)                    ok
  7: source file not found                           ok
  8: temporary path invalid                          ok
  9: use of full path for cobc                       ok
 10: C Compiler optimizations                        ok
 11: invalid cobc option                             ok
 12: cobcrun help and information                    ok
 13: cobcrun validation                              ok
 14: cobcrun -M DSO entry argument                   ok
 15: cobcrun -M directory/ default                   ok
 [...]

There is no automatic way to update tests, when their output has changed. Every test has to be updated manually.
In case of error, it is not always easy to rerun a specific test within its directory.

With autofonce, we tried to solve all of these issues...

Main Features of Autofonce

autofonce is written in a modern language, OCaml, so that it can handle a large testsuite much faster than GNU Autoconf. Since we do not expect users to have an OCaml environment set up, we provide binary versions of autofonce for both Linux (static executable) and Windows (cross-compiled executable) on Github.

autofonce does not use m4, instead, it has a limited support for a small set of predefined m4 macros, typically supporting m4 escape sequences (quadrigraphs), but not the addition of new m4 macros, and neither the execution of shell commands outside of these macros (yes, testsuites in GNU Autoconf are actually sh shell scripts with m4 macros...). In the case of GnuCOBOL, we were lucky enough that the testsuite was well written and avoided such problems (we had to fix only a few of them, such as including shell commands into AT_CHECK macros). The syntax of tests is documented here.

Some interesting features of autofonce are :

autofonce executes the tests in parallel by default, using as many cores as available. Only failed tests are printed, so that the developer can immediately start investigating them;
autofonce can be run from any directory in the project. A .autofonce file has to be present at the root of the project, to describe where the tests are located and in which environment they should be executed;
autofonce makes it easy to re-execute a specific test that failed, by generating, within the test sub-directory, a script for every step of the test;
autofonce provides many options to filter which tests should be executed. Tests can be specified by number, range of numbers, keywords, or negative keywords. The complete list of options is easily printable using autofonce run --help for example;

Additionnally, autofonce implements a powerful promotion mechanism to update tests, with the autofonce promote sub-command. For example, if you update a warning message in the compiler, you would like all tests where this message appears to be modified. With autofonce, it is as easy as:

# Run all tests at least once
autofonce run
# Print the patch that would be applied in case of promotion
autofonce promote
# Apply the patch above
autofonce promote --apply
# Combine running and promotion 10 times:
autofonce run --auto-promote 10

The last command iterates promotion 10 times: indeed, since a test may have multiple checks, and only the first failed check of the test will be updated during one iteration (because the test aborts at the first failed check), as many iterations as the maximal number of failed checks within a test may be needed.

Also, as for GNU Autoconf, autofonce generates a final log file containing the results with a full log of errors and files needed to reproduce the error. This file can be uploaded into the artefacts of a CI system to easily debug errors after a CI failure.

Conclusion

During our work on GnuCOBOL, autofonce improved a lot our user experience of running the testsuite, especially using the auto-promotion feature to update tests after modifications.

We hope autofonce could be used for other open-source projects that already use the GNU Autoconf testsuite. Of course, it requires that the testsuite does not make heavy use of shell features and mainly relies on standard m4 macros.

We found that the format of GNU Autoconf tests to be quite powerful to easily check exit codes, standard outputs and error outputs of shell commands. autofonce could be used to help using this format in projects, that do not want to rely on an old tool like GNU Autoconf, and are looking for a much more modern test framework.

Sub-single-instruction Peano to machine integer conversion

2023-01-23T13:56:36Z

It is a rainy end of January in Paris, morale is getting soggier by the day, and the bulk of our light exposure needs are now fulfilled by our computer screens as the sun seems to have definitively disappeared behind a continuous stream of low-hanging clouds. But, all is not lost, the warm rays of comradeship pierce through the bleak afternoons, and our joyful party of adventurers once again embarked on an adventure of curiosity and rabbit-hole depth-first-searching.

Last week's quest led us to a treasure coveted by a mere handful of enlightened connoisseurs, but a treasure of tremendous nerdy-beauty, known to the academics as "Sub-single-instruction Peano to machine integer conversion" and to the locals as "How to count how many nested Some there are very very fast by leveraging druidic knowledge about unspecified, undocumented, and unstable behavior of the Rust compiler".

Our quest begins

Our whole quest started when we wanted to learn more about discriminant elision. Discriminant elision in Rust is part of what makes it practical to use Option<&T> in place of *const T. More precisely it is what allows Option<&T> to fit in as much memory as *const T, and not twice as much. To understand why, let's consider an Option<u64>. An u64 is 8 bytes in size. An Option<u64> should have at least one more bit, to indicate whether it is a None, or a Some. But bits are not very practical to deal with for computers, and hence this discriminant value -- indicating which of the two variants (Some or None) the value is -- should take up at least one byte. Because of alignment requirements (and because the size is always a multiple of the alignment) it actually ends up taking 8 bytes as well, so that the whole Option<u64> occupies twice the size of the wrapped u64.

In languages like C, it is very common to pass around pointers, and give them a specific meaning if they are null. Typically, a function like lfind which searches for an element in a array will return a pointer to the matching element, and this pointer will be null if no such element was found. In Rust however fallibility is expected to be encoded in the type system. Hence, functions like find returns a reference, wrapped in a Option. Because this kind of API is so ubiquitous, it would have been a real hindrance to Rust adoption if it took twice as much space as the C version.

This is why discriminant elision exists. In our Option<&T> example Rust can leverage the same logic as C: &T references in Rust are guaranteed to be -- among other things -- non-null. Hence Rust can choose to encode the None variant as the null value of the variable. Transparently to the user, our Option<&T> now fits on 8 bytes, the same size as a simple &T. But Rust discriminant elision mechanism goes beyond Option<&T> and works for any general type if:

The option-like value has one fieldless variant and one single-field variant
The wrapped type has so-called niche values, that is values that are statically known to never be valid for said type.

Discriminant elision remains under-specified, but more information can be found in the FFI guidelines. Note that other unspecified situations seem to benefit from niche optimization (e.g. PR#94075).

Too many options

Out of curiosity, we wanted to investigate how the Rust compiler represents a series of nested Option. It turns out that up to 255 nested options can be stored into a byte, which is also the theoretical limit. Because this mechanism is not limited to Option, we can use it with (value-level) Peano integers. Peano integers are a theoretical encoding of integer in "unary base", but it is enough for this post to consider them a fun little gimmick. If you want to go further, know that Peano integers are more often used at the type-level, to try to emulate type-level arithmetic.

In our case, we are mostly interested in Peano-integers at the value level. We define them as follows:

#![recursion_limit = "512"]
#![allow(dead_code)]

/// An empty enum, a type without inhabitants.
/// Cf: https://en.wikipedia.org/wiki/Bottom_type
enum Null {}

/// PeanoEncoder<Null> is a Peano-type able to represent integers up to 0.
/// If T is a Peano-type able to represent integers up to n
/// PeanoEncoder<T> is a Peano-type able to represent integers up to n+1
#[derive(Debug)]
enum PeanoEncoder<T> {
    Successor(T),
    Zero,
}

macro_rules! times2 {
    ($peano_2x:ident, $peano_x:ident ) => {
        type $peano_2x<T> = $peano_x<$peano_x<T>>;
    };
}
times2!(PeanoEncoder2, PeanoEncoder);
times2!(PeanoEncoder4, PeanoEncoder2);
times2!(PeanoEncoder8, PeanoEncoder4);
times2!(PeanoEncoder16, PeanoEncoder8);
times2!(PeanoEncoder32, PeanoEncoder16);
times2!(PeanoEncoder64, PeanoEncoder32);
times2!(PeanoEncoder128, PeanoEncoder64);
times2!(PeanoEncoder256, PeanoEncoder128);

type Peano0 = PeanoEncoder<Null>;
type Peano255 = PeanoEncoder256<Null>;

Note that we cannot simply go for

enum Peano {
    Succesor(Peano),
    Zero,
}

like in Haskell or OCaml because without indirection the type has infinite size, and adding indirection would break discriminant elision. What we really have is that we are actually using a type-level Peano-encoding of integers to create a type Peano256 that contains value-level Peano-encoding of integers up to 255, as a byte would.

We can define the typical recursive pattern matching based way of converting our Peano integer to a machine integer (a byte).

trait IntoU8 {
    fn into_u8(self) -> u8;
}

impl IntoU8 for Null {
    fn into_u8(self) -> u8 {
        match self {}
    }
}

impl<T: IntoU8> IntoU8 for PeanoEncoder<T> {
    fn into_u8(self) -> u8 {
        match self {
            PeanoEncoder::Successor(x) => 1 + x.into_u8(),
            PeanoEncoder::Zero => 0,
        }
    }
}

Here, according to godbolt, Peano255::into_u8 gets compiled to more than 900 lines of assembly, which resembles a binary decision tree with jump-tables at the leaves.

However, we can inspect a bit how rustc represents a few values:

println!("Size of Peano255: {} byte", std::mem::size_of::<Peano255>());
for x in [
    Peano255::Zero,
    Peano255::Successor(PeanoEncoder::Zero),
    Peano255::Successor(PeanoEncoder::Successor(PeanoEncoder::Zero)),
] {
    println!("Machine representation of {:?}: {}", x, unsafe {
        std::mem::transmute::<_, u8>(x)
    })
}

which gives

Size of Peano255: 1 byte
Machine representation of Zero: 255
Machine representation of Successor(Zero): 254
Machine representation of Successor(Successor(Zero)): 253

A pattern seems to emerge. Rustc chooses to represent Peano255::Zero as 255, and each successor as one less.

As a brief detour, let's see what happens for PeanoN with other values of N.

let x = Peano1::Zero;
println!("Machine representation of Peano1::{:?}: {}", x, unsafe {
    std::mem::transmute::<_, u8>(x)
});
for x in [
    Peano2::Successor(PeanoEncoder::Zero),
    Peano2::Zero,
] {
    println!("Machine representation of Peano2::{:?}: {}", x, unsafe {
        std::mem::transmute::<_, u8>(x)
    })
}

gives

Machine representation of Peano1::Zero: 1
Machine representation of Peano2::Successor(Zero): 1
Machine representation of Peano2::Zero: 2

Notice that the representation of Zero is not the same for each PeanoN. What we actually have -- and what is key here -- is that the representation for x of type PeanoN is the same as the representation of Succesor(x) of type PeanoEncoder<PeanoN>, which implies that the machine representation of an integer k in the type PeanoN is n-k.

That detour being concluded, we refocus on Peano255 for which we can write a very efficient conversion function

impl Peano255 {
    pub fn transmute_u8(x: u8) -> Self {
        unsafe { std::mem::transmute(u8::MAX - x) }
    }
}

Note that this function mere existence is very wrong and a sinful abomination to the eye of anything that is holy and maintainable. But provided you run the same compiler version as me on the very same architecture, you may be ok using it. Please don't use it.

In any case transmute_u8 gets compiled to

movl    %edi, %eax
notb    %al
retq

that is a simple function that applies a binary not to its argument register. And in most use cases, this function would actually be inlined and combined with operations above, making it run in less than one processor operation!

And because 255 is so small, we can exhaustively check that the behavior is correct for all values! Take that formal methods!

for i in 0_u8..=u8::MAX {
    let x = Peano255::transmute_u8(i);
    if i % 8 == 0 {
        print!("{:3} ", i)
    } else if i % 8 == 4 {
        print!(" ")
    }
    let c = if x.into_u8() == i { '✓' } else { '✗' };
    print!("{}", c);
    if i % 8 == 7 {
        println!()
    }
}

  0 ✓✓✓✓ ✓✓✓✓
  8 ✓✓✓✓ ✓✓✓✓
 16 ✓✓✓✓ ✓✓✓✓
 24 ✓✓✓✓ ✓✓✓✓
 32 ✓✓✓✓ ✓✓✓✓
 40 ✓✓✓✓ ✓✓✓✓
 48 ✓✓✓✓ ✓✓✓✓
 56 ✓✓✓✓ ✓✓✓✓
 64 ✓✓✓✓ ✓✓✓✓
 72 ✓✓✓✓ ✓✓✓✓
 80 ✓✓✓✓ ✓✓✓✓
 88 ✓✓✓✓ ✓✓✓✓
 96 ✓✓✓✓ ✓✓✓✓
104 ✓✓✓✓ ✓✓✓✓
112 ✓✓✓✓ ✓✓✓✓
120 ✓✓✓✓ ✓✓✓✓
128 ✓✓✓✓ ✓✓✓✓
136 ✓✓✓✓ ✓✓✓✓
144 ✓✓✓✓ ✓✓✓✓
152 ✓✓✓✓ ✓✓✓✓
160 ✓✓✓✓ ✓✓✓✓
168 ✓✓✓✓ ✓✓✓✓
176 ✓✓✓✓ ✓✓✓✓
184 ✓✓✓✓ ✓✓✓✓
192 ✓✓✓✓ ✓✓✓✓
200 ✓✓✓✓ ✓✓✓✓
208 ✓✓✓✓ ✓✓✓✓
216 ✓✓✓✓ ✓✓✓✓
224 ✓✓✓✓ ✓✓✓✓
232 ✓✓✓✓ ✓✓✓✓
240 ✓✓✓✓ ✓✓✓✓
248 ✓✓✓✓ ✓✓✓✓

Isn't computer science fun?

Note: The code for this blog post is available here.

Statically guaranteeing security properties on Java bytecode: Paper presentation at VMCAI 23

2023-01-12T13:56:36Z

We are excited to announce that Nicolas will present a paper at the International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI) the 16th and 17th of January.

This year, VMCAI is co-located with the Symposium on Principles of Programming Languages (POPL) conference, which, as its name suggests, is a flagship conference in the Programming Languages domain.

What's more, for its 50th anniversary edition, POPL will return back where its first edition took place: Boston! It is thus in the vicinity of the MIT and Harvard that we will meet with prominent figures of computer science research.

This paper will be presented at VMCAI'2023, colocated with POPL'2023 at Boston!

A sound technique to statically guarantee security properties on Java bytecode

Nicolas will be presenting a novel static program analysis technique dedicated to the discovery of information flows in Java bytecode. By automatically discovering such flows, the new technique allows developers and users of Java libraries to assess key security properties on the software they run.

Two prominent examples of such properties are confidentiality (stating that no single bit of secret information may be inadvertently revealed by the software), and its dual, integrity (stating that no single bit of trusted information may be tampered via untrusted data).

The technique is proven sound (i.e. it cannot miss a flow of information), and achieves state-of-the-art precision (i.e. it does not raise too many false alarms) according to evaluations using the IFSpec benchmark suite.

Try it out!

In addition to being supported by a proof, the technique has also been implemented in a tool called Guardies.

We believe this static analysis tool will naturally complement the taint tracking and dynamic analysis techniques that are usually employed to assess software security.

Reading more about it

You may already access the full paper here.

Nicolas developed this contribution while working at the University of Liverpool, in collaboration with Narges Khakpour, herself from the University of Newcastle.

Release of ocplib-simplex, version 0.5

2023-01-05T13:56:36Z

On last November, we released version 0.5 of ocplib-simplex, a generic library implementing the Simplex Algorithm in OCaml. It is a key component of the Alt-Ergo automatic theorem prover that we keep developing at OCamlPro.

Table of contents

The Simplex Algorithm
What Changed in 0.5 ?

Try ocplib-simplex before implementing your own library !

The simplex algorithm

The Simplex Algorithm is well known among linear optimization enthusiasts. Let's say you own a manufacture producing two kinds of chairs: the first is cheap, you make a small profit out of them but they are quick to produce; the second one is a bit more fancy, you make a bigger profit but they need a lot of time to build. You have a limited amount of wood and time. How many cheap and fancy chairs should you produce to optimize your profits?

You can represent this problem with a set of mathematical constraints (more precisely, linear inequalities) which is precisely the scope of the simplex algorithm. Given a set of linear inequalities, it computes a solution maximizing a given value (in our example, the total profit). If you are interested in the detail of the algorithm, you shoud definitely watch this video.

The simplex algorithm is known to be a difficult problem in terms of complexity. While the base algorithm is EXP-time, it is generally very efficient in practice.

What Changed in 0.5 ?

Among the main changes in this new version of ocplib-simplex:

Make the library's API more generic and easier to use (see the System Solving Example or the Linear Optimization Example);
All the modules are better documentated in their .mli interfaces (see coreSig.mli for example);
the build system has been switched to dune

We hope that this work of simplification will help you to integrate this library more easily in your projects!

If you want to follow this project, report an issue or contribute, you can find it on GitHub.

Please do not hesitate to contact us at OCamlPro: alt-ergo@ocamlpro.com.

The Growth of the OCaml Distribution

2023-01-02T13:56:36Z

We recently worked on a project to build a binary installer for OCaml, inspired from RustUp for Rust. We had to build binary packages of the distribution for every OCaml version since 4.02.0, and we were surprised to discover that their (compressed) size grew from 18 MB to about 200 MB. This post gives a survey of our findings.

Table of contents

Introduction
General Trends
Causes and Consequences
Inside the OCaml Installation
Conclusion

Introduction

One of the strengths of Rust is the ease with which it gets installed on a new computer in user space: with a simple command copy-pasted from a website into a terminal, you get all what you need to start building Rust projects in a few seconds. Rustup, and a set of prebuilt packages for many architectures, is the project that makes all this possible.

OCaml, on the other hand, is a bit harder to install: you need to find in the documentation the proper way for your operating system to install opam, find how to create a switch with a compiler version, and then wait for the compiler to be built and installed. This usually takes much more time.

As a winter holiday project, we worked on a project similar to Rustup, providing binary packages for most OCaml distribution versions. It builds upon our experience of opam and opam-bin, our plugin to build and share binary packages for opam.

While building binary packages for most versions of the OCaml distribution, we were surprised to discover that the size of the binary archive grew from 18 MB to about 200 MB in 10 years. Though on many high-bandwidth connexions, it is not a problem, it might become one when you go far from big towns (and fortunately, we designed our tool to be able to install from sources in such a case, compromising the download speed against the installation speed).

We decided it was worth trying to investigate this growth in more details, and this post is about our early findings.

General Trends

In 10 years, the OCaml Distribution binary archive grew by a factor 10, from 18 MB to 198 MB, corresponding to a growth from 73 MB to 522 MB after installation, and from 748 to 2433 installed files.

So, let's have a look at the evolution of the size of the binary OCaml distribution in more details. Between version 4.02.0 (Aug 2014) and version 5.0.0 (Dec 2022):

The size of the compressed binary archive grew from from 18 MB to 198 MB
The size of the installed binary distribution grew from 73 MB to 522 MB
The number of installed files grew from 748 to 2433

The OCaml Distribution source archive was much more stable, with a global growth smaller than 2.

On the other hand, the source distribution itself was much more stable:

The size of the compressed source archive grew only from 3 MB to 5 MB
The size of the sources grew from 14 MB to 26 MB
The number of source files grew from 2355 to 4084

For our project, this evolution makes the source distribution a good alternative to binary distributions for low-bandwidth settings, especially as OCaml is much faster than Rust at building itself. For the record, version 5.0.0 takes about 1 minute to build on a 16-core 64GB-RAM computer.

Interestingly, if we plot the total size of the binary distribution, and the total size with only files that were present in the previous version, we can notice that the growth is mostly caused by the increase in size of these existing files, and not by the addition of new files:

The growth is mostly caused by the increase in size of existing files, and not by the addition of new files.

Causes and Consequences

We tried to identify the main causes of this growth: the growth is linear most of the time, with sharp increases (and decreases) at some versions. We plotted the difference in size, for the total size, the new files, the deleted files and the same files, i.e. the files that made it from one version to the next one:

The difference of size between two versions is not big most of the time, but some versions exhibit huge increases or decreases.

Let's have a look at the versions with the highest increases in size:

+86 MB for 4.08.0: though there are a lot of new files (+307), they only account for 3 MB of additionnal storage. Most of the difference comes from an increase in size of both compiler libraries (probably in relation with the use of Menhir for parsing) and of some binaries. In particular:
- +13 MB for bin/ocamlobjinfo.byte (2_386_046 -> 16_907_776)
- +12 MB for bin/ocamldep.byte (2_199_409 -> 15_541_022)
- +6 MB for bin/ocamldebug (1_092_173 -> 7_671_300)
- +6 MB for bin/ocamlprof.byte (630_989 -> 7_043_717)
- +6 MB for lib/ocaml/compiler-libs/parser.cmt (2_237_513 -> 9_209_256)
+74 MB for 4.03.0: again, though there are a lot of new files (+475, mostly in compiler-libs), they only account for 11 MB of additionnal storage, and a large part is compensated by the removal of ocamlbuild from the distribution, causing a gain of 7 MB.

Indeed, most the increase in size is probably caused by the compilation with debug information (option -g), that increases considerably the size of all executables, for example:
- +12 MB for bin/ocamlopt (2_016_697 -> 15_046_969)
- +9 MB for bin/ocaml (1_833_357 -> 11_574_555)
- +8 MB for bin/ocamlc (1_748_717 -> 11_070_933)
- +8 MB for lib/ocaml/expunge (1_662_786 -> 10_672_805)
- +7 MB for lib/ocaml/compiler-libs/ocamlcommon.cma (1_713_947 -> 8_948_807)
+72 MB for 4.11.0: again, the increase almost only comes from existing files. For example:
- +16 MB for bin/ocamldebug (8_170_424 -> 26_451_049)
- +6 MB for bin/ocamlopt.byte (21_895_130 -> 28_354_131)
- +5 MB for lib/ocaml/extract_crc (659_967 -> 6_203_791)
- +5 MB for bin/ocaml (17_074_577 -> 22_388_774)
- +5 MB for bin/ocamlobjinfo.byte (17_224_939 -> 22_523_686)
Again, the increase is probably related to adding more debug information in the executable (there is a specific PR on ocamldebug for that, and for all executables more debug info is available for each allocation);
+48 MB for 5.0.0: a big difference in storage is not surprising for a change in a major version, but actually half of the difference just comes from an increase of 23 MB of bin/ocamldoc;
+34 MB for 4.02.3: this one is worth noting, as it comes at a minor version change. The increase is mostly caused by the addition of 402 new files, corresponding to cmt/cmti files for the stdlib and compiler-libs

We could of course study some other versions, but understanding the root causes of most of these changes would require to go deeper than what we can in such a blog post. Yet, these figures give good hints for experts on which versions to start investigating with.

Inside the OCaml Installation

Before concluding, it might also be worth studying which parts of the OCaml Installation take most of the space. 5.0.0 is a good candidate for such a study, as libraries have been moved to separate directories, instead of all being directly stored in lib/ocaml.

Here is a decomposition of the OCaml Installation:

Total: 529 MB
- share: 1 MB
- man: 4 MB
- bin: 303 MB
- lib/ocaml: 223 MB
  - compiler-libs: 134 MB
  - expunge: 20 MB

As we can see, a large majority of the space is used by executables. For example, all these ones are above 10 MB:

28 MB ocamldoc
26 MB ocamlopt.byte
25 MB ocamldebug
21 MB ocamlobjinfo.byte, ocaml
20 MB ocamldep.byte, ocamlc.byte
19 MB ocamldoc.opt
18 MB ocamlopt.opt
15 MB ocamlobjinfo.opt
14 MB ocamldep.opt, ocamlc.opt, ocamlcmt

There are both bytecode and native code executables in this list.

Conclusion

Our installer project would benefit from having a smaller binary OCaml distribution, but most OCaml users in general would also benefit from that: after a few years of using OCaml, OCaml developers usually end up with huge $HOME/.opam directories, because every opam switch often takes more than 1 GB of space, and the OCaml distribution takes a big part of that. opam-bin partially solves this problem by sharing equal files between several switches (when the --enable-share configuration option has been used).

Here is a short list of ideas to test to decrease the size of the binary OCaml distribution:

Use the same executable for multiple programs (ocamlc.opt, ocamlopt.opt, ocamldep.opt, etc.), using the first command argument to choose the behavior to have. Rustup, for example, only installs one binary in $HOME/.cargo/bin for cargo, rustc, rustup, etc. and actually, our tool does the same trick to share the same binary for itself, opam, opam-bin, ocp-indent and drom.
Split installed files into separate opam packages, of which only one would be installed as the compiler distribution. For example, most cmt files of compiler-libs are not needed by most users, they might only be useful for compiler/tooling developers, and even then, only in very rare cases. They could be installed as another opam package.
Remove the -linkall flag on ocamlcommon.cm[x]a libraries. In general, such a flag should only be set when building an executable that is expected to use plugins, because otherwise, this executable will contain all the modules of the library, even the ones that are not useful for its specific purpose.

WebAssembly/Wasm and OCaml

2022-12-14T13:56:36Z

The Dragon-Camel is raging at the sight of all the challenges we overcome!

In this first post about WebAssembly (Wasm) and OCaml, we introduce the work we have been doing for quite some time now, though without publicity, about our participation in the Garbage-Collection (GC) Working Group for Wasm, and two related development projects in OCaml.

WebAssembly, a fast and portable bytecode

WebAssembly is a low-level, binary format that allows compiled code to run efficiently in the browser. Its roadmap is decided by Working Groups from multiple organizations and companies, including Microsoft, Google, and Mozilla. These groups meet regularly to discuss and plan the development of WebAssembly, with the broader community of developers, academics, and other interested parties to gather feedback and ideas for the future of WebAssembly.

There are multiple projects in OCaml related to Wasm, notably Wasicaml, a production-ready port of the OCaml bytecode interpreter to Wasm . However, these projects don't tackle the domain we would like to address, and for good reasons: they target the existing version of Wasm, which is basically a very simple programming language with no data structures, but with an access to a large memory array. Almost anything can of course be compiled to something like that, but there is a big restriction: the resulting program can interact with the outside world only through the aforementioned memory buffer. This is perfectly fine if you write Command-Line Interface (CLI) tools, or workers to be deployed in a Content Delivery Network (CDN). However, this kind of interaction can become quite tedious if you need to deal with abstract objects provided by your environment, for example DOM objects in a browser to manipulate webpages. In such cases, you will need to write some wrapper access functions in JavaScript (or OCaml with js_of_ocaml of course), and you will have to be very careful about the lifetime of those objects to avoid memory leaks.

Hence the shiny new proposals to extend Wasm with various useful features that can be very convenient for OCaml. In particular, three extensions crucially matter to us, functional programmers: the Garbage Collection, Exceptions and Tail-Call proposals.

Our involvement in the GC-related Working Group

The Wasm committee has already worked on these proposals for a few years, and the Exceptions and Tail-Call proposals are now quite satisfying. However, this is not yet the case for the GC proposal. Indeed, finding a good API for a GC that is compatible with all the languages in the wild, that can be implemented efficiently, and can be used to run a program you don't trust, is all but an easy task. Multiple attempts by strong teams, for different virtual machines, have exposed limitations of past proposals. But, we must now admit that the current proposal has reached a state where it is quite impressive, being both simple and generic.

The proposal is now getting close to a feature freeze status. Thanks to the hard work of many people on the committee, including us, the particularities of functional typed languages were not forgotten in the design, and we are convinced that there should be no problem for OCaml. Now is the time to test it for real!

Targetting Wasm from the OCaml Compiler

Adding a brand new backend to a compiler to target something that is quite different from your usual assembly can be a huge work, and only a few language developers actively work on making a prototype for Wasm+GC. Yet, we think that it is important for the committee, to have as many examples as possible to validate the proposal and move it to the next step.

That's the reason why we decided to contribute to the proposal, by prototyping a backend for Wasm to the OCaml compiler.

Our experimental Wasm interpreter in OCaml

In parallel, we are also working on the development of our own Wasm Virtual Machine in OCaml, to be able to easily experiment both on the OCaml side and Wasm side, while waiting for most official Wasm VM to fully implement the new proposals.

These experimental projects and related discussions are very important design steps, although obviously far from production-ready status.

As our current work focuses on OCaml 4.14, effect handlers are left for future work. The current proposal that would make it possible to compile effect handlers to Wasm nicely is still in its earlier stages. We hope to be able to prototype it too on our Wasm VM.

Note that we are looking for sponsors to fund this work. If supporting Wasm in OCaml may impact your business, you can contact us to discuss how we can use your help!

Our next blog post in January will provide more technical details on our two prototyping efforts.

Alt-Ergo: the SMT solver with model generation

2022-11-16T13:56:36Z

The Alt-Ergo automatic theorem prover developed at OCamlPro has just been released with a major update : counterexample model can now be generated. This is now available on the next branch, and will officially be part of the 2.5.0 release, coming this year !

Alt-Ergo at a Glance

Alt-Ergo is an open source automatic theorem prover based on the SMT technology. It was born at the Laboratoire de Recherche en Informatique, Inria Saclay Ile-de-France and CNRS in 2006 and has been maintained and developed by OCamlPro since 2013.

It is capable of reasoning in a combination of several built-in theories such as:

uninterpreted equality;
integer and rational arithmetic;
arrays;
records;
algebraic data types;
bit vectors.

It also is able to deal with commutative and associative operators, quantified formulas and has a polymorphic first-order native input language. Alt-Ergo is written in OCaml. Its core has been formally proved in the Coq proof assistant.

Alt-Ergo has been involved in a qualification process (DO-178C) by Airbus Industrie. During this process, a qualification kit has been produced. It was composed of a technical document with tool requirements (TR) that gives a precise description of each part of the prover, a companion document (~ 450 pages) of tests, and an instrumented version of the tool with a TR trace mechanism.

Model Generation

When a property is false, generating a counterexample is a key that many state-of-the-art SMT-solvers should include by default. However, this is a complex problem in the first place.

The first obstacle is the decidability of the theories manipulated by the SMT solvers. In general, the complexity class (i.e. the classification of algorithmic problems) is between "NP-Hard" (for the linear arithmetic theory on integers for example) and "Undecidable" (for the polynomial arithmetic on integers for example). Then comes the quantified properties, i.e. properties prefixed with foralls and exists, adding an additional layer of complexity and undecidability. Another challenge was the core algorithm behind Alt-Ergo which does not natively support model generation. At last, an implementation of the models have to take care of Alt-Ergo's support of polymorphism.

How to use Model Generation in Alt-Ergo

There are two ways to activate model generation on Alt-Ergo.

Basic usage: simply add the option --model to your command ($ alt-ergo file --model)
Advanced usage: three options mainly impact the model generation.
- --interpretation: sets the model generation strategy. It can either be none for no model generation; first for generating the very first interpretation computed only; every for generating a model after each decision and last only generating a model when alt-ergo concludes on the formula satisfiability.
- --sat-solver: only the 'Tableaux-CDCL' sat solver is compatible with the interpretation feature
- --instantiation-heuristic: when set to normal, alt-ergo generates model faster. This is an experimental feature that sometimes generates incorrect models.
  
  Example:
  
  $ alt-ergo file --interpretation every --sat-solver Tableaux-CDCL --instantiation-heuristic auto

Warning: only the linear arithmetic and the enum model generation have been tested. Other theories are either not implemented (ADTs) or experimental (risk of crash or unsound models). We are currently still heavily testing the feature, so feel free to join us on Alt-Ergo's GitHub repository if you have questions or issues with this new feature. Note that the models generated are best-effort models; Alt-Ergo does not answer Sat when it outputs a model. In a future version, we will add a mechanism that automatically checks the model generated.

Godspeed!

Acknowledgements

We want to thank David Mentré and Denis Cousineau at Mitsubishi Electric R&D Center Europe for funding the initial work on counterexample.

Note that MERCE has been a Member of the Alt-Ergo Users’ Club for 3 years. This partnership allowed Alt-Ergo to evolve and we hope that more users will join the Club on our journey to make Alt-Ergo a must-have tool. Please do not hesitate to contact the Alt-Ergo team at OCamlPro: alt-ergo@ocamlpro.com.

Let's Encrypt Wildcard Certificates Made Easy with Agnos

2022-10-05T13:56:36Z

It is with great pleasure that we announce the first beta release of Agnos. A former personal project of our new recruit, Arthur, Agnos development is now hosted at and sponsored by OCamlPro's Rust division, Red Iron.

TL;DR: If you are familiar with ACME providers like Let's Encrypt, DNS-01 and the challenges relating to wildcard certificates, simply know that Agnos touts itself as a single-binary, API-less, provider-agnostic dns-01 client, allowing you to easily obtain wildcard certificates without having to interface with your DNS provider. To do so, it offers a user-friendly configuration and answers Let's Encrypt DNS-01 challenges on its own, bypassing the need for API calls to edit DNS zones. You may want to jump to the last section of this post, or directly join us on Agnos's github.

Agnos was born from the observation that even though wildcard certificates are in many cases more convenient and useful than their fully qualified counterparts, they are not often used in practice. As of today it is not uncommon to see certificates with multiple Subject Alternate Names (SAN) for multiple subdomains, which can become problematic and weaken infrastructure. If some situations indeed require to forego wildcard certificates, this choice is too often still a default one.

At OCamlPro, we believe that technical difficulties should not stand in the way of optimal decision making, and that compromises should only be made in the face of unsolvable challenges. By releasing this first beta of Agnos, we hope that your feedback we'll help us build a tool truly useful to the community and that together, we can open a path towards seamless wildcard certificate issuance – tossing away issues and pain-points previously encountered as a thing of the past.

This blog post describes the different ACME challenges, why DNS providers API have so far been hindering DNS-01 adoption, and how Agnos solves this issue. If you are already curious and want to run some code, let's meet on Agnos's github

Let's encrypt's mechanism and ACME challenges

The Automatic Certificate Management Environment (ACME) is the protocol behind automated certificate authority services like Let's Encrypt. At its core, this protocol requires the client asking for a certificate to provide evidence that they control a resource by having said resource display some authority-determined token.

The easiest way to do so is to serve a file on a web-server. For example serving a file containing the token at my-domain.example would prove that I control the web-server that the fully qualified domain name my-domain.example is pointing to. This, under normal circumstances proves that I somewhat control this fully qualified domain. This process is illustrated below.

The ACME client initiates the certificate issuance process and is challenged to serve the token via HTTP at the domain address. The ACME client and HTTP server can be and often are on the same machine. The token can be quickly provisioned, and the ACME client can ask the ACME server to validate the challenge and issue the certificate.

However, demonstrating that one controls an HTTP server pointed to by my-domain.example is not deemed enough by Let's Encrypt to demonstrate full control of the my-domain.example domain and all its subdomains. Hence, the user cannot be issued a wildcard certificate through this method.

To obtain a wildcard certificate, one must rely on the DNS-01 type of challenge, illustrated below. The ACME client initiates the certificate issuance process and is challenged to serve the token via a DNS TXT record. Because DNS management is often delegated to a DNS provider, the DNS server is rarely on the same machine, and the token must be provisioned via a call to the DNS provider API, if there is any. Moreover, DNS providers virtually always use multiple servers, and the new record must be propagated to all of them. The ACME client must then wait and check for the propagation to be finished before asking the ACME server to validate the challenge and issue the certificate.

The pros and cons of each of these two challenge type are summarized by Let's Encrypt's documentation as follow:

HTTP-01

Pros

It’s easy to automate without extra knowledge about a domain’s configuration.

It allows hosting providers to issue certificates for domains CNAMEd to them.

It works with off-the-shelf web servers.

Cons

It doesn’t work if your ISP blocks port 80 (this is rare, but some residential ISPs do this).

Let’s Encrypt doesn’t let you use this challenge to issue wildcard certificates.

If you have multiple web servers, you have to make sure the file is available on all of them.

DNS-01

Pros

You can use this challenge to issue certificates containing wildcard domain names.

It works well even if you have multiple web servers.

Cons

Keeping API credentials on your web server is risky.

Your DNS provider might not offer an API.

Your DNS API may not provide information on propagation times.

Agnos as the best of both worlds

By using NS records to delegate the DNS-01 challenge to Agnos itself, we can virtually remove all of DNS-01 cons. Indeed by serving its own DNS answers, Agnos:

Nullifies the need for API and API credentials
Nullifies all concerns regarding propagation times

In more details, Agnos proceeds as follows (and as illustrated below). Before any ACME transaction takes place (and only once), the ACME client user manually updates their DNS zone to delegate ACME specific subdomains to Agnos. Note that the rest of DNS functionality is still assumed by the DNS provider. To carry out the ACME transaction, the ACME client initiates the certificate issuance process and is challenged to serve the token via a DNS TXT record. Agnos does so using its own DNS functionality (leveraging Trust-dns). The ACME client can immediately ask the ACME server for validation. The ACME server asks the DNS provider for the TXT record and is replied to that the ACME specific subdomain is delegated to Agnos. The ACME server then asks Agnos-as-a-DNS-server for the TXT record which Agnos provides. Finally the certificate is issued and stored by Agnos on the client machine.

Taking Agnos for a ride

In conclusion, we hope that by switching to Agnos, or more generally to provider-agnostic DNS-01 challenge solving, individuals and organizations will benefit from the full power of DNS-01 and wildcard certificates, without having to take API-related concerns into account when choosing their DNS provider.

If this post has piqued your interest and you want to help us develop Agnos further by trying the beta out, let's meet on our github. We would very much appreciate any feedback and bug reports, so we tried our best to streamline and well document the installation process to facilitate new users. On ArchLinux for example, getting started can be as easy as:

Adding two records to your DNS zone using your provider web GUI:

agnos-ns.doma.in            A       1.2.3.4
_acme-challenge.doma.in     NS      agnos-ns.doma.in

and running on your server

# Install the agnos binary
yay -S agnos
# Allow agnos to bind the priviledge 53 port
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/agnos
# Download the example configuration file
curl 'https://raw.githubusercontent.com/krtab/agnos/v.0.1.0-beta.1/config_example.toml' > agnos_config.toml
# Edit it to suit your need
vim agnos_config.toml
# Launch agnos 🚀
agnos agnos_config.toml

Until then, happy hacking!

opam 2.1.3 is released!

2022-08-12T13:56:36Z

Feedback on this post is welcomed on Discuss!

We are pleased to announce the minor release of opam 2.1.3.

This opam release consists of backported fixes:

Fix opam init and opam init --reinit when the jobs variable has been set in the opamrc or the current config. (#5056)
opam var no longer fails if no switch is set (#5025)
Setting a variable with option --switch <sw> fails instead of writing an invalid switch-config file (#5027)
Handle external dependencies when updating switch state pin status (all pins), instead as a post pin action (only when called with opam pin (#5046)
Remove windows double printing on commands and their output (#4940)
Stop Zypper from upgrading packages on updates on OpenSUSE (#4978)
Clearer error message if a command doesn't exist (#4112)
Actually allow multiple state caches to co-exist (#4554)
Fix some empty conflict explanations (#4373)
Fix an internal error on admin repository upgrade from OPAM 1.2 (#4965)

and improvements:

When inferring a 2.1+ switch invariant from 2.0 base packages, don't filter out pinned packages as that causes very wide invariants for pinned compiler packages (#4501)
Some optimisations to opam list --installable queries combined with other filters (#4311)
Improve performance of some opam list combinations (e.g. --available, --installable) (#4999)
Improve performance of opam list --conflicts-with when combined with other filters (#4999)
Improve performance of opam show by as much as 300% when the package to show is given explicitly or is unique (#4997)(#4172)
When a field is defined in switch and global scope, try to determine the scope also by checking switch selection (#5027)

You can also find API changes in the release note.

Opam installation instructions (unchanged):

From binaries: run
```
$ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.3"
```
or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.
From source, using opam:
```
$ opam update; opam install opam-devel
```
(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)
From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

OCamlPro at the JFLA2022 Conference

2022-07-12T13:56:36Z

Domaine d'Essendiéras , located in French Region Perigord, where the JFLA2022 took place!

In today's article, we share our contributions to the 2022 JFLAs, the French-Speaking annual gathering on Application Programming Languages, mainly Functional Languages such as OCaml (Journées Francophones des Langages Applicatifs).

This much awaited event is organised by Inria, the French National Institute for Research in Science and Digital Technologies.

This is always the best occasion for us to directly share our latest works and contributions with this diverse community of researchers, professors, students and industrial actors alike. Moreover, it allows us to meet up with all our long known peers and get in contact with an ever changing pool of actors in the fields of functional languages in general, formal methods and everything OCaml!

This year the three papers we submitted were selected, and this is what this article is about!

Table of contents

Mikino, formal verification made accessible

Connecting Software Heritage with the OCaml ecosystem

Alt-Ergo-Fuzz, hunting the bugs of the bug hunter

Mikino, formal verification made accessible

Mikino and all correlated content mentionned in this article was made by Adrien Champion

If you follow our Blog, you may have already read our Mikino blogpost, but if you haven't here's a quick breakdown and a few pointers... In case you wish to play around or maybe contribute to the project. ;)

So what is Mikino ?

Mikino is a simple induction engine over transition systems. It is written in Rust, with a strong focus on ergonomics and user-friendliness.

Depending on what your needs are, you could either be interested in the Mikino Api or the Mikino Binary or just, for purely theoretical reasons, want to undergo our Verification for Dummies: SMT and Induction tutorial which is specifically tailored to appeal to the newbies of formal verification!

Have a go at it, learn and have fun!

For further reading: OCamlPro's paper for the JFLA2022 (Mikino) (French-written article describing the entire project).

Connecting Software Heritage with the OCaml ecosystem

The archiving of OCaml packages into the SWH architecture, the release of swhid library and the integration of SWH into opam was done by Léo Andrès, Raja Boujbel, Louis Gesbert and Dario Pinto

Once again, if you follow our Blog, you must have seen Software Heritage (SWH) mentioned in our yearly review article.

Now you can also look at SWH paper by OCamlPro for the JFLA2022 (French) if you are looking for a detailed explanation of how important Software Heritage is to free software as a whole, and in what manner OCamlPro contributed to this gargantuan long-term endeavour of theirs.

This great collaboration was one of the highlights of last year from which arose an OCaml library called swhid and the guaranteed perennity of all the packages found on opam.

The work we did to achieve this was to:

add a few modules to the SWH architecture in order to store all the OCaml packages found on opam in the Library of Alexandria of open source software.
release a library used for computing SWH identifiers
add support in opam in order to allow a fallback on SWH architecture if a given package is missing from the opam repository
patch the opam repository in order to detect already missing packages

Alt-Ergo-Fuzz, hunting the bugs of the bug hunter

The fuzzing of the SMT-Solver Alt-Ergo was done by Hichem Rami Ait El Hara, Guillaume Bury and Steven de Oliveira

As the last entry of OCamlPro's papers that have made it to this year's JFLA: a rundown of Hichem's work, guided by Guillaume and Steven, on developping a Fuzzer for Alt-Ergo.

When it comes to critical systems, and industry-borne software, there are no limits to the requirements in safety, correctness, testing that would prove a program's reliability.

This is what SMT (Satisfiability Modulo Theory)-Solvers like Alt-Ergo are for: they use a complex mix of theory and implementation in order to prove, given a set of input theories, whether a program is acceptable... But SMT-Solvers, like any other program in the world, has to be searched for bugs or unwanted behaviours - this is the harsh reality of development.

With that in mind, Hichem sought to provide a fuzzer for Alt-Ergo to help hunt the bugs of the bug hunter: Alt-Ergo-Fuzz.

This tool has helped identify several bugs of unsoundness and crashes:

#474 - Crash
#475 - Crash
#476 - Unsoundness
#477 - Unsoundness
#479 - Unsoundness
#481 - Crash
#482 - Crash

More details in OCamlPro's paper for the JFLA2022 (Alt-Ergo-Fuzz).

2021 at OCamlPro

2022-01-31T13:56:36Z

Passing from one year to another is a great time to have a look back!

OCamlPro was created in 2011 to advocate the adoption of the OCaml language and Formal Methods in general in the industry. 2021 was a very special year as we celebrated our 10th anniversary! While building a team of highly-skilled engineers, we navigated through our expertise domains, programming languages design, compilation and analysis, advanced developer tooling, formal methods, blockchains and high-value software prototyping.

In this article, as every year (see last year's post), we review some of the work we did during 2021, in many different worlds.

Table of contents

Newcomers at OCamlPro

Real Life Modern Applications

Modernizing the French Income Tax System
A First Step in the COBOL Universe
Auditing a High-Scale Genealogy Application
Improving an ecotoxicology platform

Contributions to OCaml

Flambda Code Optimizer
Opam Package Manager
LearnOCaml and TryOCaml
OCaml Documentation Hub
Plugging Opam into Software Heritage

Tooling for Formal Methods

Alt-Ergo Development
Alt-Ergo Users’ Club and R&D Projects
Dolmen Library for Automated Deduction Languages

Rust Developments

SMT, Induction and Mikino
Matla, a Project Manager for TLA+/TLC
Rust Training at Onera
Audit of a Rust Blockchain Node

Scaling and Verifying Blockchains

From Dune Network to FreeTON/EverScale
A Why3 Framework for Solidity

Participations in Public Events

Open Source Experience 2021
OCaml Workshop at ICFP 2021
Joining the Why3 Consortium at the ProofInUse Seminar

Towards 2022

As always, we warmly thank all our clients, partners, and friends, for their support and collaboration during this peculiar year!

Newcomers at OCamlPro

Some of the new and old members of the team: Pierre Chambart, Dario Pinto, Léo Andrès, Fabrice Le Fessant, Louis Gesbert, Artemiy Rozovyk, Muriel Shan Sei Fan, Nicolas Berthier, Vincent Laviron, Steven De Oliveira and Keryan Didier.

A company is nothing without its employees. This year, we have been delighted to welcome a great share of newcomers:

Hichem Rami Ait El Hara recently completed his master's degree in Computer Science. After an internship at OCamlPro, during which he developed a fuzzer for Alt-Ergo, he joined OCamlPro to work on Alt-Ergo and the verification of smart contracts. He will soon start a PhD on SMT solving.
Nicolas Berthier holds a PhD on synchronous programming for resource-constrained systems. With many years experience on model-checking, abstract interpretation, and software analysis, he joined OCamlPro to work on programming language compilation and analysis.
Julien Blond is a senior OCaml developer with a strong experience in formal verification of security software. He joined OCamlPro as both a project manager and a Coq expert.
Keryan Didier joined the team as a R&D engineer. He holds a PhD from University Pierre et Marie Curie, during which he developed an automated implementation method for hard real-time applications. Previously, he studied functional programming and language design at University Paris-Diderot. Keryan has been involved in the MLang project as well as the flambda2 project within OCamlPro's Compilation team.
Mohamed Hernouf recently completed his master's degree in Computer Science. After an internship at OCamlPro, working on the OCaml Documentation Hub, he joined OCamlPro and continues to work on the documentation hub and other OCaml applications.
Dario Pinto is a student at the 42Paris School of Computer Science. He joined OCamlPro in a work-study contract for two years.
Artemiy Rozovyk recently completed his master's degree in Computer Science. He joined OCamlPro to work on the development of applications for the EverScale and Avalanche blockchains.

Real Life Modern Applications

Modernizing the French Income Tax System

The M language, designed in the 80s for the Income Tax, is now being rewritten and extended in OCaml.

The M language is a very old programming language developed by the French tax administration to compute income taxes. Recently, Denis Merigoux and Raphael Monat have implemented a new compiler in OCaml for the M language. This new compiler shows better performance, clearer semantics, and achieves greater maintainability than the former compiler. OCamlPro is now involved in strengthening this new compiler, to put it in production and eventually compute the taxes of more than 30 million French families.

A First Step in the COBOL Universe

Recent studies still estimate that COBOL has the highest amount of lines of code running.

Born more than 60 years ago, COBOL is still said to be the most used language in the world, in terms of the number of lines running in computers, though many people forecast it would disappear at the edge of the 21st century. With more than 300 reserved keywords, it is also one of the most complex languages to parse and analyse. It's not enough to scare the developers at OCamlPro: while helping one of the biggest COBOL users in France to translate its programs into the GNUCobol open-source compiler, OCamlPro built a strong expertise of COBOL and mainframes, and developed a powerful parser of COBOL that will help us bring modern development tools to the COBOL developers.

Auditing a High-Scale Genealogy Application

Geneweb was developed in the 90s to manage family trees... and is still managing them!

Geneweb is one of the most powerful software to manage and share genealogical data to date. Written in OCaml more than 20 years ago, it contains a web server and complex algorithms to compute information on family trees. It is used by Geneanet, which is one of the leading companies in the genealogy field, to store more than 800,000 family trees and more than 7 billion names of ancestors. OCamlPro is now working with Geneanet to improve Geneweb and make it scale to even larger data sets.

Improving an ecotoxicology platform

Mosaic is used by ecotoxicologists and regulators to obtain advanced and innovative methods for environmental risks assessment.

The Mosaic platform helps researchers, industrials actors and regulators in the field of ecotoxicology by providing an easy way to run various statistical analyses. All the user has to do is to enter some data on the web interface, then computations are run on the server and the results are displayed. The platform is fully written in OCaml and takes care of calling the mathematical model which is written in R. OCamlPro modernised the project in order to ease maintainance and new contributions. In the process, we discovered bugs introduced by new R versions (without any kind of warning). Then we developped a new interface for data input, it's similar to a spreadsheet and much more convenient than having to write raw CSV. During this work, we had the opportunity to contribute to some other OCaml packages such as leveldb or write new ones such as agrid.

Contributions to OCaml

Flambda Code Optimizer

Flambda2 is a powerful code optimizer for the OCaml compiler.

OCamlPro is proud to be working on Flambda2, an ambitious OCaml optimizing compiler project, initiated with Mark Shinwell from Jane Street, our long-term partner and client. Flambda focuses on reducing the runtime cost of abstractions and removing as many short-lived allocations as possible. Jane Street has launched large-scale testing of flambda2, and on our side, we have documented the design of some key algorithms. In 2021, the Flambda team grew bigger with Keryan. Along with the considerable amount of fixes and improvements, this will allow us to publish Flambda2 in the coming months!

In other OCaml compiler news, 2021 saw the long-awaited merge of the multicore branch into the official development branch. This was thanks to the amazing work of many people, including our own, Damien Doligez. This is far from the end of the story though, and we're looking forward to both further contributing to the compiler (fixing bugs, re-enabling support for all platforms) and making use of the features in our own programs.

This work is allowed thanks to Jane Street’s funding.

Opam Package Manager

A large set of new features have been implemented in Opam in 2021.

Opam is the OCaml source-based package manager. The first specification draft was written in early 2012 and went on to become OCaml’s official package manager — though it may be used for other languages and projects, since Opam is language-agnostic! If you need to install, upgrade and manage your compiler(s), tools and libraries easily, Opam is meant for you. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.

Opam development and maintenance is a collaboration between OCamlPro, with Raja & Louis, and OCamlLabs, with David Allsopp & Kate Deplaix.

Our 2021 work on opam lead to the final release of the long-awaited opam 2.1, three versions of opam 2.0 and two versions of opam 2.1 with small fixes.

Opam 2.1 introduced several new features:

Integration of system dependencies (formerly the opam-depext plugin)
Creation of lock files for reproducible installations (formerly the opam-lock plugin)
Switch invariants, replacing the "base packages" in opam 2.0 and allowing for easier compiler upgrades
Improved option configurations
CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
opam root readability by newer and older versions, even if the format changed
Performance improvements to opam-update, conflict messages, and many other areas

Take a stroll through the blog post for a closer look.

In 2021, we also prepared the soon to-be alpha release of opam 2.2 version. It will provide a better handling of the Windows ecosystem, integration of Software Heritage archive fallback, better UI on user interactions, recursively pinning of projects, fetching optimisations, etc.

This work is greatly helped by Jane Street’s funding and support.

LearnOCaml and TryOCaml

We have also been active in the maintainance of Learn-ocaml. What was originally designed as the platform for the OCaml MOOC is now a tool in the hands of OCaml teachers worldwide, managed and funded by the OCaml Foundation.

The work included a well overdue port to OCaml 4.12; generation of portable executables (automatic through CI) for much easier deployment and use of the command-line client; as well as many quality-of-life and usability improvements following from two-way conversations with many teachers.

On a related matter, we also reworked our on-line OCaml editor and toplevel TryOCaml, improving its design and adding features like code snippet sharing. We were glad to see that, in these difficult times, these tools proved useful to both teachers and students, and look forward to improving them further.

OCaml Documentation Hub

The OCaml Documentation Hub includes browsable documentation and sources for more than 2000 Opam packages.

As one of the biggest user of OCaml, OCamlPro aims at facilitating daily use of OCaml by developing a lot of open-source tooling.

One of our main contributions to the OCaml ecosystem in 2021 was probably the OCaml Documentation Hub at docs.ocaml.pro.

The OCaml Documentation Hub is a website that provides documentation for more than 2000 OPAM packages, among which of course the most popular ones, with inter-package documentation links! The website also contains browsable sources for all these packages, and a search engine to discover useful OCaml functions, modules, types and classes.

All this documentation is generated using our custom tool Digodoc. Though it's not worth a specific section, we also kept maintaining Drom, our layer on Dune and Opam that most of our recent projects use.

Pluging Opam into Software Heritage

Svalbard Global Seed Vault in Norway.

Last year has also seen the long awaited collaboration between Software Heritage and OCamlPro happen.

Thanks to a grant by the Alfred P. Sloan Foundation, OCamlPro has been able to collaborate with our partners at Software Heritage and manage to further expand the coverage of this gargantuan endeavour of theirs by archiving 3516 opam packages. In effect, the main benefits of this Open Source collaboration have been:

The addition of several modules to the Software Heritage architecture, allowing the archiving of said opam packages;
The publication of an OCaml library allowing to work with SWHIDs;
An implementation of a possible fallback onto Software Heritage if a given package on opam is no longer available;
A fix for the official opam repository in order to identify already missing packages.

Not long after Software was at last acknowledged by Unesco as part of the World Heritage, we were thrilled to be part of this great and meaningful initiative. We could feel how true passion remained throughout our interactions and long after the work was done.

Tooling for Formal Methods

Avionics, blockchains, cyber-security, cloud, etc... formal methods are spreading in the computer industry.

Alt-Ergo Development

OCamlPro develops and maintains Alt-Ergo, an automatic solver of mathematical formulas designed for program verification and based on Satisfiability Modulo Theories (SMT). Alt-Ergo was initially created within the VALS team at University of Paris-Saclay.

In 2021, we continued to focus on the maintainability of our solver. We released versions 2.4.0 and 2.4.1 in January and July respectively, with 2.4.1 containing a bugfix as well as some performance improvements.

In order to increase our test coverage, we instrumented Alt-Ergo so that we could run it using afl-fuzz. Although this is a proof of concept, and has yet to be integrated into Alt-ergo's continuous integration, it has already helped us find a few bugs, such as this.

Alt-Ergo Users’ Club and R&D Projects

We thank our partners from the Alt-Ergo Users’ Club, Adacore, CEA List, MERCE (Mitsubishi Electric R&D Centre Europe), Thalès, and Trust-In-Soft, for their trust. Their support allows us to maintain our tool.

The club was launched in 2019 and the third annual meeting of the Alt-Ergo Users’ Club was held in early April 2021. Our annual meeting is the perfect place to review each partner’s needs regarding Alt-Ergo. This year, we had the pleasure of receiving our partners to discuss the roadmap for future Alt-Ergo features and enhancements. If you want to join us for the next meeting (coming soon), contact us!

Finally, we will be able to merge into the main branch of Alt-Ergo some of the work we did in 2020. Thanks to our partner MERCE (Mitsubishi Electric R&D Centre Europe), we worked on the SMT model generation. Alt-Ergo is now (partially) able to output a model in the smt-lib2 format. Thanks to the Why3 team from University of Paris-Saclay, we hope that this work will be available in the Why3 platform to help users in their program verification efforts. OCamlPro was very happy to join the Why3 Consortium this year, for even more collaborations to come!

This work is funded in part by the FUI R&D Project LCHIP, MERCE, Adacore and with the support of the Alt-Ergo Users’ Club.

Dolmen Library for Automated Deduction Languages

Dolmen is a powerful library providing flexible parsers and typecheckers for many languages used in automated deduction.

The ongoing work on using the Dolmen library as frontend for Alt-Ergo has progressed considerably, both on the side of dolemn which has been extended to support Alt-Ergo's native language in this PR, and on Alt-Ergo's side to add dolmen as a frontend that can be chosen in this PR. Once these are merged, Alt-Ergo will be able to read input problems in new languages, such as TPTP!

Rust Developments

Rust is a very good complement to OCaml for performance critical applications.

SMT, Induction and Mikino

A few months ago, we published a series of posts: verification for dummies: SMT and induction. These posts introduce and discuss SMT solvers, the notion of induction and that of invariant strengthening. They rely on mikino, a simple software we wrote that can analyze simple transition systems and perform SMT-based induction checks (as well as BMC, i.e. bug-finding). We wrote mikino in Rust with readability and ergonomics in mind: mikino showcases the basics of writing an SMT-based model checker performing induction. The posts are very hands-on and leverage mikino's high-quality output to discuss induction and invariant strengthening, with examples that readers can run and edit themselves.

Matla, a Project Manager for TLA+/TLC

During 2021 we ended up using the TLA+ language and its associated TLC verification engine in several completely unrelated projects. TLC is an amazing tool, but it is not suited to handle a TLA+ project with many modules (files), regression tests, etc. In particular, TLA+ is not a typed language. This means that TLA+ code tends to have many checks (dynamic assertions) checking that quantities have the expected type. This is fine, albeit a bit tedious, to some extent, but as the code grows bigger the analysis conducted by TLC can become very, very expensive. Eventually it is not reasonable to keep assert-type-checking everything since it contributes to TLC's analysis exploding.

As TLA+/TLC users, we are currently developing matla which manages TLA+ projects. Written in Rust, matla is heavily inspired by the Rust ecosystem, in particular cargo. Matla has not been publicly released yet as we are waiting for more feedback from early users. We do use it internally however as its various features make our TLA+ projects much simpler:

handling the TLA toolchain (download, PATH, updates...) for the user;
provide a Matla module with "debug assertions" helpers: these assertions are active in debug mode, which is the default when running matla run. Passing --release to matla's run mode however compiles all debug assertions away; this allows to type-check everything when debugging while making sure release runs do not pay the price of these checks;
handle integration testing: matla projects have a tests directory where users can write tests (TLA files with a .tla and .cfg files) and specify if they are expected to be successful or to fail (and how);
understand and transform TLC's output to improve user feedback, in particular when TLC yields an error (not good enough yet and is the reason we have not released yet); matla also parses and prettifies TLC's counterexample traces by formatting values, formatting states (aggregation of values), and render traces of states graphically using ASCII art.

Rust Training at Onera

The ongoing pandemic is undoubtingly impacting our professional training activities. Still, we had the opportunity to set up a Rust training session with applied researchers at ONERA during the summer. The session spanned over a week (about seven hours a day) and was our first fully remote Rust training session. We still believe on-site training (when possible) is better, full remote offers some flexibility (spreading out the training over several weeks for instance) and our experience with ONERA shows that it can work in practice with the right technology. Interestingly, it turns out that some aspects of the session actually work better with remote: hands-on exercises and projects for instance benefit from screen sharing. Discussing code with one participant is done with screen sharing, meaning all participants can follow along if they so chose.

Long story short, fully remote training is something we now feel confident proposing to our clients as a flexible alternative to on-site training.

Audit of a Rust Blockchain Node

We participated in a contest aiming at writing a high-level specification of the (compiler for) the TON VM assembler, in particular its instructions and how they are compiled. This contest was a first step towards applying Formal Methods, and in particular formal verification, to the TON VM. We are happy to report that we finished first in this context, and are looking forward to future contests pushing Formal Methods further in the Everscale blockchain.

Scaling and Verifying Blockchains

OCamlPro is involved in several projects with high-throughput blockchains, such as EverScale and Avalanche.

From Dune Network to FreeTON/EverScale

In 2019-2020, we concentrated our efforts on the development of blockchains on adding new programming languages to the Dune Network ecosystem, in collaboration with Origin Labs. You can read more about Love and Solidity for Dune.

At the end of 2020, it became clear that high-throughput was becoming a major requirement for blockchain adoption in real applications, and that the Tezos-based technology behind Dune Network could not compete with high-performance blockchains such as Solana or Avalanche. Following this observation, the Dune Network community decided to merge with the FreeTON community early in 2021. Initially developed by Telegram, the TON project was stopped under legal threats, but another company, TONLabs, restarted the project from its open-source code under the FreeTON name, and the blockchain was launched mid-2020. FreeTON, now renamed EverScale, is today the fastest blockchain in the world, with around 55,000 transactions per second on an open network sustained during several days.

EverScale uses a very unique community-driven development process: contests are organized by thematic sub-governances (subgov) to improve the ecosystem, and contestants win prices in tokens to reward their high-quality work. During 2021, OCamlPro got involved in several of these sub-governances, both as a jury, in the Formal Methods subgov and the Developer Experience subgov, and a contestant winning multiple prices for the development of smart contracts (zksnarks use-cases, auctions and recurring payments), the audit of several smart contracts (TrueNFT audit, Smart Majority Voting audit and a DEX audit), and the specification of some Rust components in the node (the Assembler module).

This work in the EverScale ecosystem gave us the opportunity to develop some interesting OCaml contributions:

We improved our ocaml-solidity parser to support all the extensions of the Solidity language required to parse EverScale contracts;
We developed an OCaml binding for the EverScale Rust SDK;
We developed a command line wallet called ft to help developers easily deploy the contracts and interact with them;
We developed a bridge between Dune Network and EverScale to swap DUN tokens into EVER tokens.

This work was funded by the EverScale community through contests.

A Why3 Framework for Solidity

Our most recent work on the EverScale blockchain has been targetted into the development of a Why3 framework to formally verify EverScale Solidity contracts. At the same time, we have been involved in the specification of several big smart contract projects, and we plan to use this framework in practice on these projects as soon as their formal verification starts.

We hope to be able to extend this work to EVM based Solidity contracts, as available on Ethereum and Avalanche and many other blockchains. By comparison with other frameworks that work directly on the EVM bytecode, this work focused directly on the Solidity language should make the verification much higher-level and so more straight-forward.

Participations in Public Events

Open Source Experience 2021

Stéfane Fermigier (Abilian) and Pierre Baudracco (BlueMind) from Systematic Open Source Hub meet Amélie de Montchalin (French Minister of Public Service) in front of OCamlPro's booth.

We were present at the new edition of the Open Source Experience in Paris! Our booth welcomed our visitors to discuss tailor-made software solutions. Fabrice had the opportunity to give a presentation on FreeTON (Now EverScale) (Watch the video), the high speed blockchain he is working on. We were delighted to meet the open source community. Moreover, Amélie de Montchalin, French Minister of Transformation and Public Service, was present to the Open Source Experience to thank all the free software actors. A very nice experience for us, we can't wait to be back in 2022!

OCaml Workshop at ICFP 2021

We participated in the programming competition organized by the International Conference on Functional Programming (ICFP). 3 talks we submitted to the OCaml Workshop were accepted!

Fabrice, Mohamed and Louis presented Digodoc, our new tool that builds a graph of an opam switch, associating files, libraries and opam packages into a cyclic graph of inclusions and dependencies;
Fabrice spoke about Opam-bin, a plugin that builds binary opam packages on the fly;
Lastly, Steven and David presented Love, a smart contract language embedded in the Dune Network blockchain. It was an opportunity to present our tools and projects, and above all to discuss with the OCaml community. We're delighted to take part in this adventure every year!

Joining the Why3 Consortium at the ProofInUse seminar

We were very happy to join the Why3 Consortium while participating the ProofinUse joint lab seminar on counterexamples on October the 1st. Many thanks to Claude Marché for his role as scientific shepherd.

Towards 2022

Though 2022 is just starting, it already sounds like a great year with many new interesting and innovative projects for OCamlPro.

After a phase of adaptation to the health context in 2020 and a year of growth in 2021, we are motivated to start the year 2022 with new and very enriching projects, new professional encounters, leading to the growth of our teams. If you want to be part of a passionate team, we would love to hear from you! We are currently actively hiring. Check the available job positions and follow the application instructions!

All our amazing achievements are the result of incredible people and teamwork, kudos to Fabrice, Pierre, Louis, Vincent, Damien, Raja, Steven, Guillaume, David, Adrien, Léo, Keryan, Mohamed, Hichem, Dario, Julien, Artemiy, Nicolas, Elias, Marla, Aurore and Muriel.

Verification for Dummies: SMT and Induction

2021-10-14T13:56:36Z

Adrien Champion adrien.champion@ocamlpro.com
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

These posts broadly discusses induction as a formal verification technique, which here really means formal program verification. I will use concrete, runnable examples whenever possible. Some of them can run directly in a browser, while others require to run small easy-to-retrieve tools locally. Such is the case for pretty much all examples dealing directly with induction.

The next chapters discuss the following notions:

formal logics and formal frameworks;
SMT-solving: modern, low-level verification building blocks;
declarative transition systems;
transition system unrolling;
BMC and induction proofs over transition systems;
candidate strengthening.

The approach presented here is far from being the only one when it comes to program verification. It happens to be relatively simple to understand, and I believe that familiarity with the notions discussed here makes understanding other approaches significantly easier.

This book thus hopes to serve both as a relatively deep dive into the specific technique of SMT-based induction, as well as an example of the technical challenges inherent to both developing and using automated proof engines.

Some chapters contain a few pieces of Rust code. Usually to provide a runnable version of a system under discussion, or to serve as example of actual code that we want to encode and verify. Some notions of Rust could definitely help in places, but this is not mandatory (probably).

Generating static and portable executables with OCaml

2021-09-02T13:56:36Z

Distributing OCaml software on opam is great (if I dare say so myself), but sometimes you need to provide your tools to an audience outside of the OCaml community, or just without recompilations or in a simpler way.

However, just distributing the locally generated binaries requires that the users have all the needed shared libraries installed, and a compatible libc. It's not something you can assume in general, and even if you don't need any C shared library or are confident enough it will be installed everywhere, the libc issue will arise for anyone using a distribution based on a different kind, or a little older than the one you used to build.

There is no built-in support for generating static executables in the OCaml compiler, and it may seem a bit tricky, but it's not in fact too complex to do by hand, something you may be ready to do for a release that will be published. So here are a few tricks, recipes and advice that should enable you to generate truly portable executables with no external dependency whatsoever. Both Linux and macOS will be treated, but the examples will be based on Linux unless otherwise specified.

Example

I will take as an example a trivial HTTP file server based on Dream.

Sample code

fserv.ml

let () = Dream.(run @@ logger @@ static ".")

fserv.opam

opam-version: "2.0"
depends: ["ocaml" "dream"]

dune-project

(lang dune 2.8)
(name fserv)

The relevant part of our dune file is just:

(executable
  (public_name fserv)
  (libraries dream))

This is how we check the resulting binary:

$ dune build fserv.exe
      ocamlc .fserv.eobjs/byte/dune__exe__Fserv.{cmi,cmo,cmt}
    ocamlopt .fserv.eobjs/native/dune__exe__Fserv.{cmx,o}
    ocamlopt fserv.exe
$ file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1991bb9f1d67807411c93f6fb6ec46b4a0ee8ed5, for GNU/Linux 3.2.0, with debug_info, not stripped
$ ldd _build/default/fserv.exe
        linux-vdso.so.1 (0x00007ffe97690000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fd6cc636000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fd6cc342000)
        libev.so.4 => /usr/lib/x86_64-linux-gnu/libev.so.4 (0x00007fd6cc330000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd6cc30e000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd6cc1ca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd6cc1c4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd6cbffd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd6cced7000)

(on macOS, replace ldd with otool -L; dune output is obtained with (display short) in ~/.config/dune/config)

So let's see how to change this result. Basically, here, libev, libssl and libcrypto are required shared libraries that may not be installed on every system, while all the others are part of the core system:

linux-vdso, libdl and ld-linux are concerned with the dynamic loading of shared objects ;
libm and libpthread are extensions of the core libc that are tightly bound to it, and always installed.

Statically linking the libraries

In simple cases, static linking can be turned on as easily as passing the -static flag to the C compiler: through OCaml you will need to pass -cclib -static. We can add that to our dune file:

(executable
  (public_name fserv)
  (flags (:standard -cclib -static))
  (libraries dream))

... which gives:

$ dune build fserv.exe
      ocamlc .fserv.eobjs/byte/dune__exe__Fserv.{cmi,cmo,cmt}
    ocamlopt .fserv.eobjs/native/dune__exe__Fserv.{cmx,o}
    ocamlopt fserv.exe
/usr/bin/ld: /usr/lib/gcc/x86_64-linuxgnu/10/../../../x86_64-linux-gnu/libcrypto.a(dso_dlfcn.o): in function `dlfcn_globallookup':
(.text+0x13): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ~/.opam/4.11.0/lib/ocaml/libunix.a(initgroups.o): in function `unix_initgroups':
initgroups.c:(.text.unix_initgroups+0x1f): warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[...]
$ file _build/default/fserv.exe 
_build/default/fserv.exe: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=9ee3ae1c24fbc291d1f580bc7aaecba2777ee6c2, for GNU/Linux 3.2.0, with debug_info, not stripped
$ ldd _build/default/fserv.exe
        not a dynamic executable

The executable was generated... and the result seems OK, but we shouldn't skip all these ld warnings. Basically, what ld is telling us is that you shouldn't statically link glibc (it internally uses dynlinking, to libraries that also need glibc functions, and will therefore still need to dynlink a second version from the system 🤯).

Indeed here, we have been statically linking a dynamic linking engine, among other things. Don't do it.

Linux solution: linking with musl instead of glibc

The easiest workaround at this point, on Linux, is to compile with musl, which is basically a glibc replacement that can be statically linked. There are some OCaml and gcc variants to automatically use musl (comments welcome if you have been successful with them!), but I have found the simplest option is to use a tiny Alpine Linux image through a Docker container. Here we'll use OCamlPro's minimal Docker images but anything based on musl should do.

$ docker run --rm -it ocamlpro/ocaml:4.12
[...]
~/fserv $ sudo apk add openssl-libs-static
(1/1) Installing openssl-libs-static (1.1.1l-r0)
OK: 161 MiB in 52 packages
~/fserv $ opam switch create . --deps ocaml-system
[...]
~/fserv $ eval $(opam env)
~/fserv $ dune build fserv.exe
~/fserv $ file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
~/fserv $ ldd _build/default/fserv.exe
        /lib/ld-musl-x86_64.so.1 (0x7ff41353f000)

Almost there! We see that we had to install extra packages with apk add: the static libraries might not be already installed and in this case are in a separate package (you would get bin/ld: cannot find -lssl). The last remaining dynamic loader in the output of ldd is because static PIE executable were not supported until recently. To get rid of it, we just need to add -cclib -no-pie (note: a previous revision of this blogpost mentionned -static-pie instead, which may work with recent compilers, but didn't seem to give reliable results):

(executable
  (public_name fserv)
  (flags (:standard -cclib -static -cclib -no-pie))
  (libraries dream))

And we are good!

~/fserv $ file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
~/fserv $ ldd _build/default/fserv.exe
/lib/ld-musl-x86_64.so.1: _build/default/fserv.exe: Not a valid dynamic program

Trick: short script to compile through a Docker container

Passing the context to a Docker container and getting the artefacts back can be bothersome and often causes file ownership issues, so I use the following snippet to pipe them to/from it using tar:
git ls-files -z | xargs -0 tar c | 
docker run --rm -i ocamlpro/ocaml:4.12 
  sh -uexc 
    '{ tar x &&
       opam switch create . ocaml-system --deps-only --locked &&
       opam exec -- dune build --profile=release @install;
     } >&2 && tar c -hC _build/install/default/bin .' | 
tar vx

The other cases: turning to manual linking

Sometimes you can't use the above: the automatic linking options may need to be tweaked for static libraries, your app may still need dynlinking support at some point, or you may not have the musl option. On macOS, for example, the libc doesn't have a static version at all (and the -static option of ld is explicitely "only used building the kernel"). Let's get our hands dirty and see how to use a mixed static/dynamic linking scheme. First, we examine how OCaml does the linking usually:

The linking options are passed automatically by OCaml, using information that is embedded in the cm(x)a files, for example:

$ ocamlobjinfo $(opam var lwt:lib)/unix/lwt_unix.cma |head
File ~/.opam/4.11.0/lib/lwt/unix/lwt_unix.cma
Force custom: no
Extra C object files: -llwt_unix_stubs -lev -lpthread
Extra C options:
Extra dynamically-loaded libraries: -llwt_unix_stubs
Unit name: Lwt_features
Interfaces imported:
        c21c5d26416461b543321872a551ea0d        Stdlib
        1372e035e54f502dcc3646993900232f        Lwt_features
        3a3ca1838627f7762f49679ce0278ad1        CamlinternalFormatBasics

Now the linking flags, here -llwt_unix_stubs -lev -lpthread let the C compiler choose the best way to link; in the case of stubs, they will be static (using the .a files — unless you make special effort to use dynamic ones), but -lev will let the system linker select the shared library, because it is generally preferred. Gathering these flags by hand would be tedious: my preferred trick is to just add the -verbose flag to OCaml (for the lazy, you can just set — temporarily — OCAMLPARAM=_,verbose=1):

(executable
  (public_name fserv)
  (flags (:standard -verbose))
  (libraries dream))

$ dune build
      ocamlc .fserv.eobjs/byte/dune__exe__Fserv.{cmi,cmo,cmt}
    ocamlopt .fserv.eobjs/native/dune__exe__Fserv.{cmx,o}
+ as  -o '.fserv.eobjs/native/dune__exe__Fserv.o' '/tmp/build8eb7e5.dune/camlasm91a0b9.s'
    ocamlopt fserv.exe
+ as  -o '/tmp/build8eb7e5.dune/camlstartupc9267f.o' '/tmp/build8eb7e5.dune/camlstartup1d9915.s'
+ gcc -O2 -fno-strict-aliasing -fwrapv -Wall -Wdeclaration-after-statement -fno-common -fexcess-precision=standard -fno-tree-vrp -ffunction-sections -D_FILE_OFFSET_BITS=64 -D_REENTRANT -DCAML_NAME_SPACE  -Wl,-E -o 'fserv.exe'  '-L~/.opam/4.11.0/lib/bigstringaf' '-L~/.opam/4.11.0/lib/ocaml' '-L~/.opam/4.11.0/lib/ocaml' '-L~/.opam/4.11.0/lib/ocaml' '-L~/.opam/4.11.0/lib/lwt/unix' '-L~/.opam/4.11.0/lib/cstruct' '-L~/.opam/4.11.0/lib/mirage-crypto' '-L~/.opam/4.11.0/lib/mirage-crypto-rng/unix' '-L~/.opam/4.11.0/lib/mtime/os' '-L~/.opam/4.11.0/lib/digestif/c' '-L~/.opam/4.11.0/lib/bigarray-overlap/stubs' '-L~/.opam/4.11.0/lib/ocaml' '-L~/.opam/4.11.0/lib/ssl' '-L~/.opam/4.11.0/lib/ocaml'  '/tmp/build8eb7e5.dune/camlstartupc9267f.o' '~/.opam/4.11.0/lib/ocaml/std_exit.o' '.fserv.eobjs/native/dune__exe__Fserv.o' '~/.opam/4.11.0/lib/dream/dream.a' '~/.opam/4.11.0/lib/dream/sql/dream__sql.a' '~/.opam/4.11.0/lib/dream/http/dream__http.a' '~/.opam/4.11.0/lib/dream/websocketaf/websocketaf.a' '~/.opam/4.11.0/lib/dream/httpaf-lwt-unix/httpaf_lwt_unix.a' '~/.opam/4.11.0/lib/dream/httpaf-lwt/httpaf_lwt.a' '~/.opam/4.11.0/lib/dream/h2-lwt-unix/h2_lwt_unix.a' '~/.opam/4.11.0/lib/dream/h2-lwt/h2_lwt.a' '~/.opam/4.11.0/lib/dream/h2/h2.a' '~/.opam/4.11.0/lib/psq/psq.a' '~/.opam/4.11.0/lib/dream/httpaf/httpaf.a' '~/.opam/4.11.0/lib/dream/hpack/hpack.a' '~/.opam/4.11.0/lib/dream/gluten-lwt-unix/gluten_lwt_unix.a' '~/.opam/4.11.0/lib/lwt_ssl/lwt_ssl.a' '~/.opam/4.11.0/lib/ssl/ssl.a' '~/.opam/4.11.0/lib/dream/gluten-lwt/gluten_lwt.a' '~/.opam/4.11.0/lib/faraday-lwt-unix/faraday_lwt_unix.a' '~/.opam/4.11.0/lib/faraday-lwt/faraday_lwt.a' '~/.opam/4.11.0/lib/dream/gluten/gluten.a' '~/.opam/4.11.0/lib/faraday/faraday.a' '~/.opam/4.11.0/lib/dream/localhost/dream__localhost.a' '~/.opam/4.11.0/lib/dream/graphql/dream__graphql.a' '~/.opam/4.11.0/lib/ocaml/str.a' '~/.opam/4.11.0/lib/graphql-lwt/graphql_lwt.a' '~/.opam/4.11.0/lib/graphql/graphql.a' '~/.opam/4.11.0/lib/graphql_parser/graphql_parser.a' '~/.opam/4.11.0/lib/re/re.a' '~/.opam/4.11.0/lib/dream/middleware/dream__middleware.a' '~/.opam/4.11.0/lib/yojson/yojson.a' '~/.opam/4.11.0/lib/biniou/biniou.a' '~/.opam/4.11.0/lib/easy-format/easy_format.a' '~/.opam/4.11.0/lib/magic-mime/magic_mime_library.a' '~/.opam/4.11.0/lib/fmt/fmt_tty.a' '~/.opam/4.11.0/lib/multipart_form/lwt/multipart_form_lwt.a' '~/.opam/4.11.0/lib/dream/pure/dream__pure.a' '~/.opam/4.11.0/lib/hmap/hmap.a' '~/.opam/4.11.0/lib/multipart_form/multipart_form.a' '~/.opam/4.11.0/lib/rresult/rresult.a' '~/.opam/4.11.0/lib/pecu/pecu.a' '~/.opam/4.11.0/lib/prettym/prettym.a' '~/.opam/4.11.0/lib/bigarray-overlap/overlap.a' '~/.opam/4.11.0/lib/bigarray-overlap/stubs/overlap_stubs.a' '~/.opam/4.11.0/lib/base64/rfc2045/base64_rfc2045.a' '~/.opam/4.11.0/lib/unstrctrd/parser/unstrctrd_parser.a' '~/.opam/4.11.0/lib/unstrctrd/unstrctrd.a' '~/.opam/4.11.0/lib/uutf/uutf.a' '~/.opam/4.11.0/lib/ke/ke.a' '~/.opam/4.11.0/lib/fmt/fmt.a' '~/.opam/4.11.0/lib/base64/base64.a' '~/.opam/4.11.0/lib/digestif/c/digestif_c.a' '~/.opam/4.11.0/lib/stdlib-shims/stdlib_shims.a' '~/.opam/4.11.0/lib/dream/graphiql/dream__graphiql.a' '~/.opam/4.11.0/lib/dream/cipher/dream__cipher.a' '~/.opam/4.11.0/lib/mirage-crypto-rng/lwt/mirage_crypto_rng_lwt.a' '~/.opam/4.11.0/lib/mtime/os/mtime_clock.a' '~/.opam/4.11.0/lib/mtime/mtime.a' '~/.opam/4.11.0/lib/duration/duration.a' '~/.opam/4.11.0/lib/mirage-crypto-rng/unix/mirage_crypto_rng_unix.a' '~/.opam/4.11.0/lib/mirage-crypto-rng/mirage_crypto_rng.a' '~/.opam/4.11.0/lib/mirage-crypto/mirage_crypto.a' '~/.opam/4.11.0/lib/eqaf/cstruct/eqaf_cstruct.a' '~/.opam/4.11.0/lib/eqaf/bigstring/eqaf_bigstring.a' '~/.opam/4.11.0/lib/eqaf/eqaf.a' '~/.opam/4.11.0/lib/cstruct/cstruct.a' '~/.opam/4.11.0/lib/caqti-lwt/caqti_lwt.a' '~/.opam/4.11.0/lib/lwt/unix/lwt_unix.a' '~/.opam/4.11.0/lib/ocaml/threads/threads.a' '~/.opam/4.11.0/lib/ocplib-endian/bigstring/ocplib_endian_bigstring.a' '~/.opam/4.11.0/lib/ocplib-endian/ocplib_endian.a' '~/.opam/4.11.0/lib/mmap/mmap.a' '~/.opam/4.11.0/lib/ocaml/bigarray.a' '~/.opam/4.11.0/lib/ocaml/unix.a' '~/.opam/4.11.0/lib/logs/logs_lwt.a' '~/.opam/4.11.0/lib/lwt/lwt.a' '~/.opam/4.11.0/lib/caqti/caqti.a' '~/.opam/4.11.0/lib/uri/uri.a' '~/.opam/4.11.0/lib/angstrom/angstrom.a' '~/.opam/4.11.0/lib/bigstringaf/bigstringaf.a' '~/.opam/4.11.0/lib/bigarray-compat/bigarray_compat.a' '~/.opam/4.11.0/lib/stringext/stringext.a' '~/.opam/4.11.0/lib/ptime/ptime.a' '~/.opam/4.11.0/lib/result/result.a' '~/.opam/4.11.0/lib/logs/logs.a' '~/.opam/4.11.0/lib/ocaml/stdlib.a' '-lssl_stubs' '-lssl' '-lcrypto' '-lcamlstr' '-loverlap_stubs_stubs' '-ldigestif_c_stubs' '-lmtime_clock_stubs' '-lrt' '-lmirage_crypto_rng_unix_stubs' '-lmirage_crypto_stubs' '-lcstruct_stubs' '-llwt_unix_stubs' '-lev' '-lpthread' '-lthreadsnat' '-lpthread' '-lunix' '-lbigstringaf_stubs' '~/.opam/4.11.0/lib/ocaml/libasmrun.a' -lm -ldl

There is a lot of noise, but the interesting part is at the end, the -l* options before the standard ocaml/libasmrun -lm -ldl:

  '-lssl_stubs' '-lssl' '-lcrypto' '-lcamlstr' '-loverlap_stubs_stubs' '-ldigestif_c_stubs' '-lmtime_clock_stubs' '-lrt' '-lmirage_crypto_rng_unix_stubs' '-lmirage_crypto_stubs' '-lcstruct_stubs' '-llwt_unix_stubs' '-lev' '-lpthread' '-lthreadsnat' '-lpthread' '-lunix' '-lbigstringaf_stubs'

Manually linking with glibc (Linux)

To link these statically, but the glibc dynamically:

we disable the automatic generation of linking flags by OCaml with -noautolink
we pass directives to the linker through OCaml and the C compiler, using -cclib -Wl,xxx. -Bstatic makes static linking the preferred option
we escape the linking flags we extracted above through -cclib

(executable
  (public_name fserv)
  (flags (:standard
          -noautolink
          -cclib -Wl,-Bstatic
          -cclib -lssl_stubs                    -cclib -lssl
          -cclib -lcrypto                       -cclib -lcamlstr
          -cclib -loverlap_stubs_stubs          -cclib -ldigestif_c_stubs
          -cclib -lmtime_clock_stubs            -cclib -lrt
          -cclib -lmirage_crypto_rng_unix_stubs -cclib -lmirage_crypto_stubs
          -cclib -lcstruct_stubs                -cclib -llwt_unix_stubs
          -cclib -lev                           -cclib -lthreadsnat
          -cclib -lunix                         -cclib -lbigstringaf_stubs
          -cclib -Wl,-Bdynamic
          -cclib -lpthread))
  (libraries dream))

Note that -lpthread and -lm are tightly bound to the libc and can't be static in this case, so we moved -lpthread to the end, outside of the static section. The part between the -Bstatic and the -Bdynamic is what will be statically linked, leaving the defaults and the libc dynamic. Result:

$ dune build fserv.exe && ldd _build/default/fserv.exe
      ocamlc .fserv.eobjs/byte/dune__exe__Fserv.{cmi,cmo,cmt}
    ocamlopt .fserv.eobjs/native/dune__exe__Fserv.{cmx,o}
    ocamlopt fserv.exe
$ file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=31c93085284da5d74002218b1d6b61c0efbdefe4, for GNU/Linux 3.2.0, with debug_info, not stripped
$ ldd _build/default/fserv.exe
        linux-vdso.so.1 (0x00007ffe207c5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f49d5e56000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f49d5d12000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f49d5d0c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f49d5b47000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f49d69bf000)

The remaining are the base of the dynamic linking / shared object systems, but we got away with libssl, libcrypto and libev, which were the ones possibly absent from target systems. The resulting executable should work on any glibc-based Linux distribution that is recent enough; on older ones you will likely get missing GLIBC symbols.

If you need to distribute that way, it's a good idea to compile on an old release (like Debian 'oldstable' or 'oldoldstable') for maximum portability.

Manually linking on macOS

Unfortunately, the linker on macOS doesn't seem to have options to select the static versions of the libraries; the only solution is to get our hands even dirtier, and link directly to the .a files, instead of using -l arguments.

Most of the flags just link with stubs, we can keep them as is: -lssl_stubs -lcamlstr -loverlap_stubs_stubs -ldigestif_c_stubs -lmtime_clock_stubs -lmirage_crypto_rng_unix_stubs -lmirage_crypto_stubs -lcstruct_stubs -llwt_unix_stubs -lthreadsnat -lunix -lbigstringaf_stubs

That leaves us with: -lssl -lcrypto -lev -lpthread

lpthread is built-in, we can ignore it

for the others, we need to lookup the .a file: I use e.g.

$ echo $(pkg-config libssl --variable libdir)/libssl.a
~/brew/Cellar/openssl@1.1/1.1.1k/lib/libcrypto.a

Of course you don't want to hardcode these paths, but let's test for now:

(executable
  (public_name fserv)
  (flags (:standard
          -noautolink
          -cclib -lssl_stubs           -cclib -lcamlstr
          -cclib -loverlap_stubs_stubs -cclib -ldigestif_c_stubs
          -cclib -lmtime_clock_stubs   -cclib -lmirage_crypto_rng_unix_stubs
          -cclib -lmirage_crypto_stubs -cclib -lcstruct_stubs
          -cclib -llwt_unix_stubs      -cclib -lthreadsnat
          -cclib -lunix                -cclib -lbigstringaf_stubs
          -cclib ~/brew/Cellar/openssl@1.1/1.1.1k/lib/libssl.a
          -cclib ~/brew/Cellar/openssl@1.1/1.1.1k/lib/libcrypto.a
          -cclib ~/brew/Cellar/libev/4.33/lib/libev.a))
  (libraries dream))

$ dune build fserv.exe
      ocamlc .fserv.eobjs/byte/dune__exe__Fserv.{cmi,cmo,cmt}
    ocamlopt .fserv.eobjs/native/dune__exe__Fserv.{cmx,o}
    ocamlopt fserv.exe
$ file _build/default/fserv.exe
_build/default/fserv.exe: Mach-O 64-bit executable x86_64
$ otool -L _build/default/fserv.exe
_build/default/fserv.exe:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

This is as good as it will get!

Cleaning up the build system

We have until now been adding the linking flags manually in the dune file; you probably don't want to do that and be restricted to static builds only! Not counting the non-portable link options we have been using...

The quick&dirty way

Don't use this in your build system! But for quick testing you can conveniently pass flags to the OCaml compilers using the OCAMLPARAM variable. Combined with the tar/docker snippet above, we get a very simple static-binary generating command:

git ls-files -z | xargs -0 tar c | 
docker run --rm -i ocamlpro/ocaml:4.12 
  sh -uexc '{
    tar x &&
    sudo apk add openssl-libs-static &&
    opam switch create . ocaml-system --deps-only --locked &&
    OCAMLPARAM=_,cclib=-static,cclib=-no-pie opam exec -- dune build --profile=release @install;
  } >&2 && tar c -hC _build/install/default/bin .' | 
tar vx

Note that, for releases, you may also want to strip the generated binaries.

Making it an option of the build system (with dune)

For something you will want to commit, I recommend to generate the flags in a separate file linking-flags-fserv.sexp:

(executable
  (public_name fserv)
  (flags (:standard (:include linking-flags-fserv.sexp)))
  (libraries dream))

The linking flags will depend on the chosen linking mode and on the OS. For the OS, it's easiest to generate them through a script ; for the linking mode, I use an environment variable to optionally turn static linking on.

(rule
  (with-stdout-to linking-flags-fserv.sexp
    (run ./gen-linking-flags.sh %{env:LINKING_MODE=dynamic} %{ocaml-config:system})))

This will use the following gen-linking-flags.sh script to generate the file, passing it the value of $LINKING_MODE and defaulting to dynamic. Doing it this way also ensures that dune will properly recompile when the value of the environment variable changes.

#!/bin/sh
set -ue

LINKING_MODE="$1"
OS="$2"
FLAGS=
CCLIB=

case "$LINKING_MODE" in
    dynamic)
        ;; # No extra flags needed
    static)
        case "$OS" in
            linux) # Assuming Alpine here
                CCLIB="-static -no-pie";;
            macosx)
                FLAGS="-noautolink"
                CCLIB="-lssl_stubs -lcamlstr -loverlap_stubs_stubs
                       -ldigestif_c_stubs -lmtime_clock_stubs
                       -lmirage_crypto_rng_unix_stubs -lmirage_crypto_stubs
                       -lcstruct_stubs -llwt_unix_stubs -lthreadsnat -lunix
                       -lbigstringaf_stubs"
                LIBS="libssl libcrypto libev"
                for lib in $LIBS; do
                    CCLIB="$CCLIB $(pkg-config $lib --variable libdir)/$lib.a"
                done;;
            *)
                echo "No known static compilation flags for '$OS'" >&2
                exit 1
        esac;;
    *)
        echo "Invalid linking mode '$LINKING_MODE'" >&2
        exit 2
esac

echo '('
for f in $FLAGS; do echo "  $f"; done
for f in $CCLIB; do echo "  -cclib $f"; done
echo ')'

Then you'll only have to run LINKING_MODE=static dune build fserv.exe to generate the static executable (wrapped in the Docker script above, in the case of Alpine), and can include that in your CI as well.

For real-world examples, you can check learn-ocaml or opam.

Related topics

reproducible builds should be a goal when you intend to distribute pre-compiled binaries.

opam-bundle is a different, heavy-weight approach to distributing opam software to non-OCaml developers, that retains the "compile all from source" policy but provides one big package that bootstraps OCaml, opam and all the dependencies with a single command.-

opam 2.1.0 is released!

2021-08-04T13:56:36Z

Feedback on this post is welcomed on Discuss!

We are happy to announce the release of opam 2.1.0.

Many new features made it in (see the pre-release changelogs or release notes for the details), but here are a few highlights.

What's new in opam 2.1?

Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
Creation of lock files for reproducible installations (formerly the opam-lock plugin)
Switch invariants, replacing the "base packages" in opam 2.0 and allowing for easier compiler upgrades
Improved options configuration (see the new option and expanded var sub-commands)
CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
opam root readability by newer and older versions, even if the format changed
Performance improvements to opam-update, conflict messages, and many other areas

Seamless integration of System dependencies handling (a.k.a. "depexts")

opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:

You no longer have to remember to run opam depext, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy as opam install ., without having to do the dance of opam pin add -n/depext/install. Just one command now for the common case!
The solver is only called once, which both saves time and also stabilises the behaviour of opam in cases where the solver result is not stable. It was possible to get one package solution for the opam depext stage and a different solution for the opam install stage, resulting in some depexts missing.
opam now has full knowledge of depexts, which means that packages can be automatically selected based on whether a system package is already installed. For example, if you have neither MariaDB nor MySQL dev libraries installed, opam install mysql will offer to install conf-mysql and mysql, but if you have the MariaDB dev libraries installed, opam will offer to install conf-mariadb and mysql.

Hint: You can set OPAMCONFIRMLEVEL=unsafe-yes or --confirm-level=unsafe-yes to launch non interactive system package commands.

opam lock files and reproducibility

When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 19500 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.

While this works well for libraries, we need a different strategy for projects that need to test and ship using a fixed set of dependencies. To satisfy this use-case, opam 2.0.0 shipped with support for using project.opam.locked files. These are normal opam files but with exact versions of dependencies. The lock file can be used as simply as opam install . --locked to have a reproducible package installation.

With opam 2.1.0, the creation of lock files is also now integrated into the client:

opam lock will create a .locked file for your current switch and project, that you can check into the repository.
opam switch create . --locked can be used by users to reproduce your dependencies in a fresh switch.

This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).

Hint: You can export the full configuration of a switch with opam switch export new options, --full to have all packages metadata included, and --freeze to freeze all VCS to their current commit.

Switch invariants

In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam's solver. This was a much bigger concern early on in opam 2.0's development, but is less of a problem with the default mccs solver.

However, it's a problem for system compilers. opam would detect that your system compiler version had changed, but be unable to upgrade the ocaml-system package unless you went through a slightly convoluted process with --unlock-base.

In opam 2.1, base packages have been replaced by switch invariants. The switch invariant is a package formula which must be satisfied on every upgrade and install. All existing switches' base packages could just be expressed as package1 & package2 & package3 etc. but opam 2.1 recognises many existing patterns and simplifies them, so in most cases the invariant will be "ocaml-base-compiler" {= "4.11.1"}, etc. This means that opam switch create my_switch ocaml-system now creates a switch invariant of "ocaml-system" rather than a specific version of the ocaml-system package. If your system OCaml package is updated, opam upgrade will seamlessly switch to the new package.

This also allows you to have switches which automatically install new point releases of OCaml. For example:

opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop

Creates a switch with OCaml 4.11.0 (the --repos= was just to select a version of opam-repository from before 4.11.1 was released). Now issue:

opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade

and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There's not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.

Hint: You can set up a default invariant that will apply for all new switches, via a specific opamrc. The default one is ocaml >= 4.05.0

Configuring opam from the command-line

Configuring opam is not a simple task: you need to use an opamrc at init stage, or hack global/switch config file, or use opam config var for additional variables. To ease that step, and permit a more consistent opam config tweaking, a new command was added : opam option.

For example:

opam option download-jobs gives the global download-jobs value (as it exists only in global configuration)
opam option jobs=6 --global will set the number of parallel build jobs opam is allowed to run (along with the associated jobs variable)
opam option depext-run-commands=false disables the use of sudo for handling system dependencies; it will be replaced by a prompt to run the installation commands
opam option depext-bypass=m4 --global bypass m4 system package check globally, while opam option depext-bypass=m4 --switch myswitch will only bypass it in the selected switch

The command opam var is extended with the same format, acting on switch and global variables.

Hint: to revert your changes use opam option <field>=, it will take its default value.

CLI Versioning

A new --cli switch was added to the first beta release, but it's only now that it's being widely used. opam is a complex enough system that sometimes bug fixes need to change the semantics of some commands. For example:

opam show --file needed to change behaviour
The addition of new controls for setting global variables means that the opam config was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1 still works in opam 2.0, but it's really an OPAM 1.2.2 syntax.

Changing the CLI is exceptionally painful since it can break scripts and tools which themselves need to drive opam. CLI versioning is our attempt to solve this. The feature is inspired by the (lang dune ...) stanza in dune-project files which has allowed the Dune project to rename variables and alter semantics without requiring every single package using Dune to upgrade their dune files on each release.

Now you can specify which version of opam you expected the command to be run against. In day-to-day use of opam at the terminal, you wouldn't specify it, and you'll get the latest version of the CLI. For example: opam var --global is the same as opam var --cli=2.1 --global. However, if you issue opam var --cli=2.0 --global, you will told that --global was added in 2.1 and so is not available to you. You can see similar things with the renaming of opam upgrade --unlock-base to opam upgrade --update-invariant.

The intention is that --cli should be used in scripts, user guides (e.g. blog posts), and in software which calls opam. The only decision you have to take is the oldest version of opam which you need to support. If your script is using a new opam 2.1 feature (for example opam switch create --formula=) then you simply don't support opam 2.0. If you need to support opam 2.0, then you can't use --formula and should use --packages instead. opam 2.0 does not have the --cli option, so for opam 2.0 instead of --cli=2.0 you should set the environment variable OPAMCLI to 2.0. As with all opam command line switches, OPAMCLI is simply the equivalent of --cli which opam 2.1 will pick-up but opam 2.0 will quietly ignore (and, as with other options, the command line takes precedence over the environment).

Note that opam 2.1 sets OPAMCLI=2.0 when building packages, so on the rare instances where you need to use the opam command in a package build: command (or in your build system), you must specify --cli=2.1 if you're using new features.

Since 2.1.0~rc2, CLI versioning applies to opam environment variables. The previous behavior was to ignore unknown or wrongly set environment variable, while now you will have a warning to let you know that the environment variable won't be handled by this version of opam.

To ensure not breaking compatibility of some widely used deprecated options, a default CLI is introduced: when no CLI is specified, those deprecated options are accepted. It concerns opam exec and opam var subcommands.

There's even more detail on this feature in our wiki. We're hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.

Note: For opam libraries users, since 2.1 environment variable are no more loaded by the libraries, only by opam client. You need to load them explicitly.

opam root portability

opam root format changes during opam life-cycle, new field are added or removed, new files are added ; an older opam version sometimes can no longer read an upgraded or newly created opam root. opam root format has been updated to allow new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots). It is a read-only best effort access, any attempt to modify the opam root fails.

Hint: for opam libraries users, you can safely load states with OpamStateConfig load functions.

Tremendous thanks to all involved people, who've developed, tested & retested, helped with issue reports, comments, feedback...

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam directory.

The upgrade instructions are unchanged:

Either from binaries: run

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

opam 2.0.9 release

2021-08-03T13:56:36Z

Feedback on this post is welcomed on Discuss!

We are pleased to announce the minor release of opam 2.0.9.

This new version contains some back-ported fixes.

New features

Back-ported ability to load upgraded roots read-only; allows applications compiled with opam-state 2.0.9 to load a root which has been upgraded to opam 2.1 [#4636]
macOS sandbox now supports OPAM_USER_PATH_RO for adding a custom read-only directory to the sandbox [#4589, #4609]
OPAMROOT and OPAMSWITCH now reflect the --root and --switch parameters in the package build [#4668]
When built with opam-file-format 2.1.3+, opam-format 2.0.x displays better errors for newer opam files [#4394]

Bug fixes

Linux sandbox now mounts host $TMPDIR read-only, then sets the sandbox $TMPDIR to a new separate tmpfs. Hardcoded /tmp access no longer works if TMPDIR points to another directory [#4589]
Stop clobbering DUNE_CACHE in the sandbox script [#4535, fixing ocaml/dune#4166]
Ctrl-C now correctly terminates builds with bubblewrap; sandbox now requires bubblewrap 0.1.8 or later [#4400]
Linux sandbox script no longer makes PWD read-write on remove actions [#4589]
Lint W59 and E60 no longer trigger for packages flagged conf [#4549]
Reduce the length of temporary file names for pin caching to ease pressure on Windows [#4590]
Security: correct quoting of arguments when removing switches [#4707]
Stop advertising the removed option --compiler when creating local switches [#4718]
Pinning no longer fails if the archive's opam file is malformed [#4580]
Fish: stop using deprecated ^ syntax to fix support for Fish 3.3.0+ [#4736]

Installation instructions (unchanged):

From binaries: run

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.9"

or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

From source, using opam:

opam update; opam install opam-devel

(then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

Detecting identity functions in Flambda

2021-07-16T13:56:36Z

In some discussions among OCaml developers around the empty type (PR#9459), some people mused about the possibility of annotating functions with an attribute telling the compiler that the function should be trivial, and always return a value strictly equivalent to its argument.
Curious about the feasibility of implementing this feature, we advertised an internship with our compiler team aimed at exploring this subject.
We welcomed Léo Boitel during three months to work on this topic, with Vincent Laviron as mentor, and we're proud to let him show off what he has achieved in this post.

The problem at hand

OCaml's strong typing system is one of its main perks: it allows to write safe code thanks to the abstraction it provides. Most of the basic design mistakes will directly result into a typing error and the user cannot mess up the memory as it is automatically handled by the compiler and runtime.

However, these perks keep a power-user from implementing some optimizations, in particular those linked to memory representation as they cannot be accessed directly.

A good example would be this piece of code:

type return = Ok of int | Failure
let id = function
| Some x -> Ok x
| None -> Failure

In terms of memory representation, this function is indeed the identity function. Some x and Ok x share the same representation (and so do None and Failure). However, this identity is invisible to the user. Even if the user knows the representation is the same, they would need to use this function to avoid a typing error.

Another good example would be this one:

type record = { a:int; b:int }
let id (x,y) = { a = x; b = y }

Even if those functions are the identity, they come with a cost: not only do they cost a function call, they reallocate the result instead of just returning their argument directly. Detecting those functions would allow us to produce interesting optimizations.

Hurdles

If we want to detect identities, we quickly hit the problem of recursive functions: how does one recognize identity in those cases? Can a function be an identity if it doesn't always terminate, or if it never does?

Once we have a good definition of what exactly an identity function is, we still need to prove that an existing function fits the definition. Indeed, we want to ensure the user that this optimization will not change the observable behavior of the program.

We also want to avoid breaking type safety. As an example, the following function:

let rec fake_id = function
| [] -> 0
| t::q -> fake_id (t::q)

A naive induction proof would allow us to replace this function with the identity, as [] and 0 share the same memory representation. However, this is unsafe as applying this to a non-empty list would return a list even if this function has an int type (we'll talk more about it later).

To tackle those challenges, we started the internship with a theoretical study that lasted for three fourths of the allocated time and lastly implemented a practical solution in the Flambda representation of the compiler.

Theoretical results

We worked on extensions of lambda-calculus (implemented in OCaml) in order to gradually test our ideas in a simpler framework than the full Flambda.

Pairs

We started with a lambda-calculus to which we only added the concept of pairs. To prove identities, every function has to be annotated as identity or not. We then prove these annotations by β-reducing the function bodies. After each recursive reduction, we apply a rule that states that a pair made of the first and second projection of a variable is equal to that variable. We do not reduce applications, but we replace them by the argument if the concerned function is annotated as identity.

Using this method, we maintain a reasonable complexity compared to a full β-reduction which would be unrealistic on a big program.

We then add higher-order capabilities by allowing annotations of the form Annotation → Annotation. Functions as List.map can that way be abstracted as Id → Id. Even though this solution doesn't cover every case, most real-world usage are recognized by these patterns.

Tuple reconstruction

We then move from just pairs to tuples of arbitrary size. This adds a new problem: if we make a pair out of the first two fields of a variable, this is no longer necessarily that variable, as it may have more than two fields.

We then have two solutions: we can first annotate every projection with the size of the involved tuple to know if we are indeed reconstructing the entire variable. As an example, if we make a pair from the fields of a triplet, we know there is no way to simplify this reconstruction.

Another solution, more ambitious, is to adopt a less restrictive definition of equality and to allow the replacement of (x,y) by (x,y,z). Indeed, if the variable was typed as a pair, we are guaranteed that the third field will never be accessed. The behavior of the program will therefore never be affected by this extension.

Though this allows to avoid a lot of allocations, it may also increase memory usage in some cases: if the triplet ceases to be used, it won't be deallocated by the Garbage Collector (GC) and the field z will be kept in memory as long as (x,y) is still accessible.

This approach remains interesting to us, as long as it is manually enabled by the user for some specific blocks.

Recursion

We now add recursive definitions to our language, through the use of a fixpoint operator.

To prove that a recursive function is the identity, we have to use induction. The main difficulty is to prove that the function indeed terminates to ensure the validity of the induction.

We can separate this into three different levels of safety. The first option is to not prove termination at all, and let the user state which function they know will return. We can then assume the function is the identity and replace its body on that hypothesis. This approach is enough for most practical cases, but its main problem lies in the fact that it allows to write unsafe code (as we've already seen).

Our second option is to limit our induction hypothesis to recursive applications on "smaller" elements than the argument. An element is defined as smaller if it is a projection of the argument or a projection of a smaller element. This is not enough to prove that the function will terminate (the argument might be cyclic, for example) but is enough to ensure type safety. The reason is that any possibly returned value is constructed (as it cannot directly come from a recursive call) and have therefore a defined type. Typing would fail if the function was to return a value that cannot be identified to its argument.

Finally, we may want to establish a perfect equivalence between the function and the identity function before simplifying it. In that case, we propose to create a special annotation for functions that are the identity when applied to a non-cyclical object. We can prove they have this property with the already described induction. The difficulty now lies into applying the simplification only to valid applications: if an object is immutable, wasn't recursively defined and is made of components that also have that property, we can declare that object inductive and simplify applications on it. The inductive state of variables can be propagated during our recursive pass of optimization.

Block reconstruction

The representation of blocks in Flambda provides interesting challenges in terms of equality detection, which is often crucial to prove an identity. It is very hard to detect an identical block reconstruction.

Blocks in Flambda

Variants

The blocks in Flambda come from the existence of variants in OCaml: one type may have several different constructors, as we can see in

type choice = A of int | B of int

When OCaml is compiled to Flambda, the information used by the constructor is lost and replaced by a tag. The tag is a number contained in the header of the object's memory representation between 0 and 255 that represents which constructor was used. As an example, an element of type choice would have tag 0 for the A constructor, and 1 for B.

That tag will be kept at runtime, which will allow for example to implement pattern matching as a simple switch in Flambda, that executes simple comparisons on the tag to know which branch to execute next.

This system complicates our task as Flambda's typing doesn't inform us which type the constructor is supposed to have, and therefore keeps us from easily knowing if two variants are indeed equal.

Tag generalization

To complicate things, tags are actually used for any block, meaning tuples, modules or functions (as a matter of fact, almost anything but constant constructors and integers). If the object doesn't have variants, it will usually have tag 0. This tag is never read (as there are no variants to differentiate) but keeps us from simply comparing two tuples, because Flambda will simply see two blocks of unknown tag.

Inlining

Finally, this system is optimized by inlining tuples: if a variant has a shape Pair of int * int, it will be often be flattened into a tuple (tag Pair, int, int).

This also means that variants can have an arbitrary size, which is also unknown in Flambda.

Existing approach

A partial solution to the problem already existed in a Pull Request (PR) you can read here.

The chosen approach in this PR is the natural one: we use the switch to gain information on the tag of a block, depending on the branch taken. The PR also allows to know the mutability and size of the block in each branch, starting from OCaml (where this information is known as it is explicit in the pattern matching) and propagating the knowledge to Flambda.

This allows to register every block on which a switch is done with their tag, size and mutability. We can then detect if one of them is reconstructed with the use of a Pmakeblock primitive.

Unfortunately, this path has its limits as there are numerous cases where the tag and size could be known without performing a switch on the value. As an example, this doesn't allow the simplification of tuple reconstruction.

New solution

Our new solution will have to propagate more information from OCaml into Flambda. This propagation is based on two PRs that already existed for Flambda 2, which annotated in the lambda representation each projection (Pfield) with typing informations. We add block mutability and tag and finally size.

Our first contribution was to translate these PRs to Flambda 1, and to propagate from lambda to Flambda correctly.

We then had access to every necessary information to detect and prove block reconstruction: not only we have a list of blocks that were pattern-matched, we can make a list of partially immutable blocks, meaning blocks for which we know that some fields are immutable.

Here's how we use it:

Block discovery

As soon as we find a projection, we verify whether it is done on an immutable block of known size. If so, we add that block to the list of partial blocks. We verify that the information we have on the tag and size are compatible with the already known projections. If all of the fields of the block are known, the block is added to the list of simplifiable blocks.

Of course, we also keep track of known blocks though switches.

Simplification

This part is similar to the original PR: when an immutable block is met, we check whether this block is known as simplifiable. In that case we avoid a reallocation.

Compared to the original approach, we also reduced the asymptotic complexity (from quadratic to linear) by registering the association of every projection variable to its index and original block. We also modified some implementation details that could have triggered a bug when associated with our PR.

Example

Let's consider this function:

type typ1 = A of int | B of int * int
type typ2 = C of int | D of {x:int; y:int}
let id = function
  | A n -> C n
  | B (x,y) -> D {x; y}

The current compiler would produce the resulting Flambda output:

End of middle end:
let_symbol
  (camlTest__id_21
    (Set_of_closures (
      (set_of_closures id=Test.8
        (id/5 = fun param/7 ->
          (switch*(0,2) param/7
           case tag 0:
            (let
              (Pmakeblock_arg/11 (field 0<{../../test.ml:4,4-7}> param/7)
               Pmakeblock/12
                 (makeblock 0 (int)<{../../test.ml:4,11-14}>
                   Pmakeblock_arg/11))
              Pmakeblock/12)
           case tag 1:
            (let
              (Pmakeblock_arg/15 (field 1<{../../test.ml:5,4-11}> param/7)
               Pmakeblock_arg/16 (field 0<{../../test.ml:5,4-11}> param/7)
               Pmakeblock/17
                 (makeblock 1 (int,int)<{../../test.ml:5,17-23}>
                   Pmakeblock_arg/16 Pmakeblock_arg/15))
              Pmakeblock/17)))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__id_5_closure (Project_closure (camlTest__id_21, id/5)))
  (camlTest (Block (tag 0,  camlTest__id_5_closure)))
End camlTest

Our optimization allows to detect that this function reconstructs a similar block and therefore can simplify it:

End of middle end:
let_symbol
  (camlTest__id_21
    (Set_of_closures (
      (set_of_closures id=Test.7
        (id/5 = fun param/7 ->
          (switch*(0,2) param/7
           case tag 0 (1): param/7
           case tag 1 (2): param/7))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__id_5_closure (Project_closure (camlTest__id_21, id/5)))
  (camlTest (Block (tag 0,  camlTest__id_5_closure)))
End camlTest

Possible improvements

Equality relaxation

We can use observational equality studied in the theoretical part for block equality in order to avoid more allocations. The implementation is simple:

When a block is created, to know if it will be allocated, the normal course of action is to check if all of its fields are the known projections of another block, with the same index, and if the block sizes are the same. We can just remove that last check.

Implementing this was a bit more tricky because of several practical details. First, we want that optimization to be only triggered on user-annotated blocks, we had to propagate that annotation to Flambda.

Additionally, if we only implement that optimization, numerous optimization cases will be ignored because unused variables are simplified before our optimization pass. As an example, if a function looks like

let loose_id (a,b,c) = (a,b)

The c variable will be simplified away before reaching Flambda, and there will be no way to prove that (a,b,c) is immutable as its third field could not be. This problem is being solved on Flambda2 thanks to a PR that propagates mutability information for every block, but we didn't have the time necessary to migrate it on Flambda 1.

Detecting recursive identities

Now that we can detect block reconstruction, we're left with solving the problem of recursive functions.

Unsafe approach

We began the implementation of a pass that contains no termination proof. The idea is to add the proof later, or to authorize non-terminating functions to be simplified as long as they type correctly (see previously in the theory part).

For now, we trust the user to verify these properties manually.

Hence, we modified the function simplification procedure: when a function with a single argument is modified, we first assume that this function is the identity before simplifying its body. We then check whether the result is equivalent to an identity by recursively going through it, so as to cover as many cases as possible (for example in conditional branchings). If it is the case, the function will be replaced by the identity ; otherwise, we go back to a normal simplification, without the induction hypothesis.

Constant propagation

We took some time to improve our code that checks whether the body of a function is an identity or not, so as to handle constant values. It propagates identity information we have on an argument during conditional branching.

This way, on a function like

type truc = A | B | C
let id = function
  | A -> A
  | B -> B
  | C -> C

or even

let id x = if x=0 then 0 else x

We can successfully detect identity.

Examples

Recursive functions

We can now detect recursive identities:

let rec listid = function
  | t::q -> t::(listid q)
  | [] -> []

Used to compile to:

End of middle end:
let_rec_symbol
  (camlTest__listid_5_closure
    (Project_closure (camlTest__set_of_closures_20, listid/5)))
  (camlTest__set_of_closures_20
    (Set_of_closures (
      (set_of_closures id=Test.11
        (listid/5 = fun param/7 ->
          (if param/7 then begin
            (let
              (apply_arg/13 (field 1<{../../test.ml:9,4-8}> param/7)
               apply_funct/14 camlTest__listid_5_closure
               Pmakeblock_arg/15
                 *(apply*&#091;listid/5]<{../../test.ml:9,15-25}> apply_funct/14
                    apply_arg/13)
               Pmakeblock_arg/16 (field 0<{../../test.ml:9,4-8}> param/7)
               Pmakeblock/17
                 (makeblock 0<{../../test.ml:9,12-25}> Pmakeblock_arg/16
                   Pmakeblock_arg/15))
              Pmakeblock/17)
            end else begin
            (let (const_ptr_zero/27 Const(0a)) const_ptr_zero/27) end))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
let_symbol (camlTest (Block (tag 0,  camlTest__listid_5_closure)))
End camlTest

But is now detected as being the identity:

End of middle end:
let_symbol
  (camlTest__set_of_closures_20
    (Set_of_closures (
      (set_of_closures id=Test.13 (listid/5 = fun param/7 -> param/7)
        free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__listid_5_closure
    (Project_closure (camlTest__set_of_closures_20, listid/5)))
  (camlTest (Block (tag 0,  camlTest__listid_5_closure)))
End camlTest

Unsafe example

However, we can use the unsafety of the feature to go around the typing system and access a memory address as if it was an integer:

type bugg = A of int*int | B of int
let rec bug = function
  | A (a,b) -> (a,b)
  | B x -> bug (B x)
  
let (a,b) = (bug (B 42))
let _ = print_int b

This function will be simplified to the identity even though the bugg type is not compatible with tuples; trying to project on the second field of variant b will access an undefined part of memory:

$ ./unsafe.out
47423997875612

Possible improvements - short term

Function annotation

A theoretically simple thing to add would be to let the choice of applying unsafe optimizations to the user. We lacked the time to do the work of propagating the information to Flambda, but it would not be hard to implement.

Order on arguments

For a safer optimization, we could use the idea developed in the theoretical part to make the optimization correct on non-cyclical objects and more importantly give us typing guarantees to avoid the problem we just saw.

To get this guarantee, we would have to change the simplification pass by adding an optional pair of function-argument to the environment. When this option exists, the pair indicates that we are in the body in the process of simplification and that applications on smaller elements can be simplified as identity. Of course, the pass would need to be modified to remember which elements are not smaller than the previous argument.

Possible improvements - long term

Exclusion of cyclical objects

As described in the theoretical part, we could recursively deduce which objects are cyclical and attempt to remove them from our optimization. The problem is then that instead of having to replace functions by the identity, we need to add a special annotation that represents IdRec.

This amounts to a lot of added implementation complexity when compiling over several files, as we need access to the interface of already compiled files to know when the optimization can be used.

A possibility would be to use .cmx files to store this information when the file is compiled, but that kind of work would have taken too long to be achieved during the internship. Moreover, the practicality of that choice is far from obvious: it would complexify the optimization pass for a small improvement with respect to a version that would be correct on non-cyclical objects and activated through annotations.

Détection de fonctions d’identité dans Flambda

2021-07-15T13:56:36Z

Au cours de discussions parmi les développeurs OCaml sur le type vide (PR#9459), certains caressaient l’idée d’annoter des fonctions avec un attribut indiquant au compilateur que la fonction devrait être triviale, et toujours renvoyer une valeur strictement équivalente à son argument. Nous étions curieux de voir si l’implémentation d’une telle fonctionnalité serait possible et nous avons publié une offre de stage pour explorer ce sujet. L’équipe Compilation d’OCamlPro a ainsi accueilli Léo Boitel durant trois mois pour se consacrer à ce sujet, avec Vincent Laviron pour encadrant. Nous sommes fiers des résultats auxquels Léo a abouti !

Voici ce que Léo en a écrit 🙂

Description du problème

Le typage fort d’OCaml est un de ses grands avantages : il permet d’écrire du code plus sûr grâce à la capacité d’abstraction qu’il offre. La plupart des erreurs de conception se traduiront directement en erreur de typage, et l’utilisateur ne peut pas faire d’erreur avec la manipulation de la mémoire puisqu’elle est entièrement gérée par le compilateur.

Cependant, ces avantages empêchent l’utilisateur de faire certaines optimisations lui-même, en particulier celles liées aux représentations mémoires puisqu’il n’y accède pas directement.

Un cas classique serait le suivant :

type return = Ok of int | Failure
let id = function
| Some x -> Ok x
| None -> Failure

Cette fonction est une identité, car la représentation mémoire de Some x et de Ok x est la même (idem pour None et Failure). Cependant, l’utilisateur ne le voit pas, et même s’il le voyait, il aurait besoin de cette fonction pour conserver un typage correct.

Un autre exemple serait le suivant: Another good example would be this one:

type record = { a:int; b:int }
let id (x,y) = { a = x; b = y }

Même si ces fonctions sont des identités, elles ont un coût : en plus de nous coûter un appel, elles réallouent le résultat au lieu de nous retourner leur argument directement. C’est pourquoi leur détection permettrait des optimisations intéressantes.

Difficultés

Si on veut pouvoir détecter les identités, on se heurte rapidement au problème des fonctions récursives : comment définir l’identité pour ces dernières ? Est-ce qu’une fonction peut-être l’identité si elle ne termine pas toujours, voire jamais ?

Une fois qu’on a défini l’identité, le problème est la preuve qu’une fonction est bien l’identité. En effet, on veut garantir à l’utilisateur que cette optimisation ne changera pas le comportement observable du programme.

On veut aussi éviter d’ajouter des failles de sûreté au typage. Par exemple, si on a une fonction de la forme suivante:

let rec fake_id = function
| [] -> 0
| t::q -> fake_id (t::q)

Une preuve naïve par induction nous ferait remplacer cette fonction par l’identité, car [] et 0 ont la même représentation mémoire. C’est dangereux car le résultat d’une application à une liste non-vide sera une liste alors qu’il est typé comme un entier (voir exemples plus bas).

Pour résoudre ces problèmes, nous avons commencé par une partie théorique qui a occupé les trois quarts du stage, pour finir par une partie pratique d’implémentation dans Flambda.

Résultats théoriques

Pour cette partie, nous avons travaillé sur des extensions de lambda-calcul, implémentées en OCaml, pour pouvoir tester nos idées au fur et à mesure dans un cadre plus simple que Flambda.

Paires

Nous avons commencé par un lambda calcul auquel on ajoute seulement des paires. Pour effectuer nos preuves, on annote toutes les fonctions comme des identités ou non. On prouve ensuite ces annotations en β-réduisant le corps des fonctions. Après chaque réduction récursive, on applique une règle qui dit qu’une paire composée des deux projections d’une variable est égale à la variable. On ne réduit pas les applications, mais on les remplace par l’argument si la fonction est annotée comme une identité.

On garde ainsi une complexité raisonnable par rapport à une β-réduction complète qui serait évidemment irréaliste pour de gros programmes.

On passe ensuite à l’ordre supérieur en permettant des annotations de la forme Annotation → Annotation. Les fonctions comme List.map peuvent donc être représentées comme Id → Id. Bien que cette solution ne soit pas complète, elle couvre la grande majorité des cas d’utilisation.

Reconstruction de tuples

On passe ensuite des paires aux tuples de taille arbitraire. Cela complexifie le problème : si on construit une paire à partir des projections des deux premiers champs d’une variable, ce n’est pas forcément la variable, puisqu’elle peut avoir plus de champs.

On a alors deux solutions : tout d’abord, on peut annoter les projections avec la taille du tuple pour savoir si on reconstruit la variable en entier. Par exemple, si on reconstruit une paire avec deux projections d’un triplet, on sait qu’on ne peut pas simplifier cette reconstruction.

L’autre solution, plus ambitieuse, est d’adopter une définition moins stricte de l’égalité, et de dire qu’on peut remplacer, par exemple, (x,y) par (x,y,z). En effet, si la variable a été typée comme une paire, on a la garantie qu’on accédera jamais au champ z de toute façon. Le comportement du programme sera donc le même si on étend la variable avec des champs supplémentaires.

Utiliser l’égalité observationnelle permet d’éviter beaucoup d’allocations, mais elle peut utiliser plus de mémoire dans certains cas : si le triplet cesse d’être utilisé, il ne sera pas désalloué par le Garbage Collector (GC), et le champ z restera donc en mémoire pour rien tant que (x,y) est utilisé.

Cette approche reste intéressante, au moins si on donne la possibilité à l’utilisateur de l’activer manuellement pour certains blocs.

Récursion

On ajoute maintenant les définitions récursives à notre langage, par le biais d’un opérateur de point fixe.

Pour prouver qu’une fonction récursive est l’identité, on doit procéder par induction. La difficulté est alors de prouver que la fonction termine, pour que l’induction soit correcte.

On peut distinguer trois niveaux de preuve : la première option est de ne pas prouver la terminaison, et de laisser l’utilisateur choisir les fonctions dont il est sûr qu’elles terminent. On suppose donc que la fonction est l’identité, et on simplifie son corps avec cette hypothèse. Cette approche est suffisante pour la plupart des cas pratiques, mais son problème principal est qu’elle autorise à écrire du code qui casse la sûreté du typage, comme discuté ci-dessus.

La seconde option est de faire notre hypothèse d’induction uniquement sur des applications de la fonction sur des éléments plus “petits” que l’argument. Un élément est défini comme tel s’il est une projection de l’argument, ou une projection d’un élément plus petit. Cela n’est pas suffisant pour prouver que la fonction termine (par exemple si l’argument est cyclique), mais c’est assez pour avoir un typage sûr. En effet, cela implique que toutes les valeurs de retour possibles de la fonction sont construites (puisqu’elles ne peuvent provenir directement d’un appel récursif), et ont donc un type défini. Le typage échouerait donc si la fonction pouvait renvoyer une valeur qui n’est pas identifiable à son argument.

Finalement, on peut vouloir une équivalence observationnelle parfaite entre la fonction et l’identité pour la simplifier. Dans ce cas, la solution que nous proposons est de créer une annotation spéciale pour les fonctions qui sont l’identité quand elles sont appliquées à un objet non cyclique. On peut prouver qu’elles ont cette propriété avec l’induction décrite ci-dessus. La difficulté est ensuite de faire la simplification sur les bonnes applications : si un objet est immutable, n’est pas défini récursivement, et que tous ses sous-objets satisfont cette propriété, on le dit inductif et on peut simplifier les applications sur lui. On propage le statut inductif des objets lors de notre passe récursive d’optimisation.

###Reconstruction de blocs

La représentation des blocs dans Flambda pose des problèmes intéressants pour détecter leur égalité, ce qui est souvent nécessaire pour prouver une identité. En effet, il est difficile de détecter la reconstruction d’un bloc à l’identique.

Blocs dans Flambda

Variants

The blocks in Flambda come from the existence of variants in OCaml: one type may have several different constructors, as we can see in

type choice = A of int | B of int

Quand OCaml est compilé vers Flambda, l’information du constructeur utilisé par un objet est perdue, et est remplacée par un tag. Le tag est un nombre contenu dans un entête de la représentation mémoire de l’objet, et est un nombre entre 0 et 255 représentant le constructeur de l’objet. Par exemple, un objet de type choice aurait le tag 0 si c’est un A et 1 si c’est un B.

Le tag est ainsi présent dans la mémoire à l’exécution, ce qui permet par exemple d’implémenter le pattern matching de OCaml comme un switch en Flambda, qui fait de simples comparaisons sur le tag pour décider quelle branche prendre.

Ce système nous complique la tâche puisque le typage de Flambda ne nous dit pas quel type de constructeur contient un variant, et empêche donc de décider facilement si deux variants sont égaux.

Généralisation des tags

Pour plus de complexité, les tags sont en faits utilisés pour tous les blocs, c’est à dire les tuples, les modules, les fonctions (en fait presque toutes les valeurs sauf les entiers et les constructeurs constants). Quand l’objet n’est pas un variant, on lui donne généralement un tag 0. Ce tag n’est donc jamais lu par la suite (puisqu’on ne fait pas de match sur l’objet), mais nous empêche de comparer simplement deux tuples, puisqu’on verra simplement deux objets de tag inconnu en Flambda.

Inlining

Enfin, on optimise ce système en inlinant les tuples : si on a un variant de type Pair of int*int, au lieu d’être représenté comme le tag de Pair et une adresse mémoire pointant vers un couple (donc un tag 0 et les deux entiers), le couple est inliné et l’objet est de la forme (tag Pair, entier, entier).

Cela implique que les variants sont de taille arbitraire, qui est aussi inconnue dans Flambda.

Approche existante

Une solution partielle au problème existait déjà dans une Pull Request (PR) disponible ici.

L’approche qui y est adoptée est naturelle : on y utilise les switchs pour gagner de l’information sur le tag d’un bloc, en fonction de la branche prise. La PR permet aussi de connaître la mutabilité et la taille du bloc dans chaque branche, en partant de OCaml (où l’information est connue puisque le constructeur est explicite dans le match), et propageant l’information jusqu’à Flambda.

Cela permet d’enregistrer tous les blocs sur lesquels on a fait un switch dans l’environnement, avec leur tag, taille et mutabilité. On peut ensuite détecter si on reconstruit l’un d’entre eux avec la primitive Pmakeblock.

Cette approche est malheureusement limitée puisqu’ils existe de nombreux cas où on pourrait connaître le tag et la taille du bloc sans faire de switch dessus. Par exemple, on ne pourra jamais simplifier une reconstruction de tuple avec cette solution.

Nouvelle approche

Notre nouvelle approche commence donc par propager plus d’information depuis OCaml. La propagation est fondée sur deux PR qui existaient sur Flambda 2, et qui annotent dans lambda chaque projection (Pfiel) avec des informations dérivées du typage OCaml. Une ajoute la mutabilité du bloc et l’autre son tag et enfin sa taille.

Notre première contribution a été d’adapter ces PRs à Flambda 1 et de les propager de lambda à Flambda correctement.

Nous avons ensuite les informations nécessaires pour détecter les reconstructions de blocs : en plus d’avoir une liste de blocs sur lesquels on a switché, on crée une liste de blocs partiellement immutables, c’est à dire dont on sait que certains champs sont immutables.

On l’utilise ainsi :

Découverte de blocs

Dès qu’on voit une projection, on regarde si elle est faite sur un bloc immutable de taille connue. Si c’est le cas, on ajoute le bloc correspondant aux blocs partiels. On vérifie que l’information qu’on a sur le tag et la taille est compatible avec celle des projections de ce bloc vues précédemment. Si on connaît maintenant tous les champs du bloc, on l’ajoute à notre liste de blocs connus sur lesquels on peut faire des simplifications.

On garde aussi les informations sur les blocs qu’on connaît grâce aux switchs.

Simplification

Cette partie est similaire à celle de la PR originale : quand on construit un bloc immutable, on vérifie si on le connaît, et le cas échéant on ne le réalloue pas.

Par rapport à l’approche originale, nous avons aussi réduit la complexité de la PR originale (de quadratique à linéaire), en enregistrant l’association de chaque variable de projection à son index et bloc original. Nous avons aussi modifié des détails de l’implémentation originale qui auraient pu créer un bug lorsque associés à notre PR.

Exemple

Considérons cette fonction:

type typ1 = A of int | B of int * int
type typ2 = C of int | D of {x:int; y:int}
let id = function
  | A n -> C n
  | B (x,y) -> D {x; y}

Le compilateur actuel produirait le Flambda suivant:

End of middle end:
let_symbol
  (camlTest__id_21
    (Set_of_closures (
      (set_of_closures id=Test.8
        (id/5 = fun param/7 ->
          (switch*(0,2) param/7
           case tag 0:
            (let
              (Pmakeblock_arg/11 (field 0<{../../test.ml:4,4-7}> param/7)
               Pmakeblock/12
                 (makeblock 0 (int)<{../../test.ml:4,11-14}>
                   Pmakeblock_arg/11))
              Pmakeblock/12)
           case tag 1:
            (let
              (Pmakeblock_arg/15 (field 1<{../../test.ml:5,4-11}> param/7)
               Pmakeblock_arg/16 (field 0<{../../test.ml:5,4-11}> param/7)
               Pmakeblock/17
                 (makeblock 1 (int,int)<{../../test.ml:5,17-23}>
                   Pmakeblock_arg/16 Pmakeblock_arg/15))
              Pmakeblock/17)))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__id_5_closure (Project_closure (camlTest__id_21, id/5)))
  (camlTest (Block (tag 0,  camlTest__id_5_closure)))
End camlTest

Notre amélioration permet de détecter que cette fonction reconstruit des blocs similaires et donc la simplifie:

End of middle end:
let_symbol
  (camlTest__id_21
    (Set_of_closures (
      (set_of_closures id=Test.7
        (id/5 = fun param/7 ->
          (switch*(0,2) param/7
           case tag 0 (1): param/7
           case tag 1 (2): param/7))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__id_5_closure (Project_closure (camlTest__id_21, id/5)))
  (camlTest (Block (tag 0,  camlTest__id_5_closure)))
End camlTest

Pistes d’amélioration

Relâchement de l’égalité

On peut utiliser l’égalité observationnelle étudiée dans la partie théorique pour l’égalité de blocs, afin d’éviter plus d’allocations. L’implémentation est simple :

Quand on crée un bloc, pour voir si il est alloué, l’approche normale est de regarder si chacun de ses champs est une projection connue d’un autre bloc, a le même index et si les deux blocs sont de même taille. On peut simplement supprimer la dernière vérification.

L’implémentation a été un peu plus difficile que prévu à cause de détails pratiques. Tout d’abord, on veut appliquer cette optimisation uniquement sur certains blocs annotés par l’utilisateur. Il faut donc propager l’annotation jusqu’à Flambda.

De plus, si on se contente d’implémenter l’optimisation, beaucoup de cas seront ignorés car les variables inutilisées sont simplifiées avant notre passe. Par exemple, prenons une fonction de la forme suivante :

let loose_id (a,b,c) = (a,b)

La variable c sera simplifiée avant d’atteindre Flambda, et on ne pourra donc plus prouver que (a,b,c) est immutable car son troisième champ pourrait ne pas l’être. Ce problème est en passe d’être résolu sur Flambda 2 grâce à une PR qui propage l’information de mutabilité pour tous les blocs, mais nous n’avons pas eu le temps nécessaire pour l’adapter à Flambda 1.

Détection d’identités récursives

Maintenant que nous pouvons détecter les reconstructions de blocs, reste à résoudre le problème des fonctions récursives.

Approche sans garanties

Nous avons commencé par implémenter une approche qui ne comporte pas de preuve de terminaison. L’idée est de rajouter la preuve ensuite, ou d’autoriser les fonctions qui ne terminent pas toujours à être simplifiées à condition qu’elles soient correctes au niveau du typage (voir section 7 dans la partie théorique).

Ici, on fait confiance à l’utilisateur pour vérifier ces propriétés manuellement.

Nous avons donc modifié la simplification de fonction : quand on simplifie une fonction à un seul argument, on commence par supposer que cette fonction est l’identité avant de simplifier son corps. On vérifie ensuite si le résultat est équivalent à une identité en le parcourant récursivement, pour couvrir le plus de cas possible (par exemple les branchements conditionnels). Si c’est le cas, la fonction est remplacée par l’identité ; sinon, on revient à une simplification classique, sans hypothèse d’induction.

Propagation de constantes

Nous avons ensuite amélioré notre fonction qui détermine si le corps d’une fonction est une identité ou non, pour gérer les constantes. Il propage les informations d’égalité qu’on gagne sur l’argument lors des branchements conditionnels.

Ainsi, si on a une fonction de la forme

type truc = A | B | C
let id = function
  | A -> A
  | B -> B
  | C -> C

ou même

let id x = if x=0 then 0 else x

on détectera bien que c’est l’identité.

Exemples

Fonctions récursives

Nous pouvons maintenant détecter les identités récursives :

let rec listid = function
  | t::q -> t::(listid q)
  | [] -> []

compilait avant ainsi:

End of middle end:
let_rec_symbol
  (camlTest__listid_5_closure
    (Project_closure (camlTest__set_of_closures_20, listid/5)))
  (camlTest__set_of_closures_20
    (Set_of_closures (
      (set_of_closures id=Test.11
        (listid/5 = fun param/7 ->
          (if param/7 then begin
            (let
              (apply_arg/13 (field 1<{../../test.ml:9,4-8}> param/7)
               apply_funct/14 camlTest__listid_5_closure
               Pmakeblock_arg/15
                 *(apply*&#091;listid/5]<{../../test.ml:9,15-25}> apply_funct/14
                    apply_arg/13)
               Pmakeblock_arg/16 (field 0<{../../test.ml:9,4-8}> param/7)
               Pmakeblock/17
                 (makeblock 0<{../../test.ml:9,12-25}> Pmakeblock_arg/16
                   Pmakeblock_arg/15))
              Pmakeblock/17)
            end else begin
            (let (const_ptr_zero/27 Const(0a)) const_ptr_zero/27) end))
         free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
let_symbol (camlTest (Block (tag 0,  camlTest__listid_5_closure)))
End camlTest

On détecte maintenant que c’est l’identité :

End of middle end:
let_symbol
  (camlTest__set_of_closures_20
    (Set_of_closures (
      (set_of_closures id=Test.13 (listid/5 = fun param/7 -> param/7)
        free_vars={ } specialised_args={}) direct_call_surrogates={ }
        set_of_closures_origin=Test.1])))
  (camlTest__listid_5_closure
    (Project_closure (camlTest__set_of_closures_20, listid/5)))
  (camlTest (Block (tag 0,  camlTest__listid_5_closure)))
End camlTest

Exemple non sûr

En revanche, on peut profiter de l’absence de garanties pour contourner le typage, et accéder à une adresse mémoire comme à un entier :

type bugg = A of int*int | B of int
let rec bug = function
  | A (a,b) -> (a,b)
  | B x -> bug (B x)
  
let (a,b) = (bug (B 42))
let _ = print_int b

Cette fonction va être simplifiée vers l’identité alors que le type bugg n’est pas compatible avec le type tuple ; quand on essaie de projeter sur le second champ du variant b, on accède à une partie de la mémoire indéfinie :

$ ./unsafe.out
47423997875612

Pistes d’améliorations – court terme

Annotation des fonctions

Une amélioration simple en théorie, serait de laisser le choix à l’utilisateur des fonctions sur lesquelles il veut appliquer ces optimisations qui ne sont pas toujours correctes. Nous n’avons pas eu le temps de faire le travail de propagation de l’information jusqu’à Flambda, mais il ne devrait pas y avoir de difficultés d’implémentation.

Ordre sur les arguments

Pour avoir une optimisation plus sûre, on voudrait pouvoir utiliser l’idée développée dans la partie théorique, qui rend l’optimisation correcte sur les objets non cycliques, et surtout qui nous redonne les garanties du typage pour éviter le problème vu dans l’exemple ci-dessus.

Afin d’avoir cette garantie, on veut changer la passe de simplification pour que son environnement contienne une option de couple fonction – argument. Quand cette option existe, le couple indique que nous sommes dans le corps d’une fonction, en train de la simplifier, et donc que les applications de la fonction sur des éléments plus petits que l’argument peuvent être simplifiés en une identité. Bien sûr, on devrait aussi modifier la passe pour se rappeler des éléments qui ne sont pas plus petits que l’argument.

Pistes d’améliorations – long terme

Exclusion des objets cycliques

Comme décrit dans la partie théorique, on pourrait déduire récursivement quels objets sont cycliques et tenter de les exclure de notre optimisation. Le problème est alors qu’au lieu de remplacer les fonctions par l’identité, on doit avoir une annotation spéciale qui représente IdRec.

Cela devient bien plus complexe à implémenter quand on compile entre plusieurs fichiers, puisqu’on doit alors avoir cette information dans l’interface des fichiers déjà compilés pour pouvoir faire l’optimisation quand c’est nécessaire.

Une piste serait d’utiliser les fichiers .cmx pour enregistrer cette information quand on compile un fichier, mais ce genre d’implémentation était trop longue pour être réalisée pendant le stage. De plus, il n’est même pas évident qu’elle soit un bon choix pratique : elle complexifierait beaucoup l’optimisation pour un avantage faible par rapport à une version correcte sur les objets non cycliques et activée par une annotation de l’utilisateur.

opam 2.1.0~rc2 released

2021-06-23T13:56:36Z

Feedback on this post is welcomed on Discuss!

The opam team has great pleasure in announcing opam 2.1.0~rc2!

The focus since beta4 has been preparing for a world with more than one released version of opam (i.e. 2.0.x and 2.1.x). The release candidate extends CLI versioning further and, under the hood, includes a big change to the opam root format which allows new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots).

Please do take this release candidate for a spin! It is available in the Docker images at ocaml/opam on Docker Hub as the opam-2.1 command (or you can sudo ln -f /usr/bin/opam-2.1 /usr/bin/opam in your Dockerfile to switch to it permanently). The release candidate can also be tested via our installation script (see the wiki for more information).

Thank you to anyone who noticed the unannounced first release candidate and tried it out. Between tagging and what would have been announcing it, we discovered an issue with upgrading local switches from earlier alpha/beta releases, and so fixed that for this second release candidate.

Assuming no showstoppers, we plan to release opam 2.1.0 next week. The improvements made in 2.1.0 will allow for a much faster release cycle, and we look forward to posting about the 2.2.0 plans soon!

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam directory.

The upgrade instructions are unchanged:

Either from binaries: run

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0~rc2"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

We hope there won't be any, but please report any issues to the bug-tracker. Thanks for trying it out, and hoping you enjoy!

Tutorial: Format Module of OCaml

2021-05-06T13:56:36Z

The Format module of OCaml is an extremely powerful but unfortunately often poorly used module.

It combines two distinct elements:

pretty-print boxes
semantic tags

This tutorial aims to demystify much of this module and explain the range of things that you can do with it.

Réunion annuelle du Club des utilisateurs d’Alt-Ergo 2021

2021-04-29T13:56:36Z

La troisième réunion annuelle du Club des utilisateurs d’Alt-Ergo a eu lieu le 1er avril ! Cette réunion annuelle est l’endroit idéal pour passer en revue les besoins de chaque partenaire concernant Alt-Ergo. Nous avons eu le plaisir de recevoir nos partenaires pour discuter de la feuille de route concernant les développements et les améliorations futures d’Alt-Ergo.

Alt-Ergo est un démonstrateur automatique de formules mathématiques, créé au LRI et développé par OCamlPro depuis 2013. Pour en savoir plus ou rejoindre le Club, visitez le site https://alt-ergo.ocamlpro.com.

Notre Club a plusieurs objectifs. Son objectif principal est de garantir la pérennité d’Alt-Ergo en favorisant la collaboration entre les membres du Club et en tissant des liens avec les utilisateurs de méthodes formelles, telle que la communauté Why3. L’une de nos priorités est de définir les besoins des utilisateurs de solveurs de contraintes en étendant Alt-Ergo à de nouveaux domaines tels que le Model Checking, tout en concurrençant les autres solveurs de l’état de l’art au cours de compétitions internationales. Enfin, le dernier objectif du Club est de trouver de nouveaux projets ou contrats pour le développement de fonctionnalités à long terme.

Nous tenons à remercier tous nos membres pour leur soutien : Mitsubishi Electric R&D Centre Europe, AdaCore et le CEA List. Nous souhaitons également mettre en lumière l’équipe de développement Why3 avec laquelle nous travaillons pour améliorer nos outils.

Cette année, de nouveaux points d’intérêts ont été soulevés par nos membres. Dans un premier temps, la génération de modèles, ajoutée à Alt-Ergo suite à la dernière édition, a été utile à la majorité des membres du club. Les points techniques souhaités à présent sont de pouvoir raffiner les contraintes et étudier comment les propager. Dans un second temps a eu lieu la présentation de Dolmen, le parseur/typer qui permettra de ne typer qu’une seule fois les fichiers SMT2 et d’être prêt pour le SMT3. Son intégration à Alt-Ergo est en cours, l’avis des membres du club est enthousiaste sur les apports futurs de l’outil Dolmen à la communauté des solveurs SMT !

Ces fonctionnalités sont désormais nos principales priorités, retrouvez les planches présentées à la réunion du Club édition 2021. Pour suivre nos avancement et les nouveautés, n’hésitez pas à lire nos articles sur notre blog.

New Try-Alt-Ergo

2021-03-29T13:56:36Z

Have you heard about our Try-Alt-Ergo website? Created in 2014 (see our blogpost), the first objective was to facilitate access to our performant SMT Solver Alt-Ergo. Try-Alt-Ergo allows you to write and run your problems in your browser without any server computation.

This playground website has been maintained by OCamlPro for many years, and it's high time to bring it back to life with new updates. We are therefore pleased to announce the new version of the Try-Alt-Ergo website! In this article, we will first explain what has changed in the back end, and what you can use if you are interested in running your own version of Alt-Ergo on a website, or in an application! And then we will focus on the new front-end of our website, from its interface to its features through its tutorial about the program.* *

Try-Alt-Ergo 2014

Try-Alt-Ergo was designed to be a powerful and simple tool to use. Its interface was minimalist. It offered three panels, one panel (left) with a text area containing the problem to prove. The centered panel was composed of a button to run Alt-Ergo, load examples, set options. The right panel showed these options, examples and other information. This design lacked some features that have been added to our solver through the years. Features such as models (counter-examples), unsat-core, more options and debug information was missing in this version.

Try-Alt-Ergo did not offer a proper editor (with syntax coloration), a way to save the file problem nor an option to limit the run of the solver with a time limit. Another issue was about the thread. When the solver was called the webpage froze, that behavior was problematic in case of the long run because there was no way to stop the solver.

Alt-Ergo 1.30

The 1.30 version of Alt-Ergo was the version used in the back-end to prove problems. Since this version, a lot of improvements have been done in Alt-Ergo. To learn more about these improvements, see our changelog in the documentation.

Over the years we encountered some difficulties to update the Alt-Ergo version used in Try-Alt-Ergo. We used Js_of_ocaml to compile the OCaml code of our solver to be runnable as a JavaScript code. Some libraries were not available in JavaScript and we needed to manually disable them. The lack of automatism leads to a lack of time to update the JavaScript version of Alt-Ergo in Try-Alt-Ergo.

In 2019 we switched our build system to dune which opens the possibility to ease the cross-compilation of Alt-Ergo in JavaScript.

New back-end

With some simple modification, we were able to compile Alt-Ergo in JavaScript. This modification is simple enough that this process is now automated in our continuous integration. This will enable us to easily provide a JavaScript version of our Solver for each future version.

Two ways of using our solver in JavaScript are available:

alt-ergo.js, a JavaScript version of the Alt-Ergo CLI. It can be runned with node: node alt-ergo.js <options> <file>. Note that this code is slower than the natively compiled CLI of Alt-Ergo.In our effort to open the SMT world to more people, an npm package is the next steps of this work.
alt-ergo-worker.js, a web worker of Alt-Ergo. This web worker needs JSON file to input file problem, options into Alt-Ergo and to returns its answers:
- Options are sent as a list of couple name:value like:{"debug":true,"input_format":"Native","steps_bound":100,"sat_solver": "Tableaux","file":"test-file"}You can specify all options used in Alt-Ergo. If some options are missing, the worker uses the default value for these options. For example, if debug is not specified the worker will use its defaults value :false.- Input file is sent as a list of string, with the following format:{ "content": [ "goal g: true"] }
- Alt-Ergo answers can be composed with its results, debug information, errors, warnings …{ "results": [ "File "test-file", line 1, characters 9-13: Valid (0.2070) (0 steps) (goal g) ] ,``"debugs": [ "[Debug][Sat_solver]", "use Tableaux-like solver"] }like the options, a result value like debugs does not contains anything, "debugs": [...] is not returned.
- See the Alt-Ergo web-worker documentation to learn more on how to use it.

New Front-end

The Try-Alt-Ergo has been completely reworked and we added some features:

The left panel is still composed in an editor and answers area
- Ace editor with custom syntax coloration (both native and smt-lib2) is now used to make it more pleasant to write your problems.
A top panel that contains the following buttons:
- Ask Alt-Ergo which retrieves content from the editor and options, launch the web worker and print answers in the defined areas.
- Load and Save files.
- Documentation, that sends users to the newly added native syntax documentation of Alt-Ergo.
- Tutorial, that opens an interactive tutorial to introduce you to Alt-Ergo native syntax and program verification.

A right panel composed of tabs:
- Start and About that contains general information about Alt-Ergo, Try-Alt-Ergo and how to use it.
- Outputs prints more information than the basic answer area under the editor. In these tabs you can find debugs (long) outputs, unsat-core or models (counter-example) generated by Alt-Ergo.
- Options contains every option you can use, such as the time limit / steps limit or to set the format of the input file to prove .
- Statistics is still a basic tab that only output axioms used to prove the input problem.
- Examples contains some basic examples showing the capabilities of our solver.

We hope you will enjoy this new version of Try-Alt-Ergo, we can't wait to read your feedback!

This work was done at OCamlpro.

opam 2.0.8 release

2021-02-08T13:56:36Z

We are pleased to announce the minor release of opam 2.0.8.

This new version contains some backported fixes:

Critical for fish users! Don't add . to PATH. [#4078]
Fix sandbox script for newer ccache versions. [#4079 and #4087]
Fix sandbox crash when ~/.cache is a symlink. [#4068]
User modifications to the sandbox script are no longer overwritten by opam init. [#4020 & #4092]
macOS sandbox script always mounts /tmp read-write, regardless of TMPDIR [#3742, addressing ocaml/opam-repository#13339]
pre- and post-session hooks can now print to the console [#4359]
Switch-specific pre/post sessions hooks are now actually run [#4472]
Standalone opam-installer now correctly builds from sources [#4173]
Fix arch variable detection when using 32bit mode on ARM64 and i486 [#4462]

A more complete release note is available.

Installation instructions (unchanged):

From binaries: run

$~ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.8"

From source, using opam:

$~ opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com, and published in discuss.ocaml.org.

2020 at OCamlPro

2021-02-02T13:56:36Z

OCamlPro was created in 2011 to advocate the adoption of the OCaml language and formal methods in general in the industry. While building a team of highly-skilled engineers, we navigated through our expertise domains, delivering works on the OCaml language and tooling, training companies to the use of strongly-typed languages like OCaml but also Rust, tackling formal verification challenges with formal methods, maintaining the SMT solver Alt-Ergo, designing languages and tools for smart contracts and blockchains, and more!

In this article, as every year (see 2019 at OCamlPro for last year's post), we review some of the work we did during 2020, in many different worlds.

Table of contents

In the World of OCaml

Flambda & Compilation Team
Opam, the OCaml Package Manager
Encouraging OCaml Adoption: Trainings and Resources for OCaml
Open Source Tooling and Libraries for OCaml
Supporting the OCaml Software Foundation
Events

In the World of Formal Methods

Alt-Ergo Development
Alt-Ergo Users’ Club and R&D Projects
Alt-Ergo’s Roadmap

In the World of Rust

In the World of Blockchain Languages

We warmly thank all our partners, clients and friends for their support and collaboration during this peculiar year!

The first lockdown was a surprise and we took advantage of this special moment to go over our past contributions and sum it up in a timeline that gives an overview of the key events that made OCamlPro over the years. The timeline format is amazing to reconnect with our history and to take stock in our accomplishments.

Now this will turn into a generic timeline edition tool on the Web, stay tuned if you are interested in our internal project to be available to the general public! If you think that a timeline would fit your needs and audience, we designed a simplistic tool, tailored for users who want complete control over their data.

In the World of OCaml

Flambda & Compilation Team

Work by Pierre Chambart, Vincent Laviron, Guillaume Bury, Pierrick Couderc and Louis Gesbert

OCamlPro is proud to be working on Flambda2, an ambitious work on an OCaml optimizing compiler, in close collaboration with Mark Shinwell from our long-term partner and client Jane Street. Flambda focuses on reducing the runtime cost of abstractions and removing as many short-lived allocations as possible. In 2020, the Flambda team worked on a considerable number of fixes and improvements, transforming Flambda2 from an experimental prototype to a version ready for testing in production!

This year also marked the conclusion of our work on the pack rehabilitation (see our two recent posts Part 1 and Part 2, and a much simpler Version in 2011). Our work aimed to give them a new youth and utility by adding the possibility to generate functors or recursive packs. This improvement allows programmers to define big functors, functors that are split among multiple files, resulting in what we can view as a way to implement some form of parameterized libraries.

This work is allowed thanks to Jane Street’s funding.

Opam, the OCaml Package Manager

Work by Raja Boujbel, Louis Gesbert and Thomas Blanc

Our 2020 work on Opam led to the release of two versions of opam 2.0 with small fixes, and the release of three alphas and two betas of Opam 2.1!

Opam 2.1.0 will soon go to release candidate and will introduce a seamless integration of depexts (system dependencies handling), dependency locking, pinning sub-directories, invariant-based definition for Opam switches, the configuration of Opam from the command-line without the need for a manual edition of the configuration files, and the CLI versioning for better handling of CLI evolutions.

This work is greatly helped by Jane Street’s funding and support.

Encouraging OCaml Adoption: Trainings and Resources for OCaml

Work by Pierre Chambart, Vincent Laviron, Adrien Champion, Mattias, Louis Gesbert and Thomas Blanc

OCamlPro is also a training centre. We organise yearly training sessions for programmers from multiple companies in our offices: from OCaml to OCaml tooling to Rust! We can also design custom and on-site trainings to meet specific needs.

We released a brand new version of TryOCaml, a tool born from our work on Learn-OCaml! Try OCaml has been highly praised by professors at the beginning of the Covid lockdown. Even if it can be used as a personal sandbox, it’s also possible to adapt its usage for classes. TryOCaml is a hassle-free tool that lowers significantly the barriers to start coding in OCaml, as no installation is required.

We regularly release cheat sheets for developers: in 2020, we shared the long-awaited Opam 2.0 cheat sheet, with a new theme! In just two pages, you’ll have in one place the everyday commands you need as an Opam user. We also shine some light on unsung features which may just change your coding life.

2020 was also an important year for the OCaml language itself: we were pleased to welcome OCaml 4.10! One of the highlights of the release was the “Best-fit” Garbage Collector Strategy. We had an in-depth look at this exciting change.

This work is self-funded by OCamlPro as part of its effort to ease the adoption of OCaml.

Open Source Tooling and Libraries for OCaml

Work by Fabrice Le Fessant, Léo Andrès and David Declerck

OCamlPro has a long history of developing open source tooling and libraries for the community. 2020 was no exception!

drom is a simple tool to create new OCaml projects that will use best OCaml practices, i.e. Opam, Dune and tests. Its goal is to provide a cargo-like user experience and helps onboarding new developers in the community. drom is available in the official opam repository.

directories is a new OCaml Library that provides configuration, cache and data paths (and more!). The library follows the suitable conventions on Linux, MacOS and Windows.

opam-bin is a framework to create and use binary packages with Opam. It enables you to create, use and share binary packages easily with opam, and to create as many local switches as you want spending no time, no disk space! If you often use Opam, opam-bin is a must-have!

We also released a number of libraries, focused on making things easy for developers… so we named them with an ez_ prefix: ez_cmdliner provides an Arg-like interface for cmdliner, ez_file provides simple functions to read and write files, ez_subst provides easily configurable string substitutions for shell-like variable syntax, ez_config provides abstract options stored in configuration files with an OCaml syntax. There are also a lot of ezjs-* libraries, that are bindings to Javascript libraries that we used in some of our js_of_ocaml projects.

*This work was self-funded by OCamlPro as part of its effort to improve the OCaml ecosystem.*

Supporting the OCaml Software Foundation

OCamlPro was proud and happy to initiate the OCaml User Survey 2020 as part of the mission of the [OCaml Software Foundation]. The goal of the survey was to better understand the community and its needs. The final results have not yet been published by the Foundation, we are looking forward to reading them soon!

Events

Though the year took its toll on our usual tour of the world conferences and events, OCamlPro members still took part in the annual 72-hour team programming competition organised by the International Conference on Functional Programming (ICFP). Our joint team “crapo on acid” went through the final!

In the World of Formal Methods

Work by Albin Coquereau, Mattias, Sylvain Conchon, Guillaume Bury and Louis Rustenholz

Sylvain Conchon joined OCamlPro as Formal Methods Chief Scientific Officer in 2020!

Alt-Ergo Development

In 2020, we focused on the maintainability of our solver. The first part of this work was to maintain and fix issues within the already released version. The 2.3.0 (released in 2019) had some issues that needed to be fixed minor releases.

The second part of the maintainability work on Alt-Ergo contains more major features. All these features were released in the new version 2.4.0 of Alt-Ergo. The main goal of this release was to focus on the user experience and the documentation. This release also contains bug fixes and many other improvements. Alt-Ergo is on its way towards a new documentation and in particular a new documentation on its native syntax.

We also tried to improve the command line experience of our tools with the use of the cmdliner library to parse Alt-Ergo options. This library allows us to improve the manpage of our tool. We tried to harmonise the debug messages and to improve all of Alt-Ergo’s outputs to make it clearer for the users.

Alt-Ergo Users’ Club and R&D Projects

We thank our partners from the Alt-Ergo Users’ Club, Adacore, CEA List, MERCE (Mitsubishi Electric R&D Centre Europe) and Trust-In-Soft, for their trust. Their support allows us to maintain our tool.

The club was launched in 2019 and the second annual meeting of the Alt-Ergo Users’ Club was held in mid-February 2020. Our annual meeting is the perfect place to review each partner’s needs regarding Alt-Ergo. This year, we had the pleasure of receiving our partners to discuss the roadmap for future Alt-Ergo developments and enhancements. If you want to join us for the next meeting (coming soon), contact us!

We also want to thank our partners from the FUI R&D Project LCHIP. Thanks to this project, we were able to add a new major feature in Alt-Ergo: the support for incremental commands (push, pop and check-sat-assuming) from the smt-lib2 standard.

Alt-Ergo’s Roadmap

Some of the work we did in 2020 is not yet available. Thanks to our partner MERCE (Mitsubishi Electric R&D Centre Europe), we worked on the SMT model generation. Alt-Ergo is now (partially) able to output a model in the smt-lib2 format. Thanks to the Why3 team from University of Paris-Saclay, we hope that this work will be available in the Why3 platform to help users in their program verification efforts.

Another project was launched in 2020 but is still in early development: the complete rework of our Try-Alt-Ergo website with new features such as model generation. Try Alt-Ergo current version allows users to use Alt-Ergo directly from their browsers (Firefox, Chromium) without the need of a server for computations.

This work needed a JavaScript compatible version of Alt-Ergo. We have made some work to build our solver in two versions, one compatible with Node.js and another as a webworker. We hope that this work can make it easier to use our SMT solver in web applications.

This work is funded in part by the FUI R&D Project LCHIP, MERCE, Adacore and with the support of the Alt-Ergo Users’ Club.

In the World of Rust

Work by Adrien Champion

As OCaml-ians, we naturally saw in the Rust language a beautiful complement to our approach. One opportunity to explore this state-of-the art language has been to pursue our work on ocp-memprof and build Memthol, a visualizer and analyzer to profile OCaml programs. It works on memory dumps containing information about the size and (de)allocation date of part of the allocations performed by some execution of a program.

Between lockdowns, we’ve also been able to hold our Rust training. It’s designed as a highly-modular vocational course, from 1 to 4 days. The training covers a beginner introduction to Rust’s basics features, crucial features and libraries for real-life development and advanced features, all through complex use-cases one would find in real life.

This work was self-funded by OCamlPro as part of our exploration of other statically and strongly typed functional languages.

In the World of Blockchain Languages

Work by David Declerck and Steven de Oliveira

One of our favourite activities is to develop new programming languages, specialized for specific domains, but with nice properties like clear semantics, strong typing, static typing and functional features. In 2020, we applied our skills in the domain of blockchains and smart contracts, with the creation of a new language, Love, and work on a well-known language, Solidity.

In 2020, our blockchain experts released Love, a type-safe language with an ML syntax and suited for formal verification. In a few words, Love is designed to be expressive for fast development, efficient in execution time and cheap in storage, and readable in terms of smart contracts auditability. Yet, it has a clear and formal semantics and a strong type system to detect bugs. It allows contracts to use other contracts as libraries, and to call viewers on other contracts. Contracts developed in Love can also be formally verified.

We also released a Solidity parser and printer written in OCaml using Menhir, and used it to implement a full interpreter directly in a blockchain. Solidity is probably the most used language for smart contracts, it was first born on Ethereum but many other blockchains provide it as a way to easily onboard new developers coming from the Ethereum ecosystem. In the future, we plan to extend this work with formal verification of Solidity smart contracts.

This is a joint effort with Origin Labs, the company created to tackle blockchain-related challenges.

##Towards 2021##

Adaptability and continuous improvement, that’s what 2020 brought to OCamlPro!

We will remember 2020 as a complicated year, but one that allowed us to surpass ourselves and challenge our projects. We are very proud of our team who all continued to grow, learn, and develop our projects in this particular context. We are more motivated than ever for the coming year, which marks our tenth year anniversary! We’re excited to continue sharing our knowledge of the OCaml world and to accompany you in your own projects.

Release of Alt-Ergo 2.4.0

2021-01-22T13:56:36Z

A new release of Alt-Ergo (version 2.4.0) is available.

You can get it from Alt-Ergo's website. The associated opam package will be published in the next few days.

This release contains some major novelties:

Alt-Ergo supports incremental commands (push/pop) from the smt-lib standard.
We switched command line parsing to use cmdliner. You will need to use --<option name> instead of -<option name>. Some options have also been renamed, see the manpage or the documentation.
We improved the online documentation of your solver, available here.

This release also contains some minor novelties:

.mlw and .why extension are depreciated, the use of .ae extension is advised.
Add --input (resp --output) option to manually set the input (resp output) file format
Add --pretty-output option to add better debug formatting and to add colors
Add exponentiation operation, ** in native Alt-Ergo syntax. The operator is fully interpreted when applied to constants
Fix --steps-count and improve the way steps are counted (AdaCore contribution)
Add --instantiation-heuristic option that can enable lighter or heavier instantiation
Reduce the instantiation context (considered foralls / exists) in CDCL-Tableaux to better mimic the Tableaux-like SAT solver
Multiple bugfixes

The full list of changes is available here. As usual, do not hesitate to report bugs, to ask questions, or to give your feedback!

opam 2.1.0~beta4 released

2021-01-13T13:56:36Z

Feedback on this post is welcomed on Discuss!

On behalf of the opam team, it gives me great pleasure to announce the third beta release of opam 2.1. Don’t worry, you didn’t miss beta3 - we had an issue with a configure script that caused beta2 to report as beta3 in some instances, so we skipped to beta4 to avoid any further confusion!

We encourage you to try out this new beta release: there are instructions for doing so in our wiki. The instructions include taking a backup of your ~/.opam root as part of the process, which can be restored in order to wind back. Please note that local switches which are written to by opam 2.1 are upgraded and will need to be rebuilt if you go back to opam 2.0. This can either be done by removing _opam and repeating whatever you use in your build process to create the switch, or you can use opam switch export switch.export to backup the switch to a file before installing new packages. Note that opam 2.1 shouldn’t upgrade a local switch unless you upgrade the base packages (i.e. the compiler).

What’s new in opam 2.1?

Switch invariants
Improved options configuration (see the new option and expanded var sub-commands)
Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
Creation of lock files for reproducible installations (formerly the opam-lock plugin)
CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
Performance improvements to opam-update, conflict messages, and many other areas
New plugins: opam-compiler and opam-monorepo

Switch invariants

In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam’s solver. This was a much bigger concern early on in opam 2.0’s development, but is less of a problem with the default mccs solver.

However, it’s a problem for system compilers. opam would detect that your system compiler version had changed, but be unable to upgrade the ocaml-system package unless you went through a slightly convoluted process with --unlock-base.

In opam 2.1, base packages have been replaced by switch invariants. The switch invariant is a package formula which must be satisfied on every upgrade and install. All existing switches’ base packages could just be expressed as package1 & package2 & package3 etc. but opam 2.1 recognises many existing patterns and simplifies them, so in most cases the invariant will be "ocaml-base-compiler" {= 4.11.1}, etc. This means that opam switch create my_switch ocaml-system now creates a switch invariant of "ocaml-system" rather than a specific version of the ocaml-system package. If your system OCaml package is updated, opam upgrade will seamlessly switch to the new package.

This also allows you to have switches which automatically install new point releases of OCaml. For example:

$~ opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
$~ opam install utop

Creates a switch with OCaml 4.11.0 (the --repos= was just to select a version of opam-repository from before 4.11.1 was released). Now issue:

$~ opam repo set-url old git+https://github.com/ocaml/opam-repository
$~ opam upgrade

and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There’s not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.

opam depext integration

You no longer have to remember to run opam depext, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy as opam install ., without having to do the dance of opam pin add -n/depext/install. Just one command now for the common case!
The solver is only called once, which both saves time and also stabilises the behaviour of opam in cases where the solver result is not stable. It was possible to get one package solution for the opam depext stage and a different solution for the opam install stage, resulting in some depexts missing.
opam now has full knowledge of depexts, which means that packages can be automatically selected based on whether a system package is already installed. For example, if you have neither MariaDB nor MySQL dev libraries installed, opam install mysql will offer to install conf-mysql and mysql, but if you have the MariaDB dev libraries installed, opam will offer to install conf-mariadb and mysql.

opam lock files and reproducibility

When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 18000 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.

With opam 2.1.0, the creation of lock files is also now integrated into the client:

opam lock will create a .locked file for your current switch and project, that you can check into the repository.
opam switch create . --locked can be used by users to reproduce your dependencies in a fresh switch.

This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).

CLI Versioning

A new --cli switch was added to the first beta release, but it’s only now that it’s being widely used. opam is a complex enough system that sometimes bug fixes need to change the semantics of some commands. For example:

opam show --file needed to change behaviour
The addition of new controls for setting global variables means that the opam config was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1 still works in opam 2.0, but it’s really an OPAM 1.2.2 syntax.

Now you can specify which version of opam you expected the command to be run against. In day-to-day use of opam at the terminal, you wouldn’t specify it, and you’ll get the latest version of the CLI. For example: opam var --global is the same as opam var --cli=2.1 --global. However, if you issue opam var --cli=2.0 --global, you will told that --global was added in 2.1 and so is not available to you. You can see similar things with the renaming of opam upgrade --unlock-base to opam upgrade --update-invariant.

The intention is that --cli should be used in scripts, user guides (e.g. blog posts), and in software which calls opam. The only decision you have to take is the oldest version of opam which you need to support. If your script is using a new opam 2.1 feature (for example opam switch create --formula=) then you simply don’t support opam 2.0. If you need to support opam 2.0, then you can’t use --formula and should use --packages instead. opam 2.0 does not have the --cli option, so for opam 2.0 instead of --cli=2.0 you should set the environment variable OPAMCLI to 2.0. As with all opam command line switches, OPAMCLI is simply the equivalent of --cli which opam 2.1 will pick-up but opam 2.0 will quietly ignore (and, as with other options, the command line takes precedence over the environment).

There’s even more detail on this feature in our wiki. We’re still finalising some details on exactly how opam behaves when --cli is not given, but we’re hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.

What’s new since the last beta?

opam now uses CLI versioning (#4385)
opam now exits with code 31 if all failures were during fetch operations (#4214)
opam install now has a --download-only flag (#4036), allowing opam’s caches to be primed
opam init now advises the correct shell-specific command for eval $(opam env) (#4427)
post-install hooks are now allowed to modify or remove installed files (#4388)
New package variable opamfile-loc with the location of the installed package opam file (#4402)
opam update now has --depexts flag (#4355), allowing the system package manager to update too
depext support NetBSD and DragonFlyBSD added (#4396)
The format-preserving opam file printer has been overhauled (#3993, #4298 and #4302)
pins are now fetched in parallel (#4315)
os-family=ubuntu is now treated as os-family=debian (#4441)
opam lint now checks that strings in filtered package formulae are booleans or variables (#4439)

and many other bug fixes as listed on the release page.

New Plugins

Several features that were formerly plugins have been integrated into opam 2.1.0. We have also developed some new plugins that satisfy emerging workflows from the community and the core OCaml team. They are available for use with the opam 2.1 beta as well, and feedback on them should be directed to the respective GitHub trackers for those plugins.

opam compiler

The opam compiler plugin can be used to create switches from various sources such as the main opam repository, the ocaml-multicore fork, or a local development directory. It can use Git tag names, branch names, or PR numbers to specify what to install.

Once installed, these are normal opam switches, and one can install packages in them. To iterate on a compiler feature and try opam packages at the same time, it supports two ways to reinstall the compiler: either a safe and slow technique that will reinstall all packages, or a quick way that will just overwrite the compiler in place.

opam monorepo

The opam monorepo plugin lets you assemble standalone dune workspaces with your projects and all of their opam dependencies, letting you build it all from scratch using only Dune and OCaml. This satisfies the “monorepo” workflow which is commonly requested by large projects that need all of their dependencies in one place. It is also being used by projects that need global cross-compilation for all aspects of a codebase (including C stubs in packages), such as the MirageOS unikernel framework.

Next Steps

This is anticipated to be the final beta in the 2.1 series, and we will be moving to release candidate status after this. We could really use your help with testing this release in your infrastructure and projects and let us know if you run into any blockers. If you have feature requests, please also report them on our issue tracker -- we will be planning the next release cycle once we ship opam 2.1.0 shortly.

Memthol: exploring program profiling

2020-12-01T13:56:36Z

Memthol is a visualizer and analyzer for program profiling. It works on memory dumps containing information about the size and (de)allocation date of part of the allocations performed by some execution of a program.

For information regarding building memthol, features, browser compatibility… refer to the memthol github repository. *Please note that Memthol, as a side project, is a work in progress that remains in beta status for now. *

Memthol's background

The Memthol work was started more than a year ago (we had published a short introductory paper at the JFLA2020). The whole idea was to use the previous work originally achieved on ocp-memprof, and look for some extra funding to achieve a usable and industrial version.Then came the excellent memtrace profiler by Jane Street's team (congrats!)Memthol is a self-funded side project, that we think it still is worth giving to the OCaml community. Its approach is valuable, and can be complementary. It is released under the free GPL licence v3.

Memthol's versatility: supporting memtrace's dump format

The memtrace format is nicely designed and polished enough to be considered a future standard for other tools.This is why Memthol supports Jane Street's dumper format, instead of our own dumper library's.

Why choose Rust to implement Memthol?

We've been exploring the Rust language for more than a year now.The Memthol work was the opportunity to further explore this state-of-the-art language. We are open to extra funding, to deepen the Memthol work should industrial users be interested.

Memthol's How-to

The following steps are from the Memthol Github howto.

1. Introduction

2. Basics

3. Charts

4. Global Settings

5. Callstack Filters

Introduction

This tutorial deals with the BUI ( Browser User Interface) aspect of the profiling. How the dumps are generated is outside of the scope of this document. Currently, memthol accepts memory dumps produced by [Memtrace](https://blog.janestreet.com/finding-memory-leaks-with-memtrace) (github repository here). A memtrace dump for a program execution is a single Common Trace Format (CTF) file.

This tutorial uses CTF files from the memthol repository. All paths mentioned in the examples are from its root.

Memthol is written in Rust and is composed of

a server, written in pure Rust, and
a client, written in Rust and compiled to web assembly.

The server contains the client, which it will serve at some address on some port when launched.

Running Memthol

Memthol must be given a path to a CTF file generated by memtrace.

> ls rsc/dumps/ctf/flamba.ctf
rsc/dumps/ctf/flamba.ctf
> memthol rsc/dumps/ctf/flamba.ctf
|===| Starting
| url: http://localhost:7878
| target: `rsc/dumps/ctf/flamba.ctf`
|===|

Basics

Our running example in this section will be rsc/dumps/mini_ae.ctf:

❯ memthol --filter_gen none rsc/dumps/ctf/mini_ae.ctf
|===| Starting
| url: http://localhost:7878
| target: `rsc/dumps/ctf/mini_ae.ctf`
|===|

Notice the odd --filter_gen none passed to memthol. Ignore it for now, it will be discussed later in this section.

Once memthol is running, http://localhost:7878/ (here) will lead you to memthol's BUI, which should look something like this:

Click on the orange everything tab at the bottom left of the screen.

Memthol's interface is split in three parts:

the central, main part displays charts. There is only one here, showing the evolution of the program's total memory size over time based on the memory dump.
the header gives statistics about the memory dump and handles general settings. There is currently only one, the time window.- the footer controls your filters (there is only one here), which we are going to discuss right now.

Filters

Filters allow to split allocations and display them separately. A filter is essentially a set of allocations. Memthol has two built-in filters. The first one is the everything filter. You cannot really do anything with it except for changing its name and color using the filter settings in the footer.

Notice that when a filter is modified, two buttons appear in the top-left part of the footer. The first reverts the changes while the second one saves them. Let's save these changes.

The everything filter always contains all allocations in the memory dump. It cannot be changed besides the cosmetic changes we just did. These changes are reverted in the rest of the section.

Custom Filters

Let's create a new filter using the + add button in the top-right part of the footer.

Notice that, unlike everything, the settings for our new filter have a Catch allocation if … (empty) section with a + add button. Let's click on that.

This adds a criterion to our filter. Let's modify it so that the our filter catches everything of size greater than zero machine words, rename the filter, and save these changes.

The tab for our filter now shows (3) next to its name, indicating that this filter catches 3 allocations, which is all the allocations of the (tiny) dump.

Now, create a new filter and modify it so that it catches allocations made in file weak.ml. This requires

creating a filter,
adding a criterion to that filter,
switching it from size to callstack
removing the trailing ** (anything) by erasing it,
write weak.ml as the last file that should appear in the callstack.>

After saving it, you should get the following.

Sadly, this filter does not match anything, although some allocations fit this filter. This is because a custom filter F “catches" an allocation if

all of the criteria of F are true for this allocation, and
the allocation is not caught by any custom filter at the left of F (note that the everything filter is not a custom filter).

In other words, all allocations go through the list of custom filters from left to right, and are caught by the first filter such that all of its criteria are true for this allocation. As such, it is similar to switch/case and pattern matching.

Let's move our new filter to the left by clicking the left arrow next to it, and save the change.

Nice.

You can remove a filter by selecting it and clicking the - remove button in the top-right part of the footer, next to the + add filter button. This only works for custom filters, you cannot remove built-in filters.

Now, remove the first filter we created (size ≥ 0), which should give you this:

Out of nowhere, we get the second and last built-in filter: catch-all. When some allocations are not caught by any of your filters, they will end up in this filter. Catch-all is not visible when it does not catch any allocation, which is why it was (mostly) not visible until now. The filter we wrote previously where catching all the allocations.

In the switch/case analogy, catch-all is the else/default branch. In pattern matching, it would be a trailing wildcard _.

So, weak.ml only catches one of the three allocations: catch-all appears and indicates it matches the remaining two.

It is also possible to write filter criteria over allocations' callstacks. This is discussed in the Callstack Filters Section.

Filter Generation

When we launched this section's running example, we passed --filter_gen none to memthol. This is because, by default, memthol will run automatic filter generation which scans allocations and generates filters. The default (and currently only) one creates one filter per allocation-site file.

For more details, in particular filter generation customization, run memthol --filter_gen help.

If we relaunch the example without --filter_gen none

❯ memthol rsc/dumps/ctf/mini_ae.ctf
|===| Starting
| url: http://localhost:7878
| target: `rsc/dumps/ctf/mini_ae.ctf`
|===|

we get something like this (actual colors may vary):

Charts

This section uses the same running example as the last section.

❯ memthol rsc/dumps/ctf/mini_ae.ctf
|===| Starting
| url: http://localhost:7878
| target: `rsc/dumps/ctf/mini_ae.ctf`
|===|

Filter Toggling

The first way to interact with a chart is to (de)activate filters. Each chart has its own filter tabs allowing to toggle filters on/off.

From the initial settings

click on all filters but everything to toggle them off.

Let's create a new chart. The only kind of chart that can be constructed currently is total size over time, so click on create chart below our current, lone chart.

Deactivate everything in the second chart.

Nice. We now have the overall total size over time in the first chart, and the details for each filter in the second one.

Next, notice that both charts have, on the left of their title, a down (first chart) and up (second chart) arrow. This moves the charts up and down.

On the right of the title, we have a settings ... buttons which is discussed below. The next button collapses the chart. If we click on the collapse* button of the first chart, it collapses and the button turns into an expand button.

The last button in the chart header removes the chart.

Chart Settings

Clicking the settings ... button in the header of any chart display its settings. (Clicking on the button again hides them.)

Currently, these chart settings only allow to rename the chart and change its display mode.

Display Mode

In memthol, a chart can be displayed in one of three ways:

normal, the one we used so far,
stacked area, where the values of each filter are displayed on top of each other, and
stacked area percent, same as stacked area but values are displayed as percents of the total.

Here is the second chart from our example displayed as stacked area for instance:

Global Settings

This section uses the same running example as the last section.

❯ memthol rsc/dumps/ctf/mini_ae.ctf
|===| Starting
| url: http://localhost:7878
| target: `rsc/dumps/ctf/mini_ae.ctf`
|===|

There is currently only one global setting: the time window.

Time Window

The time window global setting controls the time interval displayed by all the charts.

In our example,

not much is happening before (roughly) 0.065 seconds. Let's have the time window start at that point:

Similar to filter edition, we can apply or cancel this change using the two buttons that appeared in the bottom-left corner of the header.

Saving these changes yields

Here is the same chart but with the time window upper-bound set at 0.074.

Callstack Filters

Callstack filters are filters operating over allocation properties that are sequences of strings (potentially with some other data). Currently, this means allocation callstacks, where the strings are file names with line/column information.

String Filters

A string filter can have three shapes: an actual string value, a regex, or a match anything / wildcard filter represented by the string "...". This wildcard filter is discussed in its own section below.

A string value is simply given as a value. To match precisely the string "file_name", one only needs to write file_name. So, a filter that matches precisely the list of strings [ "file_name_1", "file_name_2" ] will be written


string list	contains	`[ file_name_1 file_name_2 ]`

A regex on the other hand has to be written between #" and "#. If we want the same filter as above, but want to relax the first string description to be file_name_<i> where <i> is a single digit, we write the filter as


string list	contains	`[ #"file_name_[0-9]"# file_name_2 ]`

The Wildcard Filter

The wildcard filter, written ..., lazily (in general, see below) matches a repetition of any string-like element of the list. To break this definition down, let us separate two cases: the first one is when ... is not followed by another string-like filter, and second one is when it is followed by another filter.

In the first case, ... simply matches everything. Consider for instance the filter


string list	contain	`[ #"file_name_[0-9]"# ... ]`

This filter matches any list of strings that starts with a string accepted by the first regex filter. The following lists of strings are all accepted by the filter above.

[ file_name_0 ]
[ file_name_7 anything at all ]
[ file_name_3 file_name_7 ]

Now, there is one case when ... is not actually lazy: when the n string-filters after it are not .... In this case, all elements of the list but the n last ones will be skipped, leaving them for the n last string filters.

For this reason


string list	contain	`[ … #"file_name_[0-9]"# ]`

does work as expected. For example, on the string list

[ "some_file_name" "file_name_7" "another_file_name" "file_name_0" ]

a lazy behavior would not match. First, ... would match anything up to and excluding a string recognized by #"file_name_[0-9]"#. So ... would match some_file_name, but that's it since file_name_7 is a match for #"file_name_[0-9]"#. Hence the filter would reject this list of strings, because there should be nothing left after the match for #"file_name_[0-9]"#. But there are still another_file_name and file_name_0 left.

Instead, the filter works as expected. ... discards all elements but the last one file_name_0, which is accepted by #"file_name_[0-9]"#.

Callstack (Location) Filters

Allocation callstack information is a list of tuples containing:

the name of the file,
the line in the file,
a column range.

Currently, the range information is ignored. The line in the file is not, and one can specify a line constraint while writing a callstack filter. The normal syntax is

<string-filter>:<line-filter>

Now, a line filter has two basic shapes

_: anything,
<number>: an actual value.

It can also be a range:

[<basic-line-filter>, <basic-line-filter>]: a potentially open range.

Line Filter Examples


`_`	matches any line at all
`7`	matches line 7
`[50, 102]`	matches any line between `50` and `102`
`[50, _]`	matches any line greater than `50`
`[_, 102]`	matches any line less than `102`
`[_, _]`	same as `_` (matches any line)

Callstack Filter Examples

Whitespaces are inserted for readability but are not needed:


`src/main.ml : _`	matches any line of `src/main.ml`
`#".*/main.ml"# : 107`	matches line 107 of any `main.ml` file regardless of its path

Rehabilitating Packs using Functors and Recursivity, part 2.

2020-09-30T13:56:36Z

This blog post and the previous one about functor packs covers two RFCs currently developed by OCamlPro and Jane Street. We previously introduced functor packs, a new feature adding the possiblity to compile packs as functors, allowing the user to implement functors as multiple source files or even parameterized libraries.

In this blog post, we will cover the other aspect of the packs rehabilitation: allowing anyone to implement recursive compilation units using packs (as described formally in the RFC#20). Our previous post introduced briefly how packs were compiled and why we needed some bits of closure conversion to effectively implement big functors. Once again, to implement recursive packs we will need to encode modules through this technique, as such we advise the reader to check at least the introduction and the compilation part of functor packs.

Recursive modules through recursive packs

Recursive modules are a feature long available in the compiler, but restricted to modules, not compilation units. As such, it is impossible to write two files that depend on each other, except by using scripts that tie up these modules into a single compilation file. Due to the internal representation of recursive modules, it would be difficult to implement recursive (and mutually recursive) compilation units. However, we could use packs to implement these.

One common example of recursive modules are trees whose nodes are represented by sets. To implement such a data structure with the standard library we need recursive modules: Set is a functor that takes as parameter a module describing the values embedded in the set, but in our case the type needs the already applied functor.

module rec T : sig
  type t =
      Leaf of int
    | Node of TSet.t

  val compare : t -> t -> int
end = struct
  type t =
      Leaf of int
    | Node of TSet.t

  let compare t1 t2 =
    match t1, t2 with
      Leaf v1, Leaf v2 -> Int.compare v1 v2
    | Node s1, Node s2 -> TSet.compare s1 s2
    | Leaf _, Node _ -> -1
    | Node _, Leaf _ -> 1
end

and TSet : Set.S with type elt = T.t = Set.Make(T)

With recursive pack, we can simply put T and TSet into their respective files (t.ml and tSet.ml), and tie them into one module (let's name it P). Signature of recursive modules cannot be infered, as such we also need to define t.mli and tSet.mli. Both must be compiled simultaneously since they refer to each other. The result of the compilation is the following:

ocamlopt -c -for-pack P -recursive t.mli tSet.mli
ocamlopt -c -for-pack P -pack-is-recursive P t.ml
ocamlopt -c -for-pack P -pack-is-recursive P tSet.ml
ocamlopt -o p.cmx -recursive-pack t.cmx tSet.cmx

We have three new compilation options:

-recursive indicates to the compiler to typecheck all the given mlis simultaneously, as recursive modules.
-pack-is-recursive indicates which pack(s) in the hierarchy are meant to be recursive. This is necessary since it determines how the module must be compiled (i.e if we will need to apply closure conversion).
recursive-pack generates a pack that deals with the initialization of its modules, as for recursive modules.

Recursives modules compilation

One may be wondering why we need packs to compile recursive modules. Let's take a look at how they are encoded. We will craft a naive example that is simple enough once compiled:

module rec Even : sig
  val test: int -> bool
end = struct
  let test i =
    if i-1 <= 0 then false else Odd.test (i-1)
end

and Odd : sig
  val test: int -> bool
end = struct
  let test i =
    if i-1 <= 0 then true else Even.test (i-1)
end

It defines two modules Even and Odd, that both test whether an integer is even or odd, and if that is not the case calls the test function from the other module. Not a really interesting use of recursive modules obviously. The compilation schema for recursive modules is the following:

First, it allocates empty blocks for each module according to its shape (how many values are bound and what size they need in the block, if the module is a functor and what are its values, etc).
Then these blocks are filled with the implementation.

In our case, in a pseudo-code that is a bit higher order than Lambda (the first intermediate language of ocaml) it would translate as:

module Even = <allocation of the shape of even.cmx>
module Odd = <allocation of the shape of odd.cmx>

Even := <struct let test = .. end>
Odd := <struct let test = .. end>

This ensures that every reference to Even in Odd (and vice-versa) are valid pointers. To respect this schema, we will use packs to tie recursive modules together. Without packs, this means we would generate this code when linking the units into an executable which can be tricky. The pack can simply do it as initialization code.

Compiling modules for recursive pack

If we tried to compile these modules naively, we would end up in the same situation than for the functor pack: the compilation units would refer to identifiers that do not exist at the time they are generated. Moreover, the initialization part needs to know the shape of the compilation unit to be able to allocate precisely the block that will contain the recursive module. In order to implement recursive compilation units into packs, we extends the compilation units in two ways:

The shape of the unit is computed and stored in the cmo (or cmx).
As for functor pack, we apply closure conversion on the free variables that are modules from the same pack or from packs above in the hierarchy as long as they are recursive.

As an example, we will reuse our Even / Odd example above and split it into two units even.ml and odd.ml, and compile them into a recursive pack P. Both have the same shape: a module with a single value. Even refers to a free variable Odd, which is in the same recursive pack, and vice-versa. The result of the closure conversion is a function that will take the pointer resulting from the initialization. Since the module is also recursive itself, it takes its own pointer resulting from its initialization. The result will look as something like:

(* even.cmx *)
module Even_rec (Even: <even.mli><even.mli>)(Odd: <odd.mli><odd.mli>) = ..

(* odd.cmx *)
module Odd_rec (Odd: <odd.mli><odd.mli>)(Even: <even.mli><even.mli>) = ..

(* p.cmx *)
module Even = <allocation of the shape of even.cmx>
module Odd = <allocation of the shape of odd.cmx>

Even := Even_rec(Even)(Odd)
Odd := Odd_rec(Odd)(Even)

Rejunavating packs

Under the hood, these new features come with some refactoring in the pack implementation which follows work done for RFC on the representation of symbols in the middle-end of the compiler. Packs were not really used anymore and were deprecated by module aliases, this work makes them relevant again. These RFCs improve the OCaml ecosystem in multiple ways:

Compilation units are now on par with modules, since they can be functors.
Functor packs allow developers to implement parameterized libraries, without having to rely on scripts to produce multiple libraries linked with different backends (for example, Cohttp can use Lwt or Async as backend, and provides two libraries, one for each of these).
Recursive packs allow the implementation of recursive modules into separate files.

We hope that such improvements will benefit the users and library developers. Having a way to implement parameterize libraries without having to describe big functors by hand, or use mutually recursive compilation units without using scripts to generate a unique ml file will certainly introduce new workflows.

Rehabilitating Packs using Functors and Recursivity, part 1.

2020-09-24T13:56:36Z

OCamlPro has a long history of dedicated efforts to support the development of the OCaml compiler, through sponsorship or direct contributions from Flambda Team. An important one is the Flambda intermediate representation designed for optimizations, and in the future its next iteration Flambda 2. This work is funded by JaneStreet.

Packs in the OCaml ecosystem are kind of an outdated concept (options -pack and -for-pack in the OCaml manual), and their main utility has been overtaken by the introduction of module aliases in OCaml 4.02. What if we tried to redeem them and give them a new youth and utility by adding the possibility to generate functors or recursive packs?

This blog post covers the functor units and functor packs, while the next one will be centered around recursive packs. Both RFCs are currently developed by JaneStreet and OCamlPro. This idea was initially introduced by functor packs (Fabrice Le Fessant) and later generalized by functorized namespaces (Pierrick Couderc et al.).

Packs for the masses

First of all let's take a look at what packs are, and how they fixed some issues that arose when the ecosystem started to grow and the number of libraries got quite large.

One common problem in any programming language is how names are treated and disambiguated. For example, look at this small piece of code:

let x = "something"

let x = "something else"

We declare two variables x, but actually the first one is shadowed by the second, and is now unavailable for the rest of the program. It is perfectly valid in OCaml. Let's try to do the same thing with modules:

module M = struct end

module M = struct end

The compiler rejects it with the following error:

File "m.ml", line 3, characters 0-21:
3 | module M = struct end
    ^^^^^^^^^^^^^^^^^^^^^
Error: Multiple definition of the module name M.
       Names must be unique in a given structure or signature.

This also applies with programs linking two compilation units of the same name. Imagine you are using two libraries (here lib_a and lib_b), that both define a module named Misc.

ocamlopt -o prog.asm -I lib_a -I lib_b lib_a.cmxa lib_b.cmxa prog.ml 
File "prog.ml", line 1:
Error: The files lib_a/a.cmi and lib_b/b.cmi make inconsistent assumptions
over interface Misc

At link time, the compiler will reject your program since you are trying to link two modules with the same name but different implementations. The compiler is unable to differentiate the two compilation units since they define some identical symbols, as such cannot link the program. Enforcing unique module names in the same namespace (i.e. a signature) is consistent with the inability to link two modules of the same name in the same program.

However, Misc is a common name for a module in any project. How can we avoid that? As a user of the libraries there is nothing you can do, since you cannot rename the modules (you will eventually need to link two files named misc.cmx). As the developer, you need to ensure that your module names are unique enough to be used along any other libraries. One solution would be to use prefixes for each of your compilation units, for example by naming your files mylib_misc.ml, with the drawback that you will need to use those long module names inside your library. Another solution is packing your units.

A pack is simply a generated module that appends all your compilation units into one. For example, suppose you have two files a.ml and b.ml, you can generate a pack (i.e. a module) mylib.cmx that is equivalent to:

module A = struct <content of a.ml> end
module B = struct <content of b.ml> end

As such, A and B can retain their original module name, and be accessed from the outside as Mylib.A and Mylib.B. It uses the namespacing induced by the module system. A developer can simply generate a pack for its library, assuming its library name will be unique enough to be linked with other modules without the risk of name clashing. However it has one big downside: suppose you use a library with many modules but only use one. Without packs the compiler will only link the necessary compilation units from this library, but since the pack is one big compilation unit this means your program embeds the complete library.

This problem is fixed using module aliases and the compiler option -no-alias-deps since OCaml 4.02, and the result for the user is equivalent to a pack, making them more or less deprecated.

Functorizing packs, or how to parameterize a library

Packs being modules representing libraries, a useful feature would be to be able to produce libraries that take modules as parameters, just like functors. Another usage would be to split a huge functor into multiple files. In other words, we want our pack Mylib to be compiled as:

functor (P : sig .. end) -> struct 
  module A = struct <content of a.ml> end
  module B = struct <content of b.ml> end
end

while A and B would use the parameter P as a module, and Mylib instantiated later as

module Mylib = Mylib(Some_module_with_sig_compatible_with_P)

One can notice that our pack is indeed a functor, and not simply a module that binds a functor. To be able to do that, we also extends classical compilation units to be compiled as functors. Such functors are not expressed in the language, we do not provide a syntax for that, they are a matter of options at compile-time. For example:

ocamlopt -c -parameter P m.ml

will compile m.ml as a functor that has a parameter P whose interface is described in p.cmi in the compilation path. Similarly, our pack Mylib can be produced by the following compilation steps:

ocamlopt -c -parameter-of Mylib p.mli
ocamlopt -c -for-pack "Mylib(P)" a.ml
ocamlopt -c -for-pack "MyLib(P)" b.ml
ocamlopt -pack -o mylib.cmx -parameter P a.cmx b.cmx

In details:

The parameter is compiled with the flag -parameter-of Mylib, as such it won't be used as the interface of an implementation.
The two modules packed are compiled with the flag -for-pack "MyLib(P)". Expressing the parameter of the pack is mandatory since P must be known as a functor parameter (we will see why in the next section).
The pack is compiled with -parameter P, which will indeed produce a functorized compilation unit.

Functors are not limited to a unique parameter, as such they can be compiled with multiple -parameter options and multiple arguments in -for-pack. This implementation being on the build system side, it does not need to change the syntax of the language. We expect build tools like dune to provide supports for this feature, making it maybe more easier to use. Moreover, it makes compilation units on par with modules which can have a functor type. One downside however is that we cannot express type equalities between two parameters or with the functor body type as we would do with substitutions in module types.

Functor packs under the hood

In terms of implementation, packs should be seen as a concatenation of the compilation units then a rebinding of each of them in the newly created one. For example, a pack P of two units m.cmx and n.cmx is actually compiled as something like:

module P__M = <code of m.cmx> 
module P__N = <code of n.cmx> 
module M = P__M 
module N = P__N

According to this representation, if we tried to naively implement our previous functor pack Mylib(P) we would end up with a functor looking like this:

module Mylib__A = <code of a.cmx, with references to P>
module Mylib__B = <code of b.cmx, with references to P>

functor (P : <signature of p.cmi>) -> struct
  module A = Mylib__A
  module B = Mylib__B
end

Unfortunately, this encoding of functor packs is wrong: P is free in a.cmx and b.cmx and its identifier cannot correspond to the one generated for the functor retrospectively. The solution is actually quite simple and relies on a transformation known as closure conversion. In other words we will transform our modules into functors that takes as parameters their free variables, which in our case are the parameters of the functor pack and the dependencies from the same pack. Let's do it on a concrete functor equivalent to Mylib:

module Mylib' (P : P_SIG) = struct
  module A = struct .. <references to P> end
  module B = struct .. <references to P> <references to A> end
end

Our goal here is to move A and B outside of the functor, as such out of the scope of P, which is done by transforming those two modules into functors that takes a parameter P' with the same signature as P:

module A_funct (P' : P_SIG) = struct .. <references to P as P'> end 
module B_funct (P' : P_SIG) = struct 
  module A' = A_funct(P') 
  .. 
  <references to P as P'> 
  <references to A as A'> 
end 

module Mylib' (P : P_SIG) = struct 
  module A = A_funct(P) 
  module B = B_funct(P) 
end

While this code compiles it is not semantically equivalent. A_funct is instantiated twice, its side effects are computed twice: the first time when instantiating A in the functor, and the second when instantiating B. The solution is simply to go further with closure conversion and make the result of applying A_funct to P an argument of B_funct.

module A_funct (P' : P_SIG) = struct .. <references to P as P'> end
module B_funct (P' : P_SIG)(A': module type of A_funct(P'))= struct
  ..
  <references to P as P'>
  <references to A as A'>
end

module Mylib' (P : P_SIG) = struct
  module A = A_funct(P)
  module B = B_funct(P)(A)
end

This represents exactly how our functor pack Mylib is encoded. Since we need to compile modules in a specific way if they belong to a functor pack, the compiler has to know in the argument -for-pack that the pack is a functor, and what are its parameters.

Functor packs applied to `ocamlopt`

What we described is a functional prototype of functor packs, implemented on OCaml 4.10, as described in RFC#11. In practice, we already have one usage that we could benefit of in the future: cross-compilation of native code. At the moment the compiler is configured to target the architecture which it is compiled on. The modules relative to the current architecture are linked symbolically into the backend folder and the backend is compiled as if it only supported one architecture. One downside of this approach is that changes into the interface of the backend that need some modifications in each architecture are not detected at compile time, but only for the current one. You need to reconfigure the OCaml compiler and rebuild it to check if another architecture still compiles. One interesting property is that each architecture backend defines the same set of modules with compatible interfaces. In other words, these modules could simply be parameters of a functor, that is instantiated for a given architecture.

Following this idea, we implemented a prototype of native compiler whose backend is indeed packed into a functor, and instantiated at the initialization of the compiler. With this approach, we can easily switch the targeted architecture, and moreover we can be sure that each architecture is compiled, leveraging the fact that some necessary refactoring is always done when changes happen in the backend interface. Implementing such a functor is mainly a matter of adapting the build system to produce a functor pack, writing few signatures for the functor and its parameters, and instantiating the backend at the right time.

This proof of concept shows how functor packs can ease some complicated build system and allows new workflow.

Making packs useful again

Packs were an old concept mainly outdated by module aliases. They were not practical as they are some sort of monolithic libraries shaped into a unique module containing sub modules. While they perfectly use the module system for its namespacing properties, their usage enforces the compiler to link an entire library even if only one module is actually used. This improvement allows programmers to define big functors, functors that are split among multiple files, resulting in what we can view as a way to implement some form of parameterized libraries.

In the second part, we will cover another aspect of the rehabilitation of packs: using packs to implement mutually recursive compilation units.

Comments

François Bobot (25 September 2020 at 9 h 16 min):

I believe there is a typo

module Mylib’ (P : P_SIG) = struct
module A = A_funct(P)
module B = A_funct(P)
end

The last must be B_funct(P), the next example as also the same typo.

Pierrick Couderc (25 September 2020 at 10 h 31 min):

Indeed, thank you!

Cyrus Omar (8 February 2021 at 3 h 49 min):

This looks very useful! Any updates on this work? I’d like to be able to use it from dune.

A Dune Love story: From Liquidity to Love

2020-06-09T13:56:36Z

By OCamlPro & Origin Labs

Writing smart contacts may often be a burdensome task, as you need to learn a new language for each blockchain you target. In the Dune Network team, we are willing to provide as many possibilities as possible for developers to thrive in an accessible and secure framework.

There are two kinds of languages on a blockchain: “native” languages that are directly understood by the blockchain, but with some difficulty by the developers, and “compiled” languages that are more transparent to developers, but need to be translated to a native language to run on the blockchain. For example, Solidity is a developer-friendly language, compiled to the native EVM language on the Ethereum blockchain.

Dune Network supports multiple native languages:

Love, a type-safe language with a ML syntax and suited for formal verification
Michelson, inherited from Tezos, also type-safe, much more difficult to read
Solidity, the Ethereum language, of which we are currently implementing the interpreter after releasing its parser in OCaml a few weeks ago

On the side of compiled languages, Dune Network supports:

Liquidity, a type-safe ML language suited for formal verification, that compiles to Michelson (and allows developers to decompile Michelson for auditing)
ReasonML, a JavaScript language designed by Facebook that compiles down to Michelson through Liquidity
All other Tezos languages that compile to Michelson (for example Ligo, SmartPy, Albert...)

Though Liquidity and Love are both part of the ML family, Liquidity is much more developer-friendly: types are inferred, whereas in Love they have to be explicit, and Liquidity supports the ReasonML JavaScript syntax while Love is bound to its ML syntax.

For all these reasons, we are pleased to announce a wedding: Liquidity now supports the Love language!

Liquidity now supports generating Love smart contracts

This is great news for Love, as Liquidity is easier to use, and comes with an online web editor, Try-Liquidity. Liquidity is also being targeted by the ConCert project, aiming at verifying smart contracts with the formal verification framework Coq.

The Smart Contract Framework on the Dune Network

Compiling contracts from Liquidity to Love has several benefits compared to Michelson. First, Love contracts are about 60% smaller than Michelson contracts, hence they are 60% cheaper to deploy. Also, the compiler outputs a Love contract that can be easily read and audited.

The Love compiler is part of the Liquidity project. It works as follows:

The Liquidity contract is type-checked by the Liquidity compiler. The strong type system of liquidity enforces structural & semantic properties on data.
The typed Liquidity contract is compiled to a typed Love contract. During this step, the Liquidity contract is scanned to check if it complies with the Love requirements (correct use of operators, no reentrancy, etc.).
The Love contract is type-checked. Once this step is completed, the contract is ready to be deployed on the chain!

Want to try it out? Check the Try-Liquidity website: you can now compile and deploy your Liquidity contracts in Love from the online editor directly to the Mainnet and Testnet using Dune Metal!

These are some of the resources you might find interesting when building your own smart contracts:

The Love Language Documentation: https://dune.network/docs/dune-dev-docs/love-doc/introduction.html
Try-Liquidity: https://www.liquidity-lang.org/edit/
The Liquidity Website: https://www.liquidity-lang.org/
The Dune Network Website: https://dune.network

About Origin Labs

Origin Labs is a company founded in 2019 by the former blockchain team at OCamlPro. At Origin Labs, they have been developing Dune Network, a fork of the Tezos blockchain, its ecosystem, and applications over the Dune Network platform. At OCamlPro, they developed TzScan, the most popular block explorer at the time, Liquidity, a smart contract language, and were involved in the development of the core protocol and node. Feel free to reach out by email: contact@origin-labs.com.

[Interview] Sylvain Conchon joins OCamlPro

2020-06-06T13:56:36Z

On April 2020, Sylvain Conchon joined the OCamlPro team as our Chief Scientific Officer on Formal Methods. Sylvain is a professor at University Paris-Saclay, he has also been teaching OCaml in universities for about 20 years. He is the co-author of Apprendre à programmer avec OCaml with Jean-Christophe Filliâtre, a book for students in French elitist Preparatory Schools. His field of expertise is the automated deduction for program verification and model checking of parameterized systems. He is also the co-creator of Alt-Ergo, our SMT Solver dedicated to program verification, used by Airbus and qualified for the [DO-178C](http://(https://en.wikipedia.org/wiki/DO-178C) avionic standard, of Cubicle and the very useful OCamlgraph library.

Research and Industry

Sylvain, you’ve been involved in the industrial world for a long time, what do you think about the interactions between industry and research labs?

I’ve always found interactions with industry professionals to be very rewarding. During my studies, I worked for several years in IT (SSII), and as a university professor, I have supervised students during their internships or apprenticeships in tech companies or at large industrial companies every year. I also take part in research projects that involve industrial partners, and I spent some time at Intel in Portland, which allowed me to discover the computer hardware industry from inside.

How do you establish a fruitful collaboration between academia and industry?

It’s primarily a question of mutual understanding. You can see it clearly during collaborative research projects that involve both academics and industrial partners. Tools resulting from research, no matter what they are, have to be relevant to real industrial problems. Once that’s taken care of, the software also needs to be usable by industry professionals without them needing to understand its inner workings (for instance they shouldn’t have to specify all 50 necessary options for its use, interpret its results, or its absence of results!).

This requires a significant engineering effort geared towards the end user; and this task is not part of usual research activity. So, we first need to really understand the problems and needs of the industrial partner, and then determine whether our technologies and tools can be adapted or used to prototype a relevant solution.

You’ve just joined OCamlPro, what are your first thoughts?

I am very happy to be joining such a dynamic company full of talented, motivated, friendly people, where they do both high-level engineering and top-quality research! Several of my former PhD students are also working at OCamlPro, such as Albin Coquereau, David Declerck and Mattias Roux. With Mohamed Iguernlala and Alain Mebsout at our partner Origin Labs, and with the other OCP team members, it makes our team rock-solid in formal methods tooling development.

“Tools resulting from research, no matter what they are, have to satisfy real industry needs.”

OCaml, a Cutting-Edge Language

You are well known in the OCaml community, and some of your students became fans of OCaml (and of your teaching)… What do you say to your students who are just discovering OCaml?

I tend to summarize it with one phrase: “With OCaml, you’re not learning the computer programming of the last 10 years, you’re learning the programming of the 10 coming years”. This has proven true numerous times, because a good number of OCaml’s features were to be found in mainstream languages years later. That being said, all my years of teaching this language have led me to think that some modifications to its syntax would make the language easier to tackle for some beginners.

How did you personally discover OCaml?

During my master’s thesis (maîtrise) at university: one of my teachers pointed this language to me; they believed it would help me write a compiler for another programming language. So, I discovered OCaml by myself, by reading the manual and going through examples. It wasn’t until my MASt (DEA) that I discovered the theoretical foundations of this fantastic language (semantics, typing, compilation).

Would you say OCaml is an industrial programming language?

The question needs to be clarified: what is an industrial programming language? If by industrial language you mean one that is used by industry professionals, then I’d say that OCaml needs to be used more widely to be classified as such. If the question is whether OCaml is at the same level as languages used in industry, then it absolutely is. But maybe the question is more about the OCaml ecosystem and how developed the available tooling is: certain improvements undoubtedly need to be made in order to reach the level of a widespread industrial programming language. But we’re on the right track, especially thanks to companies like OCamlPro and its projects like Opam and Try-OCaml for example.

Formal Methods as an Industrial Technique, and the Example of the Alt-Ergo Solver

Formal methods being one of OCamlPro’s areas of expertise, in what way do you think OCaml is suited for the SMT domain?

Tools like SMT solvers are mainly symbolic data manipulation software that allow you to analyze, transform, and reason about logical formulas. OCaml is made for that. There is also a more “computational” side to these tools, which requires precise programming of data structures as well as efficient memory management. OCaml, with its extremely efficient garbage collector (GC), is particularly suited for this kind of development. SMT solvers are tools that also need to be very reliable because errors are difficult to find and are potentially very harmful. OCaml’s type system contributes to the reliability of these tools.

“SMT solvers are nowadays essential in software engineering”

Can you describe Alt-Ergo in a few words?

Alt-Ergo is a software for proving logical formulas automatically (without human intervention), meaning proving whether a formula is true or false. Alt-Ergo belongs to a family of automated provers called SMT (Satisfiability Modulo Theories). It was designed to be integrated into program verification platforms. These platforms (like [Why3](https://(https://why3.lri.fr/), Frama-C, Spark…) generate logical formulas that need to be proven in order to guarantee that a program is safe. Proving these formulas by hand would be very tedious (there are sometimes tens of thousands of formulas to prove). An SMT solver such as Alt-Ergo is there to do that job in a completely automated way. It is what allows these verification platforms to be used at an industrial level.

In what way developing this software in OCaml benefits Alt-Ergo over its competitors?

It makes it more reliable, since an SMT solver, like any program, can have bugs. Most of Alt-Ergo is written in a purely functional programming style, i.e. only using immutable data structures. One of the advantages of this programming style is that it allowed us to formally prove the main components of Alt-Ergo (for example, its kernel was formalized using the Coq proof assistant, which would have been impossible with a language like C++) without sacrificing efficiency thanks to a very good garbage collector and OCaml’s very powerful persistent data structure library. We made use of OCaml’s module system, particularly functors and recursive modules, to conceive a very modular code, making it maintainable and easily extensible. OCaml allowed us to create an SMT solver just as efficient as CVC4 or Z3 for program verification, but with a total number of lines of code divided by three or four.This obviously does not guarantee that Alt-Ergo has zero bugs, but it really helps us in fixing any if they are found.

What is your opinion on SMT solvers and the current state of the art of SMT?

Today, SMT solvers are essential in software engineering. They can be found in various tools for proving, testing, model checking, abstract interpretation, and typing. The main reason for this success is that they are becoming increasingly efficient and the underlying theories are becoming more and more expressive. It is a very competitive area of research among the world’s best universities and research labs, as well as large IT companies. But there is still a lot of room for improvement, particularly in the nonlinear arithmetic domain, where user demand is growing. For now, one of my research objectives is to combine Model Checking tools with program verification. These two types of tools are based on SMT and should complement each other to offer even more automation to verification tools.

What applications can SMT techniques and Alt-Ergo have in industry?

SMT techniques can be used wherever formal methods are useful. Including, but not limited to verifying the safety of critical software in embedded systems, finding security vulnerabilities in computer systems, or resolving planning problems. They can also be found in domains of artificial intelligence, where it is crucial to guarantee neural network stability and produce formal explanations of their results.

You ended up working on Model Checking, can you tell us about how Model Checking is connected to SMT and how it is currently used?

Model Checking consists of verifying that all possible states of a system respect certain properties, regardless of the input data. This is a difficult problem because some systems (like microprocessors for example) can have hundreds of millions of states. To reach that scale, model checkers implement extremely sophisticated algorithms to visit these states quickly by storing them in a compact manner. That said, this technique reaches its limits when the input values are unbounded or when the number of system components is unknown. Imagine Internet routing algorithms where you don’t know how many machines are connected. These algorithms must be correct no matter the number of machines. This is where SMT solvers come into play. By using logical formulas, we’re able to represent sets of states of arbitrary sizes. Visiting system states becomes calculating the formulas that represent the states satisfying the desired properties, etc. Therefore, everything in Model Checking is based on logical formulas, and SMT solvers are of course there to reason about these formulas.

[Interview] Sylvain Conchon rejoint OCamlPro

2020-06-05T13:56:36Z

Sylvain Conchon vient de rejoindre OCamlPro en tant que Chief Scientific Officer Méthodes Formelles. Professeur à l’Université Paris-Saclay, il travaille dans le domaine de la démonstration automatique pour la preuve de programmes et le model checking pour systèmes paramétrés. Il est aussi le co-créateur d’Alt-Ergo.

Recherche et industrie

Sylvain, tu fréquentes de longue date le monde industriel, que penses-tu des interactions entre les industriels et les laboratoires de recherche ?

J’ai toujours trouvé très enrichissantes les interactions avec les industriels. Pendant mes études, j’ai travaillé plusieurs années en SSII, et je suis mes étudiants en stage ou en apprentissage dans des sociétés technologiques ou chez de grands industriels. Je participe également à des projets de recherche qui impliquent des industriels,et j’ai passé quelques temps chez Intel à Portland, ce qui m’a permis de découvrir l’industrie du hardware.

Comment parvenir à établir des relations fructueuses entre le monde académique et les industriels ?

C’est beaucoup une histoire de rencontre. On le voit lors des montages de projets de recherche collaboratifs qui réunissent académiques et industriels. Les outils issus de la recherche, quels qu’ils soient, doivent avant tout répondre à un besoin réel des industriels. Si c’est le cas, il faut aussi que le logiciel soit utilisable par des ingénieurs du métier sans qu’il leur soit nécessaire de comprendre son fonctionnement interne (par exemple, pour positionner les 50 options nécessaires à son utilisation, interpréter ses résultats ou ses absences de résultats!). Cela nécessite à l’évidence un travail d’ingénierie important, tourné vers l’utilisateur final et souvent éloigné des activités des chercheurs. Il faut donc comprendre les problèmes et les besoins des industriels, et ensuite déterminer si les technologies et les outils que l’on maîtrise peuvent être adaptés ou utilisés pour réaliser un prototype qui réponde à certains de ces besoins.

Tu viens de rejoindre OCamlPro, quelles sont tes premières impressions ?

Je suis heureux d’avoir rejoint une entreprise très dynamique, pleine de gens talentueux, motivés et sympathiques, où l’on fait à la fois de l’ingénierie de haut niveau et de la recherche de qualité !

“ Les outils issus de la recherche, quels qu’ils soient, doivent avant tout répondre à un besoin réel des industriels.”

OCaml, un langage de pointe

Tu es connu dans la communauté OCaml, et certains de tes étudiants sont devenus des fans d’OCaml (et de ton enseignement)… que dis-tu à tes étudiants qui découvrent OCaml ?

J’ai tendance à résumer en disant ceci : « avec OCaml, vous n’apprenez pas la programmation des 10 dernières années, mais celle des 10 prochaines années ». Cette affirmation s’est toujours vérifiée car bon nombre de traits du langage OCaml se sont retrouvés dans les langages mainstream, avec plusieurs années de décalage. Cela dit, mes années d’expérience dans l’enseignement de ce langage me laissent penser que quelques modifications dans sa syntaxe permettraient une approche plus aisée pour certains débutants.

Et toi, comment as-tu découvert OCaml ?

Pendant mes études à l’Université lors de mon projet de fin de maîtrise : un de mes enseignants m’avait orienté vers ce langage pour m’aider à réaliser un compilateur pour un langage de programmation concurrente. J’ai donc découvert ce langage par moi-même, en lisant le manuel et les exemples. Ce n’est que pendant mon DEA que j’ai découvert les fondements théoriques de ce beau langage (sémantique, typage, compilation).

OCaml, un langage industriel ou pas encore ?

Il convient de préciser la question : qu’est-ce qu’un langage industriel ? Si c’est un langage utilisé par les industriels, alors OCaml n’est hélas pas encore suffisamment utilisé dans l’industrie pour être qualifié ainsi. Si la question est de savoir s’il a le niveau des langages utilisés dans l’industrie, alors la réponse est oui, sans hésiter. Mais peut-être la question porte-t-elle davantage sur l’écosystème OCaml et la maturité de l’outillage: il y a sûrement des progrès à faire pour atteindre le niveau d’un langage très répandu dans l’industrie, mais c’est en bonne voie, en particulier grâce à des entreprises telles qu’OCamlPro.

Les méthodes formelles comme technique industrielle, et l’exemple du solveur Alt-Ergo

Les méthodes formelles sont l’un des domaines d’expertise d’OCamlPro, en quoi penses-tu qu’OCaml est adapté au domaine des SMT ?

Les outils comme les solveurs SMT sont principalement des logiciels de manipulation symbolique des données qui permettent d’analyser, de transformer et de raisonner sur des formules logiques. OCaml est fait pour ce genre de traitements. Il y a aussi une partie plus « calculatoire » dans ces outils qui nécessite une programmation fine des structures de données ainsi qu’une gestion efficace de la mémoire. OCaml est particulièrement adapté pour ce genre de développements, surtout avec son ramasse-miettes (GC) extrêmement performant. Enfin, les solveurs SMT sont des outils qui doivent avoir un grand niveau de fiabilité car les erreurs dans ces logiciels sont difficiles à trouver et leur présence peut être très préjudiciable. Le système de types d’OCaml contribue à la fiabilité de ces outils.

“Les solveurs SMT sont aujourd’hui incontournables dans le domaine de l’ingénierie du logiciel.”

Peux-tu nous parler d’Alt-Ergo en quelques mots ?

C’est un logiciel utilisé pour prouver automatiquement (sans intervention humaine) des formules logiques, c’est-à-dire savoir si ces formules sont vraies ou fausses. Alt-Ergo appartient à une famille de démonstrateurs automatiques appelée SMT (pour Satisfiabilité Modulo Théories). Il a été conçu pour être intégré dans des plate-formes de vérification de programmes. Ces outils (comme Why3, Frama-C, Spark,…) génèrent des formules logiques qu’il est nécessaire de prouver afin de garantir qu’un programme est sûr. Faire la preuve de ces formules à la main serait très fastidieux (il y a parfois plusieurs dizaines de milliers de formules à prouver). Un solveur SMT comme Alt-Ergo est là pour faire ce travail, de manière complètement automatique. C’est ce qui permet à ces plateformes de vérification d’être utilisables au niveau industriel.

En quoi le développement d’Alt-Ergo en OCaml peut-il être un avantage par rapport aux concurrents ?

Cela lui confère une plus grande sûreté, car un solveur SMT, comme n’importe quel programme peut aussi avoir des bugs. La plus grande partie d’Alt-Ergo est programmée dans un style purement fonctionnel, c’est-à-dire uniquement avec l’utilisation de structures de données immuables. L’un des avantages de ce style de programmation est qu’il nous a permis de prouver formellement ses principaux composants (par exemple, son noyau a été formalisé à l’aide de l’assistant à la preuve Coq, ce qui serait impossible à faire dans un langage comme C++), sans sacrifier son efficacité grâce au très bon ramasse-miettes et à la bibliothèque de structures de données persistantes très performantes d’OCaml. Enfin, nous avons largement bénéficié du système de modules d’OCaml, en particulier les foncteurs et les modules récursifs, pour concevoir un code très modulaire, maintenable et facilement extensible. Au final, OCaml nous a permis de concevoir un solveur SMT aussi performant que CVC4 ou Z3 pour la preuve de programmes, mais avec un nombre de lignes de code divisé par trois ou quatre. Bien sûr, cela ne garantit pas que Alt-Ergo ait zéro bugs, mais cela nous aide beaucoup à mettre le doigt dessus quand quelqu’un en trouve.

“OCaml nous a permis de concevoir un solveur SMT aussi performant que CVC4 ou Z3 pour la preuve de programmes, mais avec un nombre de lignes de code divisé par trois ou quatre.“

Quel est ton avis sur les solveurs SMT et l’état de l’art SMT actuel ?

Les solveurs SMT sont aujourd’hui incontournables dans le domaine de l’ingénierie du logiciel. On les trouve aussi bien dans des outils de preuve, de test, de model checking, d’interprétation abstraite ou encore de typage. La principale raison de ce succès est qu’ils sont de plus en plus efficaces et les théories sous-jacentes sont très expressives. C’est un domaine de recherche très concurrentiel entre les meilleures universités ou laboratoires du monde et de grandes entreprises en informatique. Mais la marge de progression de ces outils est encore très grande, en particulier dans le domaine de l’arithmétique non linéaire où la demande des utilisateurs est de plus en plus forte. Pour le moment, un de mes objectifs en recherche est de combiner les outils de Model Checking avec ceux de preuve de programmes. Ces deux familles d’outils reposent sur les SMT et elles devraient se compléter pour offrir des outils de vérification encore plus automatiques.

Quelles applications les techniques SMT et Alt-Ergo peuvent-elles avoir dans l’industrie ?

Les techniques SMT peuvent être utilisées partout où les méthodes formelles peuvent être utiles. Par exemple (mais cette liste est loin d’être exhaustive), pour vérifier la sûreté de logiciels critiques dans le domaine de l’embarqué, pour trouver des failles de sécurité dans les systèmes informatiques ou pour résoudre des problèmes de planification. On les trouve également dans le domaine de l’intelligence artificielle où il est crucial de garantir la stabilité des réseaux de neurones mais aussi de produire des explications formelles sur leurs résultats.

Tu as été amené à travailler sur le Model Checking, peux-tu nous parler des liens entre Model Checking et SMT et de son utilisation actuelle ?

Le Model Checking consiste à vérifier que tous les états possibles d’un système respectent bien certaines propriétés, et ce quelles que soient les données en entrée. C’est un problème difficile car certains systèmes (microprocesseurs par ex.) peuvent avoir des centaines de millions d’états. Pour passer à l’échelle, les model checkers implémentent des algorithmes très perfectionnés pour visiter ces états rapidement, en les stockant d’une manière très compacte. Cependant, cette technique atteint ses limites quand les valeurs prises en entrée sont non bornées ou quand le nombre de composants du système n’est pas connu. Pensez aux algorithmes de routage d’Internet où on ne connaît pas le nombre de machines sur le réseau, ces algorithmes doivent être corrects, quel que soit ce nombre de machines. C’est là que les solveurs SMT entrent en jeu. En utilisant des formules logiques, on peut représenter des ensembles d’états de taille arbitraire. Visiter les états d’un système consiste alors à calculer les formules qui représentent ces états. Vérifier que les états respectent une propriété revient à prouver que les formules qui représentent des états impliquent la propriété voulue, etc. Tout dans le Model Checking repose donc sur des formules logiques et les solveurs SMT sont évidemment là pour raisonner sur ces formules.

Tutoriel Format

2020-06-01T13:56:36Z

Article écrit par Mattias.

Le module Format d’OCaml est un module extrêmement puissant mais malheureusement très mal utilisé. Il combine notamment deux éléments distincts :

les boîtes d’impression élégante
les tags sémantiques

Le présent article vise à démystifier une grande partie de ce module afin de découvrir l’ensemble des choses qu’il est possible de faire avec.

Si tout va bien vous devriez passer de

(En réalité nous arriverons à un résultat légèrement différent car l’auteur de ce tutoriel n’aime pas tous les choix faits pour afficher les messages d’erreur en OCaml mais les différences n’auront pas de grande importance)

I. Introduction générale : `fprintf fmt "%a" pp_error e`

Si vous ne comprenez pas ce que le code dans le titre doit faire, je vous invite à lire attentivement ce qui va suivre. Sinon vous pouvez directement sauter à la deuxième partie.

I.1. Rappels sur `printf`

Pour rappel, la fonction printf est une fonction variadique (c’est-à-dire qu’elle peut prendre un nombre variable de paramètres).

Le premier paramètre est une chaîne de formattage composée de caractères et de spécificateurs de format.
- Les caractères sont affichés tels quels. printf "abc" affichera abc.
- Les spécificateurs de caractère sont des caractères précédés du caractère % (syntaxe héritée du C). Ils sont remplacés à l’exécution par un des paramètres fournis après la chaîne de formattage à la fonction et servent à indiquer de quel type doit être la valeur qui sera affichée (ainsi que d’autres informations dont les détails peuvent être trouvés dans la documentation du module Printf. printf "Test: %d" attend un entier signé et affichera Test: <d> avec <d> remplacé par l’entier fourni.
Les paramètres suivants sont les valeurs fournies à printf pour remplacer les spécificateurs de format
- printf "%d %s %c" 3 s 'a' affichera l’entier signé 3, une espace insécable, le contenu de la variable s qui doit être une chaîne de caractères, une autre espace insécable et finalement le caractère ‘a’.
- On remarque aussi qu’ici le nombre de paramètres supplémentaires fournis en plus de la chaîne de formattage correspond au nombre de spécificateurs et que ceux-ci ne peuvent être intervertis. printf "%d %c" 'a' 3 ne pourra pas être compilé/exécuté car %d attend un entier signé et le premier paramètre est un caractère. Les spécificateurs qui n’attendent qu’un argument sont des spécificateurs que j’appelle unaires et sont extrêmement faciles à utiliser, il faut seulement savoir quel caractère correspond à quel type et les donner dans le bon ordre comme illustré dans la figure ci-dessous (le chevron représentant la sortie standard)

I.2. Afficher un type défini par l’utilisateur

Arrive alors ce moment où vous commencez à définir vos propres structures de données et, malheureusement, il n’y a aucun moyen d’afficher votre expression avec les spécificateurs par défaut (ce qui semble normal). Définissons donc notre propre type et affichons-le avec les techniques déjà vues :

type error =
  | Type_Error of string * string
  | Apply_Non_Function of string

let pp_error = function
  | Type_Error (s1, s2) -> printf "Type is %s instead of %s" s1 s2
  | Apply_Non_Function s -> printf "Type is %s, this is not a function" s

Supposons maintenant que nous ayons une liste d’erreurs et que nous souhaitions les afficher en les séparant par une ligne horizontale. Une première solution serait la suivante :

let pp_list l =
  List.iter (fun e ->
      pp_error e;
      printf "\n"
    ) l

Cette façon de faire a plusieurs inconvénients (qui vont être magiquement réglés par la fonction du titre).

I.3. Afficher sur un `formatter` abstrait

Le premier inconvénient est que printf envoie son résultat vers la sortie standard alors qu’on peut vouloir l’envoyer vers un fichier ou vers la sortie d’erreur, par exemple.

La solution est fprintf (il serait de bon ton de feindre la surprise ici).

fprintf prend un paramètre supplémentaire avant la chaîne de formattage appelé formatter abstrait. Ce paramètre est du type formatter et représente un imprimeur élégant (ou pretty-printer)

c’est-à-dire l’objet vers lequel le résultat devra être envoyé. L’énorme avantage qui en découle est qu’on peut transformer beaucoup de choses en formatter. Un fichier, un buffer, la sortie standard etc. À vrai dire, printf est implémenté comme let printf = fprintf std_formatter

Pour l’utiliser on va donc modifier pp_error et lui donner un paramètre supplémentaire :

let pp_error fmt = function
  | Type_Error (s1, s2) -> fprintf fmt "Type is %s instead of %s" s1 s2
  | Apply_Non_Function s -> fprintf fmt "Type is %s, this is not a function" s

Puis on réécrit pp_list pour prendre cela en compte :

let pp_list fmt l =
  List.iter (fun e ->
      pp_error fmt e;
      fprintf fmt "\n"
    ) l

Comme on peut le voir dans la figure ci-dessous, fprintf imprime dans le formatter qui lui est fourni en paramètre et non plus sur la sortie standard.

Si on veut maintenant afficher le résultat sur la sortie standard il suffira simplement de donner pp_list std_formatter comme formatter à fprintf. Cette façon de faire n’a, en réalité, que des avantages, puisqu’elle permet d’être beaucoup plus fexible quant au formatter qui sera utilisé à l’exécution du programme.

I.4. Afficher des types complexes avec `%a`

Le deuxième problème arrivera bien assez vite si nous continuons avec cette méthode. Pour bien le comprendre, reprenons pp_error. Dans le cas de Type_error of string * string on veut écrire Type is s1 instead of s2 et on fournit donc à fprintf la chaîne de formattage "Type is %s instead of %s" avec s1 et s2 en paramètres supplémentaires. Comment devrions-nous faire si s1 et s2 étaient des types définis par l’utilisateur avec chacun leur fonction d’affichage pp_s1`` : formatter -> s1 -> unit et pp_s2 : formatter -> s2 -> unit ? En suivant la logique de notre solution jusqu’ici, nous écririons le code suivant :

let pp_error fmt = function
  | Type_Error (s1, s2) ->
    fprintf fmt "Type is ";
    pp_s1 fmt s1;
    fprintf fmt "instead of ";
    pp_s2 fmt s2
  | Apply_non_function s -> fprintf fmt "Type is %s, this is not a function" s

Il est assez facile de se rendre compte rapidement que plus nous devrons manipuler des types complexes, plus cette syntaxe s’alourdira. Tout cela parce que les spécificateurs de caractère unaires ne permettent de manipuler que les types de base d’OCaml.

C’est là qu’entre en jeu %a. Ce spécificateur de caractère est, lui, binaire (ternaire en réalité mais un de ses paramètres est déjà fourni). Ses paramètres sont :

Une fonction d’affichage de type formatter -> 'a -> unit (premier paramètre devant être fourni)
Le formatter dans lequel il doit afficher son résultat (qui ne doit pas être fourni en plus)
La valeur qu’on souhaite afficher

Il appliquera ensuite le formatter et la valeur à la fonction fournie comme premier argument et lui donner la main pour qu’elle affiche ce qu’elle doit dans le formatter qui lui a été fourni en paramètre. Lorsqu’elle aura terminé, l’impression continuera. L’exemple suivant montre le fonctionnement (avec une impression sur la sortie standard, fmt ayant été remplacé par std-formatter

Dans notre cas nous avions déjà transformé nos fonctions d’affichage pour qu’elles prennent un formatter abstrait et nous n’avons donc presque rien à modifier :

let pp_error fmt = function
  | Type_Error (s1, s2) -> fprintf fmt "Type is %s instead of %s" s1 s2
  | Apply_Non_Function s -> fprintf fmt "Type is %s, this is not a function" s

let pp_list fmt l =
  List.iter (fun e ->
      fprintf fmt "%a\n" pp_error e;
    ) l

Et, bien sûr, si s1 et s2 avaient eu leurs propres fonctions d’affichage :

let pp_error fmt = function
  | Type_Error (s1, s2) -> fprintf fmt "Type is %a instead of %a" pp_s1 s1 pp_s2 s2
  | Apply_Non_Function s -> fprintf fmt "Type is %s, this is not a function" s

Arrivé-e-s ici vous devriez être à l’aise avec les notions de formatter abstrait et de spécificateur de caractère binaire et vous devriez donc pouvoir afficher n’importe quelle structure de donnée, même récursive, sans aucun soucis. Je recommande vivement cette façon de faire afin que tout changement qui devrait succéder ne nécessite pas de changer l’intégralité du code.

II. Les boîtes d’impression élégante

Et pour justement avoir des changements qui ne nécessitent pas de tout modifier, il va falloir s’intéresser un minimum aux boîtes d’impression élégante.

Aussi appelées pretty-print boxes, je les appellerai “boîtes” dorénavant, un tutoriel existe déjà, fait par l’équipe de la bibliothèque standard.

L'idée derrière les boîtes est tout simple :

À mon niveau je m’occupe correctement de comment afficher mes éléments et je n’impose rien au-dessus.

Reprenons, par exemple, la fonction permettant d’afficher les error:

let pp_error fmt = function
  | Type_Error (s1, s2) -> fprintf fmt "Type is %s instead of %s" s1 s2
  | Apply_Non_Function s -> fprintf fmt "Type is %s, this is not a function" s

Si on ajoutait un retour à la ligne on imposerait à toute fonction nous appelant ce saut de ligne or ce n’est pas à nous d’en décider. Cette fonction, en l’état, fait parfaitement ce qu’elle doit faire.

Regardons, par contre, la fonction affichant une liste d’erreur :

let pp_list fmt l =
  List.iter (fun e ->
      fprintf fmt "%a\n" pp_error e;
    ) l

A l’issue de celle-ci un saut à ligne provenant du dernier élément est forcé. Non seulement il n’est pas recommandé d’utiliser n (ou @n ou même @.) car ce ne sont pas à proprement parler des directives de Format mais des directives systèmes qui vont donc chambouler le reste de l’impression.

Malheureusement bien trop de développeurs et développeuses ont découvert @. en même temps que Format et s’en servent sans restriction. Au risque de me répéter souvent : n’utilisez pas @. !

II.1. Le spécificateur `@`

On l’avait vu, une chaîne de formattage est composée de caractères et de spécificateurs de caractères commençant par % Les spécificateurs sont des caractères qui ne sont pas affichés et qui seront remplacés avant l’affichage final.

Format ajoute son propre spécificateur de caractère : @.

II.1.a. Le vidage (flush)

La première spécification qu’on a vue est donc celle qu’il ne faut presque jamais utiliser (ce qui pose la question de l’avoir mentionnée en premier lieu) : @.. Cette spécification indique seulement au moteur d’impression qu’à ce niveau là il faut sauter une ligne et vider l’imprimeur. Les deux autres spécifications semblables sont @n qui n’indique que le saut de ligne et @? qui n’indique que le vidage de l’imprimeur. L’inconvénient de ces trois spécificateurs est qu’ils sont trop puissants et chamboulent donc le bon fonctionnement du reste de l’impression. Je n’ai personnellement jamais utilisé @n (autant utiliser une boîte avec un spécificateur de coupure comme nous le verrons immédiatement après) et n’utilise @. que lorsque je sais qu’il ne reste rien à imprimer.

II.1.b. Les indications de coupure ou d’espace

Important :

Une indication de coupure saute à la ligne s’il le faut sinon elle ne fait rien
Une indication d’espace sécable saute à la ligne s’il le faut, sinon elle affiche une espace

Les deux sont donc des indications de saut de ligne si nécessaire, il n’existe pas d’indication d’espace par défaut ou rien s’il n’y a pas assez d’espace (utiliser affichera toujours une espace).

Les indications sont au nombre de trois (et leur fonctionnement sera bien plus clair lorsque vous verrez les boîtes) :

@, : indication de coupure (c’est-à-dire rien prioritairement ou un saut à la ligne s’il le faut)
@⎵ : indique une espace sécable (c’est-à-dire une espace prioritairement ou un saut à la ligne s’il le faut) (Il faut bien évidemment comprendre le caractère ⎵ comme l’espace blanc habituel)
@;<n o> : indique n espaces sécables ou une coupure indentée de o (c’est-à-dire n espaces sécables prioritairement ou un saut à la ligne avec une indentation supplémentaire de o s’il le faut)

D’après ce que je viens d’écrire il devrait être évident maintenant que le caractère est une espace insécable qui ne provoquera donc pas de saut à la ligne quand bien même on dépasserait les limites de celle-ci. Contrairement à nos espaces de traitement de texte habituel qui sont des espaces sécables (pouvant provoquer des sauts de ligne), il faut spécifier quels espaces sont sécables lorsqu’on utilise Format.

On écrira par exemple fprintf fmt "let rec f =@ %a" pp_expr e car on ne veut pas que let rec f = soit séparé en plusieurs lignes mais on met bien @⎵ avant %a car l’expression sera soit sur la même ligne si suffisament petite soit à la ligne suivante si trop grande (on devrait même écrire @;<1 2> pour que l’expression soit indentée si on saute à la ligne suivante mais, on va le voir immédiatement, c’est là que les boîtes nous permettent d’automatiser ce genre de comportement)

II.1.c. Les boîtes

La deuxième spécification est celle permettant d’ouvrir et de fermer des boîtes.

Une boîte se commence par @[ et se termine par @]. Entre ces deux bornes, on fait ce qu’on veut (sauf utiliser @., @? ou @\n !). Tout ce qui se passe à l’intérieur de la boîte reste (et doit rester) à l’intérieur de celle-ci. Indentation, coupures, boîtes verticales, horizontales, les deux, l’une ou l’autre, toutes ces options sont accessibles une fois qu’une boîte a été ouverte. Voyons-les rapidement (pour rappel, la version détaillée est disponible dans le tutoriel.

Une fois qu’une boîte a été ouverte on peut préciser entre deux chevrons le comportement qu’on veut qu’elle ait en cas d’indication de coupure, en voici un rapide aperçu :

<v> : Toute indication de coupure entraîne un saut à la ligne
<h> : Toute indication d’espace entraîne une espace, les indications de coupure n’ont aucun effet
<hv>: Si toute la boîte peut être imprimée sur la même ligne alors seules les indications d’espace sont prises en compte sinon seules les indications de coupure le sont et chaque élément est imprimé sur sa propre ligne
<hov> : Tant que des éléments peuvent être imprimés sur une ligne ils le sont avec leurs indications d’espace. Les indications de coupure sont utilisées lorsqu’il faut sauter une ligne.

Chacun de ces comportements peut se voir attribuer une valeur supplémentaire, sa valeur d’indentation, qui indique l’indentation par rapport au début de la boîte qui devra être ajoutée à chaque saut de ligne.

Soit le code suivant permettant d’afficher une liste d’items séparés soit par une indication de coupure @,, soit par une indication d’espace @⎵ soit par une indication d’espace ou de coupure indentée @;<2 3> (2 espaces ou une coupure indentée de trois espaces) :

open Format

let l = ["toto"; "tata"; "titi"]

let pp_item fmt s = fprintf fmt "%s" s

let pp_cut fmt () = fprintf fmt "@,"
let pp_spc fmt () = fprintf fmt "@ "
let pp_brk fmt () = fprintf fmt "@;<2 3>"


let pp_list pp_sep fmt l =
  pp_print_list pp_item ~pp_sep fmt l

Voici un récapitulatif des différents comportements de boîtes en fonction des indications de coupure/espace rencontrées :

(* Boite verticale (tout est coupure) *)
printf "------------@.";
printf "v@.";
printf "------------@.";
printf "@[<v 2>[%a]@]@." (pp_list pp_cut) l;
printf "@[<v 2>[%a]@]@." (pp_list pp_spc) l;
printf "@[<v 2>[%a]@]@." (pp_list pp_brk) l;
(* Sortie attendue:
------------
v
------------
[toto
  tata
  titi]
[toto
  tata
  titi]
[toto
     tata
     titi]
*)


(* Boîte horizontale (pas de coupure) *)
printf "------------@.";
printf "h@.";
printf "------------@.";
printf "@[<h 2>[%a]@]@." (pp_list pp_cut) l;
printf "@[<h 2>[%a]@]@." (pp_list pp_spc) l;
printf "@[<h 2>[%a]@]@." (pp_list pp_brk) l;
(* Sortie attendue:
------------
h
------------
[tototatatiti]
[toto tata titi]
[toto  tata  titi]
*)


(* Boîte horizontale-verticale
  (Affiche tout sur une ligne si possible sinon boîte verticale) *)
printf "------------@.";
printf "hv@.";
printf "------------@.";
printf "@[<hv 2>[%a]@]@." (pp_list pp_cut) l;
printf "@[<hv 2>[%a]@]@." (pp_list pp_spc) l;
printf "@[<hv 2>[%a]@]@." (pp_list pp_brk) l;
(* Sortie attendue:
------------
hv
------------
[toto
  tata
  titi]
[toto
  tata
  titi]
[toto
     tata
     titi]
*)


(* Boîte horizontale ou verticale tassante
  (Affiche le maximum possible sur une ligne avant de sauter à la
   ligne suivante et recommencer) *)
printf "------------@.";
printf "hov@.";
printf "------------@.";
printf "@[<hov 2>[%a]@]@." (pp_list pp_cut) l;
printf "@[<hov 2>[%a]@]@." (pp_list pp_spc) l;
printf "@[<hov 2>[%a]@]@." (pp_list pp_brk) l;
(* Sortie attendue:
------------
hov
------------
[tototata
  titi]
[toto tata
  titi]
[toto
     tata
     titi]
*)

(*  Boîte horizontale ou verticale structurelle
   (Même fonctionnement que la boîte tassante sauf pour le dernier
    retour à la ligne qui tente de favoriser une indentation de
    niveau 0) *)
printf "------------@.";
printf "b@.";
printf "------------@.";
printf "@[<b 2>[%a]@]@." (pp_list pp_cut) l;
printf "@[<b 2>[%a]@]@." (pp_list pp_spc) l;
printf "@[<b 2>[%a]@]@." (pp_list pp_brk) l
(* Sortie attendue:
------------
b
------------
[tototata
  titi]
[toto tata
  titi]
[toto
     tata
     titi]
*)

Petite précision sur l’utilisation ici des @. alors qu’il est recommandé de ne jamais les utiliser. Il ne faut en réalité pas jamais les utiliser, il faut seulement les utiliser lorsqu’on est sûr de n’être dans aucune boîte. Ici, par exemple, on souhaite marquer distinctement les différentes impressions de boîtes, il est donc tout à fait correct d’utiliser @. étant donné qu’on est sûr d’être au dernier niveau d’impression (rien au-dessus) et de ne pas casser une passe d’impression élégante. Il serait donc bien plus précis de dire

Il ne faut pas utiliser @., @n et @? dans des impressions qui sont ou seront potentiellement imbriquées

Mais il est bien plus simple pour commencer de ne jamais les utiliser quitte à les rajouter après.

Le comportement de la boîte b (boîte structurelle) semble être le même que celui de la boîte hov (boîte tassante) mais il se trouve des cas où les deux diffèrent (généralement lorsqu’un saut de ligne réduit l’indentation courante, la boîte structurelle saute à la ligne même s’il reste de la place sur la ligne courante). Je vous invite à consulter le tutoriel pour plus de précisions (je dois aussi avouer que leur fonctionnement est très proche de ce qu’on pourrait appeler “opaque” étant donné qu’en fonction de la taille de marge le comportement attendu aura lieu ou non. L’auteur de ce tutoriel tient à préciser qu’il utilise plutôt des boîtes verticales avec une indentation nulle s’il lui arrive de vouloir obtenir le comportement des boîtes structurelles, un exemple est fourni lors de l’affichage en HTML à la fin de ce document).

II.2. Récapitulatif

Il faut utiliser des boîtes
Les indications de vidage fermant toutes les boîtes, il ne faut surtout pas les utiliser dans des fonctions d’affichage internes, il faut se limiter aux indications de coupure et d’espace
Il faut vraiment utiliser des boîtes

Vous voilà armé-e-s pour utiliser Format dans sa version la plus simple, avec des boîtes, de l’indentation, des indications de coupure et d’espace.

Reprenons notre affichage d’erreur :

let pp_error fmt = function
  | Type_Error (s1, s2) -> fprintf fmt "@[<hov 2>Type is %s@ instead of %s@]" s1 s2
  | Apply_non_function s -> fprintf fmt "@[<hov 2>Type is %s,@ this is not a function@]" s

let pp_list fmt l =
  pp_print_list pp_error fmt l

On a encapsulé l’affichage des deux erreurs dans des boîtes hov avec une indication d’espace sécable au milieu et utilisé la fonction pp_print_list du module Format

Si je tente maintenant d’afficher une liste d’erreurs dans deux environnements, un de 50 colonnes et l’autre de 25 colonnes de largeur avec le code suivant :

let () =
  let e1 = Type_Error ("int", "bool") in
  let e2 = Apply_non_function ("int") in
  let e3 = Type_Error ("int", "float") in
  let e4 = Apply_non_function ("bool") in

  let el = [e1; e2; e3; e4] in
  pp_set_margin std_formatter 50;
  fprintf std_formatter "--------------------------------------------------@.";
  fprintf std_formatter "@[<v 0>%a@]@." pp_list el;
  pp_set_margin std_formatter 25;
  fprintf std_formatter "-------------------------@.";
  fprintf std_formatter "@[<v 0>%a@]@." pp_list el;

J’obtiens le résultat suivant :

--------------------------------------------------
Type is int instead of bool
Type is int, this is not a function
Type is int instead of float
Type is bool, this is not a function
-------------------------
Type is int
  instead of bool
Type is int,
  this is not a function
Type is int
  instead of float
Type is bool,
  this is not a function

Ce qu’on rajoute en verbosité on le gagne en élégance. Et en parlant d’élégance, ça manque de couleurs.

III. Les tags sémantiques

Cette partie n’est pas présente dans le tutoriel mais dans un article tutoriel qui l’explique assez rapidement.

La troisième spécification, donc (après celles de coupure et de boîtes), est la spécification de tag sémantique : @{ pour en ouvrir un et @} pour le fermer.

III.1. Marquer son texte

Mais avant de comprendre leur fonctionnement, cherchons à comprendre leur intérêt. Que vous souhaitiez afficher dans un terminal, dans une page html ou autre, il y a de fortes chances que cette sortie accepte les marques de texte comme l’italique, la coloration etc. Utilisateur d’emacs et d’un terminal ANSI je peux modifier l’apparence de mon texte grâce aux codes ANSI :

Si je crée un programme OCaml qui affiche cette chaîne de charactère et que je l’exécute directement dans mon terminal je devrais obtenir le même résultat :

Naturellement, ça ne fonctionne pas, si l’informatique était standardisée et si tout le monde savait communiquer ça se saurait. Il s’avère que le caractère 033 est interprété en octal par les terminaux ANSI mais en décimal par OCaml (ce qui semble être l’interprétation normale). OCaml permet de représenter un caractère selon plusieurs séquences d’échappement différentes :

Séquence	Caractère résultant
`DDD`	le caractère correspondant au code ASCII `DDD` en décimal
`xHH`	le caractère correspondant au code ASCII `HH` en hexadécimal
`oOOO`	le caractère correspondant au code ASCII `OOO` en octal

On peut donc écrire au choix

let () = Format.printf "\027[36mBlue Text \027[0;3;30;47mItalic WhiteBG Black Text"
let () = Format.printf "\x1B[36mBlue Text \x1B[0;3;30;47mItalic WhiteBG Black Text"
let () = Format.printf "\o033[36mBlue Text \o033[0;3;30;47mItalic WhiteBG Black Text"

Dans tous les cas, on obtient le résultat suivant :

Que se passe-t-il, par contre, si j’exécute une de ces lignes dans un terminal non ANSI ? En testant sur TryOCaml:

On ne veut surtout pas que ce genre d’affichage puisse arriver. Il faudrait donc pouvoir s’assurer que le marquage du texte soit actif uniquement quand on le décide. L’idée de créer deux chaînes de formattage en fonction de notre capacité ou non à afficher du texte marqué n’est clairement pas une bonne pratique de programmation (changer une formulation demande de changer deux chaînes de formattage, le code est difficilement maintenable). Il faudrait donc un outil qui puisse faire un pré-traitement de notre chaîne de formattage pour lui ajouter des décorations.

Cet outil est déjà fourni par Format, ce sont les tags sémantiques.

III.2 Les tags sémantiques

Introduits par @{ et fermés par @}, comme les boîtes ils sont paramétrés par la construction <t> pour indiquer l’ouverture (et la fermeture) du tag t. Contrairement aux boîtes, les tags n’ont aucune signification pour l’imprimeur (on peut faire l’analogie avec les types de base d’OCaml que sont int, bool, float etc et les types définis par le programmeur ou la programmeuse (type t = A | B, par exemple. Les types de base ont déjà une quantité de fonctions qui leurs sont associés alors que les types définis ne signifient rien tant qu’on n’écrit pas les fonctions qui les manipuleront). L’avantage premier de ces tags est donc que, n’ayant aucune signification, ils sont tout simplement ignorés par l’imprimeur lors de l’affichage de notre chaîne de caractère finale:

Par défaut, l’imprimeur ne traite pas les tags sémantiques (ce qui permet d’avoir un comportement d’affichage aussi simple que possible par défaut). Le traitement des tags sémantiques peut être activé pour chaque formatter indépendamment avec les fonctions val pp_set_tags : formatter -> bool -> unit, val pp_set_print_tags : formatter -> bool -> unit et val pp_set_mark_tags : formatter -> bool -> unit dont on verra les effets immédiatement. Voyons déjà ce qui se passe avec la fonction générale pp_set_tags qui combine les deux suivantes :

Que s’est-il passé ?

Une fois que le traitement des tags sémantiques est activé, quatre opérations vont être effectuées à chaque ouverture et fermeture de tag :

print_open_stag suivie de mark_open_stag pour chaque tag t ouvert avec @{<t>
mark_close_stag suivie de print_close_stag pour chaque tag t fermé avec @} correspondant à la dernière ouverture @{<t>

Regardons les signatures de ces quatre opérations :

type formatter_stag_functions = {
    mark_open_stag : stag -> string;
       mark_close_stag : stag -> string;
       print_open_stag : stag -> unit;
       print_close_stag : stag -> unit;
}

Les fonctions mark_*_stag prennent un tag sémantique en paramètre et renvoie une chaîne de caractères quand les fonctions print_*_stag prennent le même paramètre mais ne renvoient rien. La raison derrière est en réalité toute simple :

Les fonctions de marquage écrivent directement dans la cible d’affichage (le terminal, le fichier ou autre)
Les fonctions d’affichage écrivent dans le formatter qui les traite comme des chaînes de caractères normales qui peuvent donc entraîner des sauts de ligne, des coupures, de nouvelles boîtes etc

Une indication de couleur pour un terminal ANSI n’apparaît pas à l’affichage, le texte se retrouve coloré, il semble donc naturel de ne pas vouloir que cette indication ait un effet sur l’impression élégante. En revanche, si on voulait avoir une sortie vers un fichier LaTeX ou HTML, cette indication de couleur apparaîtraît et devrait donc avoir une influence sur l’impression élégante.

Il est donc assez simple de savoir dans quel cas on veut utiliser print_*_stag ou mark_*_stag :

Si le tag doit avoir un impact immédiat sur l’apparence du texte affiché (couleur, taille, décorations…) et non pas son contenu, il faut utiliser mark_*_stag
Si le tag doit avoir un impact sur le contenu du texte affiché et non pas sur son apparence, il faut utiliser print_*_stag
Si le tag doit avoir un impact à la fois sur le contenu et l’apparence du texte affiché alors il faut utiliser les deux en séparant bien entre contenu géré par print_*_stag et apparence gérée par mark_*_stag

Ces quatres fonctions ont chacune un comportement par défaut que voici :

let mark_open_stag = function
  | String_tag s -> "<" ^ s ^ ">"
  | _ -> ""
let mark_close_stag = function
  | String_tag s -> "</" ^ s ^ ">"
  
let print_open_stag = ignore
let print_close_stag = ignore

Le type stag est un type somme extensible (introduits dans OCaml 4.02.0) c’est-à-dire qu’il est défini de la sorte

type stag = ..

type stag += String_tag of string

Par défaut seuls les String_tag of string sont donc reconnus comme des tags sémantiques (ce sont aussi les seuls qui peuvent être obtenus par la construction @{<t> ... @}, ici t sera traité comme String_tag t) ce qui est illustré par le comportement par défaut de mark_open_tag et mark_close_tag. Ce comportement par défaut nous permet aussi de comprendre ce qui est arrivé ici :

N’ayant pas personnalisé les opérations de manipulation des tags, leur comportement par défaut a été exécuté, ce qui revient à afficher directement le tag entre chevrons sans passer par le formatter. Il faut donc définir les comportements voulus pour nos tags (attention, ne manipulant que des chaînes de caractère, toute erreur est conséquemment difficile à identifier et corriger, il vaut mieux donc éviter les célèbres | _ -> () — il faudrait en réalité les éviter tout le temps si possible mais c’est une autre histoire).

Commençons donc par définir nos tags et ce à quoi on veut qu’ils correspondent :

open Format

type style =
  | Normal

  | Italic
  | Italic_off

  | FG_Black
  | FG_Blue
  | FG_Default

  | BG_White
  | BG_Default

let close_tag = function
  | Italic -> Italic_off
  | FG_Black | FG_Blue | FG_Default -> FG_Default

  | BG_White | BG_Default -> BG_Default

  | _ -> Normal

let style_of_tag = function
  | String_tag s -> begin match s with
      | "n" -> Normal
      | "italic" -> Italic
      | "/italic" -> Italic_off

      | "fg_black" -> FG_Black
      | "fg_blue" -> FG_Blue
      | "fg_default" -> FG_Default

      | "bg_white" -> BG_White
      | "bg_default" -> BG_Default

      | _ -> raise Not_found
    end
  | _ -> raise Not_found

Maintenant que chaque tag possible est géré, il nous faut les associer à leur valeur (ANSI dans ce cas) et implémenter nos propres fonctions de marquages (et pas d’affichage car a priori ces tags n’ont aucun effet sur le contenu du texte affiché) :

(* See https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters for some values *)
let to_ansi_value = function
  | Normal -> "0"

  | Italic -> "3"
  | Italic_off -> "23"

  | FG_Black -> "30"
  | FG_Blue -> "34"
  | FG_Default -> "39"

  | BG_White -> "47"
  | BG_Default -> "49"

let ansi_tag = Printf.sprintf "\x1B[%sm"

let start_mark_ansi_stag t = ansi_tag @@ to_ansi_value @@ style_of_tag t

let stop_mark_ansi_stag t = ansi_tag @@ to_ansi_value @@ close_tag @@ style_of_tag t

On se le rappelle, l’ouverture d’un tag ANSI se fait avec la séquence d’échappement x1B suivie de une ou plusieurs valeurs de tags séparées par ; entre [ et m. Dans notre cas chaque tag n’est associé qu’à une valeur mais il serait tout à fait possible d’avoir un Error -> "1;4;31" qui imposerait un affichage gras, souligné et en rouge. Tant que la chaîne de caractère renvoyée au terminal correspond bien à une séquence de marquage ANSI tout est possible.

Il faut ensuite faire en sorte que ces fonctions soient celles utilisées par le formatter lors de leur traitement :

let add_ansi_marking formatter =
  let open Format in
  pp_set_mark_tags formatter true;
  let old_fs = pp_get_formatter_stag_functions formatter () in
  pp_set_formatter_stag_functions formatter
    { old_fs with
      mark_open_stag = start_mark_ansi_stag;
      mark_close_stag = stop_mark_ansi_stag }

On utilise la fonction pp_set_mark_tags (au lieu de pp_set_tags) car on ne se sert pas de print_*_stags et on associe aux fonctions mark_*_stag les fonctions *_ansi_stag.

Il ne nous reste plus qu’à faire en sorte que les tags sémantiques soient traités et avec nos fonctions avant d’afficher notre chaîne de caractères :

let () =
  add_ansi_marking std_formatter;
  Format.printf "@{<fg_blue>Blue Text @}@{<italic>@{<bg_white>@{<fg_black>Italic WhiteBG BlackFG Text@}@}@}"

Et l’affichage dans le terminal sera bien celui voulu :

Si le programme doit être affiché dans un terminal non ANSI il suffit simplement d’enlever la ligne add_ansi_marking std_formatter; :

On pourrait aussi faire en sorte que notre texte puisse être envoyé vers un document HTML.

Il faut déjà changer les valeurs associées aux tags (on voit ici l’utilisation de boîtes verticales à indentation nulle mentionnée lors du paragraphe sur les boîtes structurelles) :

let to_html_value fmt =
  let fg_color c = Format.fprintf fmt {|@[<v 0>@[<v 2><span style="color:%s;">@,|} c in
  let bg_color c = Format.fprintf fmt {|@[<v 0>@[<v 2><span style="background-color:%s;">@,|} c in
  let close_span () = Format.fprintf fmt "@]@,</span>@]" in
  let default = Format.fprintf fmt in
  fun t -> match t with
    | Normal -> ()

    | Italic -> default "<i>"
    | Italic_off -> default "</i>"

    | FG_Black -> fg_color "black"
    | FG_Blue -> fg_color "blue"
    | FG_Default -> close_span ()

    | BG_White -> bg_color "white"
    | BG_Default -> close_span ()

La construction {| ... |} permet d’avoir des chaînes de caractères sans les caractères spéciaux " et `` ce qui permet d’écrire {|"This is a nice "|} sans espacer ces caractères.

De même, la construction

let fonction arg1 ... argn =
  let expr1 = ... in
  ...
  let exprn = ... in
fun argn1 ... argnm ->

Permet de définir des expressions internes à une fonction qui dépendent des arguments fournis avant et donc, dans le cas d’une application partielle, de calculer cet environnement une seule fois. Dans le cas de la fonction to_html_value je pourrai donc créer la nouvelle application partielle let to_html_value_std = to_html_value std_formatter qui contiendra donc directment les implémentations de fg_color, bg_color, close_span et default pour std_formatter.

Contrairement au cas du terminal ANSI, ce qui changera sera le contenu et non pas l’apparence du texte, nous utiliserons donc les fonctions print_*_stag. C’est pourquoi nos fonctions doivent directement écrire dans le formatter et non pas renvoyer une chaîne de caractères.

Les fonctions d’ouverture et de fermeture ne changent pas énormément :

let start_print_html_stag fmt t =
  to_html_value fmt @@ style_of_tag t

let stop_print_html_stag fmt t =
  to_html_value fmt @@ close_tag @@ style_of_tag t

On associe ensuite ces fonctions aux fonctions print_*_stag :

let add_html_printings formatter =
  let open Format in
  pp_set_mark_tags formatter false;
  pp_set_print_tags formatter true;
  let old_fs = pp_get_formatter_stag_functions formatter () in
  pp_set_formatter_stag_functions formatter
    { old_fs with
      print_open_stag = start_print_html_stag formatter;
      print_close_stag = stop_print_html_stag formatter}

On en profite pour désactiver le marquage sur le formatter passé en paramètre. Cela évite d’avoir de mauvaises surprises au cas où il aurait été activé précédemment (il aurait fallu faire de même lors du marquage pour le terminal ANSI).

Finalement, l’appel à :

let () =
  add_html_printings std_formatter;
  Format.printf "@[<v 0>@{<fg_blue>Blue Text @}@,@{<italic>@{<bg_white>@{<fg_black>Italic WhiteBG BlackFG Text@}@}@}@]@."

Nous donne le résultat attendu :

<span style="color:blue;">
  Blue Text
</span>
<i>
  <span style="background-color:white;">
     <span style="color:black;">
       Italic WhiteBG BlackFG Text
     </span>
   </span>
</i>

Conclusion

Nous voici arrivés à la fin de ce tutoriel qui, je l’espère, vous permettra d’appréhender le module Format avec bien plus de sérénité.

Dans les possibilités non présentées ici mais qu’il est intéressant d’avoir en mémoire :

Possibilité de redéfinir intégralement toutes les fonctions d’affichage définies dans l’enregistrement :

<span class="hljs-keyword">type</span>
formatter_out_functions = {
    out_string :
    <span class="hljs-built_in">string</span> -> <span class="hljs-built_in">int</span> -> <span class="hljs-built_in">int</span> -> <span class="hljs-built_in">unit</span>;
    out_flush :
    <span class="hljs-built_in">unit</span> -> <span class="hljs-built_in">unit</span>;
    out_newline :
    <span class="hljs-built_in">unit</span> -> <span class="hljs-built_in">unit</span>;
    out_spaces :
    <span class="hljs-built_in">int</span> -> <span class="hljs-built_in">unit</span>;
    out_indent :
    <span class="hljs-built_in">int</span> -> <span class="hljs-built_in">unit</span>;
}

Possibilité de transformer n’importe quel sortie en un formatter pour écrire directement dedans sans avoir à passer par des chaînes de caractère intermédiaire (notamment la fonction val formatter_of_buffer : Buffer.t -> formatter qui permet directement d’écrire dans un buffer
L’impression élégante symbolique qui imprime de façon symbolique donc permet de voir directement quelles directives seront envoyées au formatter à l’impression. Très utile pour débuguer en cas d’impression cacophonique mais aussi extrêmement puissant pour effectuer une phase de post-traitement (par exemple si on veut ajouter un symbole à chaque début de ligne)
Les fonctions utiles qu’il ne faut pas oublier d’utiliser (je sais que les devs OCaml aiment réinventer la roue mais il existe déjà des fonctions pour afficher des listes, des options et les résultats Ok _ | Error _) :

val pp_print_list : ?pp_sep:(formatter -> unit -> unit) -> (formatter -> 'a -> unit) -> formatter -> 'a list -> unit

(* Affiche une liste dont chaque élément est séparé par le séparateur par défaut `@,` ou celui fourni *)

val pp_print_option : ?none:(formatter -> unit -> unit) -> (formatter -> ‘a -> unit) -> formatter -> ‘a option -> unit

(* Affiche le contenu d’une option en cas de Some contenu et rien par défaut si None ou l’affichage fourni *)</p>

val pp_print result : ok:(formatter -> ‘a -> unit) -> error:(formatter -> ‘e -> unit) -> formatter -> (‘a, ‘e) result -> unit

(* Affiche le contenu d’un result. Les arguments ne sont ici pas optionnels et conditionnent l’affichage en cas de Ok </em> et de Error _ *)

Enfin, une pelletée de fonctions à la printf telles que, donc :
fprintf que nous avons déjà vue
dprintf qui permet de retarder l'évaluation de l'impression et donc de ne pas calculer des impressions qui ne seront jamais faites
ifprintf qui n'affiche rien (utile lorsqu'on veut avoir la même signature que fprintf mais en étant sûr que rien ne sera fait)

Sources :

Tutoriel du site OCaml
Richard Bonichon, Pierre Weis. Format Unraveled. 28ièmes Journées Francophones des LangagesApplicatifs, Jan 2017, Gourette, France. hal-01503081

Codes sources :

Code LaTeX correspondant à printf

\documentclass[tikz,border=10pt]{standalone}

\usepackage{tikz}
\usetikzlibrary{math}
\usetikzlibrary{tikzmark}

\usepackage{xcolor}

\pagecolor[rgb]{0,0,0}
\color[rgb]{1,1,1}

\colorlet{color1}{blue!50!white}
\colorlet{color2}{red!50!white}
\colorlet{color3}{green!50!black}

\begin{document}

\begin{tikzpicture}[remember picture]
\node [align=left,font=\ttfamily] at (0,0) {
    let s = "toto" in\[2em]
    printf "{color{color1}\tikzmarknode{scd}{\%d}}
        {color{color2}\tikzmarknode{scc}{\%c}}
        {color{color3}\tikzmarknode{scs}{\%s}}"
    {\color{color1}\tikzmarknode{d}{3}}
    {\color{color2}\tikzmarknode{c}{'c'}}
    {\color{color3}\tikzmarknode{s}{s}}\\[2em]
    > "3 c toto"
};
\draw[<-, color1] (scd.north) -- ++(0,0.5) -| (d);
\draw[<-, color2] (scc.south) -- ++(0,-0.4) -| (c);
\draw[<-, color3] (scs.north) -- ++(0,0.4) -| (s);
\end{tikzpicture}

end{document}

Code LaTeX correspondant à fprintf:

\documentclass[tikz,border=10pt]{standalone}

\usepackage{tikz}
\usetikzlibrary{math}
\usetikzlibrary{decorations.pathreplacing,tikzmark}

\usepackage{xcolor}

\pagecolor[rgb]{0,0,0}
\color[rgb]{1,1,1}

\colorlet{color1}{blue!50!white}
\colorlet{color2}{red!50!white}
\colorlet{color3}{green!50!black}

\begin{document}

\begin{tikzpicture}[remember picture]
\node [align=left,font=\ttfamily] at (0,0) {
    let s = "toto" in\\[2em]
    fprintf \tikzmarknode{fmt}{fmt} \tikzmarknode{str}{"{\color{color1}\tikzmarknode{scd}{\%d}}
        {\color{color2}\tikzmarknode{scc}{\%c}}
        {\color{color3}\tikzmarknode{scs}{\%s}}"}
    {\color{color1}\tikzmarknode{d}{3}}
    {\color{color2}\tikzmarknode{c}{'c'}}
    {\color{color3}\tikzmarknode{s}{s}}\\[2em]
    > \\
    (* fmt <- "3 c toto" *)
};
\draw[<-, color1] (scd.north) -- ++(0,0.5) -| (d);
\draw[<-, color2] (scc.south) -- ++(0,-0.3) -| (c);
\draw[<-, color3] (scs.north) -- ++(0,0.4) -| (s);
\draw[decorate,decoration={brace, amplitude=5pt, raise=10pt},yshift=-2cm] (str.south east) -- (str.south west) node[midway, yshift=-13pt](a){} ;

\draw[->, white] (a.south) -- ++(0,-0.1) -| (fmt);
\end{tikzpicture}

\end{document}

Code LaTeX correspondant à fprintf avec utilisation de %a

\documentclass[tikz,border=10pt]{standalone}

\usepackage{tikz}
\usetikzlibrary{math}
\usetikzlibrary{decorations.pathreplacing,tikzmark}

\usepackage{xcolor}

\pagecolor[rgb]{0,0,0}
\color[rgb]{1,1,1}

\colorlet{color1}{blue!50!white}
\colorlet{color2}{red!50!white}
\colorlet{color3}{green!50!black}

\begin{document}

\begin{tikzpicture}[remember picture]
\node [align=left,font=\ttfamily] at (0,0) {
    let s = "toto" in\\[2em]
    type expr = \{i: int; j: int\}\\
    let pp\_expr fmt {i; j} = fprintf fmt "<\%d, \%d> i j" in\\[2em]
    fprintf \tikzmarknode{fmt}{std\_formatter} \tikzmarknode{str}{"{\color{color1}\tikzmarknode{scd}{\%d}}
        {\color{color2}\tikzmarknode{sca}{\%a}}
        {\color{color3}\tikzmarknode{scs}{\%s}}"}
    {\color{color1}\tikzmarknode{d}{3}}
    {\color{color2}\tikzmarknode{ppe}{pp\_expr}}
    {\color{color2}\tikzmarknode{e}{\{i=1; j=2\}}}
    {\color{color3}\tikzmarknode{s}{s}}\\[2em]
    > "3 <1, 2> toto"
};
\draw[<-, color1] (scd.north) -- ++(0,0.5) -| (d);
\draw[<-, color2] (sca.south) -- ++(0,-0.3) -| (ppe);
\draw[<-, color2] (sca.65) -- ++(0,0.3) -| (e);
\draw[->, color2] (fmt.north) -- ++(0,0.2) -| (sca.115);
\draw[<-, color3] (scs.south) -- ++(0,-0.4) -| (s);
\draw[decorate,decoration={brace, amplitude=5pt, raise=12pt},yshift=-2cm]  (str.south east) -- (str.south west) node[midway, yshift=-13pt](a){} ;

\draw[->, white] (a.south) -- ++(0,-0.1) -| (fmt);
\end{tikzpicture}

\end{document}

A Solidity parser in OCaml with Menhir

2020-05-19T13:56:36Z

This article is cross-posted on Origin Labs’ Dune Network blog

We are happy to announce the first release of our Solidity parser, written in OCaml using Menhir. This is a joint effort with Origin Labs, the company dedicated to blockchain challenges, to implement a full interpreter for the Solidity language directly in a blockchain.

Solidity is probably the most popular language for smart-contracts, small pieces of code triggered when accounts receive transactions on a blockchain.Solidity is an object-oriented strongly-typed language with a Javascript-like syntax.

Solidity was first implemented for the Ethereum blockchain, with a compiler to the EVM, the Ethereum Virtual Machine.

Dune Network takes a different approach, as Solidity smart-contracts will be executed natively, after type-checking. Solidity will be the third native language on Dune Network, with Michelson, a low-level strongly-typed language inherited from Tezos, and Love, an higher-level strongly-typed language, also implemented jointly by OCamlPro and Origin Labs.

A first step has been accomplished, with the completion of the Solidity parser and printer, written in OCaml with Menhir.

This parser (and its printer companion) is now available as a standalone library under the LGPLv3 license with Linking Exception, allowing its integration in all projects. The source code is available at https://gitlab.com/o-labs/solidity-parser-ocaml.

Our parser should support all of Solidity 0.6, with the notable exception of inline assembly (may be added in a future release).

Example contract

Here is an example of a very simple contract that stores an integer value and allows the contract’s owner to add an arbitrary value to this value, and any other contract to read this value:

pragma solidity >=0.6.0 <0.7.0;

contract C {
    address owner;
    int x;

    constructor() public {
        owner = msg.sender;
        x = 0;
    }

    function add(int d) public {
        require(msg.sender == owner);
        x += d;
    }

    function read_x() public view returns(int) {
        return x;
    }
}

Parser Usage

Executable

Our parser comes with a small executable that demonstrates the library usage. Simply run:

./solp contract.sol

This will parse the file contract.sol and reprint it on the terminal.

Library

To use our parser as a library, add it to your program’s dependencies and use the following function:

Solidity_parser.parse_contract_file : string -> Solidity_parser.Solidity_types.module_

It takes a filename and returns a Solidity AST.

If you wish to print this AST, you may turn it into its string representation by sending it to the following function:

Solidity_parser.Printer.string_of_code : Solidity_parser.Solidity_types.module_ -> string

Conclusion

Of course, all of this is Work In Progress, but we are quite happy to share it with the OCaml community. We think there is a tremendous work to be done around blockchains for experts in formal methods. Do not hesitate to contact us if you want to use this library!

About Origin Labs

Origin Labs is a company founded in 2019 by the former blockchain team at OCamlPro. At Origin Labs, they have been developing Dune Network, a fork of the Tezos blockchain, its ecosystem, and applications over the Dune Network platform. At OCamlPro, they developed TzScan, the most popular block explorer at the time, Liquidity, a smart contract language, and were involved in the development of the core protocol and node.Do not hesitate to reach out by email: contact@origin-labs.com.

opam 2.1.0 alpha is here!

2020-04-22T13:56:36Z

We are happy to announce a alpha for opam 2.1.0, one year and a half in the making after the release of 2.0.0.

Many new features made it in (see the complete changelog or release note for the details), but here are a few highlights of this release.

Release highlights

The two following features have been around for a while as plugins and are now completely integrated in the core of opam. No extra installs needed anymore, and a more smooth experience.

Seamless integration of System dependencies handling (a.k.a. "depexts")

A number of opam packages depend on tools or libraries installed on the system, which are out of the scope of opam itself. Previous versions of opam added a specification format, and opam 2.0 already handled checking the OS and extracting the required system package names.

However, the workflow generally involved letting opam fail once, then installing the dependencies and retrying, or explicitely using the opam-depext plugin, which was invaluable for CI but still incurred extra steps.

With opam 2.1.0, depexts are seamlessly integrated, and you basically won't have to worry about them ahead of time:

Before applying its course of actions, opam 2.1.0 checks that external dependencies are present, and will prompt you to install them. You are free to let it do it using sudo, or just run the provided commands yourself.
It is resilient to depexts getting removed or out of sync.
Opam 2.1.0 detects packages that depend on stuff that is not available on your OS version, and automatically avoids them.

This is all fully configurable, and can be bypassed without tricky commands when you need it (e.g. when you compiled a dependency yourself).

Dependency locking

To share a project for development, it is often necessary to be able to reproduce the exact same environment and dependencies setting — as opposed to allowing a range of versions as opam encourages you to do for releases.

For some reason, most other package managers call this feature "lock files". Opam can handle those in the form of [foo.]opam.locked files, and the --locked option.

With 2.1.0, you no longer need a plugin to generate these files: just running opam lock will create them for existing opam files, enforcing the exact version of all dependencies (including locally pinned packages).

If you check-in these files, new users would just have run opam switch create . --locked on a fresh clone to get a local switch ready to build the project.

Pinning sub-directories

This one is completely new: fans of the Monorepo rejoice, opam is now able to handle projects in subtrees of a repository.

Using opam pin PROJECT_ROOT --subpath SUB_PROJECT, opam will look for PROJECT_ROOT/SUB_PROJECT/foo.opam. This will behave as a pinning to PROJECT_ROOT/SUB_PROJECT, except that the version-control handling is done in PROJECT_ROOT.
Use opam pin PROJECT_ROOT --recursive to automatically lookup all sub-trees with opam files and pin them.

Opam switches are now defined by invariants

Previous versions of opam defined switches based on base packages, which typically included a compiler, and were immutable. Opam 2.1.0 instead defines them in terms of an invariant, which is a generic dependency formula.

This removes a lot of the rigidity opam switch commands had, with little changes on the existing commands. For example, opam upgrade ocaml commands are now possible; you could also define the invariant as ocaml-system and have its version change along with the version of the OCaml compiler installed system-wide.

Configuring opam from the command-line

The new opam option command allows to configure several options, without requiring manual edition of the configuration files.

For example:

opam option jobs=6 --global will set the number of parallel build jobs opam is allowed to run (along with the associated jobs variable)
opam option depext-run-commands=false disables the use of sudo for handling system dependencies; it will be replaced by a prompt to run the installation commands.

The command opam var is extended with the same format, acting on switch and global variables.

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam directory.

The upgrade instructions are unchanged:

Either from binaries: run

$~ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0~alpha"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

This is still a alpha, so a few glitches or regressions are to be expected. Please report them to the bug-tracker. Thanks for trying it out, and hoping you enjoy!

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

opam 2.0.7 release

2020-04-21T13:56:36Z

We are pleased to announce the minor release of opam 2.0.7.

This new version contains backported small fixes:

Escape Windows paths on manpages [#4129 @AltGr @rjbou]
Fix opam installer opam file [#4058 @rjbou]
Fix various warnings [#4132 @rjbou @AltGr - fix #4100]
Fix dune 2.5.0 promote-install-files duplication [#4132 @rjbou]

Installation instructions (unchanged):

From binaries: run

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.7"

From source, using opam:

opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

Le nouveau GC d’OCaml 4.10 : premier aperçu de la stratégie best-fit

2020-03-24T13:56:36Z

Le GC d’OCaml oeuvre discrètement à l’efficacité de vos allocations mémoire. Tel un héros de l’ombre, il reste méconnu de la plupart des hackers OCaml. Avec l’arrivée d’OCaml 4.10, il s’enrichit d’une nouvelle stratégie apparue dans le changelog, signée de Damien Doligez.

Dans cet article nous commençons à explorer la nouvelle stratégie baptisée *best-fit *du nouveau Glaneur de Cellules dans OCaml 4.10.

En savoir plus : article en anglais.

An in-depth Look at OCaml’s new “Best-fit” Garbage Collector Strategy

2020-03-23T13:56:36Z

The Garbage Collector probably is OCaml’s greatest unsung hero. Its pragmatic approach allows us to allocate without much fear of efficiency loss. In a way, the fact that most OCaml hackers know little about it is a good sign: you want a runtime to gracefully do its job without having to mind it all the time.

But as OCaml 4.10.0 has now hit the shelves, a very exciting feature is in the changelog:

#8809, #9292: Add a best-fit allocator for the major heap; still experimental, it should be much better than current allocation policies (first-fit and next-fit) for programs with large heaps, reducing both GC cost and memory usage. This new best-fit is not (yet) the default; set it explicitly with OCAMLRUNPARAM="a=2" (or Gc.set from the program). You may also want to increase the space_overhead parameter of the GC (a percentage, 80 by default), for example OCAMLRUNPARAM="o=85", for optimal speed. (Damien Doligez, review by Stephen Dolan, Jacques-Henri Jourdan, Xavier Leroy, Leo White)

At OCamlPro, some of the tools that we develop, such as the package manager opam, the Alt-Ergo SMT solver or the Flambda optimizer, can be quite demanding in memory usage, so we were curious to better understand the properties of this new allocator.

Minor heap and Major heap: the GC in a nutshell

Not all values are allocated equal. Some will only be useful for the span of local calculations, some will last as long as the program lives. To handle those two kinds of values, the runtime uses a Generational Garbage Collector with two spaces:

The minor heap uses the Stop-and-copy principle. It is fast but has to stop the computation to perform a full iteration.
The major heap uses the Mark-and-sweep principle. It has the perk of being incremental and behaves better for long-lived data.

Allocation in the minor heap is straightforward and efficient: values are stored sequentially, and when there is no space anymore, space is emptied, surviving values get allocated in the major heap while dead values are just forgotten for free. However, the major heap is a bit more tricky, since we will have random allocations and deallocations that will eventually produce a scattered memory. This is called fragmentation, and this means that you’re using more memory than necessary. Thankfully, the GC has two strategies to counter that problem:

Compaction: a heavyweight reallocation of everything that will remove those holes in our heap. OCaml’s compactor is cleverly written to work in constant space, and would be worth its own specific article!
Free-list Allocation: allocating the newly coming data in the holes (the free-list) in memory, de-scattering it in the process.

Of course, asking the GC to be smarter about how it allocates data makes the GC slower. Coding a good GC is a subtle art: you need to have something smart enough to avoid fragmentation but simple enough to run as fast as possible.

Where and how to allocate: the 3 strategies

OCaml used to propose 2 free-list allocation strategies: next-fit, the default, and first-fit. Version 4.10 of OCaml introduces the new best-fit strategy. Let’s compare them:

Next-fit, the original and remaining champion

OCaml’s original (and default) “next-fit” allocating strategy is pretty simple:

Keep a (circular) list of every hole in memory ordered by increasing addresses;
Have a pointer on an element of that list;
When an allocation is needed, if the currently pointed-at hole is big enough, allocate in it;
Otherwise, try the next hole and so-on.

This strategy is extremely efficient, but a big hole might be fragmented with very small data while small holes stay unused. In some cases, the GC would trigger costly compactions that would have been avoidable.

First-fit, the unsuccessful contender

To counteract that problem, the “first-fit” strategy was implemented in 2008 (OCaml 3.11.0):

Same idea as next-fit, but with an extra allocation table.
Put the pointer back at the beginning of the list for each allocation.
Use the allocation table to skip some parts of the list.

Unfortunately, that strategy is slower than the previous one. This is an example of making the GC smarter ends up making it slower. It does, however, reduce fragmentation. It was still useful to have this strategy at hand for the case where compaction would be too costly (on a 100Gb heap, for instance). An application that requires low latency might want to disable compaction and use that strategy.

Best-fit: a new challenger enters!

This leads us to the brand new “best-fit” strategy. This strategy is actually composite and will have different behaviors depending on the size of the data you’re trying to allocate.

On small data (up to 32 words), segregated free lists will allow allocation in (mostly) constant time.
On big data, a general best-fit allocator based on splay trees.

This allows for the best of the two worlds, as you can easily allocate your numerous small blocks in the small holes in your memory while you take a bit more time to select a good place for your big arrays.

How will best-fit fare? Let’s find out!

Try it!

First, let us remind you that this is still an experimental feature, which from the OCaml development team means “We’ve tested it thoroughly on different systems, but only for months and not on a scale as large as the whole OCaml ecosystem”.

That being said, we’d advise you don’t use it in production code yet.

Why you should try it

Making benchmarks of this new strategy could be beneficial for you and the language at large: the dev team is hoping for feedback, the more quality feedback you give means the more the future GC will be tuned for your needs.

In 2008, the first-fit strategy was released with the hope of improving memory usage by reducing fragmentation. However, the lack of feedback meant that the developers were not aware that it didn’t meet the users’ needs. If more feedback had been given, it’s possible that work on improving the strategy or on better strategies would have happened sooner.

Choosing the allocator strategy

Now, there are two ways to control the GC behavior: through the code or through environment variables.

First method: Adding instructions in your code

This method should be used by those of us who have code that already does some GC fine-tuning. As early as possible in your program, you want to execute the following lines:

let () =
Gc.(set
  { (get()) with
    allocation_policy = 2; (* Use the best-fit strategy *)
    space_overhead = 100; (* Let the major GC work a bit less since it's more efficient *)
  })

You might also want to add verbose = 0x400; or verbose = 0x404; in order to get some GC debug information. See here for more details on how to use the GC module.

Of course, you’ll need to recompile your code, and this will apply only after the runtime has initialized itself, triggering a compaction in the process. Also, since you might want to easily switch between different allocation policies and overhead specifications, we suggest you use the second method.

Second method: setting `$OCAMLRUNPARAM`

At OCamlPro, we develop and maintain a program that any OCaml developer should want to run smoothly. It’s called Opam, maybe you’ve heard of it? Though most commands take a few seconds, some administrative-heavy commands can be a strain on our computer. In other words: those are perfect for a benchmark.

Here’s what we did to benchmark Opam:

$ opam update
$ opam switch create 4.10.0
$ opam install opam-devel # or build your own code
$ export OCAMLRUNPARAM='b=1,a=2,o=100,v=0x404'
$ cd my/local/opam-repository
$ perf stat ~/.opam/4.10.0/lib/opam-devel/opam admin check --installability # requires right to execute perf, time can do the trick

If you want to compile and run your own benchmarks, here are a few details on OCAMLRUNPARAM:

b=1 means “print the backtrace in case of uncaught exception”
a=2 means “use best-fit” (default is 0 , first-fit is 1)
o=100 means “do less work” (default is 80, lower means more work)
v=0x404 means “have the gc be verbose” (0x400 is “print statistics at exit”, 0x4 is “print when changing heap size”)

See the manual for more details on OCAMLRUNPARAM

You might want to compare how your code fares on all three different GC strategies (and fiddle a bit with the overhead to find your best configuration).

Our results on opam

Our contribution in this article is to benchmark opam with the different allocation strategies:

Strategy:	Next-fit	First-fit	Best-fit
Overhead:	80	80	80	100	120
Cycles used (Gcycle)	2,040	3,808	3,372	2,851	2,428
Maximum heap size (kb)	793,148	793,148	689,692	689,692	793,148
User time (s)	674	1,350	1,217	1,016	791

A quick word on these results. Most of opam‘s calculations are done by dose and rely heavily on small interconnected blocks. We don’t really have big chunks of data we want to allocate, so the strategy won’t give us the bonus you might have as it perfectly falls into the best-case scenario of the next-fit strategy. As a matter of fact, for every strategy, we didn’t have a single GC compaction happen. However, Best-fit still allows for a lower memory footprint!

Conclusions

If your software is highly reliant on memory usage, you should definitely try the new Best-fit strategy and stay tuned on its future development. If your software requires good performance, knowing if your performances are better with Best-fit (and giving feedback on those) might help you in the long run.

The different strategies are:

Next-fit: generally good and fast, but has very bad worst cases with big heaps.
First fit: mainly useful for very big heaps that must avoid compaction as much as possible.
Best-fit: almost the best of both worlds, with a small performance hit for programs that fit well with next-fit.

Remember that whatever works best for you, it’s still better than having to malloc and free by hand. Happy allocating!

Comments

gasche (23 March 2020 at 17 h 50 min):

What about higher overhead values than 120, like 140, 160, 180 and 200?

Thomas Blanc (23 March 2020 at 18 h 17 min):

Because 100 was the overhead value Leo advised in the PR discussion I decided to put it in the results. As 120 got the same maximum heap size as next-fit I found it worth putting it in. Higher overhead values lead to faster execution time but a bigger heap.

I don’t have my numbers at hand right now. You’re probably right that they are relevant (to you and Damien at least) but I didn’t want to have a huge table at the end of the post.

nbbb (24 March 2020 at 11 h 18 min):

Higher values would allow us to see if best-fit can reproduce the performance characteristics of next-fit, for some value of the overhead.

nbbb (24 March 2020 at 16 h 51 min):

I just realized that 120 already has a heap as bit as next-fit — so best-fit can’t get as good as next-fit in this example, and higher values of the overhead are not quite as informative. Should have read more closely the first time.

Thomas Blanc (24 March 2020 at 16 h 55 min):

Sorry that it wasn’t as clear as it could be.

Note that opam and dose are in the best-case scenario of best-fit. Your own code would probably produce a different result and I encourage you to test it and communicate about it.

New version of TryOCaml in beta!

2020-03-16T13:56:36Z

We are happy to announce that our venerable "TryOCaml" service is being retired and replaced by a new, modern version based upon our work on Learn-OCaml.

→ Try it here ←

The new interface provides an editor panel besides the familiar top-level, error and warning positions highlighting, the latest OCaml release (4.10.0), local storage of your session, and more.

The service is still in beta, so it would be helpful if you could tell us about any hiccups you may encounter on the Discuss thread.

Let's read the testimony of Sylvain Conchon about our new version of TryOCaml:

“TryOCaml saved our lives in Paris Saclay in these times of social distancing. I teach functional programming with OCaml to my Y2 Bachelor’s Degree students. With the quarantine in place, we weren’t able to host the practical assignment in the machine room as usual, so we decided the students would do the exam at home. However, many of our students use Windows on which setting up OCaml is a hassle, or otherwise encountered problems while setting up the OCaml environment. We invited our students to use try-ocaml instead! Many have and the exam went really smoothly.”

Réunion annuelle du Club des utilisateurs d’Alt-Ergo

2020-03-03T13:56:36Z

La deuxième réunion annuelle du Club des utilisateurs d’Alt-Ergo a eu lieu à la mi-février ! Notre réunion annuelle est l’endroit idéal pour passer en revue les besoins de chaque partenaire concernant Alt-Ergo. Cette année, nous avons eu le plaisir de recevoir nos partenaires pour discuter de la feuille de route concernant les développements et les améliorations futures d’Alt-Ergo.

Alt-Ergo est un démonstrateur automatique de formules mathématiques, créé au LRI et développé par OCamlPro depuis 2013. Pour en savoir plus ou rejoindre le Club, visitez le site https://alt-ergo.ocamlpro.com/.

Notre Club a plusieurs objectifs, le premier étant de garantir la pérennité d’Alt-Ergo en favorisant la collaboration entre les membres du Club et en renforçant la collaboration avec les communautés de méthodes formelles telles que Why3. L’une de nos priorités est d’augmenter le nombre d’utilisateurs de notre outil en l’étendant à de nouveaux domaines tels que le Model Checking, la participation à des compétitions internationales étant également un moyen de gagner en visibilité. Enfin, le dernier objectif du Club est de trouver de nouveaux projets ou contrats pour le développement de fonctionnalités à long terme.

Nous remercions tous nos membres pour leur soutien et souhaitons la bienvenue à Mitsubishi Electric R&D Centre Europe qui rejoint AdaCore et le CEA List en tant que membre du Club cette année. Nous souhaitons également mettre en lumière l’équipe de développement Why3 avec laquelle nous travaillons pour améliorer nos outils.

Nos membres sont particulièrement intéressés par les points suivants :

– Une meilleure génération de modèles et de contre-exemples

– L’ajout de la théorie des séquences

– L’amélioration du support de l’arithmétique non linéaire dans Alt-Ergo

Ces fonctionnalités sont maintenant nos principales priorités. Pour suivre nos avancement et les nouveautés, n’hésitez pas à lire nos articles sur ce blog.

2019 chez OCamlPro

2020-02-05T13:56:36Z

OCamlPro a pour ambition d’aider les industriels dans leur adoption du langage OCaml et des méthodes formelles. L’entreprise est passée d’1 à 21 personnes et est restée fidèle à cet objectif. L’année 2019 chez OCamlPro a été très animée, et le nombre de réalisations impressionnant, d’abord dans le monde OCaml (flambda2 & optimisations du compilateur, opam 2, notre interface Rust pour memprof, des outils comme tryOCaml, ocp-indent, et le soutien à la OCaml Software Foundation), et dans le monde des méthodes formelles (nouvelles versions de notre solveur SMT Alt-Ergo, lancement du Club des utilisateurs Alt-Ergo,lancement du langage Love, etc.)

Lire la suite (en anglais)

2019 at OCamlPro

2020-02-04T13:56:36Z

OCamlPro was created to help OCaml and formal methods spread into the industry. We grew from 1 to 21 engineers, still strongly sharing this ambitious goal! The year 2019 at OCamlPro was very lively, with fantastic accomplishments all along!

Let's quickly review the past years' works, first in the world of OCaml (flambda2 & compiler optimisations, opam 2, our Rust-based UI for memprof, tools like tryOCaml, ocp-indent, and supporting the OCaml Software Foundation), then in the world of formal methods (new versions of our SMT Solver Alt-Ergo, launch of the Alt-Ergo Users' Club, the Love language, etc.).

In the World of OCaml

Flambda/Compilation Team

Work by Pierre Chambart, Vincent Laviron, Guillaume Bury and Pierrick Couderc

Pierre and Vincent's considerable work on Flambda 2 (the optimizing intermediate representation of the OCaml compiler – on which inlining occurs), in close cooperation with Jane Street (Mark Shinwell, Leo White and their team) aims at overcoming some of flambda's limitations. We have continued our work on making OCaml programs always faster: internal types are clearer, more concise, and possible control flow transformations are more flexible. Overall a precious outcome for industrial users. In 2019, the major breakthrough was to go from the initial prototype to a complete compiler, which allowed us to compile simple examples first and then to bootstrap it.

On the OCaml compiler side, we also worked with Leo on two new features: functorized compilation units and functorized packs, and recursive packs. The former will allow any developer to implement .ml files as if they were functors and not simply modules, and more importantly generate packs that are real functors. As such, this allows to split big functors into several files or to parameterize libraries on other modules. The latter allows two distinct usages: recursive interfaces, to implement recursive types into distinct .mlis as long as they do not need any implementation; and recursive packs, whose components are typed and compiled as recursive modules.

These new features are described on the new RFC repository for OCaml (a similar idea was suggested and implemented in 2011 by Fabrice Le Fessant).
The implementation is available on GitHub for both functorized packs and recursive packs. Be aware that both are based on an old version of OCaml for now, but should be in sync with the current trunk in the near future.
See also Vincent's OCamlPro’s compiler team work update of August 2019.

This work is allowed thanks to Jane Street's funding.

Work on a formalized type system for OCaml

Work of Pierrick Couderc

At the end of 2018, Pierrick defended his PhD on "Checking type inference results of the OCaml language", leading to a formalized type systems and semantics for a large subset of OCaml, or at least its unique typed intermediate language: the Typedtree. This work led us to work on new aspects of the OCaml compiler as recursive and functorized packs described earlier, and we hope this proves useful in the future for the evolution of the language.

The OPAM package manager

Work of Raja Boujbel and Louis Gesbert

OPAM is maintained and developed at OCamlPro by Louis and Raja. Thanks to their thorough efforts the opam 2.1 first release candidate is soon to be published!

Back in 2018, the long-awaited opam 2.0 version was finally released. It embedded many changes, in opam itself as well as for the community. The opam file format was redefined to simplify and add new features. With the close collaboration of OCamlLabs and opam repository maintainers, we were able to manage a smooth transition of the repository and whole ecosystem from opam 1.2 format to the new – and richer – opam 2.0 format. Other emblematic features are e.g. for practically integrated mccs solver, sandboxing builds, for security issues (we care about your filesystem!), for usability reworking of the pin' command, etc.

While the 2.1.0 version is in preparation, the 2.0.0 version is still updated with minor releases to fix issues. The lastest 2.0.6 release is fresh from January.

In the meantime, we continued to improve opam by integrating some opam plugins (opam lock, opam depext), recursively discover opam files in the file tree when pinning, new definition of a switch compiler, the possibility to use z3 backend instead of mccs, etc.

All these new features – among others – will be integrated in the 2.1.0 release, that is betaplanned for February. The best is yet to come!

More details: on https://opam.ocaml.org
Releases on Releases on https://github.com/ocaml/opam/releases & our blog

This work is allowed thanks to Jane Street's funding.

Encouraging OCaml adoption

OCaml Expert trainings for professional programmers

We proposed in 2019 some OCaml expert training specially designed for developers who want to use advanced features and master all the open-source tools and libraries of OCaml.

The "Expert" OCaml course is for already experienced OCaml programmers to better understand advanced type system possibilities (objects, GADTs), discover GC internals, write "compiler-optimizable" code. These sessions are also an opportunity to come discuss with our OPAM & Flambda lead developers and core contributors in Paris.

Next session: 3-4 March 2020, Paris (registration)

Our cheat-sheets on OCaml, the stdlib and opam

Work of Thomas Blanc, Raja Boujbel and Louis Gesbert

Thomas announced the release of our up-to-date cheat-sheets for the OCaml language, standard library and opam. Our original cheat-sheets were dating back to 2011. This was an opportunity to update them after the many changes in the language, library and ecosystem overall.

Cheat-sheets are helpful to refer to, as an overview of the documentation when you are programming, especially when you’re starting in a new language. They are meant to be printed and pinned on your wall, or to be kept in handy on a spare screen. They come in handy when your rubber duck is rubbish at debugging your code!

More details on Thomas' blog post

Open Source Tooling and Web IDEs

And let's not forget the other tools we develop and maintain! We have tools for education such as our interactive editor OCaml-top and Try-OCaml (from the previous work on the learn-OCaml platform for the OCaml Fun MOOC) which you can use to code in your browser. Developers will appreciate tools like our indentation tool ocp-indent, and ocp-index which gives you easy access to the interface information of installed OCaml libraries for editors like Emacs and Vim.

Supporting the OCaml Software Foundation

OCamlPro was proud to be one of the first supporters of the new Inria's OCaml Software Foundation. We keep committed to the adoption of OCaml as an industrial language:

"[…] As a long-standing supporter of the OCaml language, we have always been committed to helping spread OCaml's adoption and increase the accessibility of OCaml to beginners and students. […] We value close and friendly collaboration with the main actors of the OCaml community, and are proud to be contributing to the OCaml language and tooling." (August 2019, Advisory Board of the OCSF, ICFP Berlin)

More information on the OCaml Software Foundation

In the World of Formal Methods

By Mohamed Iguernlala, Albin Coquereau, Guillaume Bury

In 2018, we welcomed five new engineers with a background in formal methods. They consolidate the department of formal methods at OCamlPro, in particular help develop and maintain our SMT solver Alt-Ergo.

Release of Alt-Ergo 2.3.0, and version 2.0.0 (free)

After the release of Alt-Ergo 2.2.0 (with a new front-end that supports the SMT-LIB 2 language, extended prenex polymorphism, implemented as a standalone library) came the version 2.3.0 in 2019 with new features : dune support, ADT / algebraic datatypes, improvement of the if-then-else and let-in support, improvement of the data types.

More information on the Alt-Ergo SMT Solver
Albin Coquereau defended his PhD thesis in Decembre 2019 "Improving performance of the SMT solver Alt-Ergo with a better integration of efficient SAT solver"
We participated in the SMT-COMP 2019 during the 22nd SAT conference. The results of the competition are detailed here.

The launch of the Alt-Ergo Users' Club

Getting closer to our users, gathering industrial and academic supporters, collecting their needs into the Alt-Ergo roadmap is key to Alt-Ergo's development and sustainability.

The Alt-Ergo Users' Club was officially launched beginning of 2019. The first yearly meeting took place in February 2019. We were happy to welcome our first members Adacore, CEA List, Trust-In-Soft, and now Mitsubishi MERCE.

More information on the Alt-Ergo Users' Club

Harnessing our language-design expertise: Love

Work by David Declerck & Steven de Oliveira

Following the launch of Dune network, the Love language for smart-contracts was born from the collaboration of OCamlPro and Origin Labs. This new language, whose syntax is inspired from OCaml and Liquidity, is an alternative to the Dune native smart contract language Michelson. Love is based on system-F, a type system requiring no type inference and allowing polymorphism. The language has successfully been integrated on the network and the first smart contracts are being written.

LOVE: a New Smart Contract Language for the Dune Network The Love Smart Contract Language: Introduction & Key Features — Part I

Rust-related activities

The OCaml & Rust combo should be a candidate for any ambitious software project!

A Rust-based UI for memprof: we started in 2019 to work in collaboration with the memprof developer team on a Rust based UI for memprof. See Pierre and Albin's exposé at the JFLA2020's "Gardez votre mémoire fraiche avec Memthol" (Pierre Chambart , Albin Coquereau and Jacques-Henri Jourdan)
Rust training : Rust borrows heavily from functional programming languages to provide very expressive abstraction mechanisms. Because it is a systems language, these mechanisms are almost always zero-cost. For instance, polymorphic code has no runtime cost compared to a monomorphic version.This concern for efficiency also means that Rust lets developers keep a very high level of control and freedom for optimizations. Rust has no Garbage Collection or any form of runtime memory inspection to decide when to free, allocate or re-use memory. But because manual memory management is almost universally regarded as dangerous, or at least very difficult to maintain, the Rust compiler has a borrow-checker which is responsible for i) proving that the input program is memory-safe (and thread-safe), and ii) generating a safe and “optimal” allocation/deallocation strategy. All of this is done at compile-time.
Next sessions: April 20-24th 2020 (registration)

OCamlPro around the world

OCamlPro's team members attended many events throughout the world:

ICFP 2019 (Berlin)
The JFLA’2019 (Les Rousses, Haut-Jura)
The POSS'2019 (Paris)
MirageOS Retreat (Marrakech)

As a committed member of the OCaml ecosystem's animation, we've organized OCaml meetups too (see the famous OUPS meetups in Paris!).

Now let's jump into the new year 2020, with a team keeping expanding, and new projects ahead: keep posted!

Past projects: blockchain-related achievements (2018-beginning of 2019)

Many people ask us about what happened in 2018! That was an incredibly active year on blockchain-related achievements, and at that time we were hoping to attract clients that would be interested in our blockchain expertise.

But that is history now! Still interested? Check the Origin Labs team and their partner The Garage on Dune Network!

For the record:

(April 2019) We had started Techelson: a testing framework for Michelson and Liquidity
(Nov 2018) An Introduction to Tezos RPCs: Signing Operations / An Introduction to Tezos RPCs: a Basic Wallet / Liquidity Tutorial: A Game with an Oracle for Random Numbers / First Open-Source Release of TzScan
(Oct 2018) OCamlPro’s TZScan grant proposal accepted by the Tezos Foundation – joint press release
(Jul 2018) OCamlPro’s Tezos block explorer TzScan’s last updates
(Feb 2018) Release of a first version of TzScan.io, a Tezos block explorer / OCamlPro’s Liquidity-lang demo at JFLA2018 – a smart-contract design language . We were developing Liquidity, a high level smart contract language, human-readable, purely functional, statically-typed, which syntax was very close to the OCaml syntax.
To garner interest and adoption, we also developed the online editor Try Liquidity. Smart-contract developers could design contracts interactively, directly in the browser, compile them to Michelson, run them and deploy them on the alphanet network of Tezos. Future plans included a full-fledged web-based IDE for Liquidity. Worth mentioning was a neat feature: decompiling a Michelson program back to its Liquidity version, whether it was generated from Liquidity code or not.

opam 2.0.6 release

2020-01-16T13:56:36Z

We are pleased to announce the minor release of opam 2.0.6.

This new version contains some small backported fixes and build update:

Don't remove git cache objects that may be used [#3831 @AltGr]
Don't include .gitattributes in index.tar.gz [#3873 @dra27]
Update FAQ uri [#3941 @dra27]
Lock: add warning in case of missing locked file [#3939 @rjbou]
Directory tracking: fix cached entries retrieving with precise tracking [#4038 @hannesm]
Build:
- Add sanity checks [#3934 @dra27]
- Build man pages using dune [#3902 ]
- Add patch and bunzip check for make cold [#4006 @rjbou - fix #3842]
Shell:
- fish: add colon for fish manpath [#3886 @rjbou - fix #3878]
Sandbox:
- Add dune cache as rw [#4019 @rjbou - fix #4012]
- Do not fail if $HOME/.ccache is missing [#3957 @mseri]
opam-devel file: avoid copying extraneous files in opam-devel example [#3999 @maroneze]

As sandbox scripts have been updated, don't forget to run opam init --reinit -ni to update yours.

Note: To homogenise macOS name on system detection, we decided to keep macos, and convert darwin to macos in opam. For the moment, to not break jobs & CIs, we keep uploading darwin & macos binaries, but from the 2.1.0 release, only macos ones will be kept.

Installation instructions (unchanged):

From binaries: run

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.6"

From source, using opam:

opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

The Opam 2.0 cheatsheet, with a new theme!

2020-01-10T13:56:36Z

Earlier, we dusted-off our Language and Stdlib cheatsheets, for teachers and students. With more time, we managed to design an Opam 2.0 cheat-sheet we are proud of. It is organized into two pages:

The everyday average Opam use:
- Installation, Configuration, Switches, Allowed URL formats, Packages, Exploring, Package pinning, Working with local pins, Sharing a dev setup, Configuring remotes.
Peculiar advanced use cases (opam-managed project, publishing, repository maintenance, etc.):
- Package definition files, Some optional fields, Expressions, External dependencies, Publishing, Repository administration.

Moreover, with the help of listings, we tried the use of colors for better readability. And we left some blank space for your own peculiar commands. Two versions are available (PDF):

The Opam cheatsheet in black & white
The Opam cheatsheet in colour.

In any case do not hesitate to send us your suggestions on github:

Louis and Raja, the lead Opam developers, designed this cheatsheet so as to shed light on some important features (some I even discovered even though I speak daily with them!). If a command you find useful is not mentioned, let us know and we’ll add it. Feel free to ask for clarification and/or expansion of the manual!

Happy hacking!

Note: If you come to one of our training sessions, you’ll get a free cheatsheet! Isn’t that a bargain?

Des nouvelles de la part de l'équipe compilateur d'OCamlPro

2019-09-30T13:56:36Z

Nous sommes heureux de présenter certains travaux en cours sur le compilateur OCaml, travaux menés en étroite collaboration avec notre partenaire et client Janestreet.

Un travail conséquent a été fait pour aboutir à un nouveau framework d’optimisation du compilateur, appelé Flambda2, dont nous espérons qu’il corrigera certains défauts apparus dans Flambda. En parallèle, l’équipe a mené à bien certaines améliorations immédiates sur Flambda, ainsi que des modifications du compilateur qui seront utiles pour Flambda2.

Voir (en anglais) : OCamlPro’s compiler team work update

Formations OCaml par OCamlPro : 5-6 et 7-8 novembre 2019

2019-09-26T13:56:36Z

OCamlPro lance un cycle de formations régulières à OCaml, en français, dans ses locaux parisiens (métro Alésia). La première session aura lieu début novembre 2019, avec 2 formations:

Formation débutant : passer à OCaml (5-6 novembre)
Formation expert : approfondir sa maîtrise du langage (7-8 novembre).

La formation expert sera l’occasion pour des programmeurs OCaml ayant déjà une certaine expérience de mieux comprendre les possibilités avancées du typage (objets, GADTs), de découvrir en détail le fonctionnement du GC et d’écrire du code optimisable par le compilateur.

Ces formations sont aussi une occasion de venir discuter avec les lead développeurs et contributeurs d’OPAM et Flambda chez OCamlPro.

Des formations en anglais peuvent aussi être organisées sur demande à contact@ocamlpro.com

OCaml expert and beginner training by OCamlPro (in French): Nov. 5-6 & 7-8

2019-09-25T13:56:36Z

In our endeavour to encourage professional programmers to understand and use OCaml, OCamlPro will be giving two training sessions, in French, in our Paris offices:

OCaml Beginner course for professional programmers (5-6 Nov)
OCaml Expertise (7-8 Nov).

The "Expert" OCaml course is for already experienced OCaml programmers to better understand advanced type system possibilities (objects, GADTs), discover GC internals, write "compiler-optimizable" code.

These sessions are also an opportunity to come discuss with OCamlPro's OPAM & Flambda lead developers and core contributors in Paris.

Training in English can also be organized, on-demand.

This complements the excellent OCaml MOOC from Université Paris-Diderot and the learn-OCaml platform of the OCaml Software Foundation.

A look back on OCaml since 2011

2019-09-20T13:56:36Z

As you already know if you’ve read our last blogpost, we have updated our OCaml cheat sheets starting with the language and stdlib ones. We know some of you have students to initiate in September and we wanted these sheets to be ready for the start of the school year! We’re working on more sheets for OCaml tools like opam or Dune and important libraries such as ~~Obj~~ Lwt or Core. Keep an eye on our blog or the repo on GitHub to follow all the updates.

Going through the documentation was a journey to the past: we have looked back on 8 years of evolution of the OCaml language and library. New feature after new feature, OCaml has seen many changes. Needless to say, upgrading our cheat sheets to OCaml 4.08.1 was a trip down memory lane. We wanted to share our throwback experience with you!

2011

Fabrice Le Fessant first published our cheat sheets in 2011, the year OCamlPro was created! At the time, OCaml was in its 3.12 version and just got its current name agreed upon. First-class modules were the new big thing, Camlp4 and Camlp5 were battling for the control of the syntax extension world and Godi and Oasis were the packaging rage.

2012

Right after 3.12 came the switch to OCaml 4.00 which brought a major change: GADTs (generalized algebraic data types). Most of OCaml’s developers don’t use their almighty typing power, but the possibilities they provide are really helpful in some cases, most notably the format overhaul. They’re also a fun way to troll a beginner asking how to circumvent the typing system on Stack Overflow. Since most of us might lose track of their exact syntax, GADTs deserve their place in the updated sheet (if you happen to be OCamlPro’s CTO, of course the writer of this blogpost remembers how to use GADTs at all times).

On the standard library side, the big change was the switch of Hashtbl to Murmur 3 and the support for seeded randomization.

2013

With OCaml 4.01 came constructor disambiguation, but there isn’t really a way to add this to the sheet. This feature allows you to avoid misguided usage of polymorphic variants, but that’s a matter of personal taste (there’s a well-known rule that if you refresh the comments section enough times, someone —usually called Daniel— will appear to explain polymorphic variants’ superiority to you). -ppx rewriters were introduced in this version as well.

The standard library got a few new functions. Notably, Printexc.get_callstack for stack inspection, the optimized application operators |> and @@ and Format.asprintf.

2014

Gabriel Scherer, on the Caml-list, end of January:

TL;DR: During the six next months, we will follow pull requests (PR) posted on the github mirror of the OCaml distribution, as an alternative to the mantis bugtracker. This experiment hopes to attract more people to participate in the extremely helpful and surprisingly rewarding activity of patch reviews.

Can you guess which change to the cheat-sheets came with 4.02? It’s a universally-loved language feature added in 2014. Still don’t know? It is exceptional! Got it?

Drum roll… it is the match with exception construction! It made our codes simpler, clearer and in some cases more efficient. A message to people who want to improve the language: please aim for that.

This version also added the {quoted|foo|quoted} syntax (which broke comments), generative functors, attributes and extension nodes, extensible data types, module aliases and, of course, immutable strings (which was optional at the time). Immutable strings is the one feature that prompted us to remove a line from the cheat sheets. More space is good. Camlp4 and Labltk moved out of the distribution.

In consequence of immutable strings, Bytes and BytesLabel were added to the library. For the great pleasure of optimization addicts, raise_notrace popped up. Under the hood, the format type was re-implemented using GADTs.

2015

This release was so big that 4.02.2 feels like a release in itself, with the adding of nonrec and #... operators.

The standard library was spared by this bug-fix themed release. Note that this is the last comparatively slow year of OCaml as the transition to GitHub would soon make features multiply, as hindsight teaches us.

2016

Speaking of a major release, we’re up to OCaml 4.03! It introduced inline records, a GADT exhaustiveness check on steroids (with -> . to denote unreachability) and standard attributes like warning, inlined, unboxed or immediate. Colors appeared in the compiler and last but not least, it was the dawn of a new option called Flambda.

The library saw a lot of useful new functions coming in: lots of new iterators for Array, an equal function in most basic type modules, Uchar, the *_ascii alternatives and, of course, Ephemeron.

4.04 was much more restrained, but it was the second release in a single year. Local opening of module with the M.{} syntax was added along with the let exception ... in construct. String.split_on_char was notably added to the stdlib which means we don’t have to rewrite it anymore.

2017

We now get to 4.05… which did not change the language. Not that the development team wasn’t busy, OCaml just got better without any change to the syntax.

On the library side however, much happened, with the adding of *_opt functions pretty much everywhere. If you’re using the OCaml compiler from Debian, this is where you might think the story ends. You’d be wrong…

…because 4.06 added a lot! My own favorite feature from this release has to be user-defined indexing operators. This is also when safe-string became the default, giving worthwhile work to every late maintainer in the community. This release also added one awesome function in the standard library: Map.update.

2018

4.07 was aimed towards solidifying the language. It added empty variants and type-based selection of GADT constructors to the mix.

On the library side, one old and two new modules were added, with the integration of Bigarray, Seq and Float.

2019

And here we are with 4.08, in the present day! We can now put exceptions under or-patterns, which is the only language change from this release we propagated to the sheet. Time will tell if we need to add custom binding operators or [@@alert]. Pervasives is now deprecated in profit of Stdlib and new modules are popping up (Int, Bool, Fun, Result… did we miss one?) while Sort made its final deprecation warning.

We did not add 4.09 to this journey to the past, as this release is still solidly in the now at the time of this blogpost. Rest assured, we will see much more awesome features in OCaml in the future! In the meantime, we are working on updating more cheat sheets: keep posted!

Comments

Micheal Bacarella (23 September 2019 at 18 h 17 min):

For a blog-post from a company called OCaml PRO this seems like a rather tone-deaf PR action.

I wanted to read this and get hyped but instead I’m disappointed and I continue to feel like a chump advocating for this language.

Why? Because this is a rather underwhelming summary of 8 years of language activity. Perhaps you guys didn’t intend for this to hit the front of Hacker News, and maybe this stuff is really exciting to programming language PhDs, but I don’t see how the average business OCaml developer would relate to many of these changes at all. It makes OCaml (still!) seem like an out-of-touch academic language where the major complaints about the language are ignored (multicore, Windows support, programming-in-the-large, debugging) while ivory tower people fiddle with really nailing type-based selection in GADTs.

I expect INRIA not to care about the business community but aren’t you guys called OCaml PRO? I thought you liked money.

You clearly just intended this to be an interesting summary of changes to your cheatsheet but it’s turned into a PR release for the language and leaves normals with the continued impression that this language is a joke.

Thomas Blanc (24 September 2019 at 14 h 57 min):

Yes, latency can be frustrating even in the OCaml realm. Thanks for your comment, it is nice to see people caring about it and trying to remedy through contributions or comments.

Note that we only posted on discuss.ocaml.org expecting to get one or two comments. The reason for this post was that while updating the CS we were surprised to see how much the language had changed and decided to write about it.

You do raise some good points though. We did work on a full windows support back in the day. The project was discontinued because nobody was willing to buy it. We also worked on memory profiling for the debugging of memory leaks (before other alternatives existed). We did not maintain it because the project had no money input. I personally worked on compile-time detection of uncaught exception until the public funding of that project ran out. We also had a proposal for namespaces in the language that would have facilitated programming-in-the-large (no funding) and worked on multicore (funding for one man for one year).

Mise à jour des Cheat Sheets : OCaml Language et OCaml Standard Library

2019-09-14T13:56:36Z

Les mémentos (cheat-sheets) OCaml lang et OCaml stdlib partagés par OCamlPro en 2011 ont été mis à jour pour OCaml 4.08.

Si vous souhaitez contribuer des améliorations: sources sur GitHub.

En savoir plus : Updated Cheat Sheets: OCaml Language and OCaml Standard Library

Updated Cheat Sheets: OCaml Language and OCaml Standard Library

2019-09-13T13:56:36Z

In 2011, we shared several cheat sheets for OCaml. Cheat sheets are helpful to refer to, as an overview of the documentation when you are programming, especially when you’re starting in a new language. They are meant to be printed and pinned on your wall, or to be kept in handy on a spare screen. We hope they will help you out when your rubber duck is rubbish at debugging your code!

Since we first shared them, OCaml and its related tools have evolved. We decided to refresh them and started with the two most-used cheat sheets—our own contribution to the start of the school year!

Download the revised version:

OCaml Language (lang) (PDF)
OCaml Standard Library (stdlib) (PDF)

You can also find the sources on GitHub. We welcome contributions, feel free to send patches if you see room for improvement! We’re working on other cheat sheets: keep an eye on our blog to see updates and brand new cheat sheets.

While we were updating them, we realized how much OCaml had evolved in the last eight years. We’ll tell you everything about our trip down memory lane very soon in another blogpost!

OCamlPro’s compiler team work update

2019-08-30T13:56:36Z

The OCaml compiler team at OCamlPro is happy to present some of the work recently done jointly with JaneStreet's team.

A lot of work has been done towards a new framework for optimizations in the compiler, called Flambda2, aiming at solving the shortcomings that became apparent in the Flambda optimization framework (see below for more details). While that work is in progress, the team also worked on some more short-term improvements, notably on the current Flambda optimization framework, as well as some compiler modifications that will benefit Flambda2.

This work is funded by JaneStreet :D

Short-term improvements

Recursive values compilation

OCaml supports quite a large range of recursive definitions. In addition to recursive (and mutually-recursive) functions, one can also define regular values recursively, as for the infinite list let rec l = 0 :: l.

Not all recursive constructions are allowed, of course. For instance, the definition let rec x = x is rejected because there is no way to actually build a value that would behave correctly.

The basic rule for deciding whether a definition is allowed or not is made under the assumption that recursive values (except for functions, mostly) are compiled by first allocating space in the heap for the recursive values, binding the recursively defined variables to the allocated (but not yet initialized) values. The defining expressions are then evaluated, yielding new values (that can contain references the non-initialized values). Finally, the fields of these new values are copied one-by-one into the corresponding fields of the initial values.

For this approach to work, some restrictions need to apply:

the compiler needs to be able to compute the size of the values beforehand (these values must be allocated values, in order to avoid defining an integer recursively),
and since during the evaluation of the defining expressions their fields are not valid, one cannot write any code that may read these fields, like pattern-matching on the value, or passing the value to some function (or storing it in a mutable field of some record).

All of those restrictions have recently been reworked and formalized based on work from Alban Reynaud during an internship at Inria, reviewed and completed by Gabriel Scherer and Jeremy Yallop.

Unfortunately, this work only covers checking whether the recursive definitions are allowed or not; actual compilation is done later in the compiler, in one place for bytecode and another for native code, and these pieces of code have not been linked with the new check so there have been a few cases where the check allowed code that wasn't actually compiled correctly.

Since we didn't want to deal with it directly in our new version of Flambda, we had started working on a patch to move the compilation of recursive values up in the compilation pipeline, before the split between bytecode and native code. After some amount of hacking (we discovered that compilation of classes creates recursive value bindings that would not pass the earlier recursive check…), we have a patch that is mostly ready for review and will soon start engaging with the rest of the compiler team with the aim of integrating it into the compiler.

Separate compilation of recursive modules, compilation units as functors

Some OCaml developers like to encapsulate each type definition in its own module, with an interface that can expose the needed types and functions, while abstracting away as much of the actual implementation as possible. It is then common to have each of these modules in its own file, to simplify management and avoid unseemly big files.

However, this breaks down when one needs to define several types that depend on each other. The usual solutions are either to use recursive modules, which have the drawback of requiring all the modules to be in the same compilation unit, leading to very big files (we have seen a real case of a more than 10,000-lines file), or make each module parametric in the other modules, translating them into functors, and then instantiate all the functors when building the outwards-facing interface.

To address these issues, we have been working on two main patches to improve the life of developers facing these problems.

The first one allows compiling several different files as mutually recursive modules, reusing the approach used to compile regular recursive modules. In practice, this will allow developers using recursive modules extensively to properly separate not only the different modules from each other, but also the implementation and interfaces into a .ml and .mli files. This would of course need some additional support from the different build tools, but we're confident we can get at least dune to support the feature.

The second one allows compiling a single compilation unit as a functor instead of a regular module. The arguments of the functor would be specified on the command line, their signature taken from their corresponding interface file. This can be useful not only to break recursive dependencies, like the previous patch (though in a different way), but also to help developers relying on multiple implementations of a same .mli interface functorize their code with minimal effort.

These two improvements will also benefit packs, whereas recursive compilation units could be packed in a single module and packs could be functorized themselves.

Small improvements to Flambda

We are still committed to maintain the Flambda part of the compiler. Few bugs have been found, so we concentrate our efforts on small features that either yield overall performance gains or allow naive code patterns to be compiled as efficiently as their equivalent but hand-optimized versions.

As an example, one optimization that we should be able to submit soon looks for cases where an immutable block is allocated but an immutable block with the same exact fields and tag already exists.

This can be demonstrated with the following example:

let result_bind f = function
  | Ok x -> f x
  | Error e -> Error e

The usual way to avoid the extra allocation of Error e is to write the clause as | (Error e) as r -> r. With this new patch, the redundant allocation will be detected and removed automatically! This can be even more interesting with inlining:

let my_f x =
  if (* some condition *)
  then Ok x
  else (* something else *)

let _ =
  (* ... *)
  let r = result_bind my_f (* some argument *) in
  (* ... *)

In this example, inlining result_bind then my_f can match the allocation Ok x in my_f with the pattern matching in result_bind. This removes an allocation that would be very hard to remove otherwise. We expect these patterns to occur quite often with some programming styles relying on a great deal of abstraction and small independent functions.

Flambda 2.0

We are building on the work done for Flambda and the experience of its users to develop Flambda 2.0, the next optimization framework.

Our goal is to build a framework for analyzing the costs and benefits of code transformations. The framework focuses on reducing the runtime cost of abstractions and removing as many short-lived allocations as possible.

The aim of Flambda 2.0 is roughly the same as the original Flambda. So why did we decide to write a new framework instead of patching the existing one? Several points led us to this decision.

An invariant on the representation of closures that ensured that every closure had a unique identifier, which was convenient for a number of reasons, turned out to be quite expensive to maintain and prevented some optimizations.
The internal representation of Flambda terms included too many different cases that were either redundant or not relevant to the optimizations we were interested in, making a lot of code more complicated than necessary.
The ANF-like representation we used was not perfect. We wanted an easier way to do control flow optimizations, which led us to choose a CPS-like representation for Flambda 2.0.
Finally, the original Flambda was thought of as an alternative to the closure conversion and inlining algorithms performed by the Closure module of the compiler, translating from the Lambda representation to Clambda. However, a number of optimizations (most importantly unboxing) are done during the next phase of compilation, Cmmgen, which translates to the Cmm representation. The original Flambda had trouble to estimate correctly which optimizations would trigger and what would their benefit be. It may be noted that correctly estimating benefit is a key in Flambda's algorithms, and we know of a number of cases where Flambda is not as good as it could be because it couldn't predict the unboxing opportunities that inlining would have allowed. Flambda 2.0 will go from Lambda to Cmm, and will handle all transformations done in both Closure and Cmmgen in a single framework.

These improvements are still very much a work in progress. We have not reached the point where other developers can try out the new framework on their codebases yet.

This does not mean there are no news to enjoy before our efforts show on the mainstream compiler! While working on Flambda 2.0, we did deploy a number of patches on the compiler both before and after the Flambda stage. We proposed all the changes independant enough to be proposed on their own. Some of these fixes have been merged already. Others are still under discussion and some, like the recursive values patch mentioned above, are still waiting for cleanup or documentation before submission.

Comments

Jon Harrop (30 August 2019 at 20 h 11 min):

What is the status of multicore OCaml?

Vincent Laviron (2 September 2019 at 16 h 22 min):

OCamlPro is not working on multicore OCaml. It is still being worked on elsewhere, with efforts concentrated around OCaml Labs, but I don’t have more information than what is publicly available. All of the work we described here is not expected to interfere with multicore.

Lindsay (25 September 2020 at 20 h 20 min):

Thanks for your continued work on the compiler and tooling! Am curious if there is any news regarding the item “Separate compilation of recursive modules”.

Release d’opam 2.0.5

2019-07-23T13:56:36Z

Nous sommes fiers d’annoncer la release (mineure) d’ opam 2.0.5. Cette nouvelle version contient des mises à jours de build et correctifs.

Plus d’information

opam 2.0.5 release

2019-07-11T13:56:36Z

We are pleased to announce the minor release of opam 2.0.5.

This new version contains build update and small fixes:

Bump src_ext Dune to 1.6.3, allows compilation with OCaml 4.08.0. [#3887 @dra27]
Support Dune 1.7.0 and later [#3888 @dra27 - fix #3870]
Bump the ocaml_mccs lib-ext, to include latest changes [#3896 @AltGr]
Fix cppo detection in configure [#3917 @dra27]
Read jobs variable from OpamStateConfig [#3916 @dra27]
Linting:
- add check upstream option [#3758 @rjbou]
- add warning for with-test in run-test field [#3765, #3860 @rjbou]
- fix misleading doc filter warning [#3871 @rjbou]
Fix typos [#3891 @dra27, @mehdid]

Note: To homogenise macOS name on system detection, we decided to keep macos, and convert darwin to macos in opam. For the moment, to not break jobs & CIs, we keep uploading darwin & macos binaries, but from the 2.1.0 release, only macos ones will be kept.

Installation instructions (unchanged):

From binaries: run

sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

From source, using opam:

opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

Résultats de la SMT-Comp 2019 pour Alt-Ergo

2019-07-10T13:56:36Z

Les résultats de la compétition SMT-COMP 2019 ont été publiés au whorkshop SMT de la 22e conférence SAT. Nous étions fiers d’y participer pour la deuxième année consécutive, surtout depuis qu’Alt-Ergo prend en charge le standard SMT-LIB 2.

Alt-Ergo est un SAT solveur open-source maintenu et distribué par OCamlPro, et financé entre autres grâce à plusieurs projets de R&D collaborative (BWare, SOPRANO, Vocal, LChip).

Si vous êtes un utilisateur d’Alt-Ergo, songez à rejoindre le Club des Utilisateurs d’Alt-Ergo! L’histoire de ce logiciel remonte à 2006, où il est né de recherches académiques conjointes entre Inria et le CNRS dans le laboratoire du LRI. Il est depuis septembre 2013 maintenu, développé & et distribué par OCamlPro (voir l’historique des versions passées).

Si vous êtes curieux des activités d’OCamlPro dans le domaine des méthodes formelles, vous pouvez lire le court témoignage d’un client heureux

Voir /blog/alt-ergo-participation-to-the-smt-comp-2019

The Alt-Ergo SMT Solver’s results in the SMT-COMP 2019

2019-07-09T13:56:36Z

The results of the SMT-COMP 2019 were released a few days ago at the SMT whorkshop during the 22nd SAT conference. We were glad to participate in this competition for the second year in a row, especially as Alt-Ergo now supports the SMT-LIB 2 standard.

Alt-Ergo is an open-source SAT-solver maintained and distributed by OCamlPro and partially funded by R&D projects. If you’re interested, please consider joining the Alt-Ergo User’s Club! Its history goes back in 2006 from early academic researches conducted conjointly at Inria & CNRS “LRI” lab, and the maintenance and development work by OCamlPro since September 2013 (see the past releases).

If you’re curious about OCamlPro’s other activities in Formal Methods, see a happy client’s feedback

SMT-COMP 2018

Our goal last year was to challenge ourselves on the community benchmarks. We wanted to compare Alt-Ergo to state-of-the-art SMT solvers. We thus selected categories close to the “deductive program verification”, as Alt-Ergo is primarily tuned for formulas coming from this application domain. Specifically, we took part in four main tracks categories: ALIA, AUFLIA, AUFLIRA, AUFNIRA. These categories are a combination of theories such as Arrays, Uninterpreted Function and Linear and Non-linear arithmetic over Integers and Reals.

Alt-Ergo’s Results at SMT-COMP 2018

For its first participation in SMT-COMP, Alt-Ergo showed that it was a competitive solver comparing to state of the art solvers such as CVC4, Vampire, VeriT or Z3.

Main Track Categories (number of participants)	Sequential Perfs	Parallel Perfs
ALIA (4)
AUFLIA (4)
AUFLIRA (4)
AUFNIRA (3)

The global results of the competition are available here.

SMT-COMP 2019

Since last year’s competition, we made some improvements on Alt-Ergo, specifically over our data structures and the support of algebraic datatypes (see post).

A few changes can be noted for this year’s competition:

A distinction between SAT and UNSAT in the scoring scheme allowed us to compete in more categories, as Alt-Ergo doesn’t send back SAT.
The aim of the 24s Scoring is to reward solvers which solve problems quickly.
The number of benchmarks in each category has changed. For each category, only the benchmarks which were not proven by every solver last year are used. For example: in the division AUFLIRA, 20011 benchmarks were used last year, of which 1683 remained this year.

Alt-Ergo only competed in the Single Query Track. We selected the same categories as last year and added UF, UFLIA, UFLRA and UFNIA. We also decided to compete over categories supporting algebraic DataTypes to test our newly support of this theory. Alt-Ergo’s expertise is over quantified problems, but we wanted to test our hand in the solver theories over some Quantifier-free categories.

Alt-Ergo’s Results at SMT-COMP 2019

We were proud to see Alt-Ergo performs within a reasonable margin on Quantifier Free problems comparing to other solvers over the UNSAT problems, even though these problems are not our solver’s primary goal. And we were happy with the performance of our solver in Datatype categories, as the support of this theory is new.

For the last categories, Alt-Ergo managed to reproduce last year’s performance, close to CVC4 (2018 and 2019 winner) and Vampire.

Single Query Categories (number of participants)	Sequential	Parallel	Unsat	24s
ALIA (8)
AUFLIA (8)
AUFLIRA (8)
AUFNIRA (5)
UF (8)
UFLIA (8)
UFLRA (8)
UFNIA (8)

This year results are available here. These results do not include Par4 a portfolio solver.

Alt-Ergo is constantly evolving, as well as our support of the SMT-LIB standard. For next year’s participation, we will try to compete in more categories and hope to cover more tracks, such as the UNSAT-Core track.

Blockchains @ OCamlPro: an Overview

2019-04-29T13:56:36Z

OCamlPro started working on blockchains in 2014, when Arthur Breitman came to us with an initial idea to develop the Tezos ledger. The idea was very challenging with a lot of innovations. So, we collaborated with him to write a specification, and to turn the specification into OCaml code. Since then, we continually improved our skills in this domain, trained more engineers, introduced the technology to students and to professionals, advised a dozen projects, developed tools and libraries, made some improvements and extensions to the official Tezos node, and conducted several private deployments of the Tezos ledger.

For an overview of OCamlPro’s blockchain activities see here

TzScan: A complete Block Explorer for Tezos

TzScan is considered today to be the best block explorer for Tezos. It’s made of three main components:

an indexer that queries the Tezos node and fills a relational database,
an API server that queries the database to retrieve various informations,
a web based user interface (a Javascript application)

We deployed the indexer and API to freely provide the community with an access to all the content of the Tezos blockchain, already used by many websites, wallets and apps. In addition, we directly use this API within our TzScan.io instance. Our deployment spans on multiple Tezos nodes, multiple API servers and a distributed database to scale and reply to millions of queries per day. We also regularly release open source versions under the GPL license, that can be easily deployed on private Tezos networks. TzScan’s development has been initiated in September 2017. It represents today an enormous investment, that the Tezos Foundation helped partially fund in July 2018.

Contact us for support, advanced features, advertisement, or if you need a private deployment of the TzScan infrastructure.

Liquidity: a Smart Contract Language for Tezos

Liquidity is the first high-level language for Tezos over Michelson. Its development began in April 2017, a few months before the Tezos fundraising in July 2017. It is today the most advanced language for Tezos: it offers OCaml-like and ReasonML-like syntaxes for writing smart contracts, compilation and de-compilation to/from Michelson, multiple-entry points, static type-checking à la ML, etc. Its online editor allows to develop smart contracts and to deploy them directly into the alphanet or mainnet. Liquidity has been used before the mainnet launch to de-compile the Foundation’s vesting smart contracts in order to review them. This smart contract language represents more than two years of work, and is fully funded by OCamlPro. It has been developed with formal verification in mind, formal verification being one of the selling points of Tezos. We have elaborated a detailed roadmap mixing model-checking and deductive program verification to investigate this feature. We are now searching for funding opportunities to keep developing and maintaining Liquidity.

See our online editor to get started ! Contact us if you need support, training, writing or in-depth analysis of your smart contracts.

Techelson: a testing framework for Michelson and Liquidity

Techelson is our newborn in the set of tools for the Tezos blockchain. It is a test execution engine for the functional properties of Michelson and Liquidity contracts. Techelson is still in its early development stage. The user documentation is available here. An example on how to use it with Liquidity is detailed in this post.

Contact us to customize the engine to suit your own needs!

IronTez: an optimized Tezos node by OCamlPro

IronTez is a tailored node for private (and public) deployments of Tezos. Among its additional features, the node adds some useful RPCs, improves storage, enables garbage collection and context pruning, allows an easy configuration of the private network, provides additional Michelson instructions (GET_STORAGE, CATCH…). One of its nice features is the ability to enable adaptive baking in private / proof-of-authority setting (eg. baking every 5 seconds in presence of transactions and every 10 minutes otherwise, etc.).

A simplified version of IronTez has already been made public to allow testing its improved storage system, Ironmin, showing a 10x reduction in storage. Some TzScan.io nodes are also using versions of IronTez. We’ve also successfully deployed it along with TzScan for a big foreign company to experiment with private blockchains. We are searching for projects and funding opportunities to keep developing and maintaining this optimized version of the Tezos node.

Don’t hesitate to contact us if you want to deploy a blockchain with IronTez, or for more information !

Comments

Kristen (3 May 2019 at 0 h 30 min):

I really wanted to keep using IronTez but I ran into bugs that have not yet been fixed, the code is out of date with upstream, and there is no real avenue for support/assistance other than email.

opam 2.0.4 release

2019-04-10T13:56:36Z

We are pleased to announce the release of opam 2.0.4.

This new version contains some backported fixes:

Sandboxing on macOS: considering the possibility that TMPDIR is unset [#3597 @herbelin - fix #3576]
display: Fix opam config var display, aligned on opam config list [#3723 @rjbou - rel. #3717]
pin:
- update source of (version) pinned directory [#3726 @rjbou - #3651]
- fix --ignore-pin-depends with autopin [#3736 @AltGr]
- fix pinnings not installing/upgrading already pinned packages (introduced in 2.0.2) [#3800 @AltGr]
opam clean: Ignore errors trying to remove directories [#3732 @kit-ty-kate]
remove wrong "mismatched extra-files" warning [#3744 @rjbou]
urls: fix hg opam 1.2 url parsing [#3754 @rjbou]
lint: update message of warning 47, to avoid confusion because of missing synopsis field internally inferred from descr [#3753 @rjbou - fix #3738]
system:
- lock & signals: don't interrupt at non terminal signals [#3541 @rjbou]
- shell: fix fish manpath setting [#3728 @gregory-nisbet]
- git: use diff.noprefix=false config argument to overwrite user defined configuration [#3788 @rjbou, #3628 @Blaisorblade - fix #3627]
dirtrack: fix precise tracking mode [#3796 @rjbou]
fix some mispellings [#3731 @MisterDA]
CI enhancement & fixes [#3706 @dra27, #3748 @rjbou, #3801 @rjbou]

Note: To homogenise macOS name on system detection, we decided to keep macos, and convert darwin to macos in opam. For the moment, to not break jobs & CIs, we keep uploading darwin & macos binaries, but from the 2.1.0 release, only macos ones will be kept.

Installation instructions (unchanged):

From binaries: run

sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

From source, using opam:

opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

opam 2.0 tips

2019-03-12T13:56:36Z

This blog post looks back on some of the improvements in opam 2.0, and gives tips on the new workflows available.

Package development environment management

Opam 2.0 has been vastly improved to handle locally defined packages. Assuming you have a project ~/projects/foo, defining two packages foo-lib and foo-bin, you would have:

~/projects/foo
|-- foo-lib.opam
|-- foo-bin.opam
`-- src/ ...

(See also about computed dependency constraints for handling multiple package definitions with mutual constraints)

Automatic pinning

The underlying mechanism is the same, but this is an interface improvement that replaces most of the opam 1.2 workflows based on opam pin.

The usual commands (install, upgrade, remove, etc.) have been extended to support specifying a directory as argument. So when working on project foo, just write:

cd ~/projects/foo
opam install .

and both foo-lib and foo-bin will get automatically pinned to the current directory (using git if your project is versioned), and installed. You may prefer to use:

opam install . --deps-only

to just get the package dependencies ready before you start hacking on it. See below for details on how to reproduce a build environment more precisely. Note that opam depext . will not work at the moment, which will be fixed in the next release when the external dependency handling is integrated (opam will still list you the proper packages to install for your OS upon failure).

If your project is versioned and you made changes, remember to either commit, or add --working-dir so that your uncommitted changes are taken into account.

Local switches

Opam 2.0 introduced a new feature called "local switches". This section explains what it is about, why, when and how to use them.

Opam switches allow to maintain several separate development environments, each with its own set of packages installed. This is particularly useful when you need different OCaml versions, or for working on projects with different dependency sets.

It can sometimes become tedious, though, to manage, or remember what switch to use with what project. Here is where "local switches" come in handy.

How local switches are handled

A local switch is simply stored inside a _opam/ directory, and will be selected automatically by opam whenever your current directory is below its parent directory.

NOTE: it's highly recommended that you enable the new shell hooks when using local switches. Just run opam init --enable-shell-hook: this will make sure your PATH is always set for the proper switch.

You will otherwise need to keep remembering to run eval $(opam env) every time you cd to a directory containing a local switch. See also how to display the current switch in your prompt

For example, if you have ~/projects/foo/_opam, the switch will be selected whenever in project foo, allowing you to tailor what it has installed for the needs of your project.

If you remove the switch dir, or your whole project, opam will forget about it transparently. Be careful not to move it around, though, as some packages still contain hardcoded paths and don't handle relocation well (we're working on that).

Creating a local switch

This can generally start with:

cd ~/projects/foo
opam switch create . --deps-only

Local switch handles are just their path, instead of a raw name. Additionally, the above will detect package definitions present in ~/projects/foo, pick a compatible version of OCaml (if you didn't explicitely mention any), and automatically install all the local package dependencies.

Without --deps-only, the packages themselves would also get installed in the local switch.

Using an existing switch

If you just want an already existing switch to be selected automatically, without recompiling one for each project, you can use opam switch link:

cd ~/projects/bar
opam switch link 4.07.1

will make sure that switch 4.07.1 is chosen whenever you are in project bar. You could even link to ../foo here, to share foo's local switch between the two projects.

Reproducing build environments

Pinnings

If your package depends on development versions of some dependencies (e.g. you had to push a fix upstream), add to your opam file:

depends: [ "some-package" ] # Remember that pin-depends are depends too
pin-depends: [
  [ "some-package.version" "git+https://gitfoo.com/blob.git#mybranch" ]
]

This will have no effect when your package is published in a repository, but when it gets pinned to its dev version, opam will first make sure to pin some-package to the given URL.

Lock-files

Dependency contraints are sometimes too wide, and you don't want to explore all the versions of your dependencies while developing. For this reason, you may want to reproduce a known-working set of dependencies. If you use:

opam lock .

opam will check what version of the dependencies are installed in your current switch, and explicit them in *.opam.locked files. opam lock is a plugin at the moment, but will get automatically installed when needed.

Then, assuming you checked these files into version control, any user can do

opam install . --deps-only --locked

to instruct opam to reproduce the same build environment (the --locked option is also available to opam switch create, to make things easier).

The generated lock-files will also contain added constraints to reproduce the presence/absence of optional dependencies, and reproduce the appropriate dependency pins using pin-depends. Add the --direct-only option if you don't want to enforce the versions of all recursive dependencies, but only direct ones.

Release : Liquidity version 1.0 !

2019-03-09T13:56:36Z

Nous sommes fiers d'annoncer la release de la première version majeure de Liquidity, le langage de smart contracts et son outillage. Parmi les fonctions phares : multiples points d'entrée, système de contrats modulaire, polymorphisme et inférence de type, syntaxe ReasonML pour une plus grande adoption, etc.

Voir cet article !

Announcing Liquidity version 1.0

2019-03-08T13:56:36Z

Liquidity version 1.0

We are pleased to announce the release of the first major version of the Liquidity smart-contract language and associated tools.

Some of the highlights of this version are detailed below.

Multiple Entry Points

In the previous versions of Liquidity, smart contracts were limited to a single entry point (named main). But traditionally smart contracts executions path depend strongly on the parameter and in most cases they are completely distinct.

Having different entry points allows to separate code that do not overlap and which usually accomplish vastly different tasks. Encoding entry points with complex pattern matching constructs before was tedious and made the code not extremely readable. This new feature gives you readability and allows to call contracts in a natural way.

Internally, entry points are encoded with sum types and pattern matching so that you keep the strong typing guarantees that come over from Michelson. This means that you cannot call a typed smart contract with the wrong entry point or the wrong parameter (this is enforced statically by both the Liquidity typechecker and the Michelson typechecker).

Modules and Contract System

Organizing, encapsulating and sharing code is not always easy when you need to write thousand lines files. Liquidity now allows to write modules (which contain types and values/functions) and contracts (which define entry points in addition). Types and non-private values of contracts and modules in scope can be accessed by other modules and contracts.

You can even compile several files at once with the command line compiler, so that you may organize your multiple smart contract projects in libraries and files.

Polymorphism and Type Inference

Thanks to a new and powerful type inference algorithm, you can now get rid of almost all type annotations in the smart contracts.

Instead of writing something like

let%entry main (parameter : bool) (storage : int) =
  let ops = ([] : operation list) in
  let f (c : bool) = if not c then 1 else 2 in
  ops, f parameter

you can now write

let%entry main parameter _ =
  let ops = [] in
  let f c = if not c then 1 else 2 in
  ops, f parameter

And type inference works with polymorhpism (also a new feature of this release) so you can now write generic and reusable functions:

type 'a t = { x : 'a set; y : 'a }

let mem_t v = Set.mem v.y v.x

Inference also works with contract types and entry points.

ReasonML Syntax

We originally used a modified version of the OCaml syntax for the Liquidity language. This made the language accessible, almost for free, to all OCaml and functional language developers. The typing discipline one needs is quite similar to other strongly typed functional languages so this was a natural fit.

However this is not the best fit for everyone. We want to bring the power of Liquidity and Tezos to the masses so adopting a seemingly familiar syntax for most people can help a lot. With this new version of Liquidity, you can now write your smart contracts in both an OCaml-like syntax or a ReasonML-like one. The latter being a lot closer to Javascript on the surface, making it accessible to people that already know the language or people that write smart contracts for other platforms like Solidity/Ethereum.

You can see the full changelog as well as download the latest release and binaries at this address.

Don't forget that you can also try all these new cool features and more directly in your browser with our online editor.

Release de Techelson, moteur de tests pour Michelson et Liquidity

2019-03-07T13:56:36Z

Nous sommes fiers d’annoncer la première release de Techelson, moteur d’exécution de tests pour Michelson. Les programmeurs Liquidity peuvent également l’utiliser.

Voir Techelson, a test execution engine for Michelson.

Techelson, a test execution engine for Michelson

2019-03-06T13:56:36Z

We are pleased to announce the first release of Techelson, available here.

Techelson is a Test Execution Engine for Michelson. It aims at testing functional properties of Michelson smart contracts. Make sure to check the user documentation to get a sense of Techelson's workflow and features.

For Liquidity programmers interested in Techelson, take a look at this blog post discussing how to write tests in Liquidity and run them using Techelson.

Techelson is still young: if you have problems, suggestions or feature requests please open an issue on the repository.

Signing Data for Smart Contracts

2019-03-05T13:56:36Z

Smart contracts calls already provide a built-in authentication mechanism as transactions (i.e. call operations) are cryptographically signed by the sender of the transaction. This is a guarantee on which programs can rely.

However, sometimes you may want more involved or flexible authentication schemes. The ones that rely on signature validity checking can be implemented in Michelson, and Liquidity provide a built-in instruction to do so. (You still need to keep in mind that you cannot store unencrypted confidential information on the blockchain).

This instruction is Crypto.check in Liquidity. Its type can be written as:

Crypto.check: key -> signature -> bytes -> bool

Which means that it takes as arguments a public key, a signature and a sequence of bytes and returns a Boolean. Crypto.check pub_key signature message is true if and only if the signature signature was obtained by signing the Blake2b hash of message using the private key corresponding to the public key pub_key.

A small smart contract snippet which implements a signature check (against a predefined public key kept in the smart contract's storage) can be tested online here.

type storage = key

let%entry main ((message : string), (signature : signature)) key =
  let bytes = Bytes.pack message in
  if not (Crypto.check key signature bytes) then
    failwith "Wrong signature";
  ([] : operation list), key

This smart contract fails if the string message was not signed with the private key corresponding to the public key key stored. Otherwise it does nothing.

This signature scheme is more flexible than the default transaction/sender one, however it requires that the signature can be built outside of the smart contract. (And more generally outside of the toolset provided by Liquidity and Tezos). On the other hand, signing a transaction is something you get for free if you use the tezos client or any tezos wallet (as is it essentially their base function).

The rest of this blog post will focus on various ways to sign data, and on getting signatures that can be used in Tezos and Liquidity directly.

Signing Using the Tezos Client

One (straightforward) way to sign data is to use the Tezos client directly. You will need to be connected to a Tezos node though as the client makes RPCs to serialize data (this operation is protocol dependent). We can only sign sequences of bytes, so the first thing we need to do is to serialize whichever data we want to sign. This can be done with the command hash data of the client.

$ ./tezos-client -A alphanet-node.tzscan.io -P 80 hash data '"message"' of type string
Raw packed data:
  0x0501000000076d657373616765
Hash:
  exprtXaZciTDGatZkoFEjE1GWPqbJ7FtqAWmmH36doxBreKr6ADcYs
Raw Blake2b hash:
  0x01978930fd2d04d0db8c2e4ef8a3f5d63b8e732177c8723135ed0dc7d99ebed3
Raw Sha256 hash:
  0x32569319f6517036949bcead23a761bfbfcbf4277b010355884a86ba09349839
Raw Sha512 hash:
  0xdfa4ea9f77db3a98654f101be1d33d56898df40acf7c2950ca6f742140668a67fefbefb22b592344922e1f66c381fa2bec48aa47970025c7e61e35d939ae3ca0
Gas remaining: 399918 units remaining

This command gives the result of hashing the data using various algorithms but what we're really interested in is the first item Raw packed data which is the serialized version of our data ("message") : 0x0501000000076d657373616765.

We can now sign these bytes using the Tezos client as well. This step can be performed completely offline, for that we need to use the option -p of the client to specify the protocol we want to use (the sign bytes command will not be available without first selecting a valid protocol). Here we use protocol 3, designated by its hash PsddFKi3.

$ ./tezos-client -p PsddFKi3 sign bytes 0x0501000000076d657373616765 for my_account
Signature:
  edsigto9QHtXMyxFPyvaffRfFCrifkw2n5ZWqMxhGRzieksTo8AQAFgUjx7WRwqGPh4rXTBGGLpdmhskAaEauMrtM82T3tuxoi8

The account my_account can be any imported account in the Tezos client. In particular, it can be an encrypted key pair (you will need to enter a password to sign) or a hardware Ledger (you will need to confirm the signature on the Ledger). The obtained signature can be used as is with Liquidity or Michelson. This one starts with edsig because it was obtained using an Ed25519 private key, but you can also get signatures starting with spsig1 or p2sig depending on the cryptographic curve that you use.

Signing Manually

In this second section we detail the necessary steps and provide a Python script to sign string messages using an Ed25519 private key. This can be easily adapted for other signing schemes.

These are the steps that will need to be performed in order to sign a string:

Assuming that the value you want to sign is a string, you first need to convert its ASCII version to hexa, for the string "message" that is 6d657373616765.
You need to produce the packed version of the corresponding Michelson expression. The binary representation can vary depending on the types of the values you want to pack but for strings it is:

| 0x | 0501 | [size of the string on 4 bytes] | [ascii string in hexa] |

for "message" (of length 7), it is

| 0x | 0501 | 00000007 | 6d657373616765 |

or 0x0501000000076d657373616765.

Hash this value using Blake2b (01978930fd2d04d0db8c2e4ef8a3f5d63b8e732177c8723135ed0dc7d99ebed3) which is 32 bytes long.
Depending on your public key, you then need to sign it with the corresponding curve (ed25519 for edpk keys), the signature is 64 bytes:

753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

Optionally convert to base58check. This is not needed because Liquidity and Michelson allow signatures (as well as keys and key hashes) to be given in hex format with a 0x:

0x753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

The following Python (3) script will do exactly this, entirely offline. Note that this is just an toy example, and should not be used in production. In particular you need to give your private key on the command line so this might not be secure if the machine you run this on is not secure.

$ pip3 install base58check pyblake2 ed25519
> python3 ./sign_string.py "message" edsk2gL9deG8idefWJJWNNtKXeszWR4FrEdNFM5622t1PkzH66oH3r
0x753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

`sign_string.py`

from pyblake2 import blake2b
import base58check
import ed25519
import sys

message = sys.argv[1]
seed_b58 = sys.argv[2]

prefix = b'x05x01'
len_bytes = (len(message)).to_bytes(4, byteorder='big')
h = blake2b(digest_size=32)
b = bytearray()
b.extend(message.encode())
h.update(prefix + len_bytes + b)
digest = h.digest()

seed = base58check.b58decode(seed_b58)[4:-4]
sk = ed25519.SigningKey(seed)
sig = sk.sign(digest)
print("0x" + sig.hex())

What's new for Alt-Ergo in 2018? Here is a recap!

2019-02-11T13:56:36Z

After the hard work done on the integration of floating-point arithmetic reasoning two years ago, 2018 is the year of polymorphic SMT2 support and efficient SAT solving for Alt-Ergo. In this post, we recap the main novelties last year, and we announce the first Alt-Ergo Users’ Club meeting.

An SMT2 front-end with prenex polymorphism

As you may know, Alt-Ergo’s native input language is not compliant with the SMT-LIB 2 input language standard, and translating formulas from SMT-LIB 2 to Alt-Ergo’ syntax (or vice-versa) is not immediate. Besides its extension with polymorphism, this native language diverges from SMT-LIB’s by distinguishing terms of type boolean from formulas (that are propositions). This distinction makes it hard, for instance, to efficiently translate let-in and if-then-else constructs that are ubiquitous in SMT-LIB 2 benchmarks.

In order to work closely with the SMT community, we designed a conservative extension of the SMT-LIB 2 standard with prenex polymorphism and implemented it as a new frontend in Alt-Ergo 2.2. This work has been published in the 2018 edition of the SMT-Workshop. An online version of the paper is available here. Experimental results showed that polymorphism is really important for Alt-Ergo, as it allows to improve both resolution rate and resolution time (see Figure 5 in the paper for more details).

Improved SAT solvers

We also worked on improving SAT-solving in Alt-Ergo last year. The main direction towards this goal was to extend our CDCL-based SAT solver to mimic some desired behaviors of the native Tableaux-like SAT engine. Generally speaking, this allows a better management of the context during proof search, which prevents from overwhelming theories and instantiation engines with useless facts. A comparison of this solver with Alt-Ergo’s old Tableaux-like solver is also done in our SMT-Workshop paper.

SMT-Comp and SMT-Workshop 2018

As emphasized above, we published our work regarding polymorphic SMT2 and SAT solving in SMT-Workshop 2018. More generally, this was an occasion for us to write the first tool paper about Alt-Ergo, and to highlight the main features that make it different from other state-of-the-art SMT solvers like CVC4, Z3 or Yices.

Thanks to our new SMT2 frontend, we were able to participate to the SMT-Competition last year. Naturally, we selected categories that are close to “deductive program verification”, as Alt-Ergo is primarily tuned for formulas coming from this application domain.

Although Alt-Ergo did not rank first, it was a positive experience and this encourages us to go ahead. Note that Alt-Ergo’s brother, Ctrl-Ergo, was not far from winning the QF-LIA category of the competition. This performance is partly due to the improvements in the CDCL SAT solver that were also integrated in Ctrl-Ergo.

Alt-Ergo for Atelier-B

Atelier-B is a framework that allows to develop formally verified software using the B Method. The framework rests on an automatic reasoner that allows to discharges thousands of mathematical formulas extracted from B models. If a formula is not discharged automatically, it is proved interactively. ClearSy (the company behind development of Atelier-B) has recently added a new backend to produce verification conditions in Why3’s logic, in order to target more automatic provers and increase automation rate. For certifiability reasons, we extended Alt-Ergo with a new frontend that is able to directly parse these verification conditions without relying on Why3.

Improved hash-consed data-structures

As said above, Alt-Ergo makes a clear distinction between Boolean terms and Propositions. This distinction prevents us from doing some rewriting and simplifications, in particular on expressions involving let-in and if-then-else constructs. This is why we decided to merge Term, Literal, and Formula in a new Expr data-structure, and remove this distinction. This allowed us to implement some additional simplification steps, and we immediately noticed performance improvements, in particular on SMT2 benchmarks. For instance, Alt-Ergo 2.3 proves 19548 formulas of AUFLIRA category in ~350 minutes, while version 2.2 proves 19535 formulas in ~1450 minutes (time limit was set to 20 minutes per formula).

Towards the integration of algebraic datatypes

Last Autumn, we also started working on the integration of algebraic datatypes reasoning in Alt-Ergo. In this first iteration, we extended Alt-Ergo’s native language to be able to declare (mutually recursive) algebraic datatypes, to write expressions with patterns matching, to handle selectors, … We then extended the typechecker accordingly and implemented a (not that) basic theory reasoner. Of course, we also handle SMT2’s algebraic datatypes. Here is an example in Alt-Ergo’s native syntax:

type ('a, 'b) t = A of {a_1 : 'a} | B of {b_11 : 'a ; b12 : 'b} | C | D | E

logic e : (int, real) t
logic n : int

axiom ax_n : n &gt;= 9

axiom ax_e:
  e = A(n) or e = B(n*n, 0.) or e = E

goal g:
  match e with
   | A(u) -> u >= 8
   | B (u,v) -> u >= 80 and v = 0.
   | E -> true
   | _ -> false
  end
  and 3 <= 2+2

What is planned in 2019 and beyond: the Alt-Ergo’s Users’ Club is born!

In 2018, we welcomed a lot of new engineers with a background in formal methods: Steven (De Oliveira) holds a PhD in formal verification from the Paris-Saclay University and the French Atomic Energy Commission (CEA). He has a master in cryptography and worked in the Frama-C team, developing open-source tools for verifying C programs. David (Declerck) obtained a PhD from Université Paris-Saclay in 2018, during which he extended the Cubicle model checker to support weak memory models and wrote a compiler from a subset of the x86 assembly language to Cubicle. Guillaume (Bury) holds a PhD from Université Sorbonne Paris Cité. He studied the integration of rewriting techniques inside SMT solvers. Albin (Coquereau) is working as a PhD student between OCamlPro, LRI and ENSTA, focusing on improving the Alt-Ergo SMT solver. Adrien is interested in verification of safety properties over software and embedded systems. He worked on higher-order functional program verification at the University of Tokyo, and on the Kind 2 model checker at the University of Iowa. All these people will consolidate the department of formal methods at OCamlPro, which will be beneficial for Alt-Ergo.

In 2019 we just launched the Alt-Ergo Users’ Club, in order to get closer to our users, collect their needs, and integrate them into the Alt-Ergo roadmap, but also to ensure sustainable funding for the development of the project. We are happy to announce the very first member of the Club is Adacore, very soon to be followed by Trust-In-Soft and CEA List. Thanks for your early support!

Interested to join? Contact us: contact@ocamlpro.com

Optimisation du stockage dans Tezos : une branche de test sur Gitlab

2019-02-05T13:56:36Z

Ce troisième article consacré à l’amélioration du stockage dans Tezos fait suite à l’annonce de la mise à disposition d’une image docker pour les beta testeurs souhaitant essayer notre système de stockage et garbage collector.

Voir Improving Tezos Storage : Gitlab branch for testers

Improving Tezos Storage : Gitlab branch for testers

2019-02-04T13:56:36Z

This article is the third post of a series of posts on improving Tezos storage. In our previous post, we announced the availability of a docker image for beta testers, wanting to test our storage and garbage collector. Today, we are glad to announce that we rebased our code on the latest version of mainnet-staging, and pushed a branch mainnet-staging-irontez on our public Gitlab repository.

The only difference with the previous post is a change in the name of the RPCs : /storage/context/gc will trigger a garbage collection (and terminate the node afterwards) and /storage/context/revert will migrate the database back to Irmin (and terminate the node afterwards).

Enjoy and send us feedback !!

Comments

AppaDude (10 February 2019 at 15 h 12 min):

I must be missing something. I compiled and issued the required rpc trigger:

/storage/context/gc with the command

~/tezos/tezos-client rpc get /storage/context/gc But I just got an empty JSON response of {} and the size of the .tezos-node folder is unchanged. Any advice is much appreciated. Thank you!

Fabrice Le Fessant (10 February 2019 at 15 h 47 min):

By default, garbage collection will keep 9 cycles of blocks (~36000 blocks). If you have fewer blocks, or if you are using Irontez on a former Tezos database, and fewer than 9 cycles have been stored in Irontez, nothing will happen. If you want to force a garbage collection, you should tell Irontez to keep fewer block (but more than 100, that’s the minimum that we enforce):

~/tezos/tezos-client rpc get ‘/storage/context/gc?keep=120’

should trigger a GC if the node has been running on Irontez for at least 2 hours.

AppaDude (10 February 2019 at 16 h 04 min):

I think it did work. I was confused because the total disk space for the .tezos-node folder remained unchanged. Upon closer inspection, I see these contents and sizes:

These are the contents of .tezos-node, can I safely delete context.backup?

4.0K config.json 269M context 75G context.backup 4.0K identity.json 4.0K lock 1.4M peers.json 5.4G store 4.0K version.json

Is it safe to delete context.backup if I do not plan to revert? (/storage/context/revert)

Fabrice Le Fessant (10 February 2019 at 20 h 51 min):

Yes, normally. Don’t forget it is still under beta-testing…

Note that /storage/context/revert works even if you remove context.backup.

Jack (23 February 2019 at 0 h 24 min):

Have there been any issues reported with missing endorsements or missing bakings with this patch? We have been using this gc version (https://gitlab.com/tezos/tezos/merge_requests/720) for the past month and ever since we switched we have been missing endorsements and missing bakings. The disk space savings is amazing, but if we keep missing ends/bakes, it’s going to hurt our reputation as a baking service.

Fabrice Le Fessant (23 February 2019 at 6 h 58 min):

Hi,

I am not sure what you are asking for. Are you using our version (https://gitlab.com/tzscan/tezos/commits/mainnet-staging-irontez), or the one on the Tezos repository ? Our version is very different, so if you are using the other one, you should contact them directly on the merge request. On our version, we got a report last week, and the branch has been fixed immediately (but not yet the docker images, should be done in the next days).

Jack (25 February 2019 at 15 h 53 min):

I was using the 720MR and experiencing issues with baking/endorsing. I understand that 720MR and IronTez are different. I was simply asking if your version has had any reports of baking/endorsing troubles.

Jack (25 February 2019 at 15 h 51 min):

Is there no way to convert a “standard node” to IronTez? I was running the official tezos-node, and my datadir is around 90G. I compiled IronTez and started it up on that same dir, then ran rpc get /storage/context/gc and nothing is happening. I thought this was supposed to convert my datadir to irontez? If not, what is the RPC to do this? Or must I start from scratch to be 100% irontez?

Fabrice Le Fessant (25 February 2019 at 16 h 24 min):

There are two ways to get a full Irontez DB:

Start a node from scratch and wait for one or two days…

Use an existing node, run Irontez on it for 2 hours, and then call rpc get /storage/context/gc?keep=100 . 100 is the number of blocks to be kept. After 2 hours, the last 120 blocks should be stored in the IronTez DB, so the old DB will not be used anymore. Note that Irontez will not delete the old DB, just rename it. You should go there and remove the file to recover the disk space.

Jack (27 February 2019 at 1 h 24 min):

Where do we send feedback/get help? Email? Slack? Reddit?

Banjo E. (3 March 2019 at 2 h 40 min):

There is a major problem for bakers who want to use the irontez branch. After garbage collection, the baker application will not start because the baker requests a rpc call for the genesis block information. That genesis block information is gone after the garbage collection. Please address this isssue soon. Thank you!

Fabrice Le Fessant (6 March 2019 at 21 h 44 min):

I pushed a new branch with a tentative fix: https://gitlab.com/tzscan/tezos/tree/mainnet-staging-irontez-fix-genesis . Unfortunately, I could not test it (I am far away from work for two weeks), so feedback is really welcome, before pushing in the irontez branch.

Tezos et OCamlPro

2019-01-31T13:56:36Z

Tezos est aujourd’hui un projet open source, un réseau international développé par des équipes sur plus de cinq continents. Dans la genèse du projet, l’entreprise française OCamlPro, qui développe encore aujourd’hui de nombreux projets liés à Tezos (TZscan, Liquidity, etc.), a joué un rôle particulièrement important. C’est en effet en son sein que des ingénieurs-chercheurs ont posé les premières pierres du code, en étroite collaboration avec Arthur Breitman, l’architecte du projet, et DLS pendant plusieurs années. Nous nous réjouissons aujourd’hui de l’essor qu’a pris le projet.

Arthur et OCamlPro (publication conjointe)

Improving Tezos Storage : update and beta-testing

2019-01-30T13:56:36Z

In a previous post, we presented some work that we did to improve the quantity of storage used by the Tezos node. Our post generated a lot of comments, in which upcoming features such as garbage collection and pruning were introduced. It also motivated us to keep working on this (hot) topic, and we present here our new results, and current state. Irontez3 is a new version of our storage system, that we tested both on real traces and real nodes. We implemented a garbage-collector for it, that is triggered by an RPC on our node (we want the user to be able to choose when it happens, especially for bakers who might risk losing a baking slot), and automatically every 16 cycles in our traces.

In the following graph, we present the size of the context database during a full trace execution (~278 000 blocks):

There is definitely quite some improvement brought to the current Tezos implementation based on Irmin+LMDB, that we reimplemented as IronTez0. IronTez0 allows an IronTez node to read a database generated by the current Tezos and switch to the IronTez3 database. At the bottom of the graph, IronTez3 increases very slowly (about 7 GB at the end), and the garbage-collector makes it even less expensive (about 2-3 GB at the end). Finally, we executed a trace where we switched from IronTez0 to IronTez3 at block 225 000. The graph shows that, after the switch, the size immediately grows much more slowly, and finally, after a garbage collection, the storage is reduced to what it would have been with IronTez3.

Now, let’s compare the speed of the different storages:

The graph shows that IronTez3 is about 4-5 times faster than Tezos/IronTez0. Garbage-collections have an obvious impact on the speed, but clearly negligible compared to the current performance of Tezos. On our computer used for the traces, a Xeon with an SSD disk, the longest garbage collection takes between 1 and 2 minutes, even when the database was about 40 GB at the beginning.

In the former post, we didn’t check the amount of memory used by our storage system. It might be expected that the performance improvement could be associated with a more costly use of memory… but such is not the case :

At the top of the graph is our IronTez0 implementation of the current storage: it uses a little more memory than the current Tezos implementation (about 6 GB), maybe because it shares data structures with IronTez3, with fields that are only used by IronTez3 and could be removed in a specialized version. IronTez3 and IronTez3 with garbage collection are at the bottom, using about 2 GB of memory. It is actually surprising that the cost of garbage collections is very limited.

On our current running node, we get the following storage:

$ du

1.4G ./context
4.9G ./store
6.3G .

Now, if we use our new RPC to revert the node to Irmin (taking a little less than 8 minutes on our computer), we get :

$ du
14.3G ./context
 4.9G ./store
19.2G .

Beta-Testing with Docker

If you are interested in these results, it is now possible to test our node: we created a docker image, similar to the ones of Tezos. It is available on Docker Hub (one image that works for both Mainnet and Alphanet). Our script mainnet.sh (http://tzscan.io/irontez/mainnet.sh) can be used similarly to the alphanet.sh script of Tezos to manage the container. It can be run on an existing Tezos database, it will switch it to IronTez3. Note that such a change is not irreversible, still it might be a good idea to backup your Tezos node directory before, as (1) migrating back might take some time, (2) this is a beta-testing phase, meaning the code might still hide nasty bugs, and (3) the official node might introduce a new incompatible format.

New RPCS

Both of these RPCs will make the node TERMINATE once they have completed. You should restart the node afterwards.

The RPC /ocp/storage/gc : it triggers a garbage collection using the RPC /ocp/storage/gc . By default, this RPC will keep only the contexts from the last 9 cycles. It is possible to change this value by using the ?keep argument, and specify another number of contexts to keep (beware that if this value is too low, you might end up with a non-working Tezos node, so we have set a minimum value of 100). No garbage-collection will happen if the oldest context to keep was stored in the Irmin database. The RPC /ocp/storage/revert : it triggers a migration of the database fron Irontez3 back to Irmin. If you have been using IronTez for a while, and want to go back to the official node, this is the way. After calling this RPC, you should not run IronTez again, otherwise, it will restart using the IronTez3 format, and you will need to revert again. This operation can take a lot of time, depending on the quantity of data to move between the two formats.

Following Steps

We are now working with the team at Nomadic Labs to include our work in the public Tezos code base. We will inform you as soon as our Pull Request is ready, for more testing ! If all testing and review goes well, we hope it can be merged in the next release !

Comments

Jack (30 January 2019 at 15 h 30 min):

Please release this as a MR on gitlab so those of us not using docker can start testing the code.

Fabrice Le Fessant (10 February 2019 at 15 h 39 min):

That was done: here

Tezos and OCamlPro

2019-01-29T13:56:36Z

A reflection on the new year… Today, Tezos is a global network and an open source project with developers spanning over five continents. In the inception of this project, the French company OCamlPro which, to this day, stills develops numerous projects around Tezos, played a particularly important role. Indeed, they were the first home of the research engineers who laid down the cornerstone of the code base, in tight collaboration with Arthur Breitman and the architect of the project, and DLS. We take some time today to remember those early days and celebrate the flourishing of this once small project.

(cross-post with Arthur Breitman, Founder of the Tezos project)

opam 2.0.3 release

2019-01-28T13:56:36Z

We are pleased to announce the release of opam 2.0.3.

This new version contains some backported fixes:

Fix manpage remaining $ (OPAMBESTEFFORT)
Fix OPAMROOTISOK handling
Regenerate missing environment file

Installation instructions (unchanged):

From binaries: run

sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

From source, using opam:

opam update; opam install opam-devel

From source, manually: see the instructions in the README.

We hope you enjoy this new major version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com.

Improving Tezos Storage

2019-01-15T13:56:36Z

Running a Tezos node currently costs a lot of disk space, about 59 GB for the context database, the place where the node stores the states corresponding to every block in the blockchain, since the first one. Of course, this is going to decrease once garbage collection is integrated, i.e. removing very old information, that is not used and cannot change anymore (PR720 by Thomas Gazagnaire, Tarides, some early tests show a decrease to 14GB ,but with no performance evaluation). As a side note, this is different from pruning, i.e. transmitting only the last cycles for “light” nodes (PR663 by Thomas Blanc, OCamlPro). Anyway, as Tezos will be used more and more, contexts will keep growing, and we need to keep decreasing the space and performance cost of Tezos storage.

As one part of our activity at OCamlPro is to allow companies to deploy their own private Tezos networks, we decided to experiment with new storage layouts. We implemented two branches: our branch IronTez1 is based on a full LMDB database, as Tezos currently, but with optimized storage representation ; our branch IronTez2 is based on a mixed database, with both LMDB and file storage.

To test these branches, we started a node from scratch, and recorded all the accesses to the context database, to be able to replay it with our new experimental nodes. The node took about 12 hours to synchronize with the network, on which about 3 hours were used to write and read in the context database. We then replayed the trace, either only the writes or with both reads and writes.

Here are the results:

The mixed storage is the most interesting: it uses half the storage of a standard Tezos node !

Again, the mixed storage is the most efficient : even with reads and writes, IronTez2 is five time faster than the current Tezos storage.

Finally, here is a graph that shows the impact of the two attacks that happened in November 2018, and how it can be mitigated by storage improvement:

The graph shows that, using mixed storage, it is possible to restore the storage growth of Tezos to what it was before the attack !

Interestingly, although these experiments have been done on full traces, our branches are completely backward-compatible : they could be used on an already existing database, to store the new contexts in our optimized format, while keeping the old data in the ancient format.

Of course, there is still a lot of work to do, before this work is finished. We think that there are still more optimizations that are possible, and we need to test our branches on running nodes for some time to get confidence (TzScan might be the first tester !), but this is a very encouraging work for the future of Tezos !

OCamlPro Feed

opam 2.3.0 release!

Try it!

Major breaking change: extra-files

Major changes

Optimisation de Geneweb, 1er logiciel français de Généalogie depuis près de 30 ans

Alt-Ergo 2.6 is Out!

Bit-vectors

Model Generation

Optimization

SMT-LIB command support

Floating-point theory

Dolmen is the new default frontend

Use of dune-site for plugins

Binary releases on GitHub

Performance

And more!

Acknowledgements

Flambda2 Ep. 3: Speculative Inlining

Welcome to a new episode of The Flambda2 Snippets!

opam 2.2.0 release!

Try it!

Changes

Major change: Windows support

Major change: opam tree / opam why

Major change: with-dev-setup

Major change: opam pin --recursive

New Options

Miscellaneous changes

Flambda2 Ep. 2: Loopifying Tail-Recursive Functions

Welcome to a new episode of The Flambda2 Snippets!

Fixing and Optimizing the GnuCOBOL Preprocessor

OCaml Backtraces on Uncaught Exceptions

Opam 102: Pinning Packages

So what exactly did opam pin do here?

Flambda2 Ep. 1: Foundational Design Decisions

Welcome to The Flambda2 Snippets!

Behind the Scenes of the OCaml Optimising Compiler Flambda2: Introduction and Roadmap

Lean 4: When Sound Programs become a Choice

Opam 101: The First Steps

Maturing Learn-OCaml to version 1.0: Gateway to the OCaml World

The latest release of Alt-Ergo version 2.5.1 is out, with improved SMT-LIB and bitvector support!

Update for bug fixes

Better SMT-LIB Support

Support For Bit-Vectors Primitives

Built-in Primitives For Mixed Integer And Real Problems

Example

Model Generation

Usage

Floating Point Support

Usage

About Alt-Ergo 2.5.0

Acknowledgements

2022 at OCamlPro

Autofonce, GNU Autotests Revisited

Use of `dune-site` for plugins

So what exactly did `opam pin` do here?