Improving Tezos Storage : Gitlab branch for testers

---

This article is the third post of a series of posts on improving Tezos storage. In our previous post, we announced the availability of a docker image for beta testers, wanting to test our storage and garbage collector. Today, we are glad to announce that we rebased our code on the latest version of mainnet-staging, and pushed a branch mainnet-staging-irontez on our public Gitlab repository.

The only difference with the previous post is a change in the name of the RPCs : /storage/context/gc will trigger a garbage collection (and terminate the node afterwards) and /storage/context/revert will migrate the database back to Irmin (and terminate the node afterwards).

Enjoy and send us feedback !!

Comments

AppaDude (10 February 2019 at 15 h 12 min):

I must be missing something. I compiled and issued the required rpc trigger:

/storage/context/gc with the command

~/tezos/tezos-client rpc get /storage/context/gc But I just got an empty JSON response of {} and the size of the .tezos-node folder is unchanged. Any advice is much appreciated. Thank you!

Fabrice Le Fessant (10 February 2019 at 15 h 47 min):

By default, garbage collection will keep 9 cycles of blocks (~36000 blocks). If you have fewer blocks, or if you are using Irontez on a former Tezos database, and fewer than 9 cycles have been stored in Irontez, nothing will happen. If you want to force a garbage collection, you should tell Irontez to keep fewer block (but more than 100, that’s the minimum that we enforce):

~/tezos/tezos-client rpc get ‘/storage/context/gc?keep=120’

should trigger a GC if the node has been running on Irontez for at least 2 hours.

AppaDude (10 February 2019 at 16 h 04 min):

I think it did work. I was confused because the total disk space for the .tezos-node folder remained unchanged. Upon closer inspection, I see these contents and sizes:

These are the contents of .tezos-node, can I safely delete context.backup?

4.0K config.json 269M context 75G context.backup 4.0K identity.json 4.0K lock 1.4M peers.json 5.4G store 4.0K version.json

Is it safe to delete context.backup if I do not plan to revert? (/storage/context/revert)

Fabrice Le Fessant (10 February 2019 at 20 h 51 min):

Yes, normally. Don’t forget it is still under beta-testing…

Note that /storage/context/revert works even if you remove context.backup.

Jack (23 February 2019 at 0 h 24 min):

Have there been any issues reported with missing endorsements or missing bakings with this patch? We have been using this gc version (https://gitlab.com/tezos/tezos/merge_requests/720) for the past month and ever since we switched we have been missing endorsements and missing bakings. The disk space savings is amazing, but if we keep missing ends/bakes, it’s going to hurt our reputation as a baking service.

Fabrice Le Fessant (23 February 2019 at 6 h 58 min):

Hi,

I am not sure what you are asking for. Are you using our version (https://gitlab.com/tzscan/tezos/commits/mainnet-staging-irontez), or the one on the Tezos repository ? Our version is very different, so if you are using the other one, you should contact them directly on the merge request. On our version, we got a report last week, and the branch has been fixed immediately (but not yet the docker images, should be done in the next days).

Jack (25 February 2019 at 15 h 53 min):

I was using the 720MR and experiencing issues with baking/endorsing. I understand that 720MR and IronTez are different. I was simply asking if your version has had any reports of baking/endorsing troubles.

Jack (25 February 2019 at 15 h 51 min):

Is there no way to convert a “standard node” to IronTez? I was running the official tezos-node, and my datadir is around 90G. I compiled IronTez and started it up on that same dir, then ran rpc get /storage/context/gc and nothing is happening. I thought this was supposed to convert my datadir to irontez? If not, what is the RPC to do this? Or must I start from scratch to be 100% irontez?

Fabrice Le Fessant (25 February 2019 at 16 h 24 min):

There are two ways to get a full Irontez DB:

• Start a node from scratch and wait for one or two days…
• Use an existing node, run Irontez on it for 2 hours, and then call rpc get /storage/context/gc?keep=100 . 100 is the number of blocks to be kept. After 2 hours, the last 120 blocks should be stored in the IronTez DB, so the old DB will not be used anymore. Note that Irontez will not delete the old DB, just rename it. You should go there and remove the file to recover the disk space.

Jack (27 February 2019 at 1 h 24 min):

Where do we send feedback/get help? Email? Slack? Reddit?

Banjo E. (3 March 2019 at 2 h 40 min):

There is a major problem for bakers who want to use the irontez branch. After garbage collection, the baker application will not start because the baker requests a rpc call for the genesis block information. That genesis block information is gone after the garbage collection. Please address this isssue soon. Thank you!

Fabrice Le Fessant (6 March 2019 at 21 h 44 min):

I pushed a new branch with a tentative fix: https://gitlab.com/tzscan/tezos/tree/mainnet-staging-irontez-fix-genesis . Unfortunately, I could not test it (I am far away from work for two weeks), so feedback is really welcome, before pushing in the irontez branch.

---

About OCamlPro:

OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.

• We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
• We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
• We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
• We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.

Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.