In search of lightning feedback loop in a large codebase

January 11, 2019

As software developers, when we touch a codebase, we want the edit-compile cycle, to be as short as possible. Studies show that feedback loop less than 1s, or 100ms will prevent devleopers getting distracted.

This seems to be possible in a small project, but as project grows to 10k files, or even more, 100k files in a large company like Google, Facebook, this seems to be extremely challenging.

In this article, we are talking about how BuckleScript - it supports both OCaml and ReasonML syntax - is trying to solve this issue.

The edit-build cycle in BuckleScript consists of two components: the compiler which does type checking and code generation and the scheduler which figures out what to rebuild and how to do it concurrently.

The importance of compiler's cold performance

In order to reduce the edit-build latency, some languages adopt an approach of in memory compiler + watch mode. We think this is not a scalable or reliable approach.

A compiler is a complex piece of software, the chance that a compiler has memory leak is not low. It is not observed in real world since compiler is used mostly in a short-lived setting, it starts up fast and dies off quickly.

However, this is decimated when compiler is put in server mode. When we have 10k or 100k files held in memory, it is very easy to observe OOM (out of memory issues).

We figured that to deliver a scalable and reliable system, it is better to decouple the compiler's complexity from the scheduler. When the compiler cold starts and dies off quickly, the operating system process mechanism serves as an obviously correct garbage collector, which increases the reliability of the whole system.

To reduce such latency in a single compiler's workload, we spent lots of time tweaking the performance of the compiler itself, for example, rewriting the hot path in C code, most BuckleScript compiler source code is written in an imperative C-style to avoid allocation.

To have a general idea of how fast BuckleScript's compiler runs:

test>cat fib.ml
let rec fib = function
  | 0 | 1 -> 1
  | n -> fib (n - 1) + fib (n - 2)

test>time /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-cmi -bs-cmj -c fib.ml

real    0m0.008s
user    0m0.004s
sys     0m0.003s

Having a compiler run fast with a cold start lays the groundwork so that it won't be the bottleneck in the whole process. This is because when the architecture decouple the compiler from the scheduler, the compiler will be invoked by the scheduler hunderds of thousands of times in a build cycle, the latency of the compiler in a single compilation unit will add up.

The art of being incremental

Having a compiler running fast in cold mode does not solve the scalability issue alone. Take a code base which contains 100K source files for example, 100ms per file would result in 10,000s latency which is not acceptable. To solve such issue, we need reduce the workload as much as possible during each edit-build cycle.

In a statically typed language, suppose we have two compilation units A , B, and B depends on A. Whenever A changes, B gets recompiled, to make it worse, the recompilation propogates, all dependencies of B get recompiled, which will result in a snowball effect. In this model, whenever we touch the non-leaf compilation unit, the latency would be large.

The key observation here for BuckleScript is that B does not really depend on the last modified time of A, it depends on the intermediate output of A (e.g, .cmj and .cmi file) which may not change even if we modify A. What we work on here is to reduce the probability of changing A's intermediate output due to changing A. With the integration of a scheduler, it can help stop the propagation as early as possible.

In BuckleScript, each compilation unit is composed of two files, the implementation file and interface file which are compiled to intermediate output .cmj and .cmi separately.

The interface builds are completely separate from implemetation, it does not depend on .cmj (implementation intermediate output) at all.

Suppose the interface is not changed, whenever we touch the implementation file, BuckleScript designs the data structure of .cmj in a way that the content of .cmj is seldom changed. (Let's say its probability is 0.05 which is rare in cases when the arity of a function changed)

If neither .cmj nor .cmi gets changed, the scheduler would stop propagation.

Suppose the .cmj file is still changed (P = 0.05), A's dependency B would get recompiled. Note B.cmi only depends on A.cmi, so it will not get compiled, only B.cmj will get recompiled, its chances that B.cmj gets changed will be even lower. In practice, the probabiity of the length of propagation chain is more than two is less than

0.05 * 0.05 = 0.0025

This means when we are in an edit-build cycle, whenever A gets changed, it may have many direct dependencies, but the longest rebuild sequence will get settled in at most two compilation units. Suppose the scheduler schedule the tasks in parallel, the longest sequence is bound by two, which means the rebuild cycle is very close to compiling two compilation units.

In implementation, we also generalized the idea of stopping propogation of .cmi changes, so whenever we are adding some comments in the interface file, it will get settled quickly.

The worst case is that the root of an interface changed. In such case, it does not mean all its dependencies will get recompiled, it depends on how many dependencies' interface depend on the root's interface. The cool thing is that since interface's dependency chain is completely decoupled from implementation's dependency, actually it used to be a subset of implementation's dependency chain, so only a subset of its dependencies will get compiled. We will see a concrete an example later.

A fast scheduler

As we said, the time spent in an edit-build cycle is mostly composed of two parts: compiling invoking the compiler, scheduling.

We are reusing the very fast scheduler provided by Ninja

Where other build systems are high-level languages, Ninja aims to be an assembler.

BuckleScript outputs assembler style instructions consumed by Ninja which does scheduling very fast and provides good parallelism. By integrating with Ninja's restat attribute, we are able to implement the idea of a specialized content based build system.

It is fast enough for 99% of use cases. For a project less than 1k files, the time spent in an edit-build cycle is dominated by the compiler, but as project grows to 10K files or even more, the time spent in compiler is stable and bounded due to our incremental design, however, the time will start be dominated by the scheduler.

For a nop build around 10k files, it takes around 700ms for Ninja to figure out nothing needs to be rebuilt.

The current Ninja model is simple, every time it is invoked, it will re-read build.ninja instructions, check stats of artifacts and do the scheduling.

Instead of making a long-lived compiler, we propose to have a long-lived scheduler. The complexity of a scheduler is significantly lower than a compiler. Having an in-memory scheduler will help reducing redundant work such as parsing build.ninja instructions which is around of size 2M for 10k files. With the integration of watch mode, it does not need stat all artifacts each time, this should help increase the scalability of scheduler to 100K files or even more.

Tests on a synthetic benchmark

Our synthetic bench is borrowed from OMake and public available here.

The benchmark has these characteristics: The task is to build n^2 libraries with n^2 modules each (for a given small number n), and the dependencies between the modules are created in a way so that we can stress both the dependency analyzer of the build utility and the ability to run commands in parallel.

We modified the benchmark to add interface file for each implementation file.

The benchmark is running on MacBook Pro 18 with CPU 2.6 GHz Intel Core i7, Memory 32 GB 2400 MHz DDR4

The test is running against n = 3,5,7,9, where the source code size would be 2*3^4=162, 2*5^4=1250, 2*7^4=4802, 2*9**4=13122.

Below is what we get for a cold build from scratch:

Source size	Clean build (ms)
0162	684
1250	5,100
4802	24,112
13122	125,248

Source size	Nop build (ms)	Touching root module (m_1_1_1_1.ml)	Touching (m_1_1_1.mli)
0162	16	59	54
1250	79	120	133
4802	266	369	367
13122	728	963	962

We can see from the table as size grows, the time spent in edit-build cycle shifts from compilation to the scheduler, this is due to the fact that Ninja scheduler does not save the work for each test.

Source size	Adding value to root module (m_1_1_1_1.ml)	Changing root interface
0162	56	70
1250	131	155
4802	370	428
13122	969	991

The verbose build log for adding values on source of size 13122 is as below:

test9>ninja -C lib/bs -v
ninja: Entering directory `lib/bs'
[1/2] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe    -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/hongbozhang/git/bsb-bench/test9/node_modules/bs-platform/lib/ocaml' -color always -c -o src/dir_1_1/m_1_1_1_1.mlast -bs-syntax-only -bs-binary-ast /Users/hongbozhang/git/bsb-bench/test9/src/dir_1_1/m_1_1_1_1.ml
[2/2] /usr/local/lib/node_modules/bs-platform/lib/bsb_helper.exe  -g 0 -MD src/dir_1_1/m_1_1_1_1.mlast
[1/5905] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I src/dir_4_6 -I src/dir_4_1 -I src/dir_4_8 -I src/dir_6_5 -I src/dir_6_2 -I src/dir_8_3 -I src/dir_8_4 -I src/dir_2_3 -I src/dir_2_4 -I src/dir_6_3 -I src/dir_4_9 -I src/dir_6_4 -I src/dir_4_7 -I src/dir_2_5 -I src/dir_2_2 -I src/dir_8_5 -I src/dir_8_2 -I src/dir_9_9 -I src/dir_3_7 -I src/dir_1_3 -I src/dir_9_7 -I src/dir_3_9 -I src/dir_1_4 -I src/dir_7_6 -I src/dir_7_1 -I src/dir_5_5 -I src/dir_7_8 -I src/dir_5_2 -I src/dir_3_8 -I src/dir_9_6 -I src/dir_1_5 -I src/dir_9_1 -I src/dir_1_2 -I src/dir_3_6 -I src/dir_9_8 -I src/dir_3_1 -I src/dir_5_3 -I src/dir_5_4 -I src/dir_7_9 -I src/dir_7_7 -I src/dir_8_7 -I src/dir_2_9 -I src/dir_8_9 -I src/dir_2_7 -I src/dir_4_2 -I src/dir_6_8 -I src/dir_4_5 -I src/dir_6_1 -I src/dir_6_6 -I src/dir_2_1 -I src/dir_2_6 -I src/dir_8_8 -I src/dir_8_1 -I src/dir_2_8 -I src/dir_8_6 -I src/dir_6_7 -I src/dir_6_9 -I src/dir_4_4 -I src/dir_4_3 -I src/dir_7_2 -I src/dir_7_5 -I src/dir_5_8 -I src/dir_5_1 -I src/dir_5_6 -I src/dir_1_9 -I src/dir_3_4 -I src/dir_3_3 -I src/dir_9_4 -I src/dir_1_7 -I src/dir_9_3 -I src/dir_5_7 -I src/dir_7_4 -I src/dir_5_9 -I src/dir_7_3 -I src/dir_9_2 -I src/dir_1_1 -I src/dir_9_5 -I src/dir_1_6 -I src/dir_3_2 -I src/dir_1_8 -I src/dir_3_5  -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/hongbozhang/git/bsb-bench/test9/node_modules/bs-platform/lib/ocaml' -color always -o src/dir_1_1/m_1_1_1_1.cmj -c  src/dir_1_1/m_1_1_1_1.mlast 
File "/Users/hongbozhang/git/bsb-bench/test9/src/dir_1_1/m_1_1_1_1.ml", line 4, characters 4-5:
Warning 32: unused value a.

Ignoring the preprocess stage, We can see there are 5906 jobs scheduled but only one job is processed. This is because adding a single value does not change m_1_1_1_1.cmj so that the change of propogation is stopped immediately.

The result is surprisingly good even if change the interface of root files, by looking at the verbose build log:

test9>ninja -C lib/bs -v
ninja: Entering directory `lib/bs'
[1/2] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe    -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/hongbozhang/git/bsb-bench/test9/node_modules/bs-platform/lib/ocaml' -color always -c -o src/dir_1_1/m_1_1_1_1.mliast -bs-syntax-only -bs-binary-ast /Users/hongbozhang/git/bsb-bench/test9/src/dir_1_1/m_1_1_1_1.mli
[2/2] /usr/local/lib/node_modules/bs-platform/lib/bsb_helper.exe  -g 0 -MD src/dir_1_1/m_1_1_1_1.mliast
[1/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-no-builtin-ppx-mli -bs-no-implicit-include  -I src/dir_4_6 -I src/dir_4_1 -I src/dir_4_8 -I src/dir_6_5 -I src/dir_6_2 -I src/dir_8_3 -I src/dir_8_4 -I src/dir_2_3 -I src/dir_2_4 -I src/dir_6_3 -I src/dir_4_9 -I src/dir_6_4 -I src/dir_4_7 -I src/dir_2_5 -I src/dir_2_2 -I src/dir_8_5 -I src/dir_8_2 -I src/dir_9_9 -I src/dir_3_7 -I src/dir_1_3 -I src/dir_9_7 -I src/dir_3_9 -I src/dir_1_4 -I src/dir_7_6 -I src/dir_7_1 -I src/dir_5_5 -I src/dir_7_8 -I src/dir_5_2 -I src/dir_3_8 -I src/dir_9_6 -I src/dir_1_5 -I src/dir_9_1 -I src/dir_1_2 -I src/dir_3_6 -I src/dir_9_8 -I src/dir_3_1 -I src/dir_5_3 -I src/dir_5_4 -I src/dir_7_9 -I src/dir_7_7 -I src/dir_8_7 -I src/dir_2_9 -I src/dir_8_9 -I src/dir_2_7 -I src/dir_4_2 -I src/dir_6_8 -I src/dir_4_5 -I src/dir_6_1 -I src/dir_6_6 -I src/dir_2_1 -I src/dir_2_6 -I src/dir_8_8 -I src/dir_8_1 -I src/dir_2_8 -I src/dir_8_6 -I src/dir_6_7 -I src/dir_6_9 -I src/dir_4_4 -I src/dir_4_3 -I src/dir_7_2 -I src/dir_7_5 -I src/dir_5_8 -I src/dir_5_1 -I src/dir_5_6 -I src/dir_1_9 -I src/dir_3_4 -I src/dir_3_3 -I src/dir_9_4 -I src/dir_1_7 -I src/dir_9_3 -I src/dir_5_7 -I src/dir_7_4 -I src/dir_5_9 -I src/dir_7_3 -I src/dir_9_2 -I src/dir_1_1 -I src/dir_9_5 -I src/dir_1_6 -I src/dir_3_2 -I src/dir_1_8 -I src/dir_3_5  -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/hongbozhang/git/bsb-bench/test9/node_modules/bs-platform/lib/ocaml' -color always -o src/dir_1_1/m_1_1_1_1.cmi -c  src/dir_1_1/m_1_1_1_1.mliast
[2/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ...  -color always -o src/dir_1_1/m_1_1_1_1.cmj -c  src/dir_1_1/m_1_1_1_1.mlast 
[3/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ...  -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/hongbozhang/git/bsb-bench/test9/node_modules/bs-platform/lib/ocaml' -color always -o src/dir_1_1/m_1_1_2_2.cmj -c  src/dir_1_1/m_1_1_2_2.mlast 
[4/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_1.cmj -c  src/dir_1_1/m_1_1_2_1.mlast 
[5/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_3.cmj -c  src/dir_1_1/m_1_1_2_3.mlast 
[6/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_9.cmj -c  src/dir_1_1/m_1_1_2_9.mlast 
[7/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_4.cmj -c  src/dir_1_1/m_1_1_2_4.mlast 
[8/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_5.cmj -c  src/dir_1_1/m_1_1_2_5.mlast 
[9/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_6.cmj -c  src/dir_1_1/m_1_1_2_6.mlast 
[10/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_7.cmj -c  src/dir_1_1/m_1_1_2_7.mlast 
[11/5906] /usr/local/lib/node_modules/bs-platform/lib/bsc.exe -bs-package-name test  -bs-package-output commonjs:lib/js/src/dir_1_1 -bs-assume-has-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include   -I ... -color always -o src/dir_1_1/m_1_1_2_8.cmj -c  src/dir_1_1/m_1_1_2_8.mlast

The log means that there are 5906 jobs to be scheduled but only 11 jobs processed. This is because there only 9 direct dependencies, the cmj/cmi changes' propogation stops in the first level of dependency, indirect dependencies are not propagated any more since the intermediate representation stays stable after that. There are 9 more jobs to do compared with changing implementation but the latency only increased by 30ms, which means the scheduler does a good job in parallelism.

The real world scenario may be a bit worse than our synthetic benchmark, since our syntethic benchmark only impose dependencies across implementation files, the dependencies across interface files are flat.

To conclude, currently BuckleScript scales very well to projects with around 10k files, and with some work in improving the scheduler, we believe it is not too difficult to scale it to 100K files or even more. To reach such enormous scalability in a reliable way, we propose to have a long-lived scheduler and fast cold compiler. The language interface design and the design of the data structure for the implementation will make the longest rebuild propogation chain bounded by two in most cases.

January 7, 2019

bs-platform 4.0.17 is a major release.

It improved incremental compilation time significantly.

A pictue is worth a thousand words, below is a large monorepo which contains 4096 modules, changing the root node which has more than 3000 dependents, it finished building within 400ms.

We will write a dedicated article explaining how we achieve this incredible build performance.

A detailed list of changes is available here

Another quite important but not client facing change is that we renovated the internal build system, it will be much easier for contribution later on. We will update the contribution guide once it gets stable.

BuckleScript 4.0.8 (Part One)

December 5, 2018

Today we released version 4.0.8 of bs-platform. A detailed list of changes is available here.

Most user-facing changes are bug fixes and small enhancements, while quite a lot of work has been done behind the scenes towards the more fundamental improvements coming down the line. This blog post refers to the BuckleScript runtime and some of the work we are doing to improve it.

The design goal of BuckleScript is to make the runtime as small as possible. This runtime is divided into two parts: the C shims, and the fundamental language feature support.

The C shims are not a strict runtime requirement: in the native backend, the functions are implemented in C, but in BuckleScript this isn't necessary. We can either implement the C shims in a polyfill style or we can just implement them in OCaml and compile via BuckleScript. Recently, we have been shifting more and more work from the runtime to the normal OCaml stdlib by patching it with conditional compilation. The benefit is obvious – they are just normal functions which do not need special compiler support – but the downside is that we might need to make more patches to the libraries which use C functions, but considering the more challenging part of maintaining the patches to the compiler, we think such overhead is worthwhile.

If we ignore the C shims, the BuckleScript runtime is very small, and it is pretty easy for experienced BuckleScript programmers to write runtime-free code which generates standalone JS code. Such code could include supporting curried calling conventions, encoding of OCaml ADT, etc.

The BuckleScript runtime is written in BuckleScript itself. The benefit of this is that it is much more maintainable than implementing in JS itself, and it is easier to keep some invariants when crossing the boundary between the runtime and the stdlib. For example, we don't need to worry about the consistency of the runtime encoding of type tuple in BuckleScript, since the runtime is also implemented in BuckleScript itself, and we get three output modules for free thanks to this "dogfooding".

However, this makes the build system pretty complicated and fragile, and the dependencies between each module are mostly hard coded. Even worse, this introduces a hard dependency between the normal libraries and the runtime binary artifacts.

In particular, one issue we want to address is to make the BuckleScript toolchain lightweight. We will continue to implement the BuckleScript runtime by using BuckleScript itself, but we want to get rid of dependencies like the support for exceptions. In the end, installation will no longer involve building the runtime: BuckleScript will simply be a bunch of generated JS files, so the complexity of the build system will not impact users at all. This is quite important given that we are committed to supporting Windows.

In the future, we will therefore be able to distribute the runtime as a normal JS library, and the BuckleScript user will only need the binary compiler and a small set of JS files. They will be able to use stdlib, Belt or anything else.

To get rid of such dependencies between stdlib and the runtime, we are going to introduce a breaking change in the future. In hindsight, our support for catching JS exceptions exposed the concrete representation of the exception encoding, in particular:

match ... with 
| OCamlException exn -> ..
| Js.Exn.Error e -> ...

In this release, we introduced a function to avoid exposing such exception constructors:

match ... with 
| OCamlException exn -> ...
| e -> 
    match Js.asJsExn e with 
    | Some jserror -> ..
    | None -> ...

We encourage you to make such changes yourself to future-proof your codebases.

Oh and by the way, one side effect of this refactoring of the BuckleScript runtime is that the compilation does not require reading of the generated .cm* files, which means faster compilation :)

BuckleScript development plans for next half

November 19, 2018

In this article we will explain what we are doing now and what we plan to improve in next half (Dec-May), we would also like to hear your feedback so that we can adjust accordingly.

Keep in mind that the development team is a very small team, so we have to prioritize things instead of working on every feature.

What we are doing

In the last couple of month we are busy upgrading the OCaml compiler from 4.02.3 to 4.06.1, the good news is that the upgrade is almost done.

We plan to ship it soon by the end of this year, at the same time, we will still maintain the current version of the compiler until we feel the new compiler is as good as the old one.

Note the upgrade is not easy work since the internals of OCaml compiler changed significantly in the last few years. Our upgrade strategy is also quite conservative, it works by conditional compilation so that the bsc compiler actually work with both versions, the benefit is that in this case, we are not in a messy state, the bug fix of bsc compiler can still benefit two branches.

But the reward is also huge, there are a bunch of optimizations and nice features coming alone in the recent releases of the OCaml language, to name a few: inline records, local exception and hex notation for floats etc. More importantly, this is a great move to engage better with OCaml ecosystem: the previous old version compiler imposed some maintenance overhead for OCaml toolchain, we can make better use of OCaml toolchain after such upgrade.

What's next

Making better use of the new compiler internals

The upgrade is divided into two stage, the first stage is focused on no regression so that we can ship it.

Afterwards, we have plans to make better use of the more info rich IR in the new release. Thanks to the flambda introduced after OCaml 4.03, more information is passed down to the lambda IR where BuckleScript take from. For example, user can annotate functions inline or not with inline attribute, we can make better use of such information.

In newer versions of OCaml compiler, the block and array is more distinguished we will investigate whether we can make use of it to provide better data representation for OCaml datatype.

Another interesting direction is to see if we can encode module in a consistent style: as an JS dictionary for both global modules and local modules.

Improving the usability of BuckleScript toolchain

BuckleScript is focused on making better use of JS ecosystem and provide values to ship JS code in production (produced by BuckleScript).

Making Bucklescript toolchain more lightweight

Currently it is still too heavy for users to provide JS libraries from BuckleScript, since clients need to install bs-platform which requires a lot of disk-space and native compilation in some platform. We will investigate if we can distribute the native compiler without using npm.

Separate the compiler from stdlib may also help draw the contribution to the stdlib/belt library.

Improving the usability of bsb

Currently the bsb is restricted by the npm directory layout, the generated JS artifacts is also restricted by it, we will see if we can relocate the JS artifacts or provide more flexibility for users.

A change of undefined behavior in BuckleScript 4.0.7

November 13, 2018

In the latest BuckleScript release, we introduced a minor change in the codegen which broken some user libraries. Note this change only broke the code in the FFI boundary(the interop between JS).

In the early days of BuckleScript, there is no built-in uncurried calling convention support, since OCaml is a curried language, which means every function has arity one, so there is no way to express that a function has arity zero, this makes some interop challenging. In the mocha unit test library, it expects its callback to be function of arity zero.

To work around this issue, before this release, we did a small codegen optimization, for a function of type unit -> unit, if its argument is not used, we remove its argument in the output.

let f : unit -> int = fun () -> 3 
let f_used : unit -> unit = fun x -> Js.log x

let f: unit => int = () => 3;
let f_used: unit => unit = x => Js.log(x);

Output JS prior to v4.0.7

function f (){
    return 3
}
function f_used (x){
    console.log(x)
}

To make this hack work, in the application side, for a curried function application, we treat the function of arity 0 and arity 1 in the same way, this still works since curried function application could only happen on the ocaml function.

This trick is unintuitive, it makes code generated less predictable and it is not relevant any more, since we added native uncurried calling convention support later.

Therefore, we generate JS code in a more consistent style in this release:

let f : unit -> int = fun () -> 3

let f: unit => int = () => 3;

function f (param){
    return 3
}

So in your FFI code, if you have a callback which is expected to be of arity zero, use unit -> unit [@bs] or unit -> unit [@bs.uncurry], it is 100% correct. Note our previous trick will only make unit -> unit work most time, but it can not provide any guarantee.

Since we removed the trick, the curried runtime does not treat function of arity 0 and arity 1 in the same way, so if you have code like this

let f : unit -> int = [%bs.raw {|function () {
    return 3
}|}]

let f: unit => int = [%bs.raw {|function () {
  return 3
}|}];

It is not correct any more, the fix would be

let f : unit -> int = [%bs.raw{|function(param){
    return 3
}|}]

let f: unit => int = [%bs.raw {|function(param) {
  return 3
}|}];

let f : unit -> int [@bs] = [%bs.raw{|function(){
    return 3
}|}]

let f: (. unit) => int = [%bs.raw {|function() {
  return 3
}|}];

FFI is a double edge sword, it is a must have to ship your product, yet it is tricky, and there may be some undefined behavior you rely on but don't recognize, it is encouarged to always test your FFI in the boundary.

BuckleScript 4.0.0 (Part Two)

July 17, 2018

bs-platform 4.0.0 introduces a new runtime representation for optionals.

While beforehand None was represented at runtime as 0 and Some("hello") as an array ["hello"], the new representation tries to unbox optionals as much as possible.

Now None is represented as undefined and Some("hello") simply as "hello".

Generally speaking, Some(v) is represented as v, i.e. unboxed. The only exception is when v itself is None or Some(...(Some(None)), in which case a special boxed representation is used.

The construction of new values Some(-), and pattern matching | Some(-) => ..., perform some case analysis to decide when to box or unbox values. In the absence of nested optionals, the result of both operations will always be the indentity.

Because of that, it's possible to use type-based optimization to avoid performing case analysis in the first place. So while the generic function (x) => Some(x) will generate code to check wheter x should be boxed, the more type-specific function (x:int) => Some(x) is just compiled as the identity function, as it's clear from the type that no boxing is required.

For a high-level formalization of the boxing and unboxing operations, as well as the polymorphic comparison functions, see this gist.

One design choice was whether to represent None as null or as undefined. The choice of undefined was made because this allows a direct mapping for optional labeled arguments. As a consequence null is never boxed, so e.g. Some(null) is represented as null.

BuckleScript 4.0.0 (Part One)

July 17, 2018

bs-platform 4.0.0 is released! It has some nice features that we want to share with you, a more detailed list of changes is available here

In this post, I will talk about a new development workflow, all toolchains are self-contained in bs-platform, Cristiano will talk about the new runtime encoding for optional.

A simple approach to accelerate feedback loop in a reliable way

For mordern day-to-day development, developers expect that whenever files are changed, the build process is re-triggered automatically and browser reloaded instantly, this feedback loop should be quick enough to make developers not get distracted.

What we have before this release is as below:

Source file changes detected by bsb watch mode, rebuild
Webpack noticed JS files modified, rebundle and update the browser state

Both bsb and webpack has a watch mode, but they are architectured in fundamentally different ways that they achive different levels of reliability.

In bsb watch mode, there is no long running memory-hungry process, so whenever a file changed, a fresh process is started very fast and dies quickly, our experiment shows that in practice, a long running bsb process can work for a week without going into bad state, and such feedback loop is still instant.

Webpack holds lots of objects in memory and running for a longtime, it results in less reliability and OOM from time to time.

Another complexity introduced by a JS bundler is that it explodes users directory structure, for beginners trying to get started with bucklescript, installing such huge amount of directories is intimidating. In a slow network, this used to result in installation failure.

We understand that existing JS bundler has a huge ecosystem and it is invaluable in production mode, but we are exploring whether we can provide similar or even more reliable development experience without introducing such complexity.

Below is a new workflow we are exploring in this release:

NodeJS module loader in browser

Instead of bundling the modules like normal bundlers, we provide a NodeJS module loader so that it simply reloads the module without bundling.

Note ideally this can be achieved using ES6 module spec, however, it is not practical due to following reasons:

Most dependencies are not strictly ES6 compilant, this is true even for libraries authored in ES6 style

import {createElement} form "react" ; // not es6 compliant
import {createElement} from "node_modules/react/index" // not es6 compilant
import {createElement} from "./node_modules/react/index.js" // correct es6 module

ES6 modules does not allow an indirection, by introducing our own NodeJS module loader, we have an indirection and more meta-data about each module, so that we can do more reflection work in the future.

Loading in clean state without packing seems to introduce some redundant work, but on the contratry, it is very fast, it used to load 200 modules under 150ms, even better, since there is no cached state in a long running process, it is much more reliable.

WebSocket integration with bsb

We need a mechanism to communciate between browser and the build system so that whenever a rebuild finished, the browser get notified.

Instead of introducing more dependencies, we implemented a minimal websocket interface so that whenever a rebuild finishes, the weboscket clients which subscribe the port will get notified.

To conclude and try it out

So the proposed new work flow is as below: whenever a source file is changed, the bsb rebuild, if it build successfully, it will notify the browser to reload the NodeJS modules directly.

All the devtools are provided by bs-platform, the good thing is that there is no long running memory-hungry process, so that we expect it will deliver a more reliable and consistent experience.

You can try it out in bs-platform@4.0.0

bsb -init test -theme react-lite
cd test
npm install
npm start 
http-server # start a http server

open localhost:port/index.html, changes the reason source code and expect the browser show the changes.

BuckleScript 3.1.4

May 23, 2018

Hey again! The release two days ago removed the deprecated Js Boolean APIs (no longer needed since we compile OCaml booleans to JS boolean since 3.0.0). But folks have voiced that the removal was too hasty, as some of their dependencies still haven't upgraded to 3.0.0 and thus still needed the deprecated APIs.

We try to be diligent with our releases; hopefully this didn't churn too many people. To remediate the situation, we're putting those calls back for this version. Finger crossed that you don't have to wait on too many dependencies!

Sorry for the small churn, and thanks for all your feedback!

BuckleScript 3.1

May 21, 2018

List of changes here! Highlights:

Print Record Keys and Variant Tags

A picture's worth a thousand words:

Before	After

Please see Better Data Structures Printing (Debug Mode) for usage.

Even Better `bs.deriving abstract`

We've further polished our new way of binding to JS objects. The record fields of a bs.deriving abstract can now accept functions.

Pipe First Improvement

Pipe First now supports piping into variant tags!

let result = name |. preprocess |. Some

let result = name |. preprocess |. Some

We turn this into:

let result = Some(preprocess(name))

let result = Some(preprocess(name))

Js.Boolean Is Gone

Since BuckleScript 3.0, OCaml bool now compile to JS boolean. It was deprecated (all the boolean conversion functions became no-ops, with warnings during build), and now completely removed. No more need for the converter functions!

BuckleScript 3.0

April 16, 2018

bs-platform 3.0.0 is released! Go get it. This is a great release.

Highlighted features:

OCaml/Reason boolean are finally compiled as JS boolean! Due to historical limitations, OCaml true/false was compiled to 1/0 in JS. This caused quite a bit of confusion for newcomers. It now compiles to JS true/false. Special thanks to Cristiano for all the hard work.
New object type feature. This is an experimental and potentially much better way to bind to JS objects that potentially obsoletes the need for a few other APIs. Please see the linked docs and help us test it!
raw now accepts a function declaration with an unsafe string body: let f = [%raw (a, b) => "return a + b"] (OCaml syntax: let f = [%raw fun a b -> "return a + b"]). This makes embedding escape-hatch raw JS code even easier for the compiler to optimize for speed and readability, as you've indicated that the raw code block is a function, with specific numbers of arguments.

The usual changelog is here.

We've been working on the BuckleScript compiler for almost four years now; meanwhile, the OCaml type checker itself has already been engineered for almost three decades. After all this of work, we believe that BuckleScript has reached a stable and reliable stage.

Below is a list of to-dos that we will work on in the future. Suggestions welcome!

Upgrade the OCaml version. OCaml is quite a stable language; there are not too many changes between BuckleScript's OCaml version and latest stable one. Nonetheless, it's good to keep up with the OCaml ecosystem.
A uniform representation for local modules/global modules. Currently local modules are compiled to array, while global modules are compiled to ES6/CommonJS/AMD modules (the cost of local modules is low though, thanks to aggressive inlining).
Continue improving Belt. Some initial nice numbers here.
Enhance FFI to allow creation of idiomatic, type safe JS classes.
Introduce a debug mode to enhance the printing of OCaml data structures.
Performance. The compiler performance and generation of more performant and readable code is always our top concern.

Thank you for all the support so far!

← Prev Next →