OrgPad logo

High performance web app in Clojure

Created by Pavel Klavík

#Clojure, #ClojureScript, #OrgPad, #architecture, #performance, #programming, #web, #web apps

High performance web app in Clojure

Pavel Klavík

pavel v obleku

High performance web app in Clojure

800px-Clojure logo.svg

https://orgpad.info/s/highperf

CTO and founder of OrgPad

799px-OrgPad Logo.svg

PhD from data visualization at Charles University, Prague

Software engineer at Google (1.5 years)

Why Clojure??

Language is really simple

Instead f(x,y,z) write (f x y z).

Interactive programming

REPL + hot code reloading

Data-oriented programming

Sharvit-DOP-HI

Just text

Shell programming. Custom serialization and parsing.

POST /cgi-bin/process.cgi HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: www.tutorialspoint.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 48
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

licenseID=string&content=string&/paramsXML=string

Nested data

Screenshot (143)

Objects

Reinventing custom language for everything.

Screenshot (142)

Best distribution platform

Browser

Horrible performance

Great and powerful prototyping

Incredibly complex environment

For example, event system is really complex.

sipky

nastavovani-stylu

Clojure is very fast

Of course, one has to understand it first. For example, the article below just says "I don't know how to write efficient Clojure."

image

https://blog.asciinema.org/post/smaller-faster/

Efficient immutable data structures

Clojure works with values which cannot be changed. But it is possible to efficiently create modified values.

Screenshot (152)

Rich Hickey created these data structures so they are efficient enought for 99% of all applications. Time to reading is 1-2x, time of writing is 1-4x. Often, the program can be in total more efficient which is counterintuitive.

Screenshot (153)

Mutable

Immutable

Single thread

Multi thread

😭

😆

Transients

2x speed improvement when doing a lot of consecutive changes in a data structure.

Using native mutable types when necessary: image data in Uint8ClampedArray

image

Atoms: dealing with state changes

image

IT architecture of OrgPad

The entire code of OrgPadu consists of 112 000 lines

image

Code survival chart according to years

stack-plot

Generated in git-of-theseus

https://github.com/erikbern/git-of-theseus

Minio S3 storage

Screenshot NodeJS server in ClojureScript

Main JVM server in Clojure

Handlers

Web socket handlers

Browser client in ClojureScript

atom admin-db/db

Computation of usage statistics

PostgreSQL db

atom cache-db/db

cpu-load

atom session/store

Sessions

Sending emails

Stripe payment gateway

Runs Chrome using Puppeteer

Screenshots of OrgPages

Printing of OrgPages

Reagent + Re-frame for most rendering

reactive atom @re-frame.db/app-db

Frame by frame rendering + animations

atom @anim-db/db

Autoresize of cells

We repeatedly draw a cell outside of the screen. For a selected width, we let the browser to compute the corresponding height. We are searching a space of possible sizes to find an ideal size.

bunky

Layout improvements using WebWorkers

atom @autoresize-db/db

one state atom for each instance

atom @anim-db/db

Locating nearby cells/links

User/outside events

Rendering each frame in requestAnimationFrame

Chunks for cells and links

Computing new animation targets

Update what is visible on the screen, what just appeared, just disappeared

Update animations for playing cells/links/etc.

some mutable caches

Spring based animations using physics

You see them all around here. For example, the cells have springs attached, attracting them to their new positions. Thanks to this, animations can be smoothly assembled and connected. This method of animation was devised by designers at Apple.

OrgPad doesn't solve these with simulation but by solving the differential equations of spring motion. The spring has two parameters:

These parameters define three types of springs:

Different formulas are used for them, into which initial position and velocity are substituted.

7CCH51r

Spring Animations in OrgPad

Mutating positions of cells in DOM

Rendering links and cached cells into 2D canvas

Letter atlas

Cached cells

Assistent for graph layout improvements

The entire process is repeated by running about 25 iterations.

  1. Forces are computed.
  2. Nodes and links are shifted according to these forces.

The computation runs in a standalone thread as a WebWorker. The result is then sent to the main thread which animates nodes and links to their new positions.

Cells and links apply repulsive forces on each other

We use Archimedian forces, so force is a change in position, not velocity. It is much easier and it gives good results.

All forces applied on a cell/link are composed and we apply only their sum.

Screenshot (126)

Cells

They have both position and size. Sizes may change as cells are opened/closed or change their content.

Forces shift cells.

Repulsive forces

When cell rectangles are close to each other or overlap, they apply repulsive forces on each other. When they are further than a fixed limit, there are no repulsive forces anymore.

odpuzování

Links

Forces bend links in their direction.

Bended links are quadratic Bezier curves. Sufficiently straight links are line segments.

Repulsive forces

When a link is near a cell to which it is not attached, they apply repulsive forces on each other. Computation is much slower for bended links than for straight ones (roughly 10x slower).

spoj

Straightening of bended links

vychyleny spoj

Repulsive forces between parallel links

Parallel links repulse each other. It works very well for a few links which is sufficient.

odpuzování spojů2

Many parallel links look interesting

image

Attractive forces for long links

Originally we applied them but they pushed cells to close to each other. It was not possible to have long links within the graph. Since we are creating an assistent which does not draw graphs from nothing, they are not that necessary.

dlouhé spoje

Packed graph

Shared for everyone and stored at the server. All nodes are closed.

image

Expanded graph

Each browser window displays its own, according to which cells are currently opened. Expanded graph is used for working but it is not shared with the server and others.

image

Multiple layouts in collaborative work

Everyone can have different cells opened so displayed layouts are different. How should we interpret changes in one layout to another layout?

Attracted to their packed positions

We try to make the expanded graph as close to the packed one as possible, otherwise changes in the expanded graph would be too different from changes in the packed one. Therefore cells are slightly pushed towards their original packed positions.

otevření

Performance improvements already implemented

Ideal speed of recomputation would be 100 to 200 ms while doing many more iterations. We can do it for small graphs but not for large ones. Till recently, the biggest bottleneck was rendering so we did not do further performance improvements.

Bended links are approximated by two line segments

Only when this bended line is near a cell, the exact distance is computed. Almost always it is not true, so it greatly improves speed of bended links.

jeden zlom

The space is divided into square chunks. Only pairs sharing a chunk are compared.

215924534 826468104729397 8806203082278002716 n