![Yuriko Hirota](https://web.dev/images/authors/yuriko-hirota.jpg) Yuriko Hirota [X](https://twitter.com/yuriko_fuuu) [LinkedIn](https://www.linkedin.com/in/yurikohirota) ![Anna Ogawa](https://web.dev/images/authors/annaogawa.png) Anna Ogawa [X](https://twitter.com/piropiroanna) [LinkedIn](https://www.linkedin.com/in/%E5%AE%89%E5%A5%88-%E5%B0%8F%E5%B7%9D-43b159242) ![Thomas Steiner](https://web.dev/images/authors/thomassteiner.jpg) Thomas Steiner [GitHub](https://github.com/tomayac) [LinkedIn](https://www.linkedin.com/in/thomassteinerlinkedin) [Mastodon](https://toot.cafe/@tomayac) [Bluesky](https://bsky.app/profile/tomayac.com) [Homepage](https://blog.tomayac.com/)

<br />

pixiv is an online community service for illustrators and illustration
enthusiasts to communicate with each other through their content. It lets people
post their own illustrations. They have over
[84 million users](https://newsroom.paypal-corp.com/2023-05-19-Pixivs-Online-Creative-Community-Continues-To-Grow-From-Japan-To-North-America)
across the globe, and more than
[120 million](https://newsroom.paypal-corp.com/2023-05-19-Pixivs-Online-Creative-Community-Continues-To-Grow-From-Japan-To-North-America)
art pieces posted as of May 2023.

pixiv Sketch is one of the services provided by pixiv. It's used to draw
artworks on the website, using fingers or styluses. It supports a variety of
features for drawing amazing illustrations including numerous types of brushes,
layers, and bucket painting, and also allows people to livestream their drawing
process.

In this case study, we'll take a look at how pixiv Sketch improved the
performance and the quality of their web app by using some new web platform
features like WebGL, WebAssembly, and WebRTC.

## Why develop a sketching app on the web?

pixiv Sketch was first released on the web and on iOS in 2015. Their target
audience for the web version was primarily the desktop, which still is the most
major platform used by the illustration community.

Here are pixiv's top two reasons for choosing to develop a web version instead
of a desktop app:

- It is very costly to create apps for Windows, Mac, Linux, and more. The web reaches any browser on the desktop.
- The web has the best reach across platforms. The web is available on desktop and mobile, and on every operating system.

## Technology

pixiv Sketch has a number of different brushes for users to choose from. Before
adopting WebGL, there was only one type of brush since the 2D canvas was too
limited to depict the complex texture of different brushes, like coarse edges of
a pencil and differing width and color intensity that changes on sketch
pressure.

### Creative types of brushes using WebGL

However, with the adoption of WebGL, they were able to add more varieties in
brush details and increase the number of available brushes to seven.

![The seven different brushes in pixiv ranging from fine to coarse, sharp to unsharp, pixelated to smooth, etc.](https://web.dev/static/case-studies/pixiv/brushes.png)

Using 2D canvas context, it was only possible to draw lines that have a simple
texture with evenly distributed width, like the following screenshot:

![Brush stroke with simple texture.](https://web.dev/static/case-studies/pixiv/line.png)

These lines were drawn by creating paths and drawing strokes, but WebGL
reproduces this using point sprites and shaders, shown in the following code
samples

The following example demonstrates a vertex shader.

    precision highp float;

    attribute vec2 pos;
    attribute float thicknessFactor;
    attribute float opacityFactor;

    uniform float pointSize;

    varying float varyingOpacityFactor;
    varying float hardness;

    // Calculate hardness from actual point size
    float calcHardness(float s) {
      float h0 = .1 * (s - 1.);
      float h1 = .01 * (s - 10.) + .6;
      float h2 = .005 * (s - 30.) + .8;
      float h3 = .001 * (s - 50.) + .9;
      float h4 = .0002 * (s - 100.) + .95;
      return min(h0, min(h1, min(h2, min(h3, h4))));
    }

    void main() {
      float actualPointSize = pointSize * thicknessFactor;
      varyingOpacityFactor = opacityFactor;
      hardness = calcHardness(actualPointSize);
      gl_Position = vec4(pos, 0., 1.);
      gl_PointSize = actualPointSize;
    }

The following example shows sample code for a fragment shader.

    precision highp float;

    const float strength = .8;
    const float exponent = 5.;

    uniform vec4 color;

    varying float hardness;
    varying float varyingOpacityFactor;

    float fallOff(const float r) {
        // w is for width
        float w = 1. - hardness;
        if (w < 0.01) {
         return 1.;
        } else {
         return min(1., pow(1. - (r - hardness) / w, exponent));
        }
    }

    void main() {
        vec2 texCoord = (gl_PointCoord - .5) * 2.;
        float r = length(texCoord);

        if (r > 1.) {
         discard;
        }

        float brushAlpha = fallOff(r) * varyingOpacityFactor * strength * color.a;

        gl_FragColor = vec4(color.rgb, brushAlpha);
    }

The use of point sprites makes it straightforward to vary thickness and shading
in response to drawing pressure, allowing the following strong and weak lines to
be expressed, like these:

![Sharp, even brush stroke with thin ends.](https://web.dev/static/case-studies/pixiv/thin.png)

![Unsharp brush stroke with more pressure applied in the middle.](https://web.dev/static/case-studies/pixiv/unsharp.png)

In addition, implementations using point sprites can now attach textures by
using a separate shader, allowing for efficient representation of brushes with
textures such as pencil and felt-tip pen.

### Stylus support on the browser

Using a digital stylus has become extremely popular for digital artists. Modern
browsers support the
[PointerEvent API](https://developer.mozilla.org/docs/Web/API/PointerEvent) that
enables users to use a stylus on their device: Use `PointerEvent.pressure`to
measure pen pressure, and use `PointerEvent.tiltX`, `PointerEvent.tiltY` to
measure the angle of the pen to the device.

In order to perform brush strokes with a point sprite, the `PointerEvent` must
be interpolated and converted into a more fine-grained event sequence. In
PointerEvent, the orientation of the stylus can be obtained in the form of polar
coordinates, but pixiv Sketch converts them to a vector representing the
orientation of the stylus before using them.

    function getTiltAsVector(event: PointerEvent): [number, number, number] {
      const u = Math.tan((event.tiltX / 180) * Math.PI);
      const v = Math.tan((event.tiltY / 180) * Math.PI);
      const z = Math.sqrt(1 / (u * u + v * v + 1));
      const x = z * u;
      const y = z * v;
      return [x, y, z];
    }

    function handlePointerDown(event: PointerEvent) {
      const position = [event.clientX, event.clientY];
      const pressure = event.pressure;
      const tilt = getTiltAsVector(event);

      interpolateAndRender(position, pressure, tilt);
    }

### Multiple drawing layers

Layers are one of the most unique concepts in digital drawing. They let users
draw different pieces of illustration on top of each other, and allow for edits
layer by layer. pixiv Sketch provides layer functions much like other digital
drawing apps do.

Conventionally, it is possible to implement layers by using several `<canvas>`
elements with `drawImage()` and compositing operations. However this is
problematic because with 2D canvas context, there is no other choice but to use
[`CanvasRenderingContext2D.globalCompositeOperation`](https://developer.mozilla.org/docs/Web/API/CanvasRenderingContext2D/globalCompositeOperation)
composition mode, which is predefined and largely limits the scalability. By
using WebGL and writing the shader, it allows the developers to use composition
modes that are not predefined by the API. In the future, pixiv Sketch will
implement the layer feature using WebGL for greater scalability and flexibility.

Here's the sample code for layer composition:

    precision highp float;

    uniform sampler2D baseTexture;
    uniform sampler2D blendTexture;
    uniform mediump float opacity;

    varying highp vec2 uv;

    // for normal mode
    vec3 blend(const vec4 baseColor, const vec4 blendColor) {
      return blendColor.rgb;
    }

    // for multiply mode
    vec3 blend(const vec4 baseColor, const vec4 blendColor) {
      return blendColor.rgb * blendColor.rgb;
    }

    void main()
    {
      vec4 blendColor = texture2D(blendTexture, uv);
      vec4 baseColor = texture2D(baseTexture, uv);

      blendColor.a *= opacity;

      float a1 = baseColor.a * blendColor.a;
      float a2 = baseColor.a * (1. - blendColor.a);
      float a3 = (1. - baseColor.a) * blendColor.a;

      float resultAlpha = a1 + a2 + a3;

      const float epsilon = 0.001;

      if (resultAlpha > epsilon) {
        vec3 noAlphaResult = blend(baseColor, blendColor);
        vec3 resultColor =
            noAlphaResult * a1 + baseColor.rgb * a2 + blendColor.rgb * a3;
        gl_FragColor = vec4(resultColor / resultAlpha, resultAlpha);
      } else {
        gl_FragColor = vec4(0);
      }
    }

### Large area painting with the bucket function

> [!NOTE]
> **Note:** WebAssembly has superseded asm.js, the approach described in this subsection. While it still works and provides significant performance benefits for pixiv, they are planning to work on migrating to WebAssembly. You shouldn't start new projects based on asm.js today.

The pixiv Sketch iOS and Android apps already provided the bucket feature, but
the web version did not. The app version of the bucket function was implemented
in C++.

With the codebase already available in C++, pixiv Sketch used Emscripten and
asm.js to implement the bucket function into the web version.

    bfsQueue.push(startPoint);

    while (!bfsQueue.empty()) {
      Point point = bfsQueue.front();
      bfsQueue.pop();
      /* ... */
      bfsQueue.push(anotherPoint);
    }

Using [asm.js](http://asm.js) enabled a performant solution. Comparing the
execution time of pure JavaScript versus [asm.js](http://asm.js), the execution
time using asm.js is shortened by 67%. This is expected to be even better when
using WASM.

**Test details:**

- **How:** Paint 1180x800px area with bucket function
- **Test device:** MacBook Pro (M1 Max)

**Execution time:**

- **Pure JavaScript:** 213.8ms
- **asm.js:** 70.3ms

Using Emscripten and [asm.js](http://asm.js), pixiv Sketch was able to
successfully release the bucket feature by reusing the codebase from the
platform-specific app version.

## Live-streaming while drawing

pixiv Sketch offers the feature to live-stream while drawing, through the pixiv
Sketch LIVE web app. This uses the WebRTC API, combining the microphone audio
track obtained from `getUserMedia()` and the `MediaStream` video track retrieved
from the `<canvas>` element.

    const canvasElement = document.querySelector('#DrawCanvas');
    const framerate = 24;
    const canvasStream = canvasElement.captureStream(framerate);
    const videoStreamTrack = canvasStream.getVideoTracks()[0];

    const audioStream = await navigator.mediaDevices.getUserMedia({
      video: false,
      audio: {},
    });
    const audioStreamTrack = audioStream.getAudioTracks()[0];

    const stream = new MediaStream();
    stream.addTrack(audioStreamTrack.clone());
    stream.addTrack(videoStreamTrack.clone());

## Conclusions

With the power of new APIs like WebGL, WebAssembly and WebRTC, you can create a
complex app on the web platform and scale it across any device. You can learn
more about the technologies introduced in this case study at the following
links:

- [WebGL](https://web.dev/articles/webgl-fundamentals)
  - Also check out [WebGPU](https://web.dev/blog/web-platform-05-2023), the successor of WebGL
- [WebAssembly](https://web.dev/explore/webassembly)
- [WebRTC](https://web.dev/articles/webrtc-basics)
- [Original article in Japanese](https://inside.pixiv.blog/nontan/2557)