Construyendo una galería de imágenes usando PixiJS y WebGL

A veces, tenemos que ir un poco más allá de HTML, CSS y JavaScript para crear la interfaz de usuario que necesitamos y, en su lugar, utilizar otros recursos, como SVG, WebGL, lienzo y otros.

Por ejemplo, los efectos más sorprendentes se pueden crear con WebGL, porque es una API de JavaScript diseñada para representar gráficos interactivos en 2D y 3D dentro de cualquier navegador web compatible, lo que permite el procesamiento de imágenes acelerado por GPU.

Dicho esto, trabajar con WebGL puede resultar muy complejo. Como tal, existe una variedad de bibliotecas que lo hacen relativamente más fácil, como PixiJS , Three.js y Babylon.js , entre otras. Vamos a trabajar con uno específico de ellos, PixiJS, para crear una galería de imágenes aleatorias inspiradas en este fragmento de una toma de Dribbble de Zhenya Rynzhuk .

Esto parece difícil, pero en realidad no es necesario tener conocimientos avanzados de WebGL o incluso de PixiJS para seguir adelante, aunque algunos conocimientos básicos de Javascript (ES6) serán útiles. Quizás incluso quieras comenzar familiarizándote con el concepto básico de sombreadores de fragmentos utilizados en WebGL, con El Libro de los sombreadores como un buen punto de partida.

Dicho esto, ¡profundicemos en el uso de PixiJS para crear este efecto WebGL!

Configuración inicial

Esto es lo que necesitaremos para comenzar:

1. Agregue la biblioteca PixiJS como un script en HTML.
2. Tener un canvaselemento (o agregarlo dinámicamente desde Javascript) para renderizar la aplicación.
3. Inicialice la aplicación con new PIXI.Application(options).

Mira, nada demasiado loco todavía. Aquí está el JavaScript que podemos usar como modelo:

// Get canvas viewconst view = document.querySelector('.view')let width, height, app// Set dimensionsfunction initDimensions () {  width = window.innerWidth  height = window.innerHeight}// Init the PixiJS Applicationfunction initApp () {  // Create a PixiJS Application, using the view (canvas) provided  app = new PIXI.Application({ view })  // Resizes renderer view in CSS pixels to allow for resolutions other than 1  app.renderer.autoDensity = true  // Resize the view to match viewport dimensions  app.renderer.resize(width, height)}// Init everythingfunction init () {  initDimensions()  initApp()}// Initial callinit()

Al ejecutar este código lo único que veremos es una pantalla negra además de un mensaje como este si abrimos la consola:
PixiJS 5.0.2 - WebGL 2 - http://www.pixijs.com/.

¡Estamos listos para empezar a dibujar en el lienzo usando PixiJS y WebGL!

Creando el fondo de la cuadrícula con un WebGL Shader

A continuación crearemos un fondo que contiene una cuadrícula, lo que nos permitirá visualizar claramente el efecto de distorsión que buscamos. Pero primero, debemos saber qué es un sombreador y cómo funciona. Anteriormente recomendé El Libro de los sombreadores como punto de partida para aprender sobre ellos y estos conceptos entrarán en juego. Si aún no lo has hecho, te recomiendo encarecidamente que revises ese material y solo después continúes aquí.

Vamos a crear un sombreador de fragmentos que imprima un fondo de cuadrícula en la pantalla:

// It is required to set the float precision for fragment shaders in OpenGL ES// More info here: https://stackoverflow.com/a/28540641/4908989#ifdef GL_ESprecision mediump float;#endif// This function returns 1 if `coord` correspond to a grid line, 0 otherwisefloat isGridLine (vec2 coord) {  vec2 pixelsPerGrid = vec2(50.0, 50.0);  vec2 gridCoords = fract(coord / pixelsPerGrid);  vec2 gridPixelCoords = gridCoords * pixelsPerGrid;  vec2 gridLine = step(gridPixelCoords, vec2(1.0));  float isGridLine = max(gridLine.x, gridLine.y);  return isGridLine;}// Main functionvoid main () {  // Coordinates for the current pixel  vec2 coord = gl_FragCoord.xy;  // Set `color` to black  vec3 color = vec3(0.0);  // If it is a grid line, change blue channel to 0.3  color.b = isGridLine(coord) * 0.3;  // Assing the final rgba color to `gl_FragColor`  gl_FragColor = vec4(color, 1.0);}

Este código se extrae de una demostración de Shadertoy , que es una gran fuente de inspiración y recursos para sombreadores.

Para utilizar este sombreador, primero debemos cargar el código del archivo en el que se encuentra y, solo después de que se haya cargado correctamente, inicializaremos la aplicación.

// Loaded resources will be hereconst resources = PIXI.Loader.shared.resources// Load resources, then init the appPIXI.Loader.shared.add([  'shaders/backgroundFragment.glsl']).load(init)

Ahora, para que nuestro sombreador funcione y podamos ver el resultado, agregaremos un nuevo elemento (un vacío Sprite) al escenario, que usaremos para definir un filtro. Esta es la forma en que PixiJS nos permite ejecutar sombreadores personalizados como el que acabamos de crear.

// Init the gridded backgroundfunction initBackground () {  // Create a new empty Sprite and define its size  background = new PIXI.Sprite()  background.width = width  background.height = height  // Get the code for the fragment shader from the loaded resources  const backgroundFragmentShader = resources['shaders/backgroundFragment.glsl'].data  // Create a new Filter using the fragment shader  // We don't need a custom vertex shader, so we set it as `undefined`  const backgroundFilter = new PIXI.Filter(undefined, backgroundFragmentShader)  // Assign the filter to the background Sprite  background.filters = [backgroundFilter]  // Add the background to the stage  app.stage.addChild(background)}

Y ahora vemos el fondo cuadriculado con líneas azules. Mire de cerca porque las líneas son un poco tenues contra el color de fondo oscuro.

El efecto de distorsión

Nuestro fondo ya está listo, así que veamos cómo podemos agregar el efecto deseado (Distorsión de lente cúbica) a todo el escenario, incluido el fondo y cualquier otro elemento que agreguemos más adelante, como imágenes. Para ello, necesitamos crear un nuevo filtro y agregarlo al escenario. ¡Sí, también podemos definir filtros que afecten a toda la etapa de PixiJS!

Esta vez, hemos basado el código de nuestro sombreador en esta increíble demostración de Shadertoy que implementa el efecto de distorsión usando diferentes parámetros configurables.

#ifdef GL_ESprecision mediump float;#endif// Uniforms from Javascriptuniform vec2 uResolution;uniform float uPointerDown;// The texture is defined by PixiJSvarying vec2 vTextureCoord;uniform sampler2D uSampler;// Function used to get the distortion effectvec2 computeUV (vec2 uv, float k, float kcube) {  vec2 t = uv - 0.5;  float r2 = t.x * t.x + t.y * t.y;  float f = 0.0;  if (kcube == 0.0) {    f = 1.0 + r2 * k;  } else {    f = 1.0 + r2 * (k + kcube * sqrt(r2));  }  vec2 nUv = f * t + 0.5;  nUv.y = 1.0 - nUv.y;  return nUv;}void main () {  // Normalized coordinates  vec2 uv = gl_FragCoord.xy / uResolution.xy;  // Settings for the effect  // Multiplied by `uPointerDown`, a value between 0 and 1  float k = -1.0 * uPointerDown;  float kcube = 0.5 * uPointerDown;  float offset = 0.02 * uPointerDown;    // Get each channel's color using the texture provided by PixiJS  // and the `computeUV` function  float red = texture2D(uSampler, computeUV(uv, k + offset, kcube)).r;  float green = texture2D(uSampler, computeUV(uv, k, kcube)).g;  float blue = texture2D(uSampler, computeUV(uv, k - offset, kcube)).b;    // Assing the final rgba color to `gl_FragColor`  gl_FragColor = vec4(red, green, blue, 1.0);}

Esta vez usaremos dos uniformes. Los uniformes son variables que pasamos al sombreador mediante JavaScript:

• uResolution: Este es un objeto JavaScript que incluye {x: width, y: height}. Este uniforme nos permite normalizar las coordenadas de cada píxel del rango [0, 1].
• uPointerDown: Se trata de un flotador en el rango [0, 1], que nos permite animar el efecto de distorsión, aumentando proporcionalmente su intensidad.

Veamos el código que tenemos que agregar a nuestro JavaScript para ver el efecto de distorsión causado por nuestro nuevo sombreador:

// Target for pointer. If down, value is 1, else value is 0// Here we set it to 1 to see the effect, but initially it will be 0let pointerDownTarget = 1let uniforms// Set initial values for uniformsfunction initUniforms () {  uniforms = {    uResolution: new PIXI.Point(width, height),    uPointerDown: pointerDownTarget  }}// Set the distortion filter for the entire stageconst stageFragmentShader = resources['shaders/stageFragment.glsl'].dataconst stageFilter = new PIXI.Filter(undefined, stageFragmentShader, uniforms)app.stage.filters = [stageFilter]

¡Ya podemos disfrutar de nuestro efecto de distorsión!

Este efecto es estático por el momento, por lo que no es muy divertido todavía. A continuación, veremos cómo podemos hacer que el efecto responda dinámicamente a eventos de puntero.

Escuchar eventos de puntero

PixiJS hace que sea sorprendentemente sencillo escuchar eventos, incluso eventos múltiples que responden por igual a las interacciones táctiles y del mouse. En este caso queremos que nuestra animación funcione igual de bien en escritorio que en dispositivo móvil, por lo que debemos escuchar los eventos correspondientes a ambas plataformas.

PixiJs proporciona un interactiveatributo que nos permite hacer precisamente eso. Lo aplicamos a un elemento y comenzamos a escuchar eventos con una API similar a jQuery:

// Start listening eventsfunction initEvents () {  // Make stage interactive, so it can listen to events  app.stage.interactive = true  // Pointer  touch events are normalized into  // the `pointer*` events for handling different events  app.stage    .on('pointerdown', onPointerDown)    .on('pointerup', onPointerUp)    .on('pointerupoutside', onPointerUp)    .on('pointermove', onPointerMove)}

A partir de aquí, comenzaremos a utilizar un tercer uniforme ( uPointerDiff), que nos permitirá explorar la galería de imágenes mediante arrastrar y soltar. Su valor será igual a la traducción de la escena mientras exploramos la galería. A continuación se muestra el código correspondiente a cada una de las funciones de manejo de eventos:

// On pointer down, save coordinates and set pointerDownTargetfunction onPointerDown (e) {  console.log('down')  const { x, y } = e.data.global  pointerDownTarget = 1  pointerStart.set(x, y)  pointerDiffStart = uniforms.uPointerDiff.clone()}// On pointer up, set pointerDownTargetfunction onPointerUp () {  console.log('up')  pointerDownTarget = 0}// On pointer move, calculate coordinates difffunction onPointerMove (e) {  const { x, y } = e.data.global  if (pointerDownTarget) {    console.log('dragging')    diffX = pointerDiffStart.x + (x - pointerStart.x)    diffY = pointerDiffStart.y + (y - pointerStart.y)  }}

Seguiremos sin ver ninguna animación si miramos nuestro trabajo, pero sí podemos empezar a ver como los mensajes que hemos definido en cada función del controlador de eventos se imprimen correctamente en la consola.

¡Pasemos ahora a implementar nuestras animaciones!

Animar el efecto de distorsión y la funcionalidad de arrastrar y soltar.

The first thing we need to start an animation with PixiJS(orany canvas-based animation) is an animation loop. It usually consists of a functionthat iscalled continuously, usingrequestAnimationFrame, which in each call renders the graphics on thecanvas element, thus producing the desired animation.

We can implement our own animation loopin PixiJS, or we can use the utilities included in the library. In this case, we will use theaddmethod ofapp.ticker, which allows us to pass a function that will be executed in each frame. At the end of theinitfunction we will add this:

// Animation loop// Code here will be executed on every animation frameapp.ticker.add(() = {  // Multiply the values by a coefficient to get a smooth animation  uniforms.uPointerDown += (pointerDownTarget - uniforms.uPointerDown) * 0.075  uniforms.uPointerDiff.x += (diffX - uniforms.uPointerDiff.x) * 0.2  uniforms.uPointerDiff.y += (diffY - uniforms.uPointerDiff.y) * 0.2})

Meanwhile, in theFilterconstructor for the background, we will pass theuniformsin thestagefilter. This allowsus to simulate the translation effect of the background with thistinymodification in the corresponding shader:

uniform vec2 uPointerDiff;void main () {  // Coordinates minus the `uPointerDiff` value  vec2 coord = gl_FragCoord.xy - uPointerDiff;  // ... more code here ...}

And now we can see the distortion effect in action,includingthe draganddrop functionality for the gridd background. Play with it!

Randomly generate a masonry grid layout

To make our UI more interesting, we can randomly generate the sizing and dimensions of the grid cells.That is, each image can have different dimensions, creating a kind of masonry layout.

Let’s useUnsplash Source, which will allow us to get random images fromUnsplashanddefine the dimensions we want. This will facilitate the task of creating a random masonry layout, since the images can have any dimension that we want, and therefore, generate the layout beforehand.

To achieve this, we will use analgorithm that executesthe following steps:

1. We will start with a list of rectangles.
2. We will select the first rectangle in the listdivide it into tworectangles with random dimensions, as long as both rectangles have dimensions equal to or greater than the minimum established limit.We’ll add a check to make sure it’s possible and, if it is,add both resulting rectangles to the list.
3. If the list is empty,we willfinishexecuting. If not,we’llgo back to steptwo.

Ithink you’ll get a much better understanding of how the algorithm works in this next demo.Use the buttonsto see how it runs:Nextwillexecutestep two,Allwillexecute the entire algorithm, andResetwillreset to step one.

Drawing solid rectangles

Now that we canproperlygenerate our random grid layout, we will use the list of rectangles generated by the algorithm to draw solid rectangles in our PixiJSapplication.That way,we can see if it works and makeadjustments before adding the images using the Unsplash Source API.

To draw those rectangles, we will generate a random grid layout that isfivetimes bigger than the viewport and position it in the center of the stage.Thatallows us tomove with some freedom to any direction in the gallery.

// Variables and settings for gridconst gridSize = 50const gridMin = 3let gridColumnsCount, gridRowsCount, gridColumns, gridRows, gridlet widthRest, heightRest, centerX, centerY, rects// Initialize the random grid layoutfunction initGrid () {  // Getting columns  gridColumnsCount = Math.ceil(width / gridSize)  // Getting rows  gridRowsCount = Math.ceil(height / gridSize)  // Make the grid 5 times bigger than viewport  gridColumns = gridColumnsCount * 5  gridRows = gridRowsCount * 5  // Create a new Grid instance with our settings  grid = new Grid(gridSize, gridColumns, gridRows, gridMin)  // Calculate the center position for the grid in the viewport  widthRest = Math.ceil(gridColumnsCount * gridSize - width)  heightRest = Math.ceil(gridRowsCount * gridSize - height)  centerX = (gridColumns * gridSize / 2) - (gridColumnsCount * gridSize / 2)  centerY = (gridRows * gridSize / 2) - (gridRowsCount * gridSize / 2)  // Generate the list of rects  rects = grid.generateRects()}

So far,we have generated the list of rectangles. To add them to the stage, it is convenient to create a container, since then we can add the images to the same container and facilitate the movement when we drag the gallery.

Creating a container in PixiJS is like this:

let container// Initialize a Container element for solid rectangles and imagesfunction initContainer () {  container = new PIXI.Container()  app.stage.addChild(container)}

Now we can now add the rectangles to the container so they can be displayed on the screen.

// Padding for rects and imagesconst imagePadding = 20// Add solid rectangles and images// So far, we will only add rectanglesfunction initRectsAndImages () {  // Create a new Graphics element to draw solid rectangles  const graphics = new PIXI.Graphics()  // Select the color for rectangles  graphics.beginFill(0xAA22CC)  // Loop over each rect in the list  rects.forEach(rect = {    // Draw the rectangle    graphics.drawRect(      rect.x * gridSize,      rect.y * gridSize,      rect.w * gridSize - imagePadding,      rect.h * gridSize - imagePadding    )  })  // Ends the fill action  graphics.endFill()  // Add the graphics (with all drawn rects) to the container  container.addChild(graphics)}

Note that we have added to the calculations a padding(imagePadding)for each rectangle. In this way the images will have some space among them.

Finally, in the animation loop, weneed toadd the following code to properly define the position for the container:

// Set position for the containercontainer.x = uniforms.uPointerDiff.x - centerXcontainer.y = uniforms.uPointerDiff.y - centerY

Andnow weget the following result:

But there are still some details to fix,likedefining limits forthedraganddropfeature.Let’s add thisto theonPointerMoveevent handler, where we effectively check the limits according to the size of the grid we have calculated:

diffX = diffX  0 ? Math.min(diffX, centerX + imagePadding) : Math.max(diffX, -(centerX + widthRest))diffY = diffY  0 ? Math.min(diffY, centerY + imagePadding) : Math.max(diffY, -(centerY + heightRest))

Another small detail thatmakes things more refined is to add an offset to the grid background. Thatkeeps the blue grid lines in tact.We just have to add the desired offset(imagePadding / 2in our case) to the background shader this way:

// Coordinates minus the `uPointerDiff` value, and plus an offsetvec2 coord = gl_FragCoord.xy - uPointerDiff + vec2(10.0);

And we will get the final design for our random grid layout:

Adding images from Unsplash Source

We have our layout ready, so we are all set to addimages to it. To add an imagein PixiJS, we need aSprite, which definesthe image asaTextureof it. There are multiple ways of doing this. In our case, we will first create an emptySpritefor each image and,only when theSpriteis inside the viewport, we will load the image, create the Textureand add it to theSprite.Sound like a lot? We’ll go through it step-by-step.

To create the empty sprites, we will modify theinitRectsAndImages function. Please pay attention to the comments for a better understanding:

// For the list of imageslet images = []// Add solid rectangles and imagesfunction initRectsAndImages () {  // Create a new Graphics element to draw solid rectangles  const graphics = new PIXI.Graphics()  // Select the color for rectangles  graphics.beginFill(0x000000)  // Loop over each rect in the list  rects.forEach(rect = {    // Create a new Sprite element for each image    const image = new PIXI.Sprite()    // Set image's position and size    image.x = rect.x * gridSize    image.y = rect.y * gridSize    image.width = rect.w * gridSize - imagePadding    image.height = rect.h * gridSize - imagePadding    // Set it's alpha to 0, so it is not visible initially    image.alpha = 0    // Add image to the list    images.push(image)    // Draw the rectangle    graphics.drawRect(image.x, image.y, image.width, image.height)  })  // Ends the fill action  graphics.endFill()  // Add the graphics (with all drawn rects) to the container  container.addChild(graphics)  // Add all image's Sprites to the container  images.forEach(image = {    container.addChild(image)  })}

So far, we only have empty sprites.Next, we will create a function that’sresponsible for downloading an image and assigningit asTextureto the correspondingSprite. This function will only be called if theSpriteis inside the viewportso that the image only downloads when necessary.

On the other hand, if the gallery is dragged and aSpriteis no longer inside the viewport during the course of the download, that request may be aborted, since we are going to use anAbortController(moreon this on MDN). In this way, we will cancel the unnecessary requests as we drag the gallery, giving priority to the requests corresponding to the sprites that are inside the viewport at every moment.

Let’s see the code to land the ideas a little better:

// To store image's URL and avoid duplicateslet imagesUrls = {}// Load texture for an image, giving its indexfunction loadTextureForImage (index) {  // Get image Sprite  const image = images[index]  // Set the url to get a random image from Unsplash Source, given image dimensions  const url = `https://source.unsplash.com/random/${image.width}x${image.height}`  // Get the corresponding rect, to store more data needed (it is a normal Object)  const rect = rects[index]  // Create a new AbortController, to abort fetch if needed  const { signal } = rect.controller = new AbortController()  // Fetch the image  fetch(url, { signal }).then(response = {    // Get image URL, and if it was downloaded before, load another image    // Otherwise, save image URL and set the texture    const id = response.url.split('?')[0]    if (imagesUrls[id]) {      loadTextureForImage(index)    } else {      imagesUrls[id] = true      image.texture = PIXI.Texture.from(response.url)      rect.loaded = true    }  }).catch(() = {    // Catch errors silently, for not showing the following error message if it is aborted:    // AbortError: The operation was aborted.  })}

Now we need to call theloadTextureForImagefunction for each image whose correspondingSpriteis intersecting with the viewport. In addition, we will cancel thefetchrequests that are no longer needed, and we will add analpha transition when the rectangles enter or leave the viewport.

// Check if rects intersects with the viewport// and loads corresponding imagefunction checkRectsAndImages () {  // Loop over rects  rects.forEach((rect, index) = {    // Get corresponding image    const image = images[index]    // Check if the rect intersects with the viewport    if (rectIntersectsWithViewport(rect)) {      // If rect just has been discovered      // start loading image      if (!rect.discovered) {        rect.discovered = true        loadTextureForImage(index)      }      // If image is loaded, increase alpha if possible      if (rect.loaded  image.alpha  1) {        image.alpha += 0.01      }    } else { // The rect is not intersecting      // If the rect was discovered before, but the      // image is not loaded yet, abort the fetch      if (rect.discovered  !rect.loaded) {        rect.discovered = false        rect.controller.abort()      }      // Decrease alpha if possible      if (image.alpha  0) {        image.alpha -= 0.01      }    }  })}

And the function that verifies if a rectangle is intersecting with the viewport is the following:

// Check if a rect intersects the viewportfunction rectIntersectsWithViewport (rect) {  return (    rect.x * gridSize + container.x = width     0 = (rect.x + rect.w) * gridSize + container.x     rect.y * gridSize + container.y = height     0 = (rect.y + rect.h) * gridSize + container.y  )}

Last, wehave to add thecheckRectsAndImagesfunction to the animation loop:

// Animation loopapp.ticker.add(() = {  // ... more code here ...  // Check rects and load/cancel images as needded  checkRectsAndImages()})

Our animation is nearly ready!

Handling changes in viewport size

When initializing theapplication, we resized the renderer so that it occupies the whole viewport, but if the viewport changes its size for any reason(forexample, the user rotatestheirmobile device), we should re-adjust the dimensions and restart theapplication.

// On resize, reinit the app (clean and init)// But first debounce the calls, so we don't call init too oftenlet resizeTimerfunction onResize () {  if (resizeTimer) clearTimeout(resizeTimer)  resizeTimer = setTimeout(() = {    clean()    init()  }, 200)}// Listen to resize eventwindow.addEventListener('resize', onResize)

Thecleanfunction will clean any residuals of the animation that we were executing before the viewport changed its dimensions:

// Clean the current Applicationfunction clean () {  // Stop the current animation  app.ticker.stop()  // Remove event listeners  app.stage    .off('pointerdown', onPointerDown)    .off('pointerup', onPointerUp)    .off('pointerupoutside', onPointerUp)    .off('pointermove', onPointerMove)  // Abort all fetch calls in progress  rects.forEach(rect = {    if (rect.discovered  !rect.loaded) {      rect.controller.abort()    }  })}

In this way, ourapplication will respond properly to the dimensions of the viewport, no matter how it changes. This gives us the full and final result of our work!

Some final thoughts

Thanks for taking this journey with me! Wewalked through a lotbut we learned a lot of concepts along the way and walked out with a pretty neat piece of UI. You can check thecode on GitHub, or play withdemos on CodePen.

If you have worked with WebGL before(withor without using other libraries),I hope you saw how nice it isworking with PixiJS.Itabstracts the complexity associated with the WebGL world in a great way, allowing us to focus on what we want to do rather than the technical details to make it work.

Bottom line is that PixiJS brings the world of WebGLcloserforfront-enddevelopersto grasp, opening up a lot of possibilities beyond HTML, CSS, and JavaScript.