Extrude 2d PNG image to Mesh

I’m attempting to extrude a 2d png image to a 3d mesh, whilst I’ve considered obtaining the contours/outline of the image by extracting the alpha channel of the image and then using the results to create an extruded polygon. I’ve gone with a more straight forward method which merely seeks to stack multiple planes of the png image, I then merge the result into a single mesh. Whilst this DOES work in the main, I’m noticing artefacts at extreme angles i.e 90 degrees, at this angle the planes that make up the mesh are clearly visible.

Is there a way to stack the planes so closely that the individual planes are not visible at any angle of the resulting mesh? Alternatively, once I have the shape of the mesh can it be filled to remove any gaps present between the planes. Please see the following playground, you’ll note that if you rotate the mesh by 90 degrees on any axis, a “plane” distortion occurs as each individual plane orientates towards the camera:

Hello and welcome!

Do you need to extrude at runtime?

This one was created from a png.
PNG → SVG → Blender import to plane → babylon.js extrude

2 Likes


neat

1 Like

Yes, it has to be in real time. Like I say, the approach I have taken works apart from the plane distortion effect you can see in the playground. I’m just looking for a way to remove that. @sebavan @Evgeni_Popov could you guys provide any pointers

I’m just gonna like the idea :grin:. I’ve no faen clue how to do it. I believe at some point you would need to turn this into a geometry (unless using just parallax occlusion or something) but it will never be 3D. May be it can be done using parts of NGE? You should have added @Deltakosh I’m sure he would appreciate participating in this challenge :grinning: :laughing:

Hello and welcome :slight_smile:

I would say, duplicating 130 times a plane in order to fake some depth is not the right way. So, instead of trying to “remove” these artifacts, I would rather advice to use another method…

For example :

2 Likes

I already wrote an image vectorizer function which converts B/W pixel data to what else if not into a GreasedLine outline :wink: Unfortunatelly I can’t open source.

@Giles_Thompson
However ChatGPT is here to help:

<canvas id="canvas" width="200" height="200"></canvas>
<script>
  const canvas = document.getElementById('canvas');
  const ctx = canvas.getContext('2d');

  // Example: Drawing a black and white image on the canvas
  ctx.fillStyle = "white";
  ctx.fillRect(0, 0, 200, 200); // White background

  ctx.fillStyle = "black";
  ctx.fillRect(50, 50, 100, 100); // Black square

  // Step 1: Extract pixel data
  const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
  const pixels = imageData.data; // This is an array with RGBA values

  // Step 2: Convert the pixel data to black-and-white threshold
  function getBWPixel(x, y) {
    const index = (y * canvas.width + x) * 4;
    const r = pixels[index];
    const g = pixels[index + 1];
    const b = pixels[index + 2];

    // Simple threshold for black and white
    return (r + g + b) / 3 < 128 ? 1 : 0; // 1 for black, 0 for white
  }

  // Step 3: Contour detection (basic scanline algorithm)
  function findContours() {
    const paths = [];
    const visited = Array(canvas.width * canvas.height).fill(false);

    for (let y = 0; y < canvas.height; y++) {
      for (let x = 0; x < canvas.width; x++) {
        if (getBWPixel(x, y) === 1 && !visited[y * canvas.width + x]) {
          const path = traceContour(x, y, visited);
          if (path.length) {
            paths.push(path);
          }
        }
      }
    }
    return paths;
  }

  // Step 4: Tracing the contour using a simple "flood fill" approach
  function traceContour(x, y, visited) {
    const stack = [[x, y]];
    const path = [];
    const directions = [
      [0, -1], // Up
      [1, 0], // Right
      [0, 1], // Down
      [-1, 0] // Left
    ];

    while (stack.length) {
      const [cx, cy] = stack.pop();
      if (cx < 0 || cy < 0 || cx >= canvas.width || cy >= canvas.height) continue;
      const index = cy * canvas.width + cx;

      if (!visited[index] && getBWPixel(cx, cy) === 1) {
        visited[index] = true;
        path.push([cx, cy]);

        // Add neighboring pixels
        for (const [dx, dy] of directions) {
          stack.push([cx + dx, cy + dy]);
        }
      }
    }

    return path;
  }

  // Step 5: Convert contours to SVG
  function convertToSVG(paths) {
    const svgPaths = paths.map(path => {
      const d = path.map((point, index) => {
        const [x, y] = point;
        return index === 0 ? `M${x},${y}` : `L${x},${y}`;
      }).join(" ") + " Z"; // Close the path
      return `<path d="${d}" fill="black" stroke="none"/>`;
    }).join("\n");

    return `
      <svg width="${canvas.width}" height="${canvas.height}" xmlns="http://www.w3.org/2000/svg">
        ${svgPaths}
      </svg>
    `;
  }

  // Process the pixel data and generate the SVG
  const contours = findContours();
  const svgString = convertToSVG(contours);
  console.log(svgString);

  // You can insert the SVG into the DOM or download it
  const svgBlob = new Blob([svgString], { type: 'image/svg+xml' });
  const url = URL.createObjectURL(svgBlob);
  const link = document.createElement('a');
  link.href = url;
  link.download = 'image.svg';
  link.innerText = 'Download SVG';
  document.body.appendChild(link);
</script>

Once you have the vectors you can use the PolygonMeshBuilder to extrude the shape.

EDIT:
@Tricotou seems that we are suggesting the same approach but you were faster :wink:

1 Like

That supposes you first need to create a spline from it (or import it along with the picture). I thought the request was more something along photogrammetry (or gaussian splating from a PNG? wouldn’t know how this can work though :thinking:).

Intriguing. Would deserve to be tested. Might be deceived though… who knows? :face_with_hand_over_mouth: :roll_eyes: :grin:

Let the OP do his homework :wink:

As far as I know you guys @Tricotou @CodingCrusader @mawa @labris (+others I forgot to mention) there will be multiple solutions available tomorrow :rofl:

1 Like

Yes, you’ll note that I did make SPECIFIC mention to how this sort of thing would be typically done, i.e obtaining the outline/contours of the image and then undertaking a shape extrusion…

In fact, I DID actually go down this path first, however this had its own problems principal of which was that the outline of the shape wasn’t always traced correctly. You can see that playground here, parts of the image that SHOULD be transparent (i.e the wings to the head of the bird in this case) are visible on some images; perhaps there is a better way to trace the contours/outline of the image. I obtained the outline by extracting the alpha channel of the image using a canvas - I’d prefer not to use an external library.
@Tricotou @roland

At this point I suppose I shouldn’t ask the question that first came to my mind: Does it really need to be a bitmap? I’m guessing it must be from some user import where you cannot set i.e. svg as a requirement, correct?

I guess he was quite SPECIFIC about using PNGs!

1 Like

@mawa Any idea why the image seems to be traced incorrectly in this playground. Here I take the other approach (i.e attempting to extract the contours/lines from the vector part of the image.) As you’ll note from the index count I did try this a number of times lol. The result is nice but, certain parts of the image that should be transparent are visible (i.e area between the birds head and wings in this case).

It’s almost correct and avoids having to stack planes like in the former approach.

Why do you ask me? I’m a simple designer. I’m barely able to read your code :sweat_smile: :rofl:
All I can tell from what I’m seing with my simple designer’s eye is that the lines are drawned vertically and horizontally from one edge to the other (like on the X-axis) instead of an ‘extrude’.
My simple brain thinks we should turn it into a spline first, close the spline and then extrude it (in the depth). But again, you should ask a real dev… not someone like me :grin: :face_with_hand_over_mouth:

1 Like

Mate, me and ChatGPT are here to help you:

Here’s an enhanced and corrected version of your code. This refactor aims to improve performance and reduce the chance of artifacts by using efficient looping and better handling of pixel data. Additionally, I applied some optimizations to ensure that the outline detection is more accurate, avoiding artifacts such as unnecessary color changes. To vectorize the outline, I also improved the pixel grouping logic for smoother transitions.

Enhanced Code:

var GetTextureOutline = function (data, canvasWidth, keepOutline = true, keepOtherPixels = true) {
    const pixelsList = [];
    const notOutlineIndices = [];

    // Iterate over pixel data
    for (let j = 0; j < data.length; j += 4) {
        const alpha = data[j + 3];

        if (alpha !== 0) {
            const x = (j / 4) % canvasWidth;
            const y = Math.floor((j / 4) / canvasWidth);

            // Neighboring pixels' alpha
            const topAlpha = (y > 0) ? data[j + 3 - (canvasWidth * 4)] : 0;
            const bottomAlpha = (y < canvasWidth - 1) ? data[j + 3 + (canvasWidth * 4)] : 0;
            const leftAlpha = (x > 0) ? data[j + 3 - 4] : 0;
            const rightAlpha = (x < canvasWidth - 1) ? data[j + 3 + 4] : 0;

            // Detect outline by checking neighboring pixels
            if (topAlpha === 0 || bottomAlpha === 0 || leftAlpha === 0 || rightAlpha === 0) {
                pixelsList.push({
                    x: x,
                    y: y,
                    color: new BABYLON.Color3(data[j] / 255, data[j + 1] / 255, data[j + 2] / 255),
                    alpha: alpha / 255
                });

                if (!keepOutline) {
                    // Mark as part of the outline (magenta for visualization)
                    data[j] = 255;
                    data[j + 1] = 0;
                    data[j + 2] = 255;
                }
            } else if (!keepOtherPixels) {
                notOutlineIndices.push(j);
            }
        }
    }

    // Remove non-outline pixels if needed
    if (!keepOtherPixels) {
        for (let i = 0; i < notOutlineIndices.length; i++) {
            const idx = notOutlineIndices[i];
            data[idx] = data[idx + 1] = data[idx + 2] = data[idx + 3] = 0; // Fully transparent
        }
    }

    return pixelsList;
};

var ExtractLinesFromPixelsList = function (pixelsList) {
    const lines = [];

    while (pixelsList.length) {
        const line = [pixelsList.shift()]; // Start with the first pixel
        let pixelAdded;

        do {
            pixelAdded = false;
            for (let i = 0; i < pixelsList.length; i++) {
                const lastPixel = line[line.length - 1];
                const currentPixel = pixelsList[i];

                // Check if pixels are adjacent (either horizontally or vertically)
                const isAdjacent =
                    (Math.abs(lastPixel.x - currentPixel.x) === 1 && lastPixel.y === currentPixel.y) ||
                    (Math.abs(lastPixel.y - currentPixel.y) === 1 && lastPixel.x === currentPixel.x);

                if (isAdjacent) {
                    line.push(currentPixel);
                    pixelsList.splice(i, 1); // Remove added pixel from the list
                    pixelAdded = true;
                    break;
                }
            }
        } while (pixelAdded); // Continue until no more adjacent pixels are found

        lines.push(line); // Add the completed line to the result
    }

    return lines;
};

Key Enhancements:

  1. Neighbor Check for Alpha:
  • Now ensures that out-of-bounds indices are correctly handled (e.g., checking for topAlpha at the first row or bottomAlpha at the last row).
  1. Pixel Processing:
  • Added logic to prevent unnecessary changes to pixels unless they are explicitly marked as outline or background pixels.
  • If keepOutline is false, the outline pixels are colored magenta (as before) for visualization. You can adjust this color or remove this step.
  1. Memory Optimization:
  • The notOutlineIndices array is used only if keepOtherPixels is false, reducing memory overhead if it’s not needed.
  1. Pixel Grouping Logic:
  • The line extraction algorithm ensures smoother transitions between adjacent pixels by using Math.abs to check both horizontal and vertical adjacency, reducing visual artifacts in the vectorized output.

This approach should reduce artifacts and make the outline extraction more precise. The image can then be vectorized by further processing these lines, depending on the application you are integrating this code into.

2 Likes

Once you extracted the contour, can’t you use the extrude shape mechanism linked by @Tricotou? It would be much easier than doing it yourself (lines 68-138).

It’s “ChatGPT and Me” to be polite :joy:

1 Like

Now the extruded mesh is completely missing :rofl: See the updated playground here

1 Like

I did try that prior to attempting to craft my own, the shape was distorted on extrusion. It looks like my approach is almost there, with the exception of the fact that the line trace appears to be incorrect. @Evgeni_Popov