assetsManager - addTextFileTask using sftp instead of http

Currently working on a react app where I am rendering objects in babylon on a canvas. To load the files and create all the meshes, I’m currently using assetsManager.addTextFileTask(), and give it the URL of my local server where it can get the txt files over http. The basic task addition code looks like this

async function add_all_tasks(assetsManager, assetMap){   
    //using raw.githubusercontent/<path to txt> works here as well
    let leftVertUrl = "http://localhost:8000/lh_verts.txt" ;
    let leftVerts = assetsManager.addTextFileTask("lh_verts", leftVertUrl);
    leftVerts.onSuccess = function(task) { 
    //split the txt file using regex, map to a number, and store the array in the asset map
       assetMap["left_verts"] = task.text.split(/\r\n|\n/).map(Number);
    } 
    let leftFacesUrl = "http://localhost:8000/lh_faces.txt";
    let leftFaces = assetsManager.addTextFileTask("lh_faces", leftFacesUrl);
    leftFaces.onSuccess = function(task) {
       assetMap["left_faces"] = task.text.split(/\r\n|\n/).map(Number);
    }
   //repeat as needed for all data
}

Elsewhere in the code I then use assetsManager.onTasksDoneObservable() to create meshes with the assetMap data, initialize the scene, etc. As is, everything loads in and renders properly.

Instead of http, I want to be able to load in the data through sftp (with sftp-ssh2-client in node). All of my searching so far seems to say you can’t simply replace the url with sftp://user@server.com/<filepath> or similar. Using the assetsManager to do this would be the way I’d like to go about it - the assetsManager already has inbuilt ways to check if all the tasks are done. So how could I do this? Is there a url format for sftp? Can I/should I make a child AbstractAssetTask class for my own sftp use? (I recently thought of this possibility, no idea how good or bad it is)

Thanks for any help

I do not think you can request data from sftp directly from an xhr request unfortunately.

One possible solution would be to use something like ssh2-sftp-client and you could wrap that in a serverless HTTP endpoint. A good reason to do the sftp server-side also is that you can protect sensitive credentials, if needed. If you are using google cloud function it would be something like:

import Client, { FileInfo } from 'ssh2-sftp-client';

/**
 * @param sftpClient Instance of Client to connect
 */
const connectClient = async (sftpClient: Client): Promise<void> => {
  await sftpClient.connect({
    host: process.env.HOST,
    port: Number(process.env.PORT ?? '22'),
    username: process.env.LOGIN,
    password: process.env.PASSWORD
  })
}

export const getFile = async (path: string): Promise<Buffer> => {
  const sftpClient = new Client();

  try {
    await connectClient(sftpClient);

    const status = await sftpClient.exists(path);
    if (!status) {
      throw new Error('not a file at path:' + path); // could send a 404 in HTTP response
    }
    console.log(`File: '${path}' found.`)

    // If dst (optional parameter) is undefined, the method will put the data into a buffer and return that buffer when the Promise is resolved.
    const contentBuffer: Buffer = await sftpClient.get(path) as Buffer;

    // we had some strange encoding doing 'utf8' here.
    // const fileContents = contentBuffer.toString('utf8')

    return contentBuffer;
  } catch (e) {
    console.error(e);
    console.error(`error while getting remote file: ${path}`);
    throw e;
  } finally {
    await sftpClient.end();
  }
}

/**
 * Responds to HTTP request (GCP calls serverless Node functions as Express routes).
 *
 * @param {!express:Request} req  Cloud Function HTTP request context.
 *                                More info: https://expressjs.com/en/api.html#req
 * @param {!express:Response} res HTTP response context.
 *                                More info: https://expressjs.com/en/api.html#res
 */
export const retrieveFromSFTP = async (req: Request, res: Response) => {
  const sftpClient = new Client();
  try {
   const {fileName, path} = getFileFromRequest(req);
    const buffer = await getFile(sftpClient, path);
    
    res.setHeader("Content-Disposition", "attachment; filename=" + fileName);
    res.status(200).write(buffer); // sets content type as application/octet-stream
  } catch (e){
    res.status(500).json({error: e.message});
  }
}

I’m not aware of any SFTP libraries that work in the browser. Also, that is just snippets of some EDI code I wrote, so may need modifications to run.

This might be along the lines of what I’m looking for, thanks! My file server handles the sftp execution similar to this example, so my server-side code doesn’t look too different, I think. The main difference is using serverless deployment. I’m going to look into how to set it up - azure & aws operate differently than google cloud, it seems.

1 Like