Rich Content Guide

Technically speaking, Rich Content is any content where a Message Part is greater than 2KB, and must be uploaded/dowloaded to cloud storage separate from sending/receiving the layer.Message. This includes text/plain content as shown in the Text Content Guide. This guide does not cover all content larger than 2KB, but rather focuses on content such as images and video which are almost always going to be larger than 2KB, and which users may want to upload from their device.

Uploading a File

A common task for an application is to provide a means of sending a file, image, etc… to other users. There are three ways of doing this:

  1. A File Upload button
  2. File Drag and Drop
  3. Generating a file in Javascript

Generating a file in Javascript will not be covered in detail in this guide, but this simple example should give you enough to get started:

// Take arbitrary data we generated in our canvas and send it in a MessagePart:
canvas.toBlob(function(blob) {
  var part = new layer.MessagePart({
    body: blob,
    mimeType: 'image/png'
  });
}, 'image/png');

The other two options are discussed below.

File Upload Button

For this example, let us assume that the user is being prompted to select an Image, type in some text, and hit a Send button.

Code for Sending

First the HTML Form:

<div class='send-bar'>
  <input type='file' id='file-button' />
  <input type='text' id='text-input' />
  <button id='send-button' onClick='sendMessage()'>Send</button>
</div>

And now for our sendMessage function:

function sendMessage() {
  // Get the javascript File instance (subclass of Blob) representing the selected file;
  // This returns undefined if the user did not select a file.
  var file = document.getElementById('file-button').files[0];

  // Get the text typed in by the user
  var text = document.getElementById('text-input').value;

  // Create our Message
  var message = conversation.createMessage();

  // Mime Type will be extracted from the file, thus a file input specifies both the
  // body property and the mimeType property
  if (file) message.addPart(new layer.MessagePart(file));

  // If there's text, add it as a MessagePart as well
  if (text) {
    message.addPart(new layer.MessagePart({
      body: text,
      type: 'text/plain'
    }));
  }

  // Only send if the user entered a file and/or text
  if (message.parts.length) message.send();
}

The above code will create two parts:

  1. the file selected by the File Upload widget
  2. the text the user typed in

It will then send it. Send may take a while, depending on how large the file is. The server will not receive any information about the fact that you are trying to send a Message until the file has been uploaded to cloud storage.

File Drag and Drop

module.exports.ConversationView = Backbone.View.extend({
  tagName: 'div',
  className: 'conversation-view',
  render: function() {
    var dom = this.$el;
    var part = this.model;

    // Tells the browser that we *can* drop on this target
    drop.addEventListener('dragover', this.onDragOver, false);
    drop.addEventListener('dragenter', this.onDragOver, false);

    drop.addEventListener('dragend', this.onDragEnd, false);
    drop.addEventListener('dragleave', this.onDragEnd, false);

    drop.addEventListener('drop', this.onFileDrop, false);

    document.addEventListener('drop', this.ignoreDrop, false);
    document.addEventListener('dragenter', this.ignoreDrop, false);
    document.addEventListener('dragover', this.ignoreDrop, false);
  },

  // There is nothing more annoying that missing the drop target and having the browser replace your page with this file
  // instead of attaching the file.  So disable the browser's default handling of dropping a file on a window.
  ignoreDrop: function(evt) {
    evt.preventDefault();
    evt.stopPropagation();
    return false;
  },

  onDragOver: function(evt) {
    $this.el.addClass('drag-over');
    evt.preventDefault();
    return false;
  },

  onDragEnd: function(evt) {
    $this.el.removeClass('drag-over');
    return false;
  },

  onFileDrop: function(evt) {
    $this.el.removeClass('drag-over');;

    // stops the browser navigating away from your app and to the file:
    evt.preventDefault();
    evt.stopPropagation();

    // Gather all of the files that were drag-n-dropped onto your page (typically only one, but possibly more)
    var dt    = evt.dataTransfer;
    var files = Array.prototype.map.call(dt.files, file => file);

    // Generate the Message
    var parts = files.map(file => new layer.MessagePart(file));
    var message = this.conversation.createMessage({
      parts: parts
    })

    // Send the message
    message.send();

    return false;
  }
});

Well, that’s a lot of code, so if you want to focus on where the WebSDK becomes relevant, just read onFileDrop(). Once we have an array of files, there is little difference between this and working with a File Upload Button.

Rendering Rich Content

Its been noted before that if your Message is greater than 2KB, it will be sent as rich content. Unfortunately, this leaves room for uncertainty; the recipient of this Message will receive a MessagePart.url rather than the full Image.

  • The url property will initially be assigned a value.
  • The url is an expiring URL that expires every few hours, so a time may come with accessing url will return an empty string
  • You must call call part.fetchStream() if url returns empty string and you want to get an updated url
  • The url will not expire if you call part.fetchContent(); however, calling fetchContent() will fetch the data even if the content is scrolled out of view, or otherwise never needs to be rendered.

Images are a simplified case of Rich Content due to the fact that a url can be used, and can be provided to an <img /> html tag and left to the browser to decide when to fetch the data, when to cache the data and when to render the image.

The following view is assumed to only be used to render MessageParts whose mimeType is image/png.

module.exports = Backbone.View.extend({
  tagName: 'img',
  className: 'message-part-image',
  render: function() {
    var part = this.model;

    // This will have a value if the url has not expired:
    if (part.url) {
      this.$el.src = part.url;
    }

    // If the Rich Content expiring URL has expired, trigger a refresh, causing:
    // 1. A new url to the content to be loaded
    // 2. The Query to trigger an event causing this View to rerender
    else {
      part.fetchStream();
    }
  }
});

Rendering Images With Orientation

Unless your site generates its own image data, or only gets curated data, you probably have some photos with EXIF data specifying an orientation. Sadly, browser <img> tags do not support EXIF. For more on this, see this Annoying Bug Report but basically, any web site that displays photos has had to implement custom solutions to work around the browser bug, and now, there are so many sites out there with custom solutions that browser vendors can’t fix it.

So, welcome to the fun of rendering images. There are many solutions to be found, the one described here uses Javascript-Load-Image to manage the orientation in a Canvas.

Now, since we need to locally manipulate images within a Canvas, we can no longer use the expiring URLs, and must fetch the content directly. Lets start with this as a simple update, which downloads the entire image into memory rather than just using the URL. The URL used in this snippet is actually a local url to a browser allocated resource created from your downloaded data.

module.exports = Backbone.View.extend({
  tagName: 'img',
  className: 'message-part-image',
  render: function() {
    var part = this.model;

    // If the part has a body, then the url will be created as a Data URL
    if (part.body) {
      this.$el.src = part.url;
    }

    // If the body has not yet been fetched, fetch it now.
    // The Query to trigger an event onec loaded, causing this View to rerender
    else {
      part.fetchContent();
    }
  }
});

Now lets add a basic Blob to Canvas conversion with EXIF parsing:

// Load the Manager
var ImageManager = window.loadImage = require("blueimp-load-image/js/load-image");

// Load the parsing Extensions
require("blueimp-load-image/js/load-image-orientation.js");
require("blueimp-load-image/js/load-image-meta.js");
require("blueimp-load-image/js/load-image-exif.js");


module.exports = Backbone.View.extend({
  tagName: 'div',
  className: 'message-part-image',
  isBuildingCanvas: false,
  render: function() {
    var part = this.model;
    var div = this.$el;

    // Make sure that multiple events from the Query don't cause us to parse this blob multiple times
    if (part.body && !this.isBuildingCanvas) {
      this.isBuildingCanvas = true;
      // Read the EXIF data
      ImageManager.parseMetaData(
        part.body,
        function (data) {
          var orientation = 1;
          var options = {
              canvas: true
          };

          if (data.imageHead && data.exif) {
              options.orientation = data.exif[0x0112];
          }

          // Write the image to a canvas with the specified orientation
          ImageManager(part.body, function(canvas) {
              while(div.firstChild) div.removeChild(div.firstChild);
              div.appendChild(canvas);
          }, options);
        }
      );
    }

    // If the body has not yet been fetched, fetch it now.
    // The Query will trigger an event once loaded, causing this View to rerender
    else {
      part.fetchContent();
    }
  }
});

Our Image View will take the Blob in part.body, feed it into the EXIF parser, set options.orientation based on the parser, and tell the ImageManager to create a canvas with the new Image correctly oriented.

Three Part Images

Ok, so thats the simple version… Layer’s Atlas components send photos using 3 MessagePart objects:

  1. The original image so that a user wanting to zoom in can open up the full resolution (image/jpeg, image/png, etc…) in a new View or window
  2. A preview image that displays nicely in a Message List (image/jpeg+preview)
  3. A JSON structure with metadata about the original image (size and orientation so that space can be reserved in the MessageList without waiting for images to download)

Obviously, this takes a bit more work to generate and to render. But it also insures smoother rendering and scrolling.

Rendering Three Part Images

The code below will take the simplifying assumption that we only ever receive images as part of the 3-MessagePart package described above, and that images are never sent alone. Some additional tests/handling is needed if they can be sent alone… or use a different View for those types of images.

The code below should render the Preview in the Message List and open the Original in a new window if its clicked.

// Tweak these numbers to suit your layout
export const MAX_HEIGHT = 400;
export const MAX_WIDTH = Math.max(document.body.clientWidth - 100 : 500); //

export function normalizeSize(sizeData) {

  if (sizeData.width > MAX_WIDTH) {
    const width = sizeData.width;
    sizeData.width = MAX_WIDTH;
    sizeData.height = sizeData.height * MAX_WIDTH / width;
  }
  if (sizeData.height > MAX_HEIGHT) {
    const height = sizeData.height;
    sizeData.height = MAX_HEIGHT;
    sizeData.width = sizeData.width * MAX_HEIGHT / height;
  }
  return {
    width: Math.round(sizeData.width),
    height: Math.round(sizeData.height)
  };
}

module.exports = Backbone.View.extend({
  tagName: 'div',
  className: 'message-part-image',
  isBuildingCanvas: false,
  initialize: function() {
    this.$el.addEventListener('click', function(evt) {
      var part = this.model;
      if (part) {
        // If the user clicked on this Image, and we have a URL, open it
        if (part.url) {
          window.open(part.url);
        } else {
          // The url was expired; refresh it and then open the new window.
          part.fetchStream(function() {
            window.open(part.url);
          });
        }
      }
    }.bind(this));
  },
  render: function() {
    var part = this.model;
    var sizesPart = this.sizesPartModel;
    var div = this.$el;
    var sizes = normalizeSize(sizesPart);
    div.style.width = sizes.width + 'px';
    div.style.height = sizes.height + 'px';

    // Make sure that multiple events from the Query don't cause us to parse this blob multiple times
    if (part.body && !this.isBuildingCanvas) {
      this.isBuildingCanvas = true;
      // Read the EXIF data
      ImageManager.parseMetaData(
        part.body,
        function (data) {
          var orientation = 1;
          var options = {
              canvas: true
          };

          if (data.imageHead && data.exif) {
              options.orientation = data.exif[0x0112];
          }

          // Write the image to a canvas with the specified orientation
          ImageManager(part.body, function(canvas) {
              while(div.firstChild) div.removeChild(div.firstChild);
              div.appendChild(canvas);
          }, options);
        }
      );
    }

    // If the body has not yet been fetched, fetch it now.
    // The Query to trigger an event onec loaded, causing this View to rerender
    else {
      part.fetchContent();
    }
  }
});

The above code has three lines that are key:

var sizes = normalizeSize(sizesPart);
div.style.width = sizes.width + 'px';
div.style.height = sizes.height + 'px';

Why is this needed? After all, the Canvas library used above will set the correct height/width. By fixing the size of the image within the MessageList we insure that behaviors such as scrolling to the bottom of the MessageList to read the latest messages can work correctly. Otherwise, the MessagePart will have zero size while the image data is loading, during which time your scrolling code would scroll to the end of the document. Then the image finishes loading, that zero size grows, and the place you scrolled to is no longer the end of the document. A MessageList with 5 images in it all doing this together would create a very jerky and messy experience.

Sending Three Part Images

On sending an image through your web application, a function such as this can be used to turn it into the Three Part Image format expected by the Atlas Clients.

function handleImage(file, callback) {
  // First MessagePart is the Original image; the rest of the code is for generating the other 2 parts of the 3 part Image
  const parts = [new layer.MessagePart(file)];

  // STEP 1: Determine the correct orientation for the image
  ImageManager.parseMetaData(
      file,
      function (data) {
        var orientation = 1;
        var options = {
            canvas: true
        };

        if (data.imageHead && data.exif) {
            options.orientation = data.exif[0x0112];
        }

        // STEP 2: Write the image to a canvas with the specified orientation
        ImageManager(file, scaleCanvas, options);

        function scaleCanvas(srcCanvas) {

            // STEP 3: Calculate the new size for our scaled image
            const originalSize = {
              width: srcCanvas.width,
              height: srcCanvas.height
            };
            const size = normalizeSize(originalSize);

            // STEP 4: Create a new canvas and scale it to the desired size
            const canvas = document.createElement('canvas');
            canvas.width = size.width;
            canvas.height = size.height;
            let context = canvas.getContext('2d');
            context.drawImage(srcCanvas, 0, 0, size.width, size.height);

            // STEP 5: Turn the canvas into a jpeg image for our Preview Image
            const binStr = atob(canvas.toDataURL('image/jpeg').split(',')[1]);
            const len = binStr.length;
            const arr = new Uint8Array(len);
            for (let i = 0; i < len; i++ ) {
              arr[i] = binStr.charCodeAt(i);
            }
            const blob = new Blob([arr], {type: 'image/jpeg'});

            // STEP 6: Create our Preview Message Part
            parts.push(new layer.MessagePart({
              body: blob,
              mimeType: 'image/jpeg+preview'
            }));

            // STEP 7: Create the Metadata Message Part
            parts.push(new layer.MessagePart({
              mimeType: 'application/json+imageSize',
              body: `{"orientation":${orientation}, "width":${originalSize.width}, "height":${originalSize.height}}`
            }));

            // FINISHED: Provide the 3 Message Parts to the caller, who can send it.
            callback(parts);
        }
      }
  );
}

Note that the input to this is a file or blob; you can get this file or blob from your user when they use a File Upload or Drag and Drop to provide you with a file.

PDF Files

PDF is a generic, relatively simple type of file; you can probably extrapolate from the handling of this file type to other types of file type handling.

For this example:

  • Most browsers have built in pdf rendering… if you load a PDF in a new window or an iframe
  • We will take a shortcut and rely upon an iframe loading a simple PDF renderer
  • We assume that you have some Message handling code that will call this View on any MessagePart of MIME Type application/pdf.

So, really all that is needed is an iframe whose src property matches the MessagePart.url – and of course we have to make sure that the url doesn’t expire.

module.exports.ConversationView = Backbone.View.extend({
  tagName: 'iframe',
  className: 'conversation-view',
  render: function() {
    var dom = this.$el;
    var part = this.model;
    if (part.url) {
      dom.src = part.url;
    } else {
      part.fetchStream();
    }
  }
});
Text Content Guide Previewing Messages