Upload and Download Multiple Binary Files using MongoDB

Upload and Download Multiple Binary Files using MongoDB

In this tutorial we are going to develop multiple file upload and file download capability using RESTful web service using JAX-RS and Jersey storing the contents of files into MongoDB Database using a powerful feature in MongoDB for managing large files called GridFS. The majority of the framework for this tutorial came from my previous tutorial on Inserting and Retrieving Binary Data with MongoDB using JAX-RS RESTful Web Service. Hence, my decision to keep the bulk of the conent here as it fully applies to this post as well.

MongoDB manages all of its data as documents with an upper limit on file size of 16MB. So what do it do when your images, video, PDFs, PPT slides or Excel spreadsheets exceed this 16MB limit? MongoDB has actually built a specification called GridFS which is supported by all of its MongoDB drivers that manages large files by splitting them up into smaller parts or chunks and storing the data in separate documents in its collections. It can then reassemble these smaller chunks automatically into the original file when asked to retrieve the contents of that file.

This functionality makes MongoDB an excellent choice for things like a queryable document repository, or content management system, or image/video warehouse with all of the associated metadata being stored in a separate MongoDB collection and associated with the GridFS system by a unique id. Best of all, GridFS does not have limitations on file size or number of documents per directory or even file naming rules. You get a truly flexible solution that can be easily ported from one system to another using MongoDB.

Saving Multiple Binary Files at once

As you will see, our example will be able to save multiple files in one fell swoop including PDF files, Excel files, Word Document files, Powerpoint files, Image files like (png, jpg, bmp, gif, tiff, svg, etc), or any other type of file BLOB available us provided we have ample amount of storage space.

Getting Started

In order to run this tutorial yourself, you will need the following:

• Java JDK 1.6 or greater
• Favorite IDE Spring Tool Suite (STS), Eclipse IDE or NetBeans (I happen to be using STS because it comes with a Tomcat server built-in)
• Tomcat 7 or greater or other popular container (Weblogic, Websphere, Glassfish, JBoss, VMWare vFabric, etc). For this tutorial I am using VMware vFabric tc Server Developer Edition which is essentially an enhanced Tomcat instance integrated with Spring STS
• Jersey JAX-RS
• Apache Commons FileUpload
• MongoDB Java driver
• log4J (for logging purposes)

Required Libraries

Copy all of the following jars to WebContent->WEB-INF->lib folder.

asm-3.1.jar
commons-fileupload-1.2.2.jar
commons-io-2.4.jar
jersey-client-1.18.jar
jersey-core-1.18.jar
jersey-json-1.18.jar
jersey-multipart-1.18.jar
jersey-server-1.18.jar
jersey-servlet-1.18.jar
jsr311-api-1.1.1.jar
log4j-1.2.17.jar
mimepull-1.6.jar
mongo-java-driver-2.11.3.jar

Complete Project Overview

I have added the project overview to give you a full view of the structure and show you all files contained in this sample project. You will notice that I had included two additional
jar files commons-fileupload-1.2.2.jar and commons-io-2.4.jar to support the Multiple file upload capability.

RESTful Web Service End Points

Collections available in our Tutorial Database in MongoDB

As you can see below, our tutorial database holds a set of collections and these collections will hold our mongoDB documents which will store the form data (filestore), the file metadata (filestore.files) and file raw binary data (filestore.chunks).

> show collections
filestore.chunks
filestore.files
system.indexes
>

Implement our File Download Service Class using the JAX-RS API

Implementing a RESTful service requires nothing more than creating a POJO and annotating using the javax.ws.rs.* annotations. Additionally, you will need to ensure your class is under the package you defined in your web descriptor as Jersey will use this package to scan your classes for the existence RESTful resources.

For the uploadFile method I am using the following four annotations: @POST which is one of several resource method designators corresponding to HTTP methods, @Path(“/upload”) is used here as a sub-resource method to identify the URI path relative to the base URI, @Consumes(MediaType.MULTIPART_FORM_DATA) annotation specifies which MIME types of data a resource can accept or consume from the client, and @Produces(“text/html”) specifies which MIME types of data a resource can produce or return to the client.

Our upload method uploadFile, takes one parameter using @HttpServletRequest object. In this method I am using @Consumes annotation with a media type of MediaType.MULTIPART_FORM_DATA which will allow us to consume multipart form data from our HTML page.

Another important method used in the RestMultiFileStoreMongoDBExample service class is the downloadFilebyID method. You will notice that I am using @Produces annotation with a media type of MediaType.APPLICATION_OCTET_STREAM. This allow us to download the file as a binary file and download it directly in your browser. This method uses the @PathParam annotation which binds the value of a URI template parameter or a path segment containing the template parameter to a resource method parameter.

The other point worth mentioning is the fact that we are performing some database checking by ensuring that the document actually exists in our MongoDB collection. If the document exists, we return it to the user and if the document is missing, we construct an appropriate error message and return it to the user in the HTML response.

Uploading a file to MongoDB Database

In this code snippet we will concentrate on the code responsible for saving the binary data to our MongoDB database by storing the actual binary data associated with the file in the GridFS collections called filestore.files and filestore.chunks. Using this code you will be able to store PDF files, Excel files, Image files, or any type of file available to you provided you have enough space available to MongoDB.

Using GridFSInputFile we create an object that is able to take the input stream coming the file we uploaded via our HTML page and save it to GridFS file structure in MongoDB.

In this code snippet, we accept multipart form data that has been submitted through the HTML form listed below and inject the data into the appropriate variables. We then open a connection to MongoDB instance and connect to our database called tutorial. We perform a query using the id we passed in to ensure that a document with that ID does not exist in our collection. If it does not, we create a BasicDBObject document object and append all of the fields we require and insert that document into our collection. The last thing we do here is use the GridFS API to create the collection and store the associated file

We will be storing the actual binary data associated with the files in the GridFS file system. We set the ID in this collection, set the filename and save the contents of the file in the collection.

@POST
@Path("/upload")
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Produces("text/html")
public Response uploadFile (
    @Context HttpServletRequest req
    ) throws UnknownHostException, Exception {

  MongoClient mongoClient = new MongoClient("localhost", 27017);
  DB mongoDB = mongoClient.getDB("tutorial");
  //Let's store the standard data in regular collection
  
  if (ServletFileUpload.isMultipartContent(req)) {
    logger.info("We have received MultiPart Request...");
    FileItemFactory fiFactory = new DiskFileItemFactory();
    ServletFileUpload fileUpload = new ServletFileUpload(fiFactory);
    
    List<FileItem> listItems = fileUpload.parseRequest(req);
    Iterator<FileItem> iter = listItems.iterator();
              
    GridFS fileStore = new GridFS(mongoDB, "filestore");        
    while (iter.hasNext()) {
			FileItem item = iter.next();
			
			if (!item.isFormField()) {  
				InputStream in=item.getInputStream();
				
				logger.info("Filename.....: " + item.getName());
				logger.info("File Size....: " + item.getSize());
				logger.info("File Type....: " + item.getContentType());       
				
				GridFSInputFile inputFile = fileStore.createFile(in);
				inputFile.setId(item.getName());
				inputFile.setFilename(item.getName());
				inputFile.save();
				in.close();
      }
    }
  }
    
  String status = "Upload has been successful";

  return Response.status(200).entity(status).build();
}

Downloading a file from MongoDB Database

In this code snippet you can see that the output we are expecting to produce is an APPLICATION_OCTET_STREAM which is essentially binary data. We will be obtaining the ID from the path parameter of the request URL. Using @PathParam annotation we will inject this id from the request URL into our String id variable which will be used as a parameter in our BasicDBObject query object to verify that our GridFS fileStore contains an ID matching our id in MongoDB database and retrieve the desired document.

Once we find the appropriate file in the GridFS collection using fileStore.findOne(query) with a matching ID we extract from the document we retrieve the binary data from the GridFS filestore. This extraction happens when we use gridFile.getInputStream(). Using this inputStream we will read in all of the bytes of streaming data and put it into our ByteArrayOutputStream which will be returned to the user as an HTTP response object.

Please note that we need to use the previously stored filename when building our response header via builder.header(“Content-Disposition”, “attachment; filename=” + filename);.

@GET
@Path("/download/file/{id}")
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response downloadFilebyID(@PathParam("id")  String id) 
          throws IOException {
  
  Response response = null;
  MongoClient mongoClient = new MongoClient("localhost", 27017);
  DB mongoDB = mongoClient.getDB("tutorial");

  logger.info("Inside downloadFilebyID...");
  logger.info("ID: " + id);

  BasicDBObject query = new BasicDBObject();
  query.put("_id", id);
  GridFS fileStore = new GridFS(mongoDB, "filestore");
  GridFSDBFile gridFile = fileStore.findOne(query);

  if (gridFile != null && id.equalsIgnoreCase((String)gridFile.getId())) {
    logger.info("ID...........: " + gridFile.getId());
    logger.info("FileName.....: " + gridFile.getFilename());
    logger.info("Length.......: " + gridFile.getLength());
    logger.info("Upload Date..: " + gridFile.getUploadDate());
    
    InputStream in = gridFile.getInputStream();
        
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    int data = in.read();
    while (data >= 0) {
      out.write((char) data);
      data = in.read();
    }
    out.flush();

    ResponseBuilder builder = Response.ok(out.toByteArray());
    
    builder.header("Content-Disposition", "attachment; filename=" 
             + gridFile.getFilename());
    response = builder.build();
    } else {
      response = Response.status(404).
        entity(" Unable to get file with ID: " + id).
        type("text/plain").
        build();
    }
      
  return response;
}

Complete Program (RestMultiFileStoreMongoDBExample.java)

package com.avaldes;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.UnknownHostException;
import java.util.Iterator;
import java.util.List;

import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.ResponseBuilder;

import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.log4j.Logger;

import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.MongoClient;
import com.mongodb.gridfs.GridFS;
import com.mongodb.gridfs.GridFSDBFile;
import com.mongodb.gridfs.GridFSInputFile;

@Path("/files")
public class RestMultiFileStoreMongoDBExample {
  static Logger logger = Logger.getLogger(
            RestMultiFileStoreMongoDBExample.class);
  
  @GET
  @Path("/status")
  @Produces(MediaType.TEXT_HTML)
  public Response status() {
    String msg = String.format("Server Status is OK");
    logger.info(msg);
     
    return Response.status(200).entity(msg).build();
  }
  
  @POST
  @Path("/upload")
  @Consumes(MediaType.MULTIPART_FORM_DATA)
  @Produces("text/html")
  public Response uploadFile (
      @Context HttpServletRequest req
      ) throws UnknownHostException, Exception {
 
    MongoClient mongoClient = new MongoClient("localhost", 27017);
    DB mongoDB = mongoClient.getDB("tutorial");
    //Let's store the standard data in regular collection
    
    if (ServletFileUpload.isMultipartContent(req)) {
      logger.info("We have received MultiPart Request...");
      FileItemFactory fiFactory = new DiskFileItemFactory();
      ServletFileUpload fileUpload = new ServletFileUpload(fiFactory);
      
      List<FileItem> listItems = fileUpload.parseRequest(req);
      Iterator<FileItem> iter = listItems.iterator();
                
      GridFS fileStore = new GridFS(mongoDB, "filestore");        
      while (iter.hasNext()) {
        FileItem item = iter.next();
        
        if (!item.isFormField()) {  
          InputStream in=item.getInputStream();
          
          logger.info("Filename.....: " + item.getName());
          logger.info("File Size....: " + item.getSize());
          logger.info("File Type....: " + item.getContentType());       
					 
          GridFSInputFile inputFile = fileStore.createFile(in);
          inputFile.setId(item.getName());
          inputFile.setFilename(item.getName());
          inputFile.save();
          in.close();
          }
      }
    }
      
    String status = "Upload has been successful";

    return Response.status(200).entity(status).build();
  }

  @GET
  @Path("/download/file/{id}")
  @Produces(MediaType.APPLICATION_OCTET_STREAM)
  public Response downloadFilebyID(@PathParam("id")  String id) 
            throws IOException {
    
    Response response = null;
    MongoClient mongoClient = new MongoClient("localhost", 27017);
    DB mongoDB = mongoClient.getDB("tutorial");

    logger.info("Inside downloadFilebyID...");
    logger.info("ID: " + id);

    BasicDBObject query = new BasicDBObject();
    query.put("_id", id);
    GridFS fileStore = new GridFS(mongoDB, "filestore");
    GridFSDBFile gridFile = fileStore.findOne(query);

    if (gridFile != null && id.equalsIgnoreCase((String)gridFile.getId())) {
      logger.info("ID...........: " + gridFile.getId());
      logger.info("FileName.....: " + gridFile.getFilename());
      logger.info("Length.......: " + gridFile.getLength());
      logger.info("Upload Date..: " + gridFile.getUploadDate());
      
      InputStream in = gridFile.getInputStream();
          
      ByteArrayOutputStream out = new ByteArrayOutputStream();
        int data = in.read();
        while (data >= 0) {
          out.write((char) data);
          data = in.read();
        }
      out.flush();
  
      ResponseBuilder builder = Response.ok(out.toByteArray());
      
      builder.header("Content-Disposition", "attachment; filename=" 
               + gridFile.getFilename());
      response = builder.build();
      } else {
        response = Response.status(404).
          entity(" Unable to get file with ID: " + id).
          type("text/plain").
          build();
      }
        
    return response;
  }
  
  @GET
  @Path("/download/details/{id}")
  @Produces(MediaType.TEXT_HTML)
  public Response showFileStoreDetails(@PathParam("id")  String id) 
              throws UnknownHostException {
    
    Response response = null;
    MongoClient mongoClient = new MongoClient("localhost", 27017);
    DB mongoDB = mongoClient.getDB("tutorial");
    
    BasicDBObject query = new BasicDBObject();
    query.put("_id", id);
    GridFS fileStore = new GridFS(mongoDB, "filestore");
    GridFSDBFile gridFile = fileStore.findOne(query);

    if (gridFile != null && id.equalsIgnoreCase((String)gridFile.getId())) {
      logger.info("ID...........: " + gridFile.getId());
      logger.info("FileName.....: " + gridFile.getFilename());
      logger.info("Length.......: " + gridFile.getLength());
      logger.info("Upload Date..: " + gridFile.getUploadDate());

      StringBuffer status = new StringBuffer(
           "<pre>Inside showHeaders: <br/><br/>");
      status.append("ID...........: ");
      status.append(gridFile.getId());
      status.append("<br/>");
      status.append("FileName.....: ");
      status.append(gridFile.getFilename());
      status.append("<br/>");
      status.append("Length.......: ");
      status.append(gridFile.getLength());
      status.append("<br/>");
      status.append("Upload Date..: ");
      status.append(gridFile.getUploadDate());
      status.append("<br/></pre>");
      
      response = Response.status(200).entity(status.toString()).build();
        } else {
          response = Response.status(404).
            entity(" Unable to get file with ID: " + id).
        type("text/plain").
        build();
        }
    return response;
  } 
}

LOG4J Configuration File (log4j.xml)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration PUBLIC "
          -//APACHE//DTD LOG4J 1.2//EN" "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">

  <!-- Appenders -->
  <appender name="console" class="org.apache.log4j.ConsoleAppender">
  <param name="Target" value="System.out" />
    <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%-5p: %c - %m%n" />
    </layout>
  </appender>
  
  <!-- Application Loggers -->
  <logger name="com.avaldes">
    <level value="info" />
  </logger>

  <!-- Root Logger -->
  <root>
    <priority value="warn" />
    <appender-ref ref="console" />
  </root>
</log4j:configuration>

Multiple File Upload — HTML Web Page (index.html)

This page is very simple having only field of type file to allow the user to choose multiple files to upload to our RESTful web service. This HTML page will use the method of POST with an encoding type of enctype=”multipart/form-data” in the HTML form element.

<html>
  <head>
     <title>
      RESTful Web Service - Multiple File Upload into MongoDB Example
     </title>
  </head>
  <body>
    <h1>RESTful Web Service - Multiple File Upload into MongoDB Example</h1>  
    <form action="/RestfulMultiFileStoreMongoDBExample/rest/files/upload" 
                                method="POST" enctype="multipart/form-data">
      <p>Select Multiple Files to Upload: <input type="file" 
                        name="files[]" value="Choose Files" multiple /></p>  
      <input type="submit" value="Submit" />  
    </form>
  </body>
</html>

Web Deployment Descriptor (web.xml)

This is a pretty straight forward deployment descriptor file – only thing you need to add is the location of you java package in the Jersey ServletContainer entry as init-param. Please ensure you add it to the web.xml file as shown below.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns="http://java.sun.com/xml/ns/javaee" 
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" 
id="WebApp_ID" version="3.0">

  <display-name>com.omega.rest</display-name>
  <welcome-file-list>
    <welcome-file>index.html</welcome-file>
    <welcome-file>index.jsp</welcome-file>
  </welcome-file-list>
  <servlet>
   <servlet-name>Jersey REST Service</servlet-name>
    <servlet-class>
       com.sun.jersey.spi.container.servlet.ServletContainer
    </servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>com.avaldes</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>Jersey REST Service</servlet-name>
    <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>
</web-app>

Testing out the Web Services

To test out the application, simply choose the files to upload using the file picker. Then click on the Submit button. This will store the files in the MongoDB database in two separate collections; one to store the file metadata (filestore.files) and the other to store the binary data in file chunks (filestore.chunks).

Saving an Image in MongoDB using JAX-RS Web Service

Using our HTML web page we will use the POST method to submit all the form elements from our form to our RESTful web service. We will also perform a file upload of our photo.jpg file and save that file into MongoDB using the GridFS file system.

Querying the file Metadata in the filestore.files collection in MongoDB

> use tutorial
switched to db tutorial
> db.filestore.files.find().pretty()
{
		"_id" : "angelina-jolie.png",
		"chunkSize" : NumberLong(262144),
		"length" : NumberLong(670520),
		"md5" : "ac0215eda50951c29331ce50e08843a8",
		"filename" : "angelina-jolie.png",
		"contentType" : null,
		"uploadDate" : ISODate("2015-11-25T23:55:26.523Z"),
		"aliases" : null
}
{
		"_id" : "jennifer-aniston.png",
		"chunkSize" : NumberLong(262144),
		"length" : NumberLong(815985),
		"md5" : "3143611d38b2267cad1f282d4fe1ac6e",
		"filename" : "jennifer-aniston.png",
		"contentType" : null,
		"uploadDate" : ISODate("2015-11-25T23:55:26.538Z"),
		"aliases" : null
}
{
		"_id" : "jennifer-lawrence.png",
		"chunkSize" : NumberLong(262144),
		"length" : NumberLong(741615),
		"md5" : "b6bfb60dd67ed87deb3363c8a550cc81",
		"filename" : "jennifer-lawrence.png",
		"contentType" : null,
		"uploadDate" : ISODate("2015-11-25T23:55:26.555Z"),
		"aliases" : null
}
{
		"_id" : "scarlett-johansson.png",
		"chunkSize" : NumberLong(262144),
		"length" : NumberLong(1107350),
		"md5" : "a12efec05bf882724e4015bf414d2f30",
		"filename" : "scarlett-johansson.png",
		"contentType" : null,
		"uploadDate" : ISODate("2015-11-25T23:55:26.571Z"),
		"aliases" : null
}
>

Downloading an Image from MongoDB using JAX-RS Web Service

Once are request is processed by the REST service the image file is returned to the use via the HTTP response object.

Displaying File Details using File ID

Displaying Error page when Binary File ID not found

Let’s view the Application Screenshots

The following screenshots give you a better picture of the running application.

Download the Code

Download Restful Multi-FileStore Using MongoDB Example

1 file(s) 2.43 MB

Download

That’s It!

I hope you enjoyed this tutorial. It was certainly a lot of fun putting it together and testing it out. Please continue to share the love and like us so that we can continue bringing you quality tutorials. Happy Coding!!!

Please Share Us on Social Media