2010-10-28

Manipulating images in App Engine's Blobstore

In my previous post I showed how to take images uploaded to the Blobstore and resize them before storing them into the Datastore. This was used on PicSoup reliably for 6 months or so. I really wanted to find a solution that could use faster image serving services like Picasa but ran into terms of service issues.

Shortly after writing that blog entry App Engine SDK 1.3.6 was released along with the new fast image serving facilities. There are some good tutorials which explain this in detail so I won't repeat that here.

What I needed for Picsoup was to use the new image serving but to be able to manipulate the images in the Blobstore. This is so I can allow the user to upload images larger than 1MB but then resize them to 800x600 to store them. Typically a JPEG at this size is less than 100KB whereas the originals tend to be around 3MB. I also wanted to provide a rotation feature so the user can correct the image orientation after uploading.

The key to this problem is how to get an image out of the Blobstore, manipulate it, and put it back. The Blobstore API has no methods for writing directly to it, you can only write by uploading data through an HTTP POST and thus creating a new Blob. So the problem breaks down into three steps:
  1. Get the image from the Blobstore and manipulate it
  2. Upload the new image to the Blobstore
  3. Update Datastore references to the new Blob and remove the old Blob

Step 1: Get the image from the Blobstore and manipulate it

BlobKey bk = new BlobKey(ce.getBlobKey());
ImagesService imagesService = ImagesServiceFactory.getImagesService();
Image oldImage = ImagesServiceFactory.makeImageFromBlob(bk);
Transform rotate = ImagesServiceFactory.makeRotate(90);
Image image = imagesService.applyTransform(rotate, oldImage, ImagesService.OutputEncoding.JPEG);
sendToBlobStore(Long.toString(ce.getId()), "save", image.getImageData());
My domain object, a competition entry (ce), has a BlobKey string property. I use the ImageService to make an image from the blob and rotate it to create a new image.

Step 2: Upload the new image to the Blobstore

This is the step that I imagine the App Engine team will get around to adding to the API at some point. It would be nice to have a function to complement makeImageFromBlob(BlobKey) called makeBlobFromImage(Image). In the mean time I have written my own multipart/form-data post routine:
private static final boolean PRODUCTION_MODE = SystemProperty.environment.value() == SystemProperty.Environment.Value.Production;
    
private static final String URL_PREFIX = PRODUCTION_MODE ? "" : "http://127.0.0.1:8888";

private void sendToBlobStore(String id, String cmd, byte[] imageBytes) throws IOException {
    String urlStr = URL_PREFIX+BlobstoreServiceFactory.getBlobstoreService().createUploadUrl("/blobimage");
    URLFetchService urlFetch = URLFetchServiceFactory.getURLFetchService();
    HTTPRequest req = new HTTPRequest(new URL(urlStr), HTTPMethod.POST, FetchOptions.Builder.withDeadline(10.0));
    
    String boundary = makeBoundary();
    
    req.setHeader(new HTTPHeader("Content-Type","multipart/form-data; boundary=" + boundary));
    
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    
    write(baos, "--"+boundary+"\r\n");
    writeParameter(baos, "id", id);
    write(baos, "--"+boundary+"\r\n");
    writeImage(baos, cmd, imageBytes);
    write(baos, "--"+boundary+"--\r\n");

    req.setPayload(baos.toByteArray());
    try {
        urlFetch.fetch(req);
    } catch (IOException e) {
        // Need a better way of handling Timeout exceptions here - 10 second deadline
        logger.error("Possible timeout?",e);
    }        
}

private static Random random = new Random();    

private static String randomString() {
    return Long.toString(random.nextLong(), 36);
}

private String makeBoundary() {
    return "---------------------------" + randomString() + randomString() + randomString();
}        

private void write(OutputStream os, String s) throws IOException {
    os.write(s.getBytes());
}

private void writeParameter(OutputStream os, String name, String value) throws IOException {
    write(os, "Content-Disposition: form-data; name=\""+name+"\"\r\n\r\n"+value+"\r\n");
}

private void writeImage(OutputStream os, String name, byte[] bs) throws IOException {
    write(os, "Content-Disposition: form-data; name=\""+name+"\"; filename=\"image.jpg\"\r\n");
    write(os, "Content-Type: image/jpeg\r\n\r\n");
    os.write(bs);
    write(os, "\r\n");
}
The sendToBlobStore method takes three arguments:
  1. id - a domain object key id used to update the datastore reference to the new blob
  2. cmd - a command string used to determine how to handle the uploaded data
  3. imageBytes - a byte array of the new image that is to be uploaded
it then creates a multipart/form-data payload to send via the URLFetchService. The Deadline has been set to 10 seconds - the current maximum - but as you can see there is still a try..catch block around urlFetch.fetch(req) to catch timeouts. More about this later.

Step 3: Update Datastore references to the new Blob and remove the old Blob

The Blobstore calls back to "/blobimage" as defined earlier in sendToBlobStore when it has finished storing the new blob. So a doPost method is required to handle the incoming callback. When we have finished processing the callback we have to send a redirect and therefore we have to have a servlet request handler ready to respond as well. A possible quirk I've noticed here is that the browser follows the redirect via a GET whereas the URLFetchService follows it with another POST request, therefore the handler has to be available for both.
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    resp.getWriter().write("SUCCESS");
}

public void doPost(HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException {
    // Handle post requests
    String qcmd = req.getParameter("qcmd");        
    if ("success".equals(qcmd)) {
        res.getWriter().write("SUCCESS");
        return;
    }
    
    // Handle upload callbacks
    Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
    if (blobs.isEmpty()) {
        throw new ServletException("UploadedBlobs map is empty");
    }
    Entry<String, BlobKey> entry = blobs.entrySet().iterator().next();
    
    String handler = entry.getKey();
    BlobKey blobKey = entry.getValue();
    
    if ("upload".equals(handler)) {
        initialUploadHandler(res, blobKey);
    } else if ("save".equals(handler)) {
        saveHandler(req, blobKey);
        res.sendRedirect(SUCCESS_RESULT);
    } else {
        throw new ServletException("Invalid handler request ["+handler+"]");
    }
}
Here you can see that my handlers for the redirects just send the word SUCCESS. My GWT code reads this and then makes further RPCs to update the front-end. The section to explain here is under the "Handle upload callbacks" comment. What I'm doing here is simply taking the first entry from the UploadedBlobs map and using the key to determine how to process the callback. The key is the "cmd" parameter we passed in earlier to the sendToBlobStore method. I have removed a few handlers from this example for brevity but you can see here how I can have different processing for an initial upload from a browser versus an internal upload following a rotate transformation.

The rotate operation we ran in Step 1 passed in the cmd "save" meaning the saveHandler is called:
private void saveHandler(HttpServletRequest req, BlobKey blobKey) {
    Long compEntryId = new Long(req.getParameter("id"));
    logger.info("Incoming image to save: ["+blobKey.getKeyString()+"] id=["+compEntryId+"]");

    CompEntry ce = dao.getCompEntry(new Key<CompEntry>(CompEntry.class, compEntryId));
    if (ce != null) {
        String oldBlobKey = ce.getBlobKey();
        
        ce.setBlobKey(blobKey.getKeyString());
        ce.setServingUrl(getServingUrl(blobKey));
        ce.setResized(true);
        dao.ofy().put(ce);
        
        // Delete the old Blob
        if (oldBlobKey != null) {
            blobstoreService.delete(new BlobKey(oldBlobKey));
        }
    }
}

private String getServingUrl(BlobKey blobKey) {
    String servingUrl = ImagesServiceFactory.getImagesService().getServingUrl(blobKey);
    // Hack for Dev mode
    if (PRODUCTION_MODE) {
        return servingUrl;
    } else {
        return servingUrl.replaceFirst("http://0.0.0.0:8888", "");                    
    }
}
In saveHandler there is a little bit of Objectify code to update the datastore object to reference the new blob. The old blob is then deleted. Note my little hack in getServingUrl to iron out a difference between the Development and Production environments.

Timeouts

I arrived at the design above following a number of experiments. The main problem that shapes the solution this way is the URLFetchService timeout. The maximum deadline at the moment is 10 seconds which seems like plenty of time but an IOException for a timeout is regularly thrown. For some reason (any explanation gratefully received) when there is only 1 instance of the app running in production the deadline is always reached. As soon as there are 2 or more instances running this stops happening. Unfortunately the exception thrown is just an IOException and not something more specific like URLFetchDeadlineExceededException which would be much nicer. On the development server this timeout is never reached.

To get around this timeout issue you just have to make sure that any critical code goes into the Blobstore callback handler. For example, I save the change to the domain object in saveHandler and not in my original call in Step 1. In my GWT front-end I have routines to check that the transformation is complete and show a spinner while waiting.

Picsoup is now using this code, go and check it out!

2010-08-05

Blogger stats + reddit

I recently switched to Blogger In Draft so I could start using the Blogger stats. It's great to see the Traffic Sources to your blog. You can find all the popular places you're linked from, brilliant!

Below is a snapshot of the stats for the last month. See if you can spot when I posted my recent blog entry: Google App Engine Image Storage? on reddit!

2010-07-31

Google App Engine Image Storage?

My App Engine project: PicSoup, is a weekly photo voting competition with a twist (read the Help tab at the site for instructions). The app is all about images. Users upload (or email in) their competition entry which is then resized and displayed on the site for everyone to vote for.

The Blobstore is used for the initial upload to get around the 1MB limit. Sadly this is not available for emails so the user is limited to 1MB for email entry. The large image is then resized and stored as a blob in the Datastore like this:

Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
BlobKey blobKey = blobs.get("upload");

// Transform the image to a small one and save it in the datastore
ImagesService imagesService = ImagesServiceFactory.getImagesService();

Image oldImage = ImagesServiceFactory.makeImageFromBlob(blobKey);

Transform resize = ImagesServiceFactory.makeResize(250, 220);

Image newImage = imagesService.applyTransform(resize, oldImage, ImagesService.OutputEncoding.JPEG);
   
Pic pic = new Pic();
pic.setImage(new Blob(newImage.getImageData()));           
dao.putPic(pic);

When it's time to serve an image the servlet code is pretty simple.

public void doGet(HttpServletRequest req, HttpServletResponse res) throws IOException {
    String picId = req.getParameter("id");
    if (picId != null) {
        Pic p = dao.getPic(Long.parseLong(picId));
        if (p != null) {
            res.setContentType("image/jpeg");
            res.setHeader("Cache-Control", "max-age=1209600"); //Cache for two weeks
            res.getOutputStream().write(p.getImage().getBytes());                
        }
    }
}

Note that I'm using the Cache-Control header with a max-age of two weeks. The image with the given id never changes so in theory this could be set to forever. This caching is very important because otherwise the app gets hit every time for the image. Users of PicSoup frequently visit the site to check for new entries .

The downside to this is that sometimes the Datastore can be very slow. I've watched images appear like they used to on an old 56k modem! Google were having some problems with Datastore performance and it is way better now but it's not as fast as accessing a static file on a dedicated server.

The performance and the Datastore usage quota put me off keeping a higher resolution image but the site really needed it. So I started developing a way to store the big images in Picasa. The documentation is really good and I soon had this working. Now when the user uploaded their image the small image would still be stored in the Datastore as above but then a task would be enqueued on the Task Queue to transform the image from the Blobstore again and then upload it to my Picasa account:

private void addToPicasa(String blobKey, String id) {
    ImagesService imagesService = ImagesServiceFactory.getImagesService();
    BlobKey bk = new BlobKey(blobKey);
    Image oldImage = ImagesServiceFactory.makeImageFromBlob(bk);
    Transform resize = ImagesServiceFactory.makeResize(1024, 768);
    Image image = imagesService.applyTransform(resize, oldImage, ImagesService.OutputEncoding.JPEG);
    logger.info("Big image bytes: "+image.getImageData().length);
            
    PicasawebService myService = new PicasawebService("PicSoup");
    try {
        myService.setUserCredentials("mypicasaaccount@gmail.com", "my.password");
        String albumid = "5495486978942507441";
        if (SystemProperty.environment.value() == SystemProperty.Environment.Value.Production) {
            albumid = "5495487073314754801";
        }
        URL albumPostUrl = new URL("http://picasaweb.google.com/data/feed/api/user/mypicasaaccount/albumid/"+albumid);

        PhotoEntry myPhoto = new PhotoEntry();
        myPhoto.setTitle(new PlainTextConstruct(id));

        MediaSource myMedia = new MediaByteArraySource(image.getImageData(), "image/jpeg");
        myPhoto.setMediaSource(myMedia);

        PhotoEntry returnedPhoto = myService.insert(albumPostUrl, myPhoto);            
        MediaContent mc = (MediaContent) returnedPhoto.getContent();

        CompEntry ce = dao.ofy().get(CompEntry.class, Long.parseLong(id));
        ce.setPicUrl(mc.getUri());
        ce.setPhotoId(returnedPhoto.getGphotoId());
        
        dao.ofy().put(ce);
    } catch (Exception e) {
        logger.error(e);
    }
    
    // Delete the hi-res
    blobstoreService.delete(bk);        
}

Before I started the UI work to display the image from Picasa I checked the Terms of Service and realised this solution may be contrary to item 5.9:
5.9 In order to use the Picasa Web Albums API with your service, all End Users on your service must have previously created their own individual Picasa Web Albums accounts. You must explicitly notify End Users that they are accessing their Picasa Web Albums accounts through your service. In other words, you may not create one or more Picasa Web Albums accounts for the purpose of storing images on behalf of users without those users creating their own individual Picasa Web Albums accounts.
Now, strictly, I'm not sure I'm storing the images on behalf of the users - they've kind of donated them to me and my app. I searched around for some clarification and found that there are plenty of people trying to do this sort of thing and the answer is always no. Have a look at this search in the forum.

So, I've removed Picasa from my app and I'm now using the Datastore to hold an 800x600 image as well. (If you go to PicSoup today [31-July-2010] only the most recent entries have the high-res view available, just click the small image). Now that the Datastore performance has improved this is not so bad.

I've looked at Flickr and Photobucket as well and they also seem to have a clause like this in their terms.

Does anyone know of a service where this is allowed?

UPDATE See my new post which explains how to use the new Blobstore based fast image serving service.

2010-07-03

About PicSoup

I've been working on PicSoup (a GWT Google App Engine application) for a few months now (on and off) and as it turns out it's quite a good show case for a lot of the App Engine Services:
  • Datastore - using Objectify instead of cumbersome JDO or JPA
  • Memcache - explicit use and Objectify annotation driven
  • Mail - both incoming picture submission and outgoing notifications
  • Images - resize the incoming pics before storing them
  • Google Accounts - for authentication, user management and admin access
  • Task Queues - used to offload competition state change processing and email notifications
  • Blobstore - uploaded images are temporarily stored here before resizing
  • Scheduled tasks - cron triggers change the competition state on a weekly basis
  • Administration Console Custom Pages - GWT admin controls
  • Appstats - used to improve performance

As well as the main front-end PicSoup has an automatically detected mobile front-end so it looks good on your Android or iPhone device. It also has a Google Gadget - see top right of this blog!

2010-05-16

App Engine email to 'admins' gotcha

I recently started adding email notifications to PicSoup (my GAEJ application). I followed the examples in the documentation and wrote a really simple function to allow me to send a simple text email to one recipient:
private void sendEmail(String recipientAddress, String recipientName, String subject, String message) {
    Properties props = new Properties();
    Session session = Session.getDefaultInstance(props, null);
    try {
        Message msg = new MimeMessage(session);
        msg.setFrom(new InternetAddress("my-admin-account@gmail.com", "PicSoup Admin"));
        msg.addRecipient(Message.RecipientType.TO, new InternetAddress(recipientAddress, recipientName));
        msg.setSubject(subject);
        msg.setText(message);
        Transport.send(msg);
        logger.info("Sent email to "+recipientAddress);
    } catch (Exception e) {
        logger.error("Failed to send email",e);
    }
}
Simple stuff, and almost identical to the documented example. Note that it uses the InternetAddress constructor that allows you to enter an address and a personal name.

This works really well and allows me to write simple calls like this:
sendEmail(p.getUser().getEmail(), p.getDisplayName(), "PicSoup info", "Your competition entry was successfully added. Good luck!");

This notifies a particular user and uses their settings to get the appropriate "personal name". I also wanted to use this call to send notifications to myself. Being the administrator I could do this using the Admins Emailed quota. To do this I thought I could use the special "admins" recipient with my sendEmail function like this:
sendEmail("admins", "PicSoup Administrators", "PicSoup info", "A new pic has been added.");

Sadly I discovered that this doesn't work. It silently fails to send the email to anyone! It turns out that this is because I have included "PicSoup Administrators" as the "personal name" in the InternetAddress object. In order to make this work I changed my sendEmail method to ignore the "personal name" for emails to admins:
private void sendEmail(String recipientAddress, String recipientName, String subject, String message) {
    Properties props = new Properties();
    Session session = Session.getDefaultInstance(props, null);
    try {
        Message msg = new MimeMessage(session);
        msg.setFrom(new InternetAddress("my-admin-account@gmail.com", "PicSoup Admin"));
        if ("admins".equals(recipientAddress)) {
            msg.addRecipient(Message.RecipientType.TO, new InternetAddress(recipientAddress));
        } else {
            msg.addRecipient(Message.RecipientType.TO, new InternetAddress(recipientAddress, recipientName));             
        }
        msg.setSubject(subject);
        msg.setText(message);
        Transport.send(msg);
        logger.info("Sent email to "+recipientAddress);
    } catch (Exception e) {
        logger.error("Failed to send email",e);
    }
}

2010-04-12

Providing for mobile web site visitors

When I'm using a mobile or an iPod Touch the web sites that detect my device and show me a mobile friendly page are more likely to be added to my favourites. For example Digg has a mobile version of their site which you are automatically taken to.

You can detect a mobile device by looking at the HTTP User Agent and Accept headers. Here are these two headers for an Android phone as captured in my application log:
INFO: Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1
INFO: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5,application/youtube-client
and for an iPod Touch:
INFO: Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16
INFO: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Detecting all the many and varied mobile devices out there would be a real pain without the help of MobileESP. This project supports multiple languages but I'm using Java on Google App Engine so all I needed to do was place the single java source file in my project.

I then wrote a servlet to handle requests to the home URL which redirects to the normal or mobile version of the site like so:
public void doGet(HttpServletRequest req, HttpServletResponse res) throws IOException {
    String userAgent = req.getHeader("User-Agent");
    String accept = req.getHeader("Accept"); 
     
    UAgentInfo info = new UAgentInfo(userAgent, accept);
       
    boolean iPhone = info.detectTierIphone();
          
    if (iPhone) {
        res.sendRedirect("/Mobile.html");
    } else {
        res.sendRedirect("/Normal.html");
    }
} 
This code simply passes the headers to the UAgentInfo object and then detects iPhone like devices. The Javadoc for detectTierIphone() says:
The quick way to detect for a tier of devices. This method detects for devices which can display iPhone-optimized web content. Includes iPhone, iPod Touch, Android, Palm WebOS, etc.

2010-03-18

Using UserPref with URL content type gadgets

Following on from my previous post I needed to add a user preference to my gadget. By following the documentation here my previous example changed to be something like this:
<?xml version="1.0" encoding="UTF-8"?>
<Module> 
      <ModulePrefs 
            height="100" 
            title="My Simple Gadget" 
            description="Test gadget" 
            author="Anonymous" 
            author_email="anonymous+gg@gmail.com"/>
      <UserPref 
            name="profileId" 
            display_name="Profile ID" 
            datatype="string" 
            urlparam="id" 
            required="true"/> 
      <Content 
            type="url" 
            href="http://yourappid.appspot.com/gadget.html" /> 
</Module>

Note that I used the urlparam attribute so that the querystring used to fetch the gadget content would be something like
http://yourappid.appspot.com/gadget.html?id=123.

My GWT code could now read this userpref like this:
public void onModuleLoad() {
    String id = Window.Location.getParameter("id");
    . . . .

Unfortunately there is a minor issue with this. It seems that some gadget containers ignore the urlparam attribute. When the gadget is hosted in the iGoogle home page it works fine but in Blogger and some other places the id parameter was not coming through in the querystring. Instead you get something like like
http://yourappid.appspot.com/gadget.html?up_profileId=123
Basically the UserPref name prefixed with up_. This meant a minor change to the GWT code:
public void onModuleLoad() {
    String id = Window.Location.getParameter("id");
    if (id == null) {
        id = Window.Location.getParameter("up_profileId"); 
    }
    . . . .

UPDATE:
It seems that Google have broken the URL type gadgets. The parameters are no longer sent in the URL to the server! See this iGoogle Developer blog post

This makes the above code rather useless. I'll be working on a new version...

UPDATE 2:
According to this forum post the problem has been fixed temporarily. I think the advice is to use HTML content type gadgets if you want to use UserPrefs. Shame.

UPDATE 3:
URL type gadgets are back for good! See this post. To summarize:
I've been talking with developers on the iGoogle team about the best way to go forward here and it looks like keeping UserPrefs in the query string for type url gadgets is the thing to do.

2010-02-27

Simple Google Gadget on GAE

I've just found how easy it is to create Google Gadgets and host them on App Engine. All you need to do is create a simple page in your App Engine application to be the content of the gadget and then host a URL content type gadget specification that points to that page.

Your Gadget specification will look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<Module> 
      <ModulePrefs 
            height="100" 
            title="My Simple Gadget" 
            description="Test gadget" 
            author="Anonymous" 
            author_email="anonymous+gg@gmail.com"/> 
      <Content 
            type="url" 
            href="http://yourappid.appspot.com/gadget.html" /> 
</Module>

In my case I'm using a very simple GWT application. Make sure that it looks good at 300 pixels wide. The height can vary but don't make it too tall because people are less likely to install it if it takes up too much space.

The URL content type approach allows you to make a normal web page using any tools you like and you can debug it like any normal page.

To put it all together simply save your gadget spec into the war directory of your GAE application and deploy it so it's available on a URL like this: http://yourappid.appspot.com/gadgetspec.xml

Now you can test it in the iGoogle Sandbox which (after you've signed up) allows you to test unpublished gadgets. Add this developer gadget to make life easy.

When you're happy with it submit your gadget and you're done.