Ahmed Live !

While working on a project, I needed to to have a feature like facebook’s “Post a link”. The user enters a URL, clicks on preview and he gets a preview of how that page will be posted on his profile. Facebook does a few tricks

1. It fetches the URL’s title and description (a small paragraph about that page’s content)
2. It provides the user with a list of images from that page so that he pick one of them
3. It also provides the user with the ability to edit the title and description if he doesn’t like what Facebook suggested

A user then posts that URL. That’s when things start to get even more interesting

1. if the link was for a video from youtube or vimeo – for example – it’ll show the youtube or vimeo player (embedded object)
2. if it’s a flickr photo page, it’ll show the image
3. if it’s not any of the websites that are handled specially, the title, description and image are shown

I needed to have the same behavior.

First, let’s handle the generic case of any webpage. Given any URL, I had to get that pages HTML content and figure out the following

• Title
  We can figure out the page’s title from any of the following
  1. The title tag <title>This is the title!</title>
  2. The meta title tag <meta content="This is the title!" name="title">
  3. The URL itself
• Description
  Again, it can come from multiple sources but I prefer to rely on the first only
  1. The meta description tag The meta title tag <meta content="This is the title!" name="description" />
  2. The first text appearing after the body. But it can never guarantee anything meaningful
• Images
  All sources of the all image tags on that page

Now to the special pages. We’ll have to keep a database of all sites that we consider special and that we are able to treat differently. For example, youtube, so that if we know it’s a youtube’s video url, we can build the proper embed object.

I felt it was too much for a simple application to handle but I really wanted to get that feature. Before implementing anything, I tried to search and even asked at stackoverflow. Although I didn’t find the answer I wanted, the answers were really helpful. It was then that I learned about

1. oEmbed
2. JSONP

oEmbed

“oEmbed is a format for allowing an embedded representation of a URL on third party sites. The simple API allows a website to display embedded content (such as photos or videos) when a user posts a link to that resource, without having to parse the resource directly.” - http://oembed.com/

Unfortunately, it was too good to be true. The list of websites that support oEmbed is very limited. oohEmbed is a project trying to address that issue by providing oEmbed to a greater number of websites.

oohEmbed proxies calls to oEmbed providers if possible and provides its own implementation when needed. The thing I like the most about oohEmbed is that it provides a single url through which you access oEmbed for various websites in contrast with the original oEmbed protocol where each website provided its own url for accessing oEmbed.

For example to get the representation for a flickr URL (http://www.flickr.com/photos/ianpollock/6707007)

• Through flickr itself http://www.flickr.com/services/oembed/?url=http://www.flickr.com/photos/ianpollock/6707007
• Through oohEmbed http://oohembed.com/oohembed/?url=http://www.flickr.com/photos/ianpollock/6707007

JSONP

It’s simply a trick so as to be able to grab JSON data bypassing the cross-domain issues. The idea is, many APIs provide JSON responses.
How can a browser make use of these APIs directly?

1. A script tag is inserted dynamically that points to the API endpoint specifying the response format as JSON
2. Normally, the endpoint will respond with a JSON object
3. JSONP calls for providing an additional parameter callback. Now, instead of returning a JSON object, the text returned represents a function call where its parameter is the JSON object.
4. The end result is that a javascript function is called whenever the script file finishes loading and this function is given the JSON response as a parameter

Flickr supports this technique. Without providing a callback, the call should return the following response

<script type=”text/javascript” src=”http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=a38b5237570253cc91d6b54ed9cf1535&place_id=Pu5_HsObApilq4vUtQ&tags=funny&format=json&nojsoncallback=1”></script>

{..}

Now with the callback

<script type=”text/javascript” src=”http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=a38b5237570253cc91d6b54ed9cf1535&place_id=Pu5_HsObApilq4vUtQ&tags=funny&format=json&jsoncallback=showPhotos”></script>

showPhotos({..})

Solving the problem

As I’m using Rails for the application I’m writing and without considering cross-domain javascript calls, the typical scenario would have been as follows

1. The user enters a URL and hits the preview button
2. Using AJAX, a request is sent to a dedicated action on the application server. That action would make an HTTP request to the URL the user provided, parse the HTML page to get the page’s info (title, description, image sources)  and convert them to JSON and submit that JSON object as a response to the browser
3. The browser then uses that JSON data to show some data about the URL the user requested

The problem with that solution is that the HTTP requests made by the server are blocking and if too many requests of that type occur, the application servers might get caught in these requests and thus will not be able to service normal requests.

Analyze This!

Learning about JSONP – as a clean solution to cross domain communication, I decided it’s best to separate the action of requesting a page and parsing its HTML into a separate application. This way the original application’s performance will not get affected. The new application is called Analyze This. In its current form it’s a simple Sinatra application that has a single action.

The kind of requests – blocking and take some time - that Analyze This handles made it a perfect candidate to use an upcoming version of NeverBlock that not only handles DB but sockets and networking as well.

For example, an HTML page requesting information about a user entered URL use a code similar to the following

$(document).ready(function(){
    $('#button').click(function(){
      var url = encodeURIComponent($('#url')[0].value);
      $('<script>').attr('type','text/javascript').attr('src','http://ANALYZE_THIS_DOMAN/analyze_this?callback=analyze_this&url='+url).appendTo('body');
    });
  });

That would result in a call to the javascript function analyze_this({..}) passing in a JSON object that has the page’s info.

To see it live, just download Analyze This, run the application and visit the sample page.

The Analyze This application itself is no big deal but the point is architecture really matters.

Recently, I’ve been involved in a project where I was creating a WAP site using Ruby On Rails that needed to display certain locations using Google Maps. Given the limited capabilities for the average WAP browsers (mainly the lack of JavaScript) I needed to figure out a way.

To my good luck, Google had Google Static Maps API which is basically a URL that you put in the src attribute of an img tag. The URL accepts parameters that will define the image you get: The most important parameters are the Longitude/Latitude of the map’s center and the zoom level. The URL looks like the following

http://maps.google.com/staticmap?key={google_maps_api_key}&size={width}x{height}&center={latitude},{longitude}&zoom={zoom}

Since Google Static Maps API is just an image provider, what if we wanted richer interaction (zooming, panning) with the map?

We’ll have to implement those features on our own.

Here’s the approach we took

1. Make a page on your website dedicated to showing the map  (http://m.example.com/map)
2. Have that page accept URL parameters to indicate the longitude, latitude and zoom level (http://m.example.com/map?lat=30.1037&lng=31.3366&zoom=14)
   The map’s page will use the given parameters to construct the Google Static Maps API’s image url
3. On the map’s page, create 6 links (2 for zooming, 2 for panning east and west, 2 for panning north and south)

Zooming
This is the easy part. The 2 links, one just increments the zoom level, the other just decrements. For example:
Current URL: http://m.example.com/map?lat=30.1037&lng=31.3366&zoom=14
Zoom in URL: http://m.example.com/map?lat=30.1037&lng=31.3366&zoom=15
Zoom out URL: http://m.example.com/map?lat=30.1037&lng=31.3366&zoom=13

Panning
This is a bit more difficult. If we want to pan east or west, we’ll have to move the map’s center a little bit right or left. How much exactly is that little bit?
To answer that question, we need a little more info on how Google Maps work. Google Maps provide the map in the form of 256x256 pixels tiles. At zoom level 0, the whole world is represented by a single tile. at zoom level 1, the whole world is represented in a 2x2 tiles. At zoom level 2, it’s 4x4 tiles.

There’s a projection that maps the earth to 2D. That is the Mercator Projection. Using that projection we can

1. Convert the Latitude/Longitude and Zoom to Pixels (X,Y)
2. Add some value (pixels) to (X,Y) that would allow us to pan east/west, north/south
3. Convert the new (X,Y) to Latitude/Longitude

I got the previous 3 steps form here. For example if we have the Latitude/Longitude and Zoom and width/height of the image in pixels:

# get the x,y of the image’s center
x = Mercator.lng_to_x(longitude, zoom)
y = Mercator.lat_to_y(latitude, zoom)

# To pan east or west we add/subtract half the image’s width (not necessarily half of course) and convert it back to longitude
x = x + 0.5 * width
longitude = Mercator.x_to_lng(x, zoom)

# To pan north or south we modify latitude similarly
y = y + 0.5 * height
latitude = Mercator.y_to_lat(y, zoom)

In the Google Maps API there’s a converter that allows converting between Latitude/Longitude and Pixels (X,Y) at different zoom levels.
The problem is – in addition to me not being an expert with all that projection stuff and its equations – is that I couldn’t take a look at the JavaScript source so that I could implement it in Ruby. Searching for that I came across an implementation in JavaScript here. I ported it to Ruby and it can be found here. There’s an issue with the code I implemented but I noted it in the gist. That issue would prevent you from using zoom levels greater than 17.

Hopefully these were enough info to get you started on using Zooming/Panning on Google Static Maps API.

While I was trying to cover a controller with some tests I faced a problem. The controller actions where protected by a filter that prompted the users for login via basic http authentication.

I found a solution in rails code here http://github.com/rails/rails/tree/master/actionpack/lib/action_controller/http_authentication.rb where it said you should do your get as follows get("/notes", nil, :authorization => ActionController::HttpAuthentication::Basic.encode_credentials(users(:x).name, users(:x).password))

This didn't work for me where basic http authentication required sending the encoded credentials in the request headers while the previous get request sent the authorization credentials in the session.

I found the following code snippet in http://snippets.dzone.com/posts/show/3785 which allowed me to set request headers class ActionController::TestRequest def set_header(name, value) @env[name] = value end end

In my tests, I now write the following @request.set_header "HTTP_AUTHORIZATION", ActionController::HttpAuthentication::Basic.encode_credentials(users(:x).email, '0000') get :show, {:user_id => users(:x).id, :format => "rss"}

And it works like a charm !

1. Text based captchas in which the user sees an image displaying letters or numbers and is asked to type what he sees
2. Image recognition captchas which display some images and asks questions about their content. Microsoft Assira is an example
3. 3D captchas which display come complex computer generated 3D graphics scene and asks about the 3D details and contents

1. recaptcha_tags which should be used in the view in your form.
2. verify_recaptcha which should be used in the controllers to verify the user's answer

  Ambethia::ReCaptcha.enabled = true
  Ambethia::ReCaptcha.public_key = '0123456789ABCDEF'
  Ambethia::ReCaptcha.private_key = '0123456789ABCDEF'

Building on my previous post about caching where the method_cache plugin was introduced, today I'd like to introduce another Rails plugin that also deals with the caching problem. This time, it's about caching views. It uses a technique called version caching. Version caching frees you from having to worry about writing code that expires your cache. For more info about version caching check Yasser Wahba's blog post explaining version caching and how we used it. I've used version caching in my last 2 projects and found it quite useful and quite easy. The problem is that we used to repeat a lot of code violating DRY. It was so clear we had to do something about it. I've taken that code, refactored it and made it available as a Rails (v2.1.0) plugin available at GitHub. Check it here.

The plugin assumes using a cache store that uses LRU (least recently used) to handle when the cache becomes full and that also supports time caching. Cutting it short, it assumes we're using memcached.

The plugin makes a couple of methods available in all controllers. These methods are

• version_cache
• time_cache
• version_cache_key_part

The first 2 are the most important. time_cache as the name implies allows for time caching for a page. it can be used as follows

class WelcomeController < ApplicationController 
  time_cache :index => {:expiry => 5, :browser_cache_enabled => true}
  .
  .
  def index
    .
    .
  end
  .
  .
end

What we've just done is that we declared our intention of caching the index page of the welcome controller for 5 minutes in our cache store. In addition to the cache store, we declared our intention that we also want the page to be cached in the browser for the same period.

version_cache is the one responsible for tying the caching of a page to a model's version. we can use it as follows

class ItemsController < ApplicationController
 
  version_cache Item, :associates => ["user"], :expiry => 5

end

By default, version_cache caches the show action unless an :action is provided. In the previous example we're saying that we want to cache the show action of the items controller.

Item: the model whose objects versions are used.
:associates: an array of members on the model whose versions need to be updated as well when the the main model's version is updated. i.e when an item is changed, its version is incremented and the item's user's version is also incremented. This is useful if we also cache users#show based on User model and changing an item reflects on the user's page. :associates is optional
:expiry => an optional maximum time for the page to expire. if not specified, expiry will happen on the normal LRU basis.

Models' versions are maintained by the means of an observer. The observer has to observer the models we use and to be also declared in the environment.rb

version_cache_key_part is another method that allows a page to have multiple cached versions. We can use it as follows

class PostsController < ApplicationController
  .
  .
  version_cache Post, :associates => ["user"], :expiry => 5
  .
  .
  def show
  .
  end
 
  def version_cache_key_part
    if logged_in_user
      "logged_in"
    else
      "guest"
    end
  end

end

What just happened here is that based on some conditions we return a string that will be part of the cache key. Now we have 2 versions cached for the show page; one for logged in users and one for guests. You can have as many versions as you want based on whatever conditions as long as they return distinct strings.

To get the plugin

ruby script/plugin install git://github.com/humanzz/version_cache.git

Then use the plugin’s generator to generate the cache observer

ruby script/generate version_cache_observer Cache Model1 Model2

The first argument “Cache” is taken to be the observer’s name. Any arguments after that are taken to be the models that the observer will observe.

Version caching is a great caching technique and hopefully with the introduction of the plugin many developers will find it appealing and easy to use.

Technorati Tags: ruby - rails - caching - version_cache - rails plugins

Building a large-scale web application can be troublesome. It leaves us facing all sorts of optimization problems. A very good – but no easy – solution is to use caching. Caching is a real saver as it can help with minimizing

1. Database queries
2. String operations

which are the two main problems that can affect the performance of a web application.

A common problem that developers always face is having to call a method regularly knowing that its results rarely change and its cost is always high. More on that in the following code

class User < ActiveRecord::Base
  def costy_method
    #some real heavy calculation that takes alot of resources
    #to complete
    #this method is called regularly and its results rarely change
  end
end

Calling user.costy_method with all its costs and behavior appears to be a point that can be optimized. How about caching its result? The user.costy_method needs to be modified so that it caches its result. A common practice for Ruby developers is to cache the result in an instance variable the following way

def costy_method
  @costy_method_result || = #the actual calculations
end

What if there were more methods like this? They'll all have to be rewritten so as to cache their results. This would be a tedious task with a lot of repeated code which violates DRY concept. Why not have a simpler solution for this particular problem? We wrote the method_cache plugin. Let's see how it works

class User < ActiveRecord::Base
  def costy_method
    #some real heavy calculation that takes alot of resources
    #to complete
    #this method is called regularly and its results rarely change
  end
  caches_method :costy_method
end

and voila, every time user.costy_method is called, you'll get the value you expect normally but this time at a much less cost.
caches_method :costy_method builds a 2 level cache(a note on that later), it caches the result in an instance variable inside the user object and it also saves a copy in your fragment_cache_store which can be of several types (a file store, a memory store or memcached). Now if the same user instance is used to call user.costy_method we'll get a huge performance gain as it will just cost u as much as it costs to retrieve any instance variable. If another user instance is used, the first time you call user.costy_method the value will be retrieved from your cache store and put for your convenience in the instance variable. If the value can't be retrieved from an instance variable or the cache store the real costy_method implementation is executed to get that value.

Another convenience method is expire_method :method_name which can be used like user.expire_method :costy_method. This method will simply clear the instance variable and cache store entry if they're available so that next time user.costy_method is called, a new value will be calculated.

Mido - Eng. Muhammed Ali - wrote this plugin. I pretty much loved the simplicity by which this will allow us to cache stuff.

More on the plugin features
We needed to be able to cache the results for certain amount of time. The problem is that the default MemCacheStore didn't support that. So again with some Ruby magic I overrided the MemCacheStore code so that I can write some stuff like the following
caches_method :costy_method, :until => :midnight
or
caches_method :costy_method, :for => 20.days

After using the plugin internally for a while we decided it's best if others can make use of it and also contribute. We've setup a project on RubyForge which can be found here.

In its current form, the plugin provides the following extensions to ActiveRecord Objects
caches_method :method_name
caches_class_method :method_name
and for expiring the cache
instance.expire_method :method_name
Class.expire_instance_method :method_name, id
Class.expire_class_method :method_name

More on the 2 level cache mentioned before, for class methods, we cached the results in static variables but this was a deal breaker once we used several mongrels to serve our application as it's impossible to synchronize the expiry of the static variables on the different mongrels so we decided it was enough to use the fragment cache store as our only store when dealing with class methods.

I think it would be nice to refactor the plugin so as to be able to use it with all classes not just ActiveRecord::Base descendants. This would require some minor changes such as defining a method that defines an instance uniquely. In case of ActiveRecord objects, it's just the id.

Update
I've updated the plugin so it uses the Rails cache store introduced in Rails 2.1. I've also moved the plugin to GitHub and it can be found here.

To install it

ruby script/plugin install git://github.com/humanzz/method_cache.git

Quite often, Ruby on Rails developers use the flash object to store objects for the following request. Flash's most common use is for displaying messages to users after redirections. Flashes are usually rendered in the application's layout.

Consider the following 2 scenarios that will stop flashes from working properly

1. AJAX requests.
   Flash will simply be ignored as it's rendered in the layout. One way to handle this is to have the AJAX response carry both the normal response + the flash message and have the JavaScript at the client figure out which is which then display the flash message and handle the normal response. This is a bit cumbersome but doable.
2. Responding with cached pages.
   What if the server responds with a cached page? The layout will never be rendered and so will the flash messages.

So, what we really need is a solution that doesn't rely on the rendering of the layout or of partial responses.

Have a cookie and be a happy - hopefully not so fat - man :)
Really, what about storing the flash messages in a cookie? Have JavaScript check that cookie and display the message accordingly?
How do we write the flash message into a cookie? What about an after_filter that writes the flash message into a cookie and then wipes the flash object clean?

So, we need to edit 2 files

1. ApplicationController
   class ApplicationController < ActionController::Base
  .
  .
  after_filter :set_flash_cookies
  private
 
  def set_flash_cookies
    if(flash[:error])
      cookiesNotification('error', flash[:error])
      flash[:alert] = nil
    elsif(flash[:note])
     cookiesNotification('note', flash[:note])
     flash[:info] = nil
    end
  end

  def cookiesNotification(type, msg)
    begin
      cookies[:notification] = {:type => type, :message => msg.to_s}.to_json
    rescue => e
      #hanlde the exception
    end
  end
end

The previous code adds the flash message into a cookie and wipes the flash object clean.
2. The application layout
   First, we'll have a function that extracts the message out of the cookie if present and display the message then delete the cookie.
   
   processNotifications = function(){
  if(Cookies.readCookie("notification")){
    //Cookies is a namespace that has a couple of function that abstracts processing cookies
    eval("notification = " + unescape(Cookies.readCookie("notification").replace(/\+/g," ")));
    //show the notification in its normal place
    showNotification(notification.type, notification.message);
    Cookies.eraseCookie("notification");
  }
}

   We need to make sure that this function is called at proper times i.e. when pages are loaded and after AJAX calls are completed. This can be achieved using the following code which uses prototype javascript library
   
   //for page loads
document.observe("dom:loaded", function(){processNotifications();});
//for AJAX calls
Ajax.Responders.register({ onComplete: function(){ processNotifications();}});

Who said cookies were bad ?!

Web standards and different browsers implementation have always been a nightmare for software developers. Joel Spolsky gives a nice explanation about the current situation.

Check his blog entry "Martian Headsets".

One of my favorite programs is FolderShare. For those who don't know, it is a small program that sits in your tray but gives you great features

1. The ability to share certain folders - Libraries in FolderShare terminology - with any number of your friends
2. Assign people who're participating in your libraries different privileges like reader, contributor or editor.
3. Synchronize different libraries across your different computers
4. Remote Access to any of your computers' file system.
   That's a real cool feature. I leave my home PC online 24x7. I can so easily access its files through a real simple interface from foldershare's website.

Microsoft bought FolderShare in 2005 and it was supposed to be integrated into Windows Live services but for the past 2 years there were no updates at all to FolderShare. That didn't bother me, I just loved the service the way it was. Today, however, the foldershare website has been updated at last to reflect the Windows Live brand and a new client is released. It can be downloaded from here. Check WL FolderShare blog.

There are no new features yet. However, I don't think that would last for much longer. FolderShare website and service appears to have been using PHP but the updated version, naturally, uses ASP.NET.

A couple of features I'd love to see

1. Integration with Windows Live ID
   Sign in once, use all WL services
2. Forget about Windows Live Messenger Sharing Folders. Do the sharing through FolderShare
3. FolderShare + SkyDrive. That would be a real interesting story. Have SkyDrive appear  as one of my computers in FolderShare so I can store Libraries on it, share with friends from it and sync my personal folders to it. That would just be great

I'm sure integration is WL ID is coming. It's the only logical thing to do next. Integration with messenger seems logical to me too but will they do it anytime soon ?!
I really hope someone is already exploring the FolderShare + SkyDrive integration. It will just rock.

Did you try Microsoft Photosynth?!
If you didn't then please go ahead. It's just a beautiful way of navigating through very large collections of images to the extent of building 3D models of places and scenes shot by thousands of different cameras and people.

Did you try Google/Windows Live/Yahoo maps?!
I enjoy all the zooming and panning through the maps. Maps work by loading images as we navigate where they load only the relevant pieces.

Have you ever wanted to do something similar with a large collection of images you own ?!
Navigate through a large collection of images while saving bandwidth and only loading the images you want ?!

Meet Deep Zoom.
It's just a fascinating piece of technology that allows you to create stunning user experiences using zoom and panning.

For some example sites check the following

• http://memorabilia.hardrock.com/
• http://www.vertigo.com/deepzoom.aspx

You should have Silverlight 2 Beta 1 runtime in order to try it.

You can give it a try and upload some pictures of yours and try it at Windows Live PhotoZoom. More on that from Liveside.net can be found here.

For developers, you can create Silverlight applications that take advantage of the Deep Zoom technology so easily

1. Grab Deep Zoom Composer from here and check the documentation
2. Grab the sample from Expression Blog (it's a real cool demo).

I was thinking about posting my demo application where I used several photos for lovely Angelina Jolie but then I searched for an example and I found a whole lot of them available. So if you struggle or something just search for Deep Zoom and I'm sure you'll get what you want easily.

Now you are ready to go.