Search usability enhancements

Following on from the search functionality enhancements to provide more efficient search indexing on CloudEngine, we’ve spent some time improving the interface and usability of the search results interface.

As increasing amount of content has been added to Cloudworks, certain search parameters are now returning many results, for example, searching for ‘learning’ on the live site returns over 2000 clouds, 300 cloudscapes and 600 users. All of these results were being presented on one page over two columns. Not only was this creating very large pages for users to scroll through, the results for user pages were appearing in the second column somewhere down the page following the cloudscape results.

Original search results interface

Original search results interface

This was clearly becoming unwieldy so we have made some changes to improve the search interface and usability. Here is a summary of the changes that have been made:

  • the results for clouds, cloudscapes and users have been split in to three separate tabs. This allows results with long titles to span the width of the page rather then split over several lines with the two column layout.
  • the total number of results is shown above the results area.
  • the number of results for clouds, cloudscapes and users is shown at the top of each tab.
  • each result has a percentage score which is the relevance of the result to the top result (which is usually scored 100%).
  • On each tab, results are paginated, with 15 results per page (customisable).
  • 200 results are retrieved with the search and are actually hidden from view but can be quickly displayed by using a link under the result set on each tab.
  • results are now presented in an ordered list, ordered by relevance in each tab, so the user can more clearly see the position of a search result within a set of results.
  • there is a link on each tab to show the full set of results for a clouds, cloudscapes and users (similar to the original search).
  • for a screen resolution of 1280 x 1024, search results usually fit on one page without needing to scroll.

Here is a screenshot of a search for ‘learning’ with the new interface:

New seach results user interface

New seach results user interface

We hope these changes make the search functionality on CloudEngine more usable and user-friendly. We hope to put these changes live on Cloudworks early during the week commencing 30th May 2011.

Posted in Uncategorized | Leave a comment

Search functionality enhancements and debug settings v1.1.2

Following reports of the search functionality on the Cloudworks site failing with an ungraceful php error message when using certain search parameters, we undertook some investigation in to the cause of the problem. It transpired that the Zend Lucene search engine was indexing all of the HTML content of the clouds, cloudscapes and user pages which included all of the recurring content such as navigation, header and footer elements. Not only was this skewing search results by giving recurring terms unfair weighting it was also causing the ungraceful error by virtue of the sheer number of results causing the search process to time out. Examples of search terms causing problems were ‘events’, ‘tags’ and ‘cloud’.

Our approach to solving the problem was to create new cut-down versions of cloud, cloudscape and user pages which were essentially ‘chrome-less’ and only output the required content to be indexed.

After the code changes were put live, a reindex of the live site was required to allow the changes to take effect. As the search index has been built up cumulatively over time as content is added to the site a reindex of the live site had never been performed. We did a test run of a full reindex on our approval server to see if the site would remain functional. After this ran successfully we ran a full index on the live site which ran for some hours but hopefully remained transparent to end users. The time out message now seems to have disappeared and meaningful results are now being returned for the previously problematic search terms. The code change should also eliminate similar problems happening with fresh installs of CloudEngine from v1.1.2.

As part of these changes, we also took the opportunity to add the Firephp debugging tool to the codebase. This relies on Firefox with the Firebug and Firephp add-ons as well as the server-side code. A setting has been added to the database that allows debug messages to be switched off, switched on for admin users or switched on for all authenticated users. This allows some debug or informational messages to be output to admin users on a live site via the Firebug console if need be but remain transparent to other users.

Posted in Uncategorized | Tagged , , | Leave a comment

Direct messaging & CloudEngine 1.1.0 beta

I’m happy to say that last Wednesday (2nd February) my colleague Richard Lovelock put the new direct messaging function live on Cloudworks. It is something that Cloudworks users have requested and we have wanted for a while. Despite a quiet launch, it has already been taken up by the Cloudworks community. And, we’re excited about its potential for fostering private discussions that can lead on to public Clouds and Cloudscapes.

Richard researched what Facebook, Linked in, Twitter and a variety of other spaces provide. He designed a system that allows private messaging between multiple participants. I’m sure you will find it quite easy and at the same time powerful to use. We are very pleased with the results – thank you Richard!

After you log in to Cloudworks, you will see a link labelled Messages in the main navigation menu. The message main-page or inbox page summarises your discussions and who was the last to reply to each thread.


The compose message page provides a neat way to search for recipients by their full name, and a list of suggestions appears as you type.


The direct messaging code is part of the open-source CloudEngine software that powers Cloudworks. You or a friendly developer can now download CloudEngine 1.1 Beta, and find out how to create your own social network. Full documentation for direct messaging is on the Wiki.

There are numerous enhancements and fixes in CloudEngine 1.1 Beta, including a handy new maintenance mode to ease system administration. And, we have implemented the new HTML5 form attributes for validation, initially on the registration form. This is most visible on Firefox 4 beta, and Opera 9.6 onwards. We will be improving other forms and integrating HTML5 form emulation for other browsers as time permits.

I’ll be at Dev8D in London next week and happy to talk more about what we’re doing with CloudEngine and Cloudworks, and how you can get involved 😉

So there you have it. Major new functionality is available on Cloudworks, and we have a new Beta release of CloudEngine. Enjoy!

Posted in Uncategorized | Tagged , , | 1 Comment

CloudEngine v1.0.1

It’s a new year, and we have an updated version of CloudEngine for you, version 1.0.1. Actually, it’s not that new, as it was released at the end of November, but things crop up.

Version 1.0 of CloudEngine was based on the initial code pushed to Bitbucket by my colleague, Juliette Culver. So the new release, while a so-called minor release, contains significant stuff.

Highlights include edits and fixes to the language template and Greek language pack, and some small accessibility and usability fixes, including for the installer. There are improvements to how email is configured and fixes to the model/database code and error handling. Full details are on the CloudEngine Wiki and issue tracker.

Go forth and enjoy!

Next, our work in progress and how to contribute.

Posted in Uncategorized | Leave a comment

Why we chose Mercurial and Bitbucket

Since the start of October we have been using Bitbucket to host the CloudEngine project. Almost as soon as we made the decision to use Bitbucket, people have asked why? So, I thought I’d reproduce an email I wrote to colleagues then. At the end I’ve noted how I feel about the decision two months on…

During September, I was tasked with deciding on where to host the free/open-source CloudEngine. By host, we mean essentially:

  • A repository of the source code (which basically keeps track of the ‘version’ of each file),
  • A way for developers to browse the source code repository online,
  • A bug and issue tracker,
  • A Wiki for documentation.

The provisional decision is to use Bitbucket, which uses the Mercurial distributed version control (repository) system (DCMS). I’ve summarised the reasons below.

The choice has been between:

Hosting the project at the OU was discounted early, as we wish to encourage the use, development and contributions to CloudEngine from the wider community. The three remaining contenders use two different version control systems, and which of the two to use – Git or Mercurial – has been a big part of the choice. Note that both Git and Mercurial are free/open-source and cross-platform software tools, with command-line and GUI alternatives. And the three hosting contenders are all free for open-source use.

To help in the decision I canvassed opinions from the Moodle and OU communities:

Some reasons to use Git (and Github) are:

  • Git is used for a lot of high profile and large projects including the Linux kernel,
  • Some projects that are interesting for the OU, Moodle and Drupal, have plans to migrate to Git (and the OU-VLE team in LTS plan to move to Git).
  • Git has the reputation for being flexible, powerful and fast.
  • Github has the most projects of the three platforms.

Some reasons to use Mercurial (and Bitbucket or Google Code) are:

  • Mercurial seems to have been designed with better cross-platform support in mind (Git was initially developed for Linux),
  • Where Git is a toolkit with about 146 sub-commands depending on the operating system (git –help –all), Mercurial is a cohesive ‘product’ with only 50 commands (hg –help). The large number of commands in Git would not be such an issue, if discussions and blog posts hadn’t pointed out that you start to need quite a number of commands for even a simple workflow. That is, Git may be complex for even simpler usage. (Eg. Still hatin’ on Git now with added actual reasons )
  • Mercurial has the reputation of having a shallower learning curve than Git – which relates to the point above.
  • Mercurial is written in one language (Python + C for ‘diff’), while Git is written in several languages
  • CodeIgniter, on which CloudEngine/Cloudworks is built, uses Bitbucket/ Mercurial (Pyrocms, also built on CI uses Github/ Git).
  • Bitbucket has more than 28,000 projects, so it isn’t small!
  • Plenty of respected projects use Mercurial/ Bitbucket/ Google Code, including Mozilla (Firefox), OpenSolaris, OpenOffice, Adium, Python (will migrate), Vim, Growl, Chromium (os Chrome browser)…

Some things to point out – these tend to level the “playing field” between Git and Mercurial:

  • It better to use any reasonably well-designed version control system, than none at all.
  • It is generally considered better to use one of the newer distributed systems like Git or Mercurial, than one of the old ones like CVS (eg. )
  • CloudEngine is fairly small, when compared to Moodle and Drupal, and personall, I’d like to keep the “core” fairly small. So we don’t necessarily need maximum flexibility if there is a cost in complexity.
  • And I was concerned with getting the initial core developers – Richard, Juliette and myself up to speed quickly.
  • In the new distributed paradigm, there will be nothing to stop someone cloning our repository from Git to Mercurial or vice versa. And we can then pick up improvements, for example as a set of patches.
  • There will be nothing to stop anyone submitting patches, whether we use Git, Mercurial … (RCS!)
  • If sufficient numbers of CloudEngine developers use Git and we use Mercurial, we may at some future point provide a mirror of CloudEngine in Git. (And we may even move to Git – nothing is set in stone.)
  • And finally, I accept that as developers, we may well have to learn both Git and Mercurial to work on various projects.

It has been a hard choice, but on balance I propose using Mercurial. And I propose Bitbucket over Google Code as:

  • While Google Code is used for some large Google projects including Chromium, it is often used for the smallest projects – for example Javascript/jQuery plugins.
  • Google Code has the feel of a repository, while Bitbucket will hopefully be a community.

My links,

Two months on…

I have no complaints! On the Mercurial front, I find the command line tool easy to use, and there are excellent tutorials, references and books (online + print). There are some neat features:

  • hg serve – type this simple command to start up a basic web server, with configurable pull and push options.
  • Mercurial makes it easy to configure hooks to run on certain commands, for example commit and update. Git also has hooks, and I can’t comment on the comparable ease-of-use.
  • Syntax highlighting, progress… and other bundled and 3rd party extensions
  • Patch queues – I haven’t looked at these yet! (Mercuarial Queues Extension.)

Note, we have Mercurial on our Linux servers, courtesy of EPEL.

I’m fairly impressed with Bitbucket too. The wiki and issue-tracker are easy to use, with some useful macros, for example, <<issues query [format]>>. We have used the latter example on the Releases page in our wiki. It’s neat having the Wiki pages in a Mecurial repository. And the Mercurial support team are fairly responsive to queries and isuses.

If I have one slight criticism, it is, why have the wiki and issue tracker per repository? I expect CloudEngine to need at least one more repository for extensions and contributed plugins – as we won’t want a separate Wiki for that repository, we’ll need some fiddly links. And the links are generally longer. Anyway, these are relatively small points, and may be partially answered by considerations like the macro feature mentioned above.

So there you have it – no complaints, and use whichever platform suits your needs.

Watch this space for more on recent and forthcoming CloudEngine releases.

Posted in Uncategorized | Tagged , , , , | 7 Comments

CloudEngine 1.0

I thought it was time to mark an official milestone – on Friday we packaged up CloudEngine 1.0 and uploaded it to Bitbucket. This is based on the initial upload by my colleague Juliette Culver.

CloudEngine installation - home page. View on Flickr.

This is a good time to mark the end of “phase 1” of the free/ open-sourcing project. Over the past 6 months or so Juliette has made the code ready for open-sourcing (including removing my “@todo” comments;-)); worked tenaciously with OSS Watch, the Open University’s legal department and our funders on licensing and trademarks; worked with Stephen Turvey on our beautiful Aurora theme, and done the hundred and one other things that a ‘launch’ requires. Thank you Juliette!

Now fellow developer, Richard Lovelock and I are getting to grips with Mercurial and Bitbucket, which I think we both quite like. And I’ve been busy adding to the Wiki and setting up Mercurial on our servers for Cloudworks.

Looking forward, Richard is busy with direct/ private messaging, which will be our next new feature. And I’m working on bug-fixes, collaborating to bring more oEmbed providers to Cloudworks and CloudEngine, a proposal for ‘universal’ popups (up-popup),  the roadmap and getting started sections in the Wiki.

We’d love your comments and ideas on all of the above. Watch this space!

Posted in Uncategorized | Leave a comment

CloudEngine Accessibility

We want CloudEngine and Cloudworks to be as accessible as possible to people with disabilities. Earlier this year, we had some help from Chetz Colwell, an accessibility expert here in the Institute of Educational Technology at The Open University in reviewing the software from an accessibility perspective.

On the whole I would say that we are more clued up about accessibility than average, and apart from a few ‘bugs’ that slipped through the net, we avoided the most common accessibility pitfalls that you come across.  Nonetheless, we still learned some new things, proving how useful it is to have your software reviewed in this manner, and I thought it might be useful to share them. We are still working on fixing some of these (and now that the code is open-source, you are very welcome to help!) and have time marked out in next year’s roadmap for accessibility improvements.

Ten things that we learned from the accessibility review

1) Check any third-party code carefully for accessibility issues

One important lesson that we learned from the review is that we need to become meticulous about checking the accessibility of any third party code that we decide to use. The chief culprits here were the text editor and date picker, but this also extends more broadly to any front-end code written by other people.

This is tricky though because sometimes choice is very limited particularly when you also have non-accessibility requirements that you want to fulfill. It is also often difficult to get good information without doing the testing yourself. However, it is still something that we need to be much more alert to and if we do find ourselves in the position of having to use code that is less accessible code than ideal, we need to at the minimum e.g. let users switch off the text editor, or give them sufficient information to add dates without using a date picker.

2) Test with high contrast settings

We had used a Colour Contrast analyser to test our pages, but we didn’t realise that we should also test with high contrast display settings. Lots of parts of the pages such as the borders of tabs and buttons became invisible or nearly invisible with these settings on.

3) Make sure you logically order the elements of pages in HTML

Some examples:

  • The Cloud page is organised into two visual columns. The first column of the page contains the title and contents of the Cloud. The name of the person who created the cloud is in the second column, so this comes after the entire contents of the cloud which might be quite long. It would make more sense for the name of the creator of the cloud to come after the title of the cloud.
  • In the comments on clouds, we put the name of the person who put the comment after the comment, whereas it would be more helpful to somebody using a screenreader if it came before.
  • On some of our form fields, as well as a field label we had some extra help text e.g. giving examples of the type of content people might want to include in the field. It is more useful if this comes before the form field rather than afterwards where we had put it.

4) Use the <hr /> tag for horizontal lines that divide up sections

I had learned to shy away from using the <hr /> tag for horizontal lines because I associate it so much with being used when CSS should be used instead. But the <hr /> tag when used semantically actually makes it easier for screenreader users to know when one section ends and another starts.

4) For HTML page titles, put the title of the page first and name of the site second

It’s easier for screenreader users if the title of a page in the HTML is e.g. ‘Create Cloud – Cloudworks’ rather than ‘Cloudworks – Create Cloud’. This way screenreader users hear the name of the actual page upfront. Although I have to say that even the BBC Website does it the other way round!

5) Make it as easy as possible for people to make user generated content accessible

User generated content always poses interesting accessibility challenges. However, we could have done much more to make it as easy as possible for people to make their content accessible. This is partly in terms of raising awareness of accessibility and providing information for people on how to make their content accessible, but there is also functionality that we could provide. For example

  • When people add embeds, we should give them option of providing an accessible alternative
  • When people add URLs, we should insist they also add a title
  • When people add images, we should make it easy for them to add alt text.
  • We need to make sure that the text editor makes it as easy as possible for people to make their content accessible

6) When using tabs, make sure the keyboard focus remains on the tab when clicked.

Tabs, such as the alphabetical tabs on the Clouds and Cloudscapes pages or the tabs for different items types on the Cloudstream pages, are difficult for screenreader users, but it is still easier if the focus remains on the tab rather than goes back to the top of the page.

7) Avoid captchas altogether on registration forms

We knew that captchas were an issue and we have always offered to register people manually. However, instead of just doing this or looking into e.g. audio captchas, we really ought to at least try turning them off altogether and trying other anti-spam measures. We haven’t tried removing the captchas on our registration form yet, but have added functionality to easily turn captchas on and off, so we might try it at some point to see to what extent captchas do actually reduce spam. Most of the spam we get is currently human-generated rather than robot-generated, but that may be because we have captchas!

8) Make sure there is not too much whitespace between a form label and form field

This makes the form difficult to navigate for users of screen magnifiers. This can easily happen if you have your form as two visual columns with the form labels in the first column and form fields in the second column (as is the case on our current registration form).  It is generally better to put the form label above the field rather than to the left of it for this reason.

9) Put form validation messages above the page heading

If there is a form validation error message, it is much easier for screen reader users if the message is above the main page heading, rather than just above the form where you might more obviously put it.

10) Be careful not to use whitespace as ‘punctation’

It can be very tempting to use whitespace to mark gaps and to forget that a screen reader will not realise this. For example, in one place we had lists of clouds with the cloud title of each cloud followed by some whitespace and then the number of comments, making it difficult to distinguish between the title and number of comments. Putting a dash, full stop or comma, instead makes it easier for screenreader users.

Posted in Uncategorized | Leave a comment