As increasing amount of content has been added to Cloudworks, certain search parameters are now returning many results, for example, searching for ‘learning’ on the live site returns over 2000 clouds, 300 cloudscapes and 600 users. All of these results were being presented on one page over two columns. Not only was this creating very large pages for users to scroll through, the results for user pages were appearing in the second column somewhere down the page following the cloudscape results.

Original search results interface

This was clearly becoming unwieldy so we have made some changes to improve the search interface and usability. Here is a summary of the changes that have been made:

• the results for clouds, cloudscapes and users have been split in to three separate tabs. This allows results with long titles to span the width of the page rather then split over several lines with the two column layout.
• the total number of results is shown above the results area.
• the number of results for clouds, cloudscapes and users is shown at the top of each tab.
• each result has a percentage score which is the relevance of the result to the top result (which is usually scored 100%).
• On each tab, results are paginated, with 15 results per page (customisable).
• 200 results are retrieved with the search and are actually hidden from view but can be quickly displayed by using a link under the result set on each tab.
• results are now presented in an ordered list, ordered by relevance in each tab, so the user can more clearly see the position of a search result within a set of results.
• there is a link on each tab to show the full set of results for a clouds, cloudscapes and users (similar to the original search).
• for a screen resolution of 1280 x 1024, search results usually fit on one page without needing to scroll.

Here is a screenshot of a search for ‘learning’ with the new interface:

New seach results user interface

We hope these changes make the search functionality on CloudEngine more usable and user-friendly. We hope to put these changes live on Cloudworks early during the week commencing 30th May 2011.

Our approach to solving the problem was to create new cut-down versions of cloud, cloudscape and user pages which were essentially ‘chrome-less’ and only output the required content to be indexed.

After the code changes were put live, a reindex of the live site was required to allow the changes to take effect. As the search index has been built up cumulatively over time as content is added to the site a reindex of the live site had never been performed. We did a test run of a full reindex on our approval server to see if the site would remain functional. After this ran successfully we ran a full index on the live site which ran for some hours but hopefully remained transparent to end users. The time out message now seems to have disappeared and meaningful results are now being returned for the previously problematic search terms. The code change should also eliminate similar problems happening with fresh installs of CloudEngine from v1.1.2.

As part of these changes, we also took the opportunity to add the Firephp debugging tool to the codebase. This relies on Firefox with the Firebug and Firephp add-ons as well as the server-side code. A setting has been added to the database that allows debug messages to be switched off, switched on for admin users or switched on for all authenticated users. This allows some debug or informational messages to be output to admin users on a live site via the Firebug console if need be but remain transparent to other users.

Richard researched what Facebook, Linked in, Twitter and a variety of other spaces provide. He designed a system that allows private messaging between multiple participants. I’m sure you will find it quite easy and at the same time powerful to use. We are very pleased with the results – thank you Richard!

After you log in to Cloudworks, you will see a link labelled Messages in the main navigation menu. The message main-page or inbox page summarises your discussions and who was the last to reply to each thread.

The compose message page provides a neat way to search for recipients by their full name, and a list of suggestions appears as you type.

The direct messaging code is part of the open-source CloudEngine software that powers Cloudworks. You or a friendly developer can now download CloudEngine 1.1 Beta, and find out how to create your own social network. Full documentation for direct messaging is on the Wiki.

There are numerous enhancements and fixes in CloudEngine 1.1 Beta, including a handy new maintenance mode to ease system administration. And, we have implemented the new HTML5 form attributes for validation, initially on the registration form. This is most visible on Firefox 4 beta, and Opera 9.6 onwards. We will be improving other forms and integrating HTML5 form emulation for other browsers as time permits.

I’ll be at Dev8D in London next week and happy to talk more about what we’re doing with CloudEngine and Cloudworks, and how you can get involved

So there you have it. Major new functionality is available on Cloudworks, and we have a new Beta release of CloudEngine. Enjoy!

Version 1.0 of CloudEngine was based on the initial code pushed to Bitbucket by my colleague, Juliette Culver. So the new release, while a so-called minor release, contains significant stuff.

Highlights include edits and fixes to the language template and Greek language pack, and some small accessibility and usability fixes, including for the installer. There are improvements to how email is configured and fixes to the model/database code and error handling. Full details are on the CloudEngine Wiki and issue tracker.

Go forth and enjoy!

Next, our work in progress and how to contribute.

During September, I was tasked with deciding on where to host the free/open-source CloudEngine. By host, we mean essentially:

• A repository of the source code (which basically keeps track of the ‘version’ of each file),
• A way for developers to browse the source code repository online,
• A bug and issue tracker,
• A Wiki for documentation.

The provisional decision is to use Bitbucket, which uses the Mercurial distributed version control (repository) system (DCMS). I’ve summarised the reasons below.

The choice has been between:

• The Open University
• Github, http://github.com/
• Bitbucket, http://bitbucket.org/
• Google Code, http://code.google.com/hosting/

Hosting the project at the OU was discounted early, as we wish to encourage the use, development and contributions to CloudEngine from the wider community. The three remaining contenders use two different version control systems, and which of the two to use – Git or Mercurial – has been a big part of the choice. Note that both Git and Mercurial are free/open-source and cross-platform software tools, with command-line and GUI alternatives. And the three hosting contenders are all free for open-source use.

To help in the decision I canvassed opinions from the Moodle and OU communities:

• Search down the Moodle forum discussion, “Why Git?” ,
• The OU web developers group thread, Views wanted on Git and Mercurial.

Some reasons to use Git (and Github) are:

• Git is used for a lot of high profile and large projects including the Linux kernel,
• Some projects that are interesting for the OU, Moodle and Drupal, have plans to migrate to Git (and the OU-VLE team in LTS plan to move to Git).
• Git has the reputation for being flexible, powerful and fast.
• Github has the most projects of the three platforms.

Some reasons to use Mercurial (and Bitbucket or Google Code) are:

• Mercurial seems to have been designed with better cross-platform support in mind (Git was initially developed for Linux),
• Where Git is a toolkit with about 146 sub-commands depending on the operating system (git –help –all), Mercurial is a cohesive ‘product’ with only 50 commands (hg –help). The large number of commands in Git would not be such an issue, if discussions and blog posts hadn’t pointed out that you start to need quite a number of commands for even a simple workflow. That is, Git may be complex for even simpler usage. (Eg. reprog.wordpress.com/ Still hatin’ on Git now with added actual reasons )
• Mercurial has the reputation of having a shallower learning curve than Git – which relates to the point above.
• Mercurial is written in one language (Python + C for ‘diff’), while Git is written in several languages
• CodeIgniter, on which CloudEngine/Cloudworks is built, uses Bitbucket/ Mercurial (Pyrocms, also built on CI uses Github/ Git).
• Bitbucket has more than 28,000 projects, so it isn’t small!
• Plenty of respected projects use Mercurial/ Bitbucket/ Google Code, including Mozilla (Firefox), OpenSolaris, OpenOffice, Adium, Python (will migrate), Vim, Growl, Chromium (os Chrome browser)…

Some things to point out – these tend to level the “playing field” between Git and Mercurial:

• It better to use any reasonably well-designed version control system, than none at all.
• It is generally considered better to use one of the newer distributed systems like Git or Mercurial, than one of the old ones like CVS (eg. http://importantshock.wordpress.com/2008/08/07/git-vs-mercurial/ )
• CloudEngine is fairly small, when compared to Moodle and Drupal, and personall, I’d like to keep the “core” fairly small. So we don’t necessarily need maximum flexibility if there is a cost in complexity.
• And I was concerned with getting the initial core developers – Richard, Juliette and myself up to speed quickly.
• In the new distributed paradigm, there will be nothing to stop someone cloning our repository from Git to Mercurial or vice versa. And we can then pick up improvements, for example as a set of patches.
  • For example, there is a Mercurial mirror of the Linux kernel (Git to Mercurial), http://kernel.org/hg/linux-2.6/
  • And a Git mirror of Moodle (CVS to Git), http://git.moodle.org/gw?p=moodle.git;a=summary
• There will be nothing to stop anyone submitting patches, whether we use Git, Mercurial … (RCS!)
• If sufficient numbers of CloudEngine developers use Git and we use Mercurial, we may at some future point provide a mirror of CloudEngine in Git. (And we may even move to Git – nothing is set in stone.)
• And finally, I accept that as developers, we may well have to learn both Git and Mercurial to work on various projects.

It has been a hard choice, but on balance I propose using Mercurial. And I propose Bitbucket over Google Code as:

• While Google Code is used for some large Google projects including Chromium, it is often used for the smallest projects – for example Javascript/jQuery plugins.
• Google Code has the feel of a repository, while Bitbucket will hopefully be a community.

My links,

• delicious.com / nfreear / versioncontrol

Two months on…

I have no complaints! On the Mercurial front, I find the command line tool easy to use, and there are excellent tutorials, references and books (online + print). There are some neat features:

• hg serve – type this simple command to start up a basic web server, with configurable pull and push options.
• Mercurial makes it easy to configure hooks to run on certain commands, for example commit and update. Git also has hooks, and I can’t comment on the comparable ease-of-use.
• Syntax highlighting, progress… and other bundled and 3rd party extensions
• Patch queues – I haven’t looked at these yet! (Mercuarial Queues Extension.)

Note, we have Mercurial on our Linux servers, courtesy of EPEL.

I’m fairly impressed with Bitbucket too. The wiki and issue-tracker are easy to use, with some useful macros, for example, <<issues query [format]>>. We have used the latter example on the Releases page in our wiki. It’s neat having the Wiki pages in a Mecurial repository. And the Mercurial support team are fairly responsive to queries and isuses.

If I have one slight criticism, it is, why have the wiki and issue tracker per repository? I expect CloudEngine to need at least one more repository for extensions and contributed plugins – as we won’t want a separate Wiki for that repository, we’ll need some fiddly links. And the links are generally longer. Anyway, these are relatively small points, and may be partially answered by considerations like the macro feature mentioned above.

So there you have it – no complaints, and use whichever platform suits your needs.

Watch this space for more on recent and forthcoming CloudEngine releases.

This is a good time to mark the end of “phase 1″ of the free/ open-sourcing project. Over the past 6 months or so Juliette has made the code ready for open-sourcing (including removing my “@todo” comments;-)); worked tenaciously with OSS Watch, the Open University’s legal department and our funders on licensing and trademarks; worked with Stephen Turvey on our beautiful Aurora theme, and done the hundred and one other things that a ‘launch’ requires. Thank you Juliette!

Now fellow developer, Richard Lovelock and I are getting to grips with Mercurial and Bitbucket, which I think we both quite like. And I’ve been busy adding to the Wiki and setting up Mercurial on our servers for Cloudworks.

Looking forward, Richard is busy with direct/ private messaging, which will be our next new feature. And I’m working on bug-fixes, collaborating to bring more oEmbed providers to Cloudworks and CloudEngine, a proposal for ‘universal’ popups (up-popup),  the roadmap and getting started sections in the Wiki.

We’d love your comments and ideas on all of the above. Watch this space!

On the whole I would say that we are more clued up about accessibility than average, and apart from a few ‘bugs’ that slipped through the net, we avoided the most common accessibility pitfalls that you come across.  Nonetheless, we still learned some new things, proving how useful it is to have your software reviewed in this manner, and I thought it might be useful to share them. We are still working on fixing some of these (and now that the code is open-source, you are very welcome to help!) and have time marked out in next year’s roadmap for accessibility improvements.

Ten things that we learned from the accessibility review

1) Check any third-party code carefully for accessibility issues

One important lesson that we learned from the review is that we need to become meticulous about checking the accessibility of any third party code that we decide to use. The chief culprits here were the text editor and date picker, but this also extends more broadly to any front-end code written by other people.

This is tricky though because sometimes choice is very limited particularly when you also have non-accessibility requirements that you want to fulfill. It is also often difficult to get good information without doing the testing yourself. However, it is still something that we need to be much more alert to and if we do find ourselves in the position of having to use code that is less accessible code than ideal, we need to at the minimum e.g. let users switch off the text editor, or give them sufficient information to add dates without using a date picker.

2) Test with high contrast settings

We had used a Colour Contrast analyser to test our pages, but we didn’t realise that we should also test with high contrast display settings. Lots of parts of the pages such as the borders of tabs and buttons became invisible or nearly invisible with these settings on.

3) Make sure you logically order the elements of pages in HTML

Some examples:

• The Cloud page is organised into two visual columns. The first column of the page contains the title and contents of the Cloud. The name of the person who created the cloud is in the second column, so this comes after the entire contents of the cloud which might be quite long. It would make more sense for the name of the creator of the cloud to come after the title of the cloud.
• In the comments on clouds, we put the name of the person who put the comment after the comment, whereas it would be more helpful to somebody using a screenreader if it came before.
• On some of our form fields, as well as a field label we had some extra help text e.g. giving examples of the type of content people might want to include in the field. It is more useful if this comes before the form field rather than afterwards where we had put it.

4) Use the <hr /> tag for horizontal lines that divide up sections

I had learned to shy away from using the <hr /> tag for horizontal lines because I associate it so much with being used when CSS should be used instead. But the <hr /> tag when used semantically actually makes it easier for screenreader users to know when one section ends and another starts.

4) For HTML page titles, put the title of the page first and name of the site second

It’s easier for screenreader users if the title of a page in the HTML is e.g. ‘Create Cloud – Cloudworks’ rather than ‘Cloudworks – Create Cloud’. This way screenreader users hear the name of the actual page upfront. Although I have to say that even the BBC Website does it the other way round!

5) Make it as easy as possible for people to make user generated content accessible

User generated content always poses interesting accessibility challenges. However, we could have done much more to make it as easy as possible for people to make their content accessible. This is partly in terms of raising awareness of accessibility and providing information for people on how to make their content accessible, but there is also functionality that we could provide. For example

• When people add embeds, we should give them option of providing an accessible alternative
• When people add URLs, we should insist they also add a title
• When people add images, we should make it easy for them to add alt text.
• We need to make sure that the text editor makes it as easy as possible for people to make their content accessible

6) When using tabs, make sure the keyboard focus remains on the tab when clicked.

Tabs, such as the alphabetical tabs on the Clouds and Cloudscapes pages or the tabs for different items types on the Cloudstream pages, are difficult for screenreader users, but it is still easier if the focus remains on the tab rather than goes back to the top of the page.

7) Avoid captchas altogether on registration forms

We knew that captchas were an issue and we have always offered to register people manually. However, instead of just doing this or looking into e.g. audio captchas, we really ought to at least try turning them off altogether and trying other anti-spam measures. We haven’t tried removing the captchas on our registration form yet, but have added functionality to easily turn captchas on and off, so we might try it at some point to see to what extent captchas do actually reduce spam. Most of the spam we get is currently human-generated rather than robot-generated, but that may be because we have captchas!

8) Make sure there is not too much whitespace between a form label and form field

This makes the form difficult to navigate for users of screen magnifiers. This can easily happen if you have your form as two visual columns with the form labels in the first column and form fields in the second column (as is the case on our current registration form).  It is generally better to put the form label above the field rather than to the left of it for this reason.

9) Put form validation messages above the page heading

If there is a form validation error message, it is much easier for screen reader users if the message is above the main page heading, rather than just above the form where you might more obviously put it.

10) Be careful not to use whitespace as ‘punctation’

It can be very tempting to use whitespace to mark gaps and to forget that a screen reader will not realise this. For example, in one place we had lists of clouds with the cloud title of each cloud followed by some whitespace and then the number of comments, making it difficult to distinguish between the title and number of comments. Putting a dash, full stop or comma, instead makes it easier for screenreader users.

Any sort of rewrite is a risky decision to make, but in this case it is one that that I am really pleased with. It took about a month for me to migrate the code over, although that time also included implementing a new design for the design for the site.  We lost a tiny bit of functionality in the move – you get lots ‘for free’ with Drupal – but nothing significant.

Before I start, I want to say that I am still a fan of Drupal for certain types of projects. It also has a great community that seems to be doing all the right things in terms of the directions that Drupal is moving forwards. In fact, I have chosen to use Drupal for other projects since. So please don’t take this as an all-out attack on Drupal, rather as some reasons why it wasn’t right for us in this context.

There are three main reasons why we moved and why I am glad that we made the decision to switch to CodeIgniter:

1) Productivity Curve

With Drupal, you can get the majority of a site built very quickly, but those last details can be agonising. Drupal has a lot of built-in functionality that you can effectively just switch on. It is also straightforward to write simple modules of your own. However the flip-side to this is that if Drupal doesn’t do exactly what you want, for example if you want a slightly different user interface or some subtle change in functionality, you often find yourself back at close to square one.  If you have a site that you just want to be ‘good enough’ and to get up and running quickly, none of this is a problem, but if those details do matter to you, the time you saved at the start is quickly eclipsed by this time fine-tuning at the end.

Using a framework such as CodeIgniter, you do have to write everything from scratch and it is a little bit more work to get things set up at the start.  However, having a framework does speed this up a great deal and writing new functionality with CodeIgniter is, if anything, quicker than writing an equivalent Drupal module. There are also lots of libraries out there that you can use with CodeIgniter, so you often aren’t starting from a completely blank slate. On the other hand, when you write some code in CodeIgniter, you can write it so that it does precisely what you want it to from the start.

Some of this difference comes down to learning curve of course. I’m certainly not a Drupal novice. I’ve written a fair few modules for Drupal and know a certain amount about how to override core behaviour in Drupal for example, but I definitely wouldn’t class myself as an expert either. I readily admit that for somebody with more knowledge than me, the difference in productivity between Drupal and CodeIgniter would be less extreme. Indeed I’d be genuinely curious to know how quickly a Drupal expert could replicate the Cloudworks site. However, any time taken learning more about Drupal is time that would have needed to come out of the time allocated to development of the site. In contrast, you could read all the CodeIgniter documentation in a day and I haven’t yet known a PHP developer who hasn’t been able to pretty much hit the ground running with CodeIgniter.

Aside from learning curve, I suspect that there are some things that are just fundamentally difficult to do in Drupal. I am in two minds about whether to go into specifics here as I don’t want to get dragged down into microissues. If you want a taster, you can see a few examples of my questions on the Drupal forums http://drupal.org/tracker/371512 before I gave up asking there! I know that that some of the issues I had have been fixed in later versions of Drupal, but I also have memories of having problems with dealing with images and configuring the registration form and user profiles quite how I wanted them and I doubt all of these have been resolved.

2) A different way of working

Working with Drupal felt a bit like building a boat by starting with a car and gradually modifying the car until it became a boat, whereas working with CodeIgniter felt like building a boat from scratch. This changes your mindset significantly when you are working on a site. Now if what you are building is very close to the ‘Drupal model’ then this isn’t too tough, but if it’s not then the mental process of figuring out how to twist Drupal to meet your requirements is much more of an uphill struggle.

Theming is a good case in point when it comes to this different way of working in Drupal versus CodeIgniter. If a designer produces some HTML and CSS for you, you can drop this straight into a site using CodeIgniter but with Drupal, you can have a major piece of work on your hands making it into a theme. Working with a designer on a Drupal version of the site was much trickier as I had to try and explain what would make a design feasible to be turned into a Drupal theme. More generally, I found it trickier discussing work on the site with other people on the project when we were using Drupal compared with when we started using CodeIgniter.

There is also has a more subtle effect. Instead of starting out thinking ‘this is what I would really like the site to be like’ you find yourself naturally looking at what modules are out there and whether they might be useful. This is quite a different way to look at building a site and means that you are less open to ideas for which there aren’t easy Drupal solutions.

3) Site structure in the database

The way Drupal works is that in order to be configurable via a nice user interface, lots of the important information about the site sits in the database.  So as well as containing the ‘site data’ such as details of all the users, the text of content and so on, it also contains the ‘site structure data’ for example specifying the types of object (nodes in Drupal-speak) for your site and the ‘views’ of the data that you are using on specific pages. It’s exactly this that makes Drupal so easy and flexible to configure for non-developers.  On a site using a framework like CodeIgniter, such information would be more ‘hardcoded’ into the structure of the database or the actual code for the specific site in question.

This has a few knock-on effects. First, pushing quite simple changes in functionality to a live Drupal site usually isn’t as simple as just updating the code. This was a little while back so there might be a nice solution to this nowadays (let me know if there is!), but in practice, to update the live Cloudworks site when I was using Drupal, I normally ended up having to write lists of all the actions that I had performed via the Drupal admin interface and repeat these on the live site and hope that I hadn’t made any mistakes and then pray that I didn’t need to rollback for some reason. Secondly, it would have made life much more complicated for making our software open-source in terms of support. It’s easy to be reasonably confident that somebody is using an unmodified version of your code, but it’s harder to be certain that they haven’t changed something vital in the Drupal admin interface. Finally, the Drupal way of having the site structure information in the database means that database queries can easily get thorny. This isn’t a problem too often, but queries that are now only moderately complex using CodeIgniter were often absolutely horrendous in Drupal.

Finally

I think the cumulative effect of all these things was that I became a much happier developer when we moved to CodeIgniter.  I somehow felt in control of my code again and I spent less time tearing my hair out trying to find some Drupal workaround! As a result, I also felt I had more energy and enthusiasm to put in the site. Overall, although the move could be regarded as slightly controversial and I was nervous about it, I have no doubt that it was the right one for us in our situation.

This is going to the place where we announce news about CloudEngine and post about why we have made certain decisions and lessons that we have learned. To kick things off, we have copied over some of the blog posts from the Cloudworks blog of a more technical nature as a start.

I’ll being going on maternity leave in a week’s time, although I hope to make a couple of more posts here before then. After that my colleague Nick Freear who has worked on several major CloudEngine features will be taking over leading the development.

My top priority before I go on maternity leave is to make the Cloudworks source-code open-source and I thought it would be worth sharing the main things that in our plan for this.

Name and branding

It is going to be confusing if the open-source version of the code is also called Cloudworks and if new installs look identical to the Cloudworks site, so we need to come up with a new name and branding. Names are hard because as well as picking a good name, you need to get them trademark-checked, you don’t want to clash with anything else with the same name that might cause confusion and you also want to be able to get a reasonably sensible domain name. We have some ideas but suggestions are very welcome! We’ve already got our graphic designer to produce a new colour scheme and site banner so we’re part of the way with the branding side of things.

Licensing

There are two main things to consider here. First, we have had to check with legal at The Open University whether we can release the code as open-source and if there are any restrictions on licences we can use, as well as check with the funding bodies, JISC and the EU, that have funded parts of the work. We’ve already done this and luckily it seems to be fairly straightforward and we can pick pretty much any licence we would like from that perspective.

Secondly, the code uses various other open-source code, for example the CodeIgniter framework, JQuery, TinyMCE and Zend Lucene. These are all covered by a variety of different licences and we need to make sure that we’re not breaking the terms of any of them when we release our code. I’m hoping that OSSWatch are going to come to the rescue here in helping me make sense of all this!

Governance Model

The OSSWatch website has lots of useful information about governance models. There are some interesting decisions for us here, especially with how the open-source work fits in with existing structures and with me going on maternity leave in October!

Installation and upgrade infrastructure

We need to write and test installation instructions for the code, as well as make sure that an ‘empty’ install of the code behaves reasonably sensibly. We also need to think ahead as to how we are going to manage upgrades and document which versions of PHP/MySQL we have tested with.

Configurability

We are going through the codebase trying to spot anywhere that things have been hardcoded in which wouldn’t make sense in the open-source version. One of the main places is the support and about pages which are currently hardcoded HTML. We also need to allow people to customise the theme and logo.

We have decided however on the whole to provide the code very much ‘as is’ in the first instance and work on improving it later. The admin interface for example is rather primitive, but we’re working on the principle that it is better to get the code out first and make those improvements afterwards.

Hosting, website and documentation

We need to decide where to host the code – at the moment, it is looking like a choice between Github and Bitbucket. We also need to put infrastructure in place for tracking bugs (and decide whether to try and import the bugs we currently have in our local bugtracker) as well as think about things like a developer mailing list, wiki and website as well as the type of documentation and guidance we want to have on them. We also need to decide how we manage reports of security vulnerabilities.

Get involved

If you’re interested in being an early guinea pig for the open-source version or getting involved in development, please do contact us on cloudworks@open.ac.uk