1

Re: Technology choices for online recording

Dear all,

As you may have read, we now have the requirements for a toolkit to help develop online recording websites and the build is set to take place over the next few months. The proposed approach is to build the "Core" of the system using PHP and MySQL since it is so ubiquitous, but it should be possible to build the web pages themselves using any web technology from PHP to Java or .NET, or even using a content management system. However, this decision would be based on an assumption that PHP and MySQL skills are more widely available within the recording community than the other options. If this is not the case then there are other alternatives which each have their own advantages.

Here's a quick summary of the things we have considered:
1) PHP - the strength of this option lies in the fact that its usage is so widespread so there are many developers available, and also hosting options are extremely cheap. All the other options would incur a greater hosting fee. PHP is perhaps not the best language for produce nicely structured code though.
2) Ruby on Rails - the strength of this option is that it is a very productive development environment. Generally things are developed more quickly in Ruby on Rails than other options. It's also got good support for multiple databases, so we would not be tied to just MySQL for example. Note that there are "frameworks" around for PHP which make the productivity closer to Ruby on Rails though.
3) Java - more powerful than PHP, but hosting is more expensive.
4) .NET - although not Open Source, the .NET development tools and languages are very powerful. Don't forget that there are very good free "Express" versions of the development tools.

For the database, the main options are MySQL and PostgreSQL or PostGIS. The advantages of the PostgreSQL approach are much better support for spatial databases, though again hosting options are not so freely available.

So, before ploughing ahead with the development, I'd like to give anyone with an opinion on the development environment we use a chance to chip in. Whether you have a strong preference for one of the tools, are aware of particular problems with a tool, know of a tool that we ought to consider that I haven't mentioned, or just have access to developers of one of these tools, please let us know. Don't forget that what we are talking about at this stage is the "back end" code - not the web pages that the end users interact with which can be developed on any development environment.

Cheers

John van Breda
Biodiverse IT

2

Re: Technology choices for online recording

Here's another one for consideration: Python/Django. This has similar pro/cons to Ruby on Rails. Python is slightly more popular than Ruby, but is arguably not quite as productive and doesn't have as large a community as Rails. Django is also supported on Google App Engine, which could be an interesting route to explore.

Ruby on Rails is a great option (and personally favored by myself), but I can see it might not be suitable for this particular project given the goals of ease-of-hosting and developer uptake. One thing to beware of in regards to PHP frameworks is that they can, potentially, consume too many resources for a simple shared host. This then challenges the ubiquity of PHP as you have to make sure your host can actually handle the application. At this stage, I don't think we quite know what kind of resource usage we're going to need from this application, no?

If a PHP framework were chosen, the next question would be, which one? The most popular ones are CakePHP, CodeIgniter, Symphony and Zend.

I'm fine with MySQL, but would like to be able to use PostgreSQL given the chance. One further point to recommend PostgreSQL is that ArcSDE 9.3 is compatible with it, thus tying it in nicely with the whole ArcGIS ecosystem. Making it as db agnostic as possible would seem to be a really good idea.

I'm interested in learning more about the relationship between the "core" and "user" elements of this. Could, for instance, a core instance be setup on a server, then a Rails app be plugged into that?

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

3

Re: Technology choices for online recording

A further consideration: what Javascript library to use? I would favour jQuery as it is popular, easy to use, well documented, and has tonnes of really useful plugins (more here, here, and here).

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

4

Re: Technology choices for online recording

Main advantages and disadvantages as I can see them:

PHP (and framework - prob. CodeIngiter?): + Ubiquity, large pool of existing developers. Hosting widely available and cheap.
- Code is not known for being especially maintainable. Not the nicest language.

Ruby (and Rails): + Built in database agnosticity. Very productive environment, well suited to agile development. Largest growing language in terms of open-source projects.
- Hosting. Need at the least a virtual server. Potentially more difficult to deploy, though there are tools like Capistrano to alleviate this.

Java (and.. Struts? Some other f/w): + Powerful. Easy transition from java gives potential developers.
- Hosting more difficult than PHP. Less likely to attract open source developers than PHP or RoR.

.Net (no idea about frameworks): + Powerful. Good development tools. Hosting relatively widely available.
- Not open source. Limit the type of development/hosting platforms (unless we build against Mono, but that probably undermines the power and makes hosting more difficult).

Personally, I think we're shooting ourselves in the foot if we choose .Net and want to attract open source developers to the project (although as somebody who runs only linux systems, I'll admit to being a little biased here.) If we were going to use a non-PHP option, I think RoR is the best choice (and this is my preferred one), both in terms of the environment and attracting developers. Whether to do that is probably a question of hosting - how many places do we see the core module being deployed, and by whom?

Just to introduce myself, as I'm new on this forum: my name is Nicholas Clarke, I've worked with Charles Copp for a few years on various projects, and will be working with John on this one.

5

Re: Technology choices for online recording

Oh, wrt Python/Django: I'd guess it has mostly the same advantages and disadvantages as Ruby on Rails, but perhaps slightly less of an advantage, so I'd advocate Rails.

6

Re: Technology choices for online recording

Ruby (and Rails): + Built in database agnosticity. Very productive environment, well suited to agile development. Largest growing language in terms of open-source projects.
- Hosting. Need at the least a virtual server. Potentially more difficult to deploy, though there are tools like Capistrano to alleviate this.

Rails has become much more cheap and easy to host in recent month due in no small part to Phusion Passenger (aka mod_rails). You can now get inexpensive, good quality shared hosting for Rails here in the UK for as little as £7 a month from the likes of Media72. This isn't much more than a decent quality PHP host. You can even get a small VPS for £10-£20 a month from the very highly rated SliceHost. Slicehost also have absolutely stellar documentation and an active, friendly community, which makes it very easy to get setup with them. If you want a UK VPS, then Brightbox seem to be good, and are not too expensive. And then there's Heroku, which is really novel and perhaps a sign of things to come (cloud computing).

Other than that I completely agree with everything else you say Nicholas (welcome to the forum, btw) :). I'd be really keen to see this project done in Rails, especially when you consider that developer productivity is an important factor. Given the very finite amount of money I imagine is available for this project, we want to get as much "bang for our buck" as it were, and I believe Rails can deliver that. Plus I believe other developers will find it very easy to pick up Rails and understand the code (because it's so structured and clean), thus encouraging participation.

One further question for consideration: given that the project is open source, where should it be hosted? I would favour either Google Code or Github. The latter of which has seen a huge surge in popularity recently with Rails itself being hosted there along with many other high-profile projects. And/or you could use something like Redmine, which is pretty easy to setup. It would be really good if, as work progresses on the project, working builds could be checked in so that those of us with an interest can get involved. This is as opposed to uploading the "final" project at launch time.

And a final point: something I didn't see mention of in the initial spec that I think should at least be considered from the start is some sort of plugin architecture. Developers are almost certainly going to want to customise this thing and it's something I think should be encouraged, so it would make sense to have some sort of official extension mechanism baked in from the start, or at least some idea of how it might be worked in at a later date.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

7

Re: Technology choices for online recording

I've just tried mod_rails, and it is indeed much easier than my previous setup!

With regards source control, either of those look good. I've mostly used subversion before, but have heard lots of good things about Git. It also seems very in the nature of this project to encourage as many people as want to to get involved during the development.

8

Re: Technology choices for online recording

More of a question than a comment....

I'm not too bothered about the flavour of code, we have a online recording system which works well with HTML and a little ASP to process the information. So long as any database is well structured...is there any problem with data pooling and interoperability?

For our system, point records are recorded using 10-digit OSGB LGRs, which can be entered directly, or by clicking on an interactive Virtual Earth map, again with ASP doing the coordinate transformation. We've recorded polygons (range of animals or extent of survey) using .KML files.

I would suggest that simplicity for the user and good feedback (maybe with plotting of data entered onto an aerial) is the most important driver for design. What am I missing?

Cheers,

Steve

[b][email=%73teve@%73u%72rey-arg.or%67.uk]Steve Langham[/email] - Reptiles Officer & Webmaster[/b]
Surrey Amphibian & Reptile Group (SARG).

9

Re: Technology choices for online recording

Hi Steve,

You are broadly thinking the same as we are - a data capture system needs good feedback to the user to help encourage them to contribute further records. So we will be including facilities to help get the data onto Google Maps, Google Earth or whatever mapping tool each individual website sees fit.

The core of the database will be fairly simple so like you say there should be very few issues with interoperability. We have to cope with other systems as well as OSGB (Irish Grids, UTM, Lat Long from GPS etc) so the design will be different, but it should all be mappable to other models and there will be upload and download tools to make this sort of thing easy.

Thanks for your input,

John van Breda
Biodiverse IT

10

Re: Technology choices for online recording

Replying to Charles' earlier comment on plugin architecture - area you referring to the individual online recording websites? If so, then I am not sure a plugin architecture is necessary at this stage because the aim of the project is to provide you with a set of reusable services and modules to build your own sites with. This means you can build anything you want and just plug in a little, or a lot, of the OPAL online recording toolkit functionality as required.

Of course once the Core is ready, some effort will be spent developing example websites which themselves might like to feature a plugin architecture, though I suspect they will be based on a content management system so will inherently be extensible.

The Core itself will be developed using CodeIgniter or Kohana (probably the latter - thanks for the idea Charles!), both frameworks include a plugin architecture so it would be easy to extend the Core in this fashion.

Best Wishes

John van Breda
Biodiverse IT