Summer of Code 2007 -- dojo.data projects

Context

What is "Summer of Code"?

• see SummerOfCode

What are the Dojo Toolkit and the Dojo Foundation?

• see the Wikipedia entry about Dojo

What is dojo.data?

The dojo.data module is one of the many modules available in the Dojo Toolkit. A JavaScript program can use the dojo.data module to get data from different data sources. A data source could be a simple text file, or a web service provided by a site like del.icio.us or Flickr, or a database like a relational database like MySQL, or an XML database.

What is a datastore?

A datastore is any JavaScript object that implements the dojo.data API. A JavaScript application can use a particular datastore object to get data from a data source. For example, you could use a dojo.data.DeliciousStore object to get data from the del.icio.us web service.

Where can I learn more about dojo.data?

• see http://dojotoolkit.org/node/98

Project 1 -- dojo.data.Overdrive

project goal

Write a dojo.data datastore implementation that uses MySQL as a server-side database. Implement the datastore so that it can deal with loosely-typed, multi-valued, semi-structured, JSON-ish, web 2.0 data, and can store that data in a rigid ugly old relational database. Maybe call the new datastore "Overdrive", with an eye towards facilitating comparisons between Ruby on Rails (TM) and Dojo on Overdrive.

mentors

We have not yet found a dojo developer to be the primary mentor for this project, but we do have a number of dojo developers who have volunteered to be mentors. For a list of available mentors, see the Volunteers list on the SummerOfCode page. Brian Skinner has volunteered to be a back-up mentor. Brian Skinner proposed this project.

background

• what is dojo.data?
  • see http://dojotoolkit.org/node/98
• what is semi-structured data?
  • see Daniela Florescu's paper, Managing Semi-Structured Data
• what is MySQL?
  • see http://en.wikipedia.org/wiki/MySQL
• can you store semi-structured data in a relational database?
  • Yes, products like the open source eZ Publish content management system are set up to store semi-structured data in relational databases. One simple technique is basically to just store all the attribute values of all the data items in a single table that has only three columns: (item_id, attribute_id, value).

programming languages needed

• JavaScript
• PHP
• SQL

what's involved?

Here are some ideas about how you might want to approach the problem...

1. Create a simple MySQL database that just has a single table. The table only needs 3 columns, for item, attribute, value.
2. By hand, populate the table with some simple sample data that matches the sample data from this movies.csv file.
3. Write a JavaScript class, dojo.data.Overdrive, that implements the dojo.data.core.Read API, and can read data items from the MySQL database. This new Overdrive class should be a subclass of the dojo.data.core.SimpleBaseStore class. You will also need to write some PHP code that runs on the server. The PHP code will take requests from your Overdrive object, read records from the MySQL database, and return JSON-format data to the Overdrive object.
4. Learn how to use the dojo unit testing framework, and write a simple unit test. Your unit test should verify that your Overdrive object returns items with attribute values that exactly match the items and attribute values returned by the existing dojo.data.CsvStore.
5. Add support for multi-valued attributes. Add a new entry in the MySQL database so that one of the movies has more than one Producer. Modify your PHP and JavaScript code so that they can return arrays of attribute values instead of just single attribute values.
6. Add support datatypes other than simple text strings. Add support for simple Numbers, Dates, and Boolean values. Extend the MySQL database schema as necessary to support the new datatypes. Add sample data to the database. Modify your PHP and JavaScript code to support the new feature.
7. Add to your code so that the Overdrive datastore supports not just the dojo.data.core.Read API, but also the dojo.data.core.Identity API.
8. Add support for references between items. Make it possible for an attribute value of one item to point to another item.
9. Write more unit tests. Write a unit test that uses the dojo.data.JsonItemStore to read this countries.json file, and verify that this data exactly matches some corresponding test data read from your Overdrive datastore. Likewise for this muppets.json file. And for this geography.opml file, available through the dojo.data.OpmlStore.
10. Add to your code so that the Overdrive datastore supports not just read-only access, but also read-write access, conforming to the dojo.data.core.Write API, and the dojo.data.core.Notification API.

Project 2 -- dojo.data.MediaWiki

project goal

Write a dojo.data datastore implementation that knows how to read semi-structured content out of a MediaWiki site by creating data item from the data in info-boxes and MediaWiki parameterized templates.

mentors

We have not yet found a dojo developer to be the primary mentor for this project, but we do have a number of dojo developers who have volunteered to be mentors. For a list of available mentors, see the Volunteers list on the SummerOfCode page. Brian Skinner has volunteered to be a back-up mentor. Brian Skinner proposed this project.

background

• what is dojo.data?
  • see http://dojotoolkit.org/node/98
• what is MediaWiki?
  • MediaWiki is the wiki engine that runs Wikipedia. See http://en.wikipedia.org/wiki/MediaWiki
• what are MediaWiki infoboxes and parameterized templates?
  • see http://en.wikipedia.org/wiki/Help:Infobox

programming languages needed

• JavaScript only

what's involved?

Here are some ideas about how you might want to approach the problem...

1. Have a look at the infobox on the right-hand-side of the Wikipedia page about Brazil. Then have a look at the raw source view of the Brazil page, to see what the raw infobox data looks like. You can also access the raw page source by using a URL like this one: http://en.wikipedia.org/w/index.php?title=Brazil&action=raw
2. Get the Wikipedia site to run some custom JavaScript code for you. Create your own Wikipedia user account, and add a page with a name like "User:Foo/monobook.js" (where "Foo" is your user name). Add a little bit of JavaScript code to your monobook.js page, to do something simple like bring up an alert box.
3. Add more code to your monobook.js page. Add code that uses an XMLHttpRequest to load the raw source of the Brazil page. Then add code that parses the Brazil page source, finds the infobox definition, and extracts all the attribute-value data (such as "area = 8,514,877") into a JavaScript object.
4. Add more code that makes the attribute-value data available through the dojo.data datastore API.
5. Add support for reading data from more than one page. Make it possible to automatically read data from all the pages in a given category (for example, all the pages in the Wikipedia category G15 nations.
6. Make a simple demo page that can display a bar graph showing the relative land areas of the G15 nations.
7. Extend your datastore so that it's not just a simple read-only datastore, but a full read-write datastore.
8. Write lots of unit tests.
9. Add more features...

Project 3 -- offline dojo.data

project goal

Get the dojo.data datastores and the dojo offline toolkit working together, so that for any app which already uses dojo.data it becomes astonishingly easy to also add offline functionality.

mentors

We have not yet found a dojo developer to be the primary mentor for this project, but we do have a number of dojo developers who have volunteered to be mentors. For a list of available mentors, see the Volunteers list on the SummerOfCode page. Brian Skinner has volunteered to be a back-up mentor. Brian Skinner proposed this project.

background

• what is dojo.offline?
  • see http://www.sitepen.com/blog/2007/01/02/the-dojo-offline-toolkit/
  • see http://www.sitepen.com/blog/category/offline/
  • see http://www.sitepen.com/blog/2007/02/26/screencast-of-dojo-offline-demo-release-download/
  • and source code:
    • http://trac.dojotoolkit.org/browser/trunk/src/off.js
    • http://trac.dojotoolkit.org/browser/trunk/src/off
    • http://trac.dojotoolkit.org/browser/trunk/src/dot
• what is dojo.data?
  • see http://dojotoolkit.org/node/98

programming languages needed

• JavaScript only

what's involved?

The dojo toolkit includes two separate modules, dojo.data and dojo.offline, which each are useful in building AJAX web applications:

• The dojo.data module is useful for reading data from a remote data source, like a web service or a MySQL database on a web server.
• The dojo.offline module is useful for making an AJAX application that continues to be available even if the computer it's running on isn't connected to the internet.

The dojo.data and dojo.offline modules each use simple data access APIs -- they each provide some basic functions that a developer can call to create new data objects, set attribute values, etc. But unfortunately the two modules use slightly differentent data access APIs, which makes it more difficult than it should be to use the two modules together in the same application.

As an example, let's say Kim is writing an AJAX application that will be used by book clubs. The people in a book club can use Kim's web site to keep track of which books the group has read, and to suggest new books to read. If the app uses dojo.offline, then the book club site can be available to a book club user even when that user's computer is offline. For example, if Pat has his laptop on a airplane with no internet connection, Pat can still visit the book club site and add a review of a new book he just read. The dojo.offline toolkit will store the review on Pat's laptop until Pat is connected to the internet again, and then the dojo.offline toolkit will send the new review to the book club site.

In order for the dojo.offline toolkit to work, it has to be able to save data to the computer's local hard drive. The dojo.offline toolkit provides a simple API for saving data and reading saved data. In order to use dojo.offline, Kim needs to write code to take the book reviews and bundle them up into the format that dojo.offline understands. Kim would also like to use a standard dojo.data datastore to access the book club server. In order to use the dojo.data datastore, Kim needs to write code to store the book reviews the way that dojo.data understands.

The goal of this project is to make it so that Kim only has to write code that uses the dojo.data API. Once Kim has the basic application working, successfully getting data from a dojo.data datatstore, then when Kim later starts to add dojo.offline features to the application, the local dojo.offline data storage should be available through the same set of data access functions calls that Kim is already using.