feedapi

Introductory Notes

These are some thoughts in progress -- I’ve been thinking these things through for probably the last few years, but things have been getting more interesting of late.

Some of the blog posts that have helped shape my thinking here include:
http://bavatuesdays.com/proud-spammer-of-open-university-courses/
http://weblogs.elearning.ubc.ca/brian/archives/044998.php
http://opencontent.org/blog/archives/464
http://www.chrislott.org/2008/02/17/confused-about-the-blog-uproar/
http://weblogs.elearning.ubc.ca/brian/archives/044813.php
http://www.funnymonkey.com/mini-edu-rss
http://www.darcynorman.net/2008/02/16/on-eduglu-part-1-background/
http://blogs.open.ac.uk/Maths/ajh59/010236.html -- this is from Tony Hirst, who has an almost overwhelming amount of great information regarding remixing content on his blog.
I've also been thinking about the work Scott Wilson has been doing with FeedForward.

Toward the end of this post, I fall short of the needed conversation when I talk about the Course and Learner sections. There’s more to be said here -- a lot more -- but the poor souls who actually persevere to that point in the post will probably agree that I’ve said enough by then already.

An Open Content and Open Learning environment

External Repository -- in this context, an external repository is a place where content is stored. In many ways, the external repository is an artificial construct that doesn’t need to exist. The single most important argument in favor of the external repository is that the external repo can provide a level of credibility that less “official” sources of information lack. For example, a piece of information coming from the MIT’s OpenCourseware will have more credibility than a YouTube video.

These external repositories, however, need to expose their content via rss/atom, or web services, something that many of them do not do. With that said, it would also be nice to see the major OCW repositories use less pdf’s to allow for easier modification.

On a technical note, Tony Hirst pointed to a Mediawiki plugin that exposes full Mediawiki articles as rss feeds. This extends Mediawiki’s flexibility by allowing Mediawiki content to be imported via rss feeds.

Planning Repository -- the planning repos are the staging grounds of course preparation. Planning repositories will import selected courses from a variety of external repositories. While a limited number of people might have access to an external repository, more people can have access to a planning repository. Within the planning repository, users can edit existing courses, add links, text, images, etc. Then, users can select individual pieces of different courses, and re-organize them into a new course. By definition, planning repositories should be messy. They are workspaces, and should be viewed as a place where people go from draft versions to more polished versions of course materials.

For example: a history department creates an departmental planning repository. Initially, they import a variety of courses from different external repositories. Then, instructors add content as needed. Once they have finished adding content, they select the lessons/material they want for their course. So, an instructor teaching a course on the Rise of Modernism could incorporate material from a course on WWI. Once the instructors have selected and organized their lessons, they export them into their courses.

On the technical side, the planning repository could be a Drupal site built using the FeedAPI. I described how to do this here, and revisited the idea here. Alan Levine (in the comments here) and Jared Stein and Patrick Gosetti-Murrayjohn (in the comments here ) ask about how to select individual pieces of content for inclusion in a course. Once you have imported content into a Drupal site, you can use Views Bookmarks, Nodequeue, or node references (part of CCK) for doing exactly that.

Once the individual lessons have been selected and organized into a course, they can be exposed via an rss feed.

Mediawiki would also make an excellent planning repository by using XFeed to aggregate external content and the WikiArticle Feeds Extension (linked to above) to generate rss feeds for curriculum.

However, here is another wrinkle: every school is already producing curriculum. Teachers generate curriculum for all of their classes. If a school used a planning repository to coordinate curriculum planning, they could export the polished curriculum to a web site that could become an external repository. In this way, schools generate their curriculum maps and provide open content as part of their ongoing course planning and development process.These planning repositories becoming external repositories would have one enormous advantage over existing content repositories: they would be fully open, with all content within them accessible via rss feeds. For all schools currently undergoing accreditation reviews, how much time are you spending collecting up curricular materials? If you build your curriculum as described in this post, you have all your curriculum ready to hand, and categorized via tags.

It’s worth noting that the technology to do this exists now, and can be built entirely using open source tools.

It’s also worth noting that, using Drupal, you can clone an entire site -- configuration, content, and even user accounts -- and move that site with minimal effort. It’s what we’ve been doing with DrupalEd for nearly a year, and with less sophisticated class sites since September of 2005.

Courses -- In this context, courses are blog based tools, and could be delivered via a tool like Wordpressor Drupal. Curricular material could be imported; Jim has shown how to do this, D’Arcy has shown how to do this , and the aggregation examples I linked to earlier show how to do this.

The feeds of learners taking the course could be added to a blogroll, or, in the case of Drupal, could be imported directly into the site. With OpenID becoming more prevalent, students could either be site members, or be granted access via their OpenID. This flexibility would allow learners to interact with the course using their preferred tools, and, if they wanted, using their pre-established online identity.

Learners -- In this context, learners are just about anyone. You don’t need to be a student to be a learner, although, for obvious reasons, most schools probably wouldn’t allow open enrollment in their courses.

For me, the interesting piece of this has to with the potential for a true PLE. While I’m not particularly enamored of the whole notion of the PLE (I see it as more of a construct than a piece of technology, and something that is better achieved via innate curiosity than lines of code, but that’s another conversation), this system of open learning solves one of the main problems inherent in most PLE implementations: how to get course content out of the course and into the PLE. In this situation, that’s not an issue, as learners use their chosen tools to contribute in their courses. As they are doing the work from their platform, they retain control of their work in a way that just isn’t possible using proprietary LMS’s, or even open source LMS’s like Moodle.

Next Steps

The next steps could include any/all of the following:

• A school, or a group of teachers, banding together to create course materials in a planning repository. Dan Meyer has called for something along these lines a while back.
• More teachers using a blog-based approach to delivering content. The WPMU work that Jim helped spearhead shows one way of doing this; and the folks at BYU have illustrated another way of doing this.
• Existing Open Content repositories could actually expose their content via rss feeds. If this happened, one of the enornous barriers to actually using the open content that has been published to date would be removed.

These thoughts are incomplete -- what's missing? What needs closer examination? What else needs to be considered here?

9:30 -- Re-read Brian Lamb's blog post.

9:33 -- Poked around Stephen Downes' site, reading over some of the documentation on Edu_Rss. Really, I'm hoping to find an OPML file. Bingo.

9:40 -- Create a database on educon20.org.

9:42 -- Go to Drupal.org -- grap a copy of the 5.7 codebase, and the following modules: FeedAPI, FeedElement Mapper, Views, Views Bonus, Tagadelic, and CCK. At a later point, if nothing blows up, I'll probably add in Similar Content.

9:50 -- untar code. Realize I'm curious how long this will actually take, and resign myself to getting less sleep than I originally hoped. So it goes.

9:59 -- upload code to the server. Crack a beer. A good one.

10:04 -- bring site live.

10:08 -- in the process of installing the modules, realize I have forgotten to download the SimplePie parser. Oy.

10:16 -- create settings for the imported feeds, and create taxonomy categories the individual posts.

10:23 -- test import with a test feed. It looks good.

10:30 -- import opml file

10:35 -- first attempt at opml import bombs. Time to increase the memory allotted to php scripts in the settings.php file. Bumping it up to 40M ought to do it. If that doesn't work, I'll break up the opml file into multiple parts. At this point, I congratulate myself on the wise choice made at 9:59. A lesser beer would offer less solace during these times of peril.

10:42 -- second attempt bombs again. Time to try a third attempt, and see if it bombs in the same place. Don't know if I'm running into a php timeout, or a malformed xml file.

10:45 -- third attempt. Fingers crossed.

10:46 -- bombs out at close to the same place. In all likelihood, a php timeout issue. Small curses.

10:57 -- finished editing the original opml file into 4 smaller opml files. The first one imports with no issues -- 100 feeds down. Now trying the second opml file, which is larger than the first.

Note: I'm doing all this via a wireless connection, which is rather silly. When I am uploading files, I prefer to use a wired connection, as there is less chance of a transfer getting munged.

11:06 -- the second opml file bombed -- edited it into two smaller opml files. Trying again now.

11:13 -- the first two opml files have imported cleanly. The third is importing now. After this, two more to go.

11:22 -- opml import complete. Now, to begin the process of importing the feeds.

11:23 -- first cron run begun. In Drupal, there are many wonderful things that occur during a cron run. It is a sign of my general disintegration that I now have an active interest in things that occur during a cron run. During the first cron run, nearly 1000 posts were imported from the various feeds.

11:26 -- second cron run begun. An additional 2000 posts imported

11:30 -- third cron run begun.

11:37 -- fourth cron run begun.

11:45 -- create default views for imported feeds, and keyword directory.

12:06 -- install Similar Terms module -- this is a lightweight content recommendation engine.

12:25 -- for the last 20 minutes or so, I've been lost reading content.

12:40 -- set up a cron job to run automatically. This will serve two main purposes: import new posts, and index the site so that the search actually works. It will probably take about half a day for the site to get fully indexed; after that point, the full text search will work pretty well.

1:00 -- clean up this post. Wonder why I didn't go to bed earlier.

As of this writing, a little over 3.5 hours from when I started, there are nearly 7500 posts imported from around 500 different feeds.

I saw this earlier today over at groups.drupal.org --

Kyle Matthews and Clint Rogers built a Drupal site in suppport of a web analytics class. The site aggregates student blogs and expert blogs; this way, everyone blogs from their chosen blogging platform, and their feed gets imported into the course site. In other words, people use whatever blogging tool they are currently using, and the software running the course (in this case, Drupal) adapts to the participant. This is a nice contrast to the usual approach, where all participants must adapt to the structure required by the LMS.

The site was built using the FeedAPI and the Feed Element Mapper. We have talked about organizing classes and building Open Educational Repositories like this in the past, and our main proof of concept site has been humming along for the last few months with no issues at all.

There has been some great development behind the FeedAPI; just last week, the folks over at Development Seed put out another screencast showing how they are extending the functionality even further.

Over the last two nights, I put some time into building out a rough proof of concept showing some of what can be accomplished via a good aggregator and Drupal's taxonomy structure.

We've been thinking about/using aggregation in a variety of ways for the last couple years, but the development of the FeedAPI has created some pretty amazing possibilities faster than we could have hoped. I've been meaning to build out a site like this for the last few months, but a couple of recent conversations stirred me into actually doing it.

What has been fun about building out this proof of concept was how quickly the site came together. It's rough, and has no graphic design component at all, but the core functionality came into place quickly.

The results are here, and I'll include the brief description from the homepage of the site.

First, the useful details:

• All content on the site can be searched via the site's search.
• Full text of all imported posts available here.
• A Directory of all keywords available here.
• The obligatory tagcloud here.
• And the feed list here.
• Posts can also be listed and filtered by keyword and author.

This site is designed to show the utility of a single location as a collection point of content from disparate sources, and how that content can then be re-organized by use of keywords to categorize the content that has been imported.

On this site, all imported content retains all keywords added to the post by the author. Additionally, new keywords are added to posts on import to allow for the content to be searched and organized in other ways.

A brief technical overview:

If you are not a geek, you can stop reading here. If you are a geek, read on!

• This site uses Drupal as the main framework.
• As this site is a proof of concept, we kept things light. The only core modules in use are Menu, Search, and Taxonomy. This site uses no path aliases, and the theme is the lightly modified Zen theme that ships with DrupalEd.
• Aggregation is handled by the FeedAPI, and extended by the Feed Element Mapper.
• The Similar By Terms module handles the content recommendations that can be seen alongside posts (see here for an example).
• The Views module generates several of the screens for displaying and navigating the imported content, and the Views Bonus module extends these views.
• Finally, CCK is installed and enabled (although, for this implementation it could probably be eliminated if necessary); and HTML Corrector is installed to clean up any unclosed tags that on imported feeds that could break the layout.

For those keeping track of such things, this site has taken a grand total of six hours to build, including this writeup. The functionality of this site is all achieved using modules and code currently available within the Drupal community.

One group of folks deserve a special mention: the team of people behind the FeedAPI module. For those interested, you can see a lot of the discussion at the RSS and Aggregation group. They planned and executed a great project, and without their work this site would not be possible.