Jalaj P. Jha Technical & Miscellaneous Ramblings

Yahoo Pipes: Build RSS for non-RSS Sites

RSS has made browsing the internet so simpler. For each site or its section that you wish to visit regularly, you just need to get the RSS feed url and subscribe it to your Feed Reader as Google Reader. After that all the updates to the sites/sections that you subscribed is like reading just a single page with all the new content. What’s more that you could have wished for? May be that all the sites could have RSS feature!!

There are a number of sites that are highly popular but were built a long ago in static html style or using the WYSIWYG editors as Dreamweaver to get the same look on all pages using templates. Unfortunately they continue to remain as such and you are eager to read it at your convenience in feed reader. Or worse you are actually owner of such a site and realize that your site is missing the RSS but don’t know how to go ahead adding it to your site. The answer lies in somehow building the RSS. If you are a site visitor the Rss will benefit as you can read it at your convenience. If you are the owner you can save the generated RSS files and link them from individual pages or if you can afford to risk biding the site again using CMS/blog as Wordpress, just import all the rss files.

Question is how to build the RSS? And answer could be Yahoo Pipes. The text ahead assumes that you are at least capable of following the instructions mentioned herein. We take here comedy-zone.net, a site that lists hundreds and thousands of jokes. Let’s start with Blonde Jokes at http://www.comedy-zone.net/jokes/laugh/blondes/index.htm. This is a page where all jokes are listed by Titles and link on each title pointing to individual pages containing the full joke. To prepare the RSS feed we will first get the list page and then get all individual page to add the entire joke to description.

From “User inputs” section add “URL Input” module. We will pass the url of list page through this. Since all such list pages are expected to be based on the same template, a single pipe would be sufficient to get feeds to different categories of joke. put Blonde Jokes list page as the default url. Use a “Fetch Page” module to get the page. If you go through this list page source code you will find that you can get entire list between <td colspan="3" bgcolor="#660000"> and </table> and you can break all jokes by using <td class="menuOption"> as the delimiter. Use these as parameters to the Fetch page module.

The result would be that the list will be broken into multiple items, each item containing the url/title of the joke and some other text we are not interested in. We will use Regex module to clean each item such that only the link remains which we will use to fetch full jokes text. The first entry doesnot contain any link (check it yourself) and to filter it off use a Filter module to permit only items that start with http://.

Now we are going to fetch each joke page but before that you may want to limit the number of items returned in the rss feed to say 10. It’s better to place this limitation here itself because fetching each page and later discarding it would be a waste.

Get the number as a parameter through “Number Input” module and pipe this to the Truncate module which will truncate the feed we are preparing.

Now add a Loop module and a Fetch page module within it. Fetch pages from URL given in item.content and limit the text to be scraped between <td width="95%" colspan="2" bgcolor="#FFEA97"> and <form. Get all pages within item.loop:fetchpage

Each of the page that we have fetched contains the title as well the description that we want to add to the rss feed. we already have the Links from list page fetching. Now all we need to do is to remove the line feed and carriage return characters as they sometimes give unexpected end results. In effect we will do this by replacing all of them to space characters replacing all occurrences to more than one continues space with single space. Once this process is complete we use the rename module to prepare the fields Title, Link & Description. A regex module added later will clean off unwanted texts (description from title and vice versa) from title/description. All these results in the final output.

This pipe is available at http://pipes.yahoo.com/jalaj/comedyzone. Replace the list urls with other categories of jokes from comedy-zone.net and see the same pipe working all over.

Want some task to be accomplished using Yahoo Pipes and featured on this blog? Use comments area to ask for it.