Scott Hanselman

Infinite Scroll WebSites via AutoPagerize - Hacky, but the beginning of something cool

April 6, '10 Comments [20] Posted in ASP.NET
Sponsored By

One of the things I like the most about Bing Image search (one of the things I prefer about it) is the "infinite scroll" feature. If you search for an image and start scrolling, it'll just keep going and going, moving the scroll bar each time and appending new images on the bottom. This concept of "infinite scroll" has been called just that, as well as endless pages, autopagerize, etc. There's even a jQuery plugin called Infinite Scroll if you want to enable something like this on your site programmatically.

oscar winners - Bing Images - Windows Internet Explorer

However, there's also been a quiet revolution on sites, and to some extent, in browsers to make infinite scroll a standard thing. At least, a de facto standard, and you can enable it on your site with minimal effort.

The general idea is that the browser notices that you're scrolling to the end and rather than making you click, it'll fetch the next page via AJAX and append it to the page you're already on. This screenshot from the AutoPagerize for Chrome extension shows it best:

How Autopagerize works

There's a few things needed and it requires a bit of dancing on your part to make it happen.

Enabling Autopagerize as a Browser of the Web

For the longest time Autopagerize has been a "Greasemonkey script." Greasemonkey is an add-on itself that enable others add-ons, via easy scripts, to dramatically change the behavior of your browser. I'm not a huge fan myself, as I have some security concerns. The main site that promotes this is a bit dodgy looking, at http://autopagerize.net/ but their Extension for FireFox works and they mean well.

Enabling Autopagerize on Firefox

You can use GreaseMonkey and the AutoPagerize userscript if you like, but I use the AutoPagerize Firefox Extension from http://autopagerize.net/.

Enabling Autopagerize on Opera

Opera supports "User JavaScript" out of the box, so you can get their oAutoPagerize script, follow some directions and you're all set. It's a modification of the standard GreaseMonkey script and it will work with Safari and GreaseKit and Chrome, although I recommend the cleaner Chrome extension.

Enabling Autopagerize on Chrome

Chrome has a Chrome Extension called, logically enough, AutoPagerize for Chrome. It has the benefit of a small colored square in the address bar that will show you if the current page is enabled for paging and the current status.

I'm still looking into a reliable way to do this on IE, but you can start with the older GreaseMonkey for IE addon.

Enabling Autopagerize as a Web Site (Blog, etc) Publisher

Here's what it gets insane. Like "horribly gross and this will never scale" insane. There's two ways. If there are children in the room who design for the web, please ask them to leave.

First, you can go to this online database of sites http://wedata.net/databases/AutoPagerize/items and add your site along with some *cough* regular expressions and XPath expressions that describe where the next page to retrieve is and what to append it to. Wow, Regular Expressions AND XPath? What, no "select * from authors"? And a centralized database. Good times.

Well, my record (and most DasBlog sites) looks like:

pageElement: id("blog-posts")

url: ^http://www\.hanselman\.com/

nextLink: //div[@class="previous-posts"]/a

It basically says, you can find the next link at the anchor after the div with the class "previous posts" and you can append it to the element with the id of "blog-posts."

So this is gross.

Second option, and more ideally, I'd say, is this microformat. I'll actually copy/paste the microformat from the GreaseMonkey script itself as it says it all:

var MICROFORMAT = {
url: '.*',
nextLink: '//a[@rel="next"] | //link[@rel="next"]',
insertBefore: '//*[contains(@class, "autopagerize_insert_before")]',
pageElement: '//*[contains(@class, "autopagerize_page_element")]',
}

It says, find either an anchor like <a href="..." rel="next"> or a link in the head like <link rel="next" href="..."> then retrieve the page. Take the element with class "autopagerize_page_element" and append it to the element with "autopagerize_insert_before."

If your site/blog just adds a few classes and this rel, it'll be automatically setup to support autopagerize. I wanted to site my site like this but I hit a wall in the extensibility of DasBlog, the blog engine I run. This would be a small change to DasBlog, but it would mean a new version.

Of course, no browser supports this out of the box yet. Opera does offer a similar feature called "Fast Forward" that extends spacebar scrolling (in all browsers you can just press the spacebar to scroll down a page) such that it will navigate to the next page when you hit the bottom. Per Opera's KB:

Fast Forward tries to analyze a page and looks for links that will take you to the next page, for example after a search with Google with several pages of search results. It looks for certain patterns that indicate a "next" link, or uses "<link rel="next">" if it is defined in the page.

Unfortunately Opera analyzes my page and gets it wrong, selecting, oddly enough, an image as the next page to go to. This would likely be solved if I added a <link rel="next"> to my page's head, although again, I'd have to do this dynamically.

As an aside, notice this comment from Opera on their KB...

Please note that Fast Forward does not use any external services to determine the next page. It only looks at the current page and tries to find things that indicate that there is a "next" page. It does not look it up from an external server or contact any site to get this info.

This means they, too, realize that an external service is folly and the only way for this to work going forward is via microformats. I fervently agree.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Wednesday, April 07, 2010 12:38:26 AM UTC
That's nice.

I think the "infinite scroll" feature is one of the most refreshing ways to provide paged data in a web UI. One of those things where I thought "why on earth isn't it used EVERYWHERE?".

By the way... really, why? Why only in Bing image search, and not in the standard Web search? :-)

Anyway, I'd love to see it also on Facebook Walls, or on twitter, where you don't get the "Next" set of data, but you have to scroll for "Older" items.
Wednesday, April 07, 2010 12:42:53 AM UTC
I like it for google and bing searches, however the firefox extension/greasemonkey script you mentioned doesn't work well with SearchPreview, the firefox extension that adds thumbnails of the websites to bing and google searches.

Anybody know if it would be possible to adjust the script to make it still get the thumbnails? There is an alternative extension called AutoPager which it works on but it's not implemented as well as this one.
Wednesday, April 07, 2010 5:10:54 AM UTC
We as developers should not rely on browser's infinite scroll. we can do that from our side by handling the browser's scroll and add some script to use the next page attribute. and another suggestion, web technologies providers such as ASP.NET/MS might add this feature as built-in feature. in this case maybe we find Page attributes like Page.IsInfiniteScrollEnabled, Page.ScrollNextPageUrland Page.ScrollPreviousPageUrl(for upper scrolling)
Wednesday, April 07, 2010 5:14:05 AM UTC
The problem with infinite scrolling is that the user's progress within the page is lost on a navigation event. ie. If it was a list of articles, after viewing one from page 10, he can't reverse back to page 10.

The other problem I can see is that currently the concept of infinite scrolling has to start from one end,, and not in the middle.

Don't forget that people WANT to deep link to a specific page of results.

That said, these are all design issues that can be solved by just thinking about the problem some more.
Marauderz
Wednesday, April 07, 2010 5:31:04 AM UTC
As I mentioned-above built-in infinite scroll feature including page attributes (Page.IsInfiniteScrollEnabled, Page.ScrollNextPageUrl and Page.ScrollPreviousPageUrl) might be added to ASP.NET 4 or even as a part of the first Service Pack.
this solution will unify the way ASP.NET web sites manipulate Infinite Scroll
Wednesday, April 07, 2010 5:33:37 AM UTC
Marauderz, I agree with you to some extent, you can however combine the 2. I think the infinite scroll mechanism is useful where you just don't want to point your mouse to the top or bottom to click on a page or next/previous link.

Wednesday, April 07, 2010 6:15:46 AM UTC
I don't get it, why infinite scroll is so popular ? It's like you point mouse towards anchor tag and hope that it shows content without clicking ? Although infinite scroll is a happening thing right now, I disagree that it is a good UI metaphor. It somehow limits user's control over actions which is main goal of WEB in general
Wednesday, April 07, 2010 9:21:20 AM UTC
hey Now Scott,

Nice, kinda reminds me of rockscroll you introduced to us to add on to Visual Studio.

Hanselminutes Fan,
Catto
Wednesday, April 07, 2010 11:00:19 AM UTC
Seems to me that HTML5 page semantics can and should help in making infinite scroll work more reliably on more sites.
Dustin Burke
Wednesday, April 07, 2010 12:23:57 PM UTC
Parag, like all UI design elements infinite scroll has an intended purpose- scrolling through an infinite (very large) set of things in a fluid way. It doesn't replace paging which is intended for navigating through a finite set of things in a bitesized way. The paging metaphor breaks down for sets of like 1000-10000 or more things but we're so used to seeing paging of search results for millions of pages that we're accustomed to it (and expect it). Infinite scrolling works for "streams" of things, like Twitter. There are other ways to improve usability and make scanning through pages of things quicker and easier - first try displaying all paged items in a fixed size area that doesn't require scrolling down so the user can see them all at a glance and then make the paging indicator and navigation visible in a static location so the user doesn't need to move their mouse when clicking through pages. How annoying is it when the "next" button moves by even 10 pixels from where you expect it to be?
Dustin Burke
Wednesday, April 07, 2010 12:49:14 PM UTC
Awesome, I added that to my search engine on my site in about 1 minute and confirmed it's working as a "nice to have". pretty cool business.
Ben
Wednesday, April 07, 2010 4:49:51 PM UTC
This just shows the power of being able to extend a browser's functionality with user scripts or extensions. I think Google Chrome has a good system that balances security and usability: You don't get full control like extensions or plugins in Firefox and IE, but instead only get access to a restricted API. Also the scripts run in a sandbox mode and have to provide a manifest indicating which websites and APIs they need access to.

I have had fun making a few user scripts mainly for resizing video on TED.com or other sites, and it really does make you wish there were similar features available in Internet Explorer.
Wednesday, April 07, 2010 5:57:29 PM UTC
Infinite scroll is cool in something like Bing, but for most sites that would drive me crazy. There are already many sites that can bring a browser to its knees with a single page of content without encouraging them to infinitely stitch together page after page until the whole thing just locks up.

Bookmarking will get messy quickly and the context of where you are on a page (in case you want to jump back to the top for some reason) will quickly get lost.

Apply carefully. There be dragons ahead.
Wednesday, April 07, 2010 6:00:21 PM UTC
I implemented an infinite scroll using jquery - it was quite straight forward, even for a middling js dev like myself.
However, something I couldn't get to work well was as follows: infinite scroll UP as well as down if landing in the middle of the infinite page. The appending of content below the viewport was fine - it didn't affect the viewport (the scrollbar would just extend) . However, adding a page ABOVE the viewport caused the viewport to be shifted down. I tried using jquery animated show together with a scroll but I coudn't get them properly synced so the pages wiggled as the new content appeared..
peter
Thursday, April 08, 2010 12:33:38 AM UTC
What about the size of the page getting longer. Is there is trick in closing the earlier pages out of the browser?
Kumar
Thursday, April 08, 2010 2:22:00 PM UTC
Ive been in the business three decades now and I fondly remember the days when we built software and features because they were useful, needed, vital, etc. Here we go again - another feature that is "cool"...
...which is of course, a moniker for completely useless. Could we please dump the "cool" and start delivering something actually useful?

Leave the "cool" to James Dean and Marlin Brando - ya know, nice guys, but both dead and buried.

Andy Foreman
Thursday, April 08, 2010 3:32:25 PM UTC
Autopager is a solution made custom for every page. And it croudsources, anyone can make a rule for autopaging. I can't understand why it's not widely used like some other extensions.
Friday, April 09, 2010 1:42:31 PM UTC
Hello Scott,
Nice post, but this infinite scroll feature conflict with the fact that you have finite RAM, and this may cause that your computer crash, specially if the content size is too large !!!

Mostafa Nageeb
Friday, April 09, 2010 7:46:48 PM UTC
I hate web sites that have this "feature". This is the primary reason I do not use Bing.
Tuesday, May 04, 2010 5:17:55 PM UTC
What about memory usage? Each of these items that are added to the DOM as you scroll is using up memory. Browsers are already memory hogs. At least paging uses a fixed amount of memory per page.
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.