Speed has been emphasized a significant amount in Caffeine discussion, and Cutts told us that page speed would likely become a ranking factor. Regardless of whether or not you are witnessing Caffeine in action yet, rest assured that it will be here sooner or later, and any edge you can give yourself in the meantime is for the good of your own site’s performance. Speed will not only supposedly help you in search going forward, but it just makes for a better user experience.
http://www.webpronews.com/topnews/2010/01/26/has-google-begun-changing-how-it-indexes-the-web