For Serendeputy, I’ve built out a series of automated checks that tell me if something seems amiss with the data. One of these monitors alerts me to sites that haven’t been updated in a while so that I can check them out, and either update them or remove them from the catalog. I caught my own site (this particular blog) this morning. That’s not a good sign. I should probably be writing more here.

I’ve been head’s down on Serendeputy for the past few weeks. I have a couple of major demos in the first week of December, so I’m probably going to be pretty quiet for the next couple of weeks, too. Lots to talk about after that, though…

Good move by the Times in opening up the Times Tags API. I can’t wait to play with it.

I’m glad they are being so clueful about this. Opening up API access to your content is the next form of Advanced SEO, and the Times is on top of it. I hope the Globe is able to follow suit.

Greg Linden talks about how Digg may be working on personalized news. This is entirely expected. Every infomediary, aggregator and publisher needs to be working on technology like this if they want to survive. By 2011, every web user will demand that all websites be intensely personalized. Like Serendeputy.

Hubris alert: I realize I haven’t launched anything yet. Working on it…

I have GMail and Yahoo Mail accounts for testing, but I’ve never relied on them as the primary email address. This is why.

Find a bug. Write a test to reproduce it. Fix it. Watch the test pass. Check in the changes.

Realize you’re one step closer to perfection.

I’m working interactively in irb, and I tweak the class to make changes. The changes won’t be reflected in the irb session unless you reload the class.


irb >> load 'document.rb'


Make sure to add the file extension. Unlike require, load needs the full filename.

Jeff Atwood and team have built a new product called Stack Overflow, a question and answer site for programmers. It’s very early, but I think it will be successful. It’s certainly been useful for me.

For Serendeputy, I’m building an index of articles relevant to me, and I need to be able to generate a unique id based on the url. I experimented with (what appeared to be) the obvious solution, but it wasn’t working. So, I asked the question: “What’s the best way to hash a url in Ruby?” Within three minutes, I got a useful answer. By morning, I received three answers, with the best answer sorted to the top.

This is incredibly useful. I have a stack of Ruby books at my desk, and I’m often flipping through them trying to find a specific answer. I generally know *what* I’m trying to accomplish; I’m just lost on the syntax and the proper Ruby idiom to use. If my experience is typical, then Stack Overflow is a very useful supplement to these references.

Stack Overflow has a ton of Google juice already. My question is number one for the relevant search query: hash a url ruby. Most of the time when I do a Google search for specific questions, I get links to the Ruby documentation (which I already have) and a decent amount of off-topic spam. If Stack Overflow gets a critical mass of specific questions with canonical answers, then Google will become much more useful for everyone.

I’m very impressed with how they’ve focused on the customer experience. I’m trying to keep the same ideals with my project. Unlike those folks at that annoying site, they have everything open and clean. They have advertising, but it’s inobtrusive. And, the value of the product was so high that I actually made a point of looking at the advertising, seeing that an advertiser who’s associating with this useful an application is probably worth checking out. It’s amazing what optimizing for the customer experience will do for you.

It reminds me of how I felt in 1999 when I switched from Alta Vista to Google.

One quibble: Requiring people to get an OpenId is a mistake. Everyone in their target market has a simple throwaway handle/password combination that they use on all these sites. Unless it’s a pragmatic choice — they don’t have to program the authentication module — I think it’s a mistake to go against the grain of what people expect for these sites. Requiring an OpenId dissuaded me from registering; I just posted as a guest.

It’s been a productive couple of weeks here at Serendeputy world headquarters. I have been heads-down trying to build the private alpha version of the site — i.e., get it running end to end on my own machine.

I hope to be able to have the private alpha done by the end of September, with an incredibly limited wider alpha by October 15th or so. I hope that it doesn’t take me two weeks to get the deployment and hosting issues squared away, but this is the weakest part of my game, so I’m trying to give myself plenty of time to get that part right.

The rails application is largely built, and the catalog application is coming along quickly. The hard part is maintaining the discipline to keep writing the tests and automating the admin pieces of the application. I keep wanting to jump ahead and build out some of the sexier features. It’s sometimes not a lot of fun being a grown up.

If you’re interested in taking an early look at the application, drop me a line, and I’ll make sure I send you the information when it’s somewhere publicly accessible.

The blog will probably be dark for a couple more weeks as I finish this sprint. The world grieves.

There’s undoubtedly a smarter way to do this, but I couldn’t figure it out. I’m doing a functional (controller) test for one of my rails pages. I want to make sure that the picture coming back is exactly 224 pixels high.

I couldn’t find a way to do this using the assert_select syntax, but this workaround seems to work:


  def test_picture_height
    assert_select "div.secondary div.lead_picture img[height]" do |height|
      # pull the tag -- this gives you the whole img tag
      @h = height.to_s
      # extract the height
      @h.gsub!(/.*height="(.*?)".*$/, '\1')
      #run the test
      assert_equal @h, "224"
    end
  end


This can be tightened up, but I left it explicit so that I could understand what I was going. The assert select makes sure that I’m setting the height. The block makes sure that it’s equal to 224.

Are there easier ways to do this?

Rails puts the log files in the same part of the tree that ends up under source control. If you don’t tell Subversion to ignore them, you’ll end up with all the log files under source control, which will drive you nuts.

If you already have them in the system:


cd YOUR_RAILS_APP/log
ls -l # Make sure you're in the right directory
svn --force rm * # Force Subversion to kill those files
cd .. # Move back to the rails root level
svn propset svn:ignore "*.log" log


Hope this helps.

I just started a new Subversion repository, taking to heart Fred Brooks’ advice to “Build one to throw away.”

Of course, I’m planning on building four or five to throw away. If things go extremely well, I’ll have something release-worthy by prototype six…

Alan is continuing to do a great job with The Big Picture. I really like the way that he’s handling sensitive pictures in this gallery covering the Georgian crisis.

Early retirement is a false idol
From 37Signals, Justin Bell writes that you shouldn’t sacrifice your entire life now for a shot at an early retirement. Enjoy life now, as you’re living it.

Dropping bombs in the newsroom
The newspaper industry really has to reinvent itself. More on this later.

Mom-and-Pop Multinationals
Interesting article from Business Week on how small business owners are taking advantage of global outsourcing. Many good ideas.

I’m thought about using some of these services since reading The Four-Hour Workweek. I’ll see if I can use any of them this fall as I’m going all-out towards launch.

How to analyze and improve the bounce rate for your website
Very good article on a very important topic. I’m trying to build a model for one of my clients on this.

25 reasons users stop using your product
Andrew Chen writes up some notes on why people give up on games and websites. I’m spending a lot of time working on Serendeputy’s out-of-the-box experience and customer lifecycle. I don’t have a ton of marketing dollars, so I need to make sure the product is keeping the customers engaged.

I’ll likely write a longer piece later about how I’m trying to incorporate the best ideas from game design to apply to more conventional websites.

Dissecting today’s Internet traffic spikes
Scaling doesn’t mean the same thing today that it did a few years ago. If I ever get to a point that this is a problem for me, I’ll likely have someone smarter than me thinking about it; still, this is really interesting information to know.

Nine steps to achieving flow
I love working from home now. It’s so much easier to get in the state of flow.

Programmer Competency Matrix
This is a smart grouping of competencies you want your programmers to have. Boy, do I have a long way to go before I’m even vaguely competent at programming.

15 questions to ask during a Ruby inteview
On the other hand, I aced this test, so I’m feeling a little bit better about my meager skills.

Editor’s note: I’m trying to give back a little to community by sharing things I find exceptional. I hope to be able to share a couple of these a week.

I love listening to Leo Laporte’s TWIT podcast (netcast?) network. I’ve been listening to these podcasts for a couple of years now and they continue to deliver each week.

These are my favorites:

This Week in Tech

This is the flagship podcast, the weekly roundup of tech news and gossip. It’s quality varies widely based on the guests, but it’s still worth a listen every week.

MacBreak Weekly

This is a really fun podcast in which the team, usually including Alex Lindsay, Scott Bourne, Andy Ihnatko and Merlin Mann, talk about the Mac, along with many many many other topics. MacBreak Weekly is like hanging out with some of my geekier friends for a couple of hours. That’s probably the highest compliment I can pay.

The weekly picks are very useful, too. Scrivener, which I’m using to write this post and which will likely be the subject of future fan mail, came to my attention via a recommendation from Andy Ihnatko.

Security Now

This is Steve Gibson’s podcast, in which they talk quite geekily about security topics. A lot of the topics fly over my head, but I’ve learned a decent amount on web security topics by listening to this one. The bi-weekly listener question shows always contain a tasty nugget or two.

Since I’ve stopped communiting into Boston each day, my podcasting time is way down. I still make sure to listen to these three each week, though.

Editor’s note: This is the first of what I hope will be many book reviews I write for this blog. I read a lot, and I want to share the books that have made me more successful, the books that have really changed the way I do things. I hope to be able to review one book or so a week, but we’ll see how it goes.

SYNOPSIS:

In The Dip, Seth Godin makes two main points: 1) To really succeed now, you need to literally be best in the world at what you do, and 2) As you pursue any goal, you will inevitably come across a dip, a period when you feel that all is lost and you need to give up. Those who quit the dips they can’t cross and persevere through those that matter are the ones who succeed.

The Dip is a very small book with a very big idea. It will take you about two hours to read.

TAKEAWAYS:

Being the best in the world

If you want to be successful, you need to be best in the world at what you do. The Internet and the power law will guarantee that those who are the best in the world at what they do with get the lion’s share of the rewards. Being really good used to be enough. No longer.

So, what do you do? You define your market as one you can dominate. If you are an accountant, and you can’t be the best accountant in the world, adjust your market until you can be the best. You might not be able to be the best accountant in the world, but you might be able to be the best accountant in the world for technology startups based in Massachusetts. You can own that market, that world.

Getting through the dip

The dip is when it’s not that much fun anymore. For doctors, it’s taking organic chemistry. For lawyers, it’s slogging through the first-year hell at law school. For developers, it’s moving beyond “hello world” and digging deep into the internals of your language.

Most people quit in the dip. This is the absolute worst thing to do. Either quit before you start, if it’s a dip not worth making it through, or just keep grinding. The good thing for you? Most people quit in the dip. That’s why the rewards are so high for those who make it through to become the best in the world at what they’re doing.

REFLECTIONS:

I recognized myself in a lot of what Godin wrote. I love to dabble, and play around with all sorts of new ideas. I don’t take enough of them as far as they could go.

The “best in the world” mantra has informed the mental framework I’m building as I’m planning out Serendeputy. I know that I need to be best in the world for a certain group of people, so I’m designing strictly for them. I’m writing off the other 95% of the world to be the absolute best I can be for my group.

As for the dip? I know that it’s coming. Right now, I’m building out the alpha version of the product. Things are moving quickly and I’m making a ton of progress each day. The commits are flying and Emacs has turned my fingers into claws. All is well. But, I’ve been doing the easy stuff. But, I know that the Dip is in front of me. It’s glaring at me.

I know that in order to differentiate the business and build some moats, I need to solve a couple of hard problems. I want to make the dip my friend, making it difficult for others to clone my work.

But, the problem with solving hard problems is that they’re hard. Knowing I need to solve them, and actually solving them, are unfortunately not the same thing. I’ve done a lot of hand-waving around the subject, but my nails are not yet dirty.

The dip is coming. And, it’s going to suck.

And, once I’m in it, I know that I’ll look at some of the cushy corporate jobs that are out there and say “Do I really need this? Maybe the most prudent thing to do is to quit.”

But, I know that path is a cul-de-sac. I’ve been a middle manager in corporate America before. I don’t enjoy it. And, I don’t want to have to go back to it solely because I lack the fortitude to make it through the dip.

I hope I can make it through. We’ll know a lot more in the next few months. If you’d like, you should be able to follow along here.

This is a good book, and a really quick read. You should get it.

Alan interviewed
Alan is getting some incredible press with his Big Picture project. Here’s an interview by Andy Baio.

Tell the advertisers when their ads suck
This is one issue I’ve been beating for ages, but have never had the power to push through: There should be no ads on any page that aren’t immediately and obviously relevant to the person reading or the content being read. If advertising is actually reliably useful, then it’s content, not an irritation.

Don’t hash secrets
Ben Adida gives some good advice on how to secure usernames and passwords (among other things). I’ve just altered the spec…

Random thoughts about the Kindle
Seth Godin thinks about the Kindle. I’m very interested, because Serendeputy is targeted to women, and most of the tricks that work for geeks won’t be in play.

7 lessons from a product launch
Dharmesh Shah gives some good lessons on how to handle a product launch. I hope to be able to follow them myself (in a couple of months or so…)

How corporations constrain productivity
I set up my home office with the best tools I could afford. Life is so much better when you love your tools.

Murdoch has a plan. Zell doesn’t
Alan Mutter dissects recent newspaper purchases.

Patterns for designing a reputation system
Another good one from the Yahoo User Interface team. I’m wrestling with this in a couple of different contexts right now, so this is very timely.

It doesn’t have to be all or nothing with a start-up
David Hansson of 37 Signals writes (and I wholeheartedly agree) that running a start-up need not entirely take over your life. I’m spending a lot of time right now designing the application and the business model so that I can run the business in a reasonable way. It may not end up that way, but that’s where I’m steering.

Joel is unhappy about the terms and conditions a community site he was about to join imposed on him.

So I own the content, but they can do anything they want with it, even in forms that haven’t been invented yet. About the only concession that they grant to my ownership rights is that the license is non-exclusive. It says later that you can delete your profiles and that removes your content from the site, but they could still use it anywhere else, perpetually.

This is a hard one. I had to write a similar document for Life Times Voice back in the day, and you have a couple of conflicting issues.

First, if you want to maintain safe harbor under the DMCA, you need to be used just for communication. If you are taking copyright on the content, then you need to police it, and you’re responsible for any copyright infringement. If you don’t claim the copyright, you’re in the clear.

You need to balance that with the need to be able to reproduce it. The copyright owner has to give you explicit right to reproduce it. The rest of the clause is to protect you if you want to also use it in some unforseen format (podcasts of the best posts of the day, for example.)

I hope to be able to find a smart way to balance each of these needs in the new project. I want to do right by the customers (without crippling the business.)

Dan Dunn leaves HP
Dan’s golden hand with acquisitions continues.

Washington Post’s hyperlocal flop
The Wall Street Journal’s massive hit job on the Washington Post’s hyperlocal effort. Interesting things to think about for Boston.com’s hyperlocal project.

Cloud service architecture
The Amazon Web Services folks on the cloud services architecture.

Once upon a time, I was talking to SmugMug about working with Boston.com on a photo project. It didn’t end up happening, but I was very impressed at how much they had their act together and how they were actually making money. In 2004, this seemed a foreign concept to the other companies I contacted.

I’ve been following them ever since, especially as they’ve become a poster child for Amazon Web Services (AWS), the cloud-computing system I’m using for Serendeputy. Don MacAskill, the CEO, just wrote an outstanding piece explaining how SmugMug is using AWS to handle all their document processing.

SkyNet [SmugMug’s main controller] is completely autonomous - it operates with with zero human interaction, either watching or providing interactive guidance. No-one at SmugMug even pays attention to it anymore (and we haven’t for many months) since it operates so efficiently. (Yes, I realize that means it’s probably well on its way to world domination. Sorry in advance to everyone killed in the forthcoming man-machine war.)

Roughly once per minute, SkyNet makes an EC2 decision: launch instance(s), terminate instance(s), or sleep. It has a lot of inputs - it checks anywhere from 30-50 pieces of data to make an informed decision. One of the reasons for that is we have a variety of different jobs coming in, some of which (uploads) are semi-predictable. We know that lots of uploads come in every Sunday evening, for example, so we can begin our prediction model there. Other jobs, though, such as watermarking an entire gallery of 10,000 photos with a single click, aren’t predictable in a useful way, and we can only respond once the load hits the queue.

I’m architecting my systems in a similar way, trying to build everything out so that it’s as decoupled and asynchronous as possible. If I can fire up only the machines I want and only when I need them, then I can bootstrap the organization far longer than I could if I had to buy the equivalent physical machines. The experiments and prototypes I’m working on would be prohibitively expensive without AWS.

Although SmugMug isn’t using it, I’m using SQS for managing all the communications between these instances. Keeping it all in one system reduces my headaches. Now, I just need Amazon to incorporate CouchDB and I’ll be really able to roll. SimpleDB is a start, but it’s not really meeting what I need.

I look forward to hearing more from Mr. MacAskill and SmugMug as they continue to innovate with AWS — especially if I can pick up more architecture hints…

Welcome to Linky Goodness. I’ll do my best to link out to interesting things I find on the web. I should post this every day or two.

How eBay scales
Excellent article about how eBay manages their systems. Their work on shifting to asynchronous processes wherever possible matches my thinking.

Adventures in Office Space
Joel Spolsky writes in Inc. Magazine about his adventures getting new Manhattan office space for his company. I entirely agree with his policy of spending money to make the developers happy and productive. (Note: beware of Inc’s horribly annoying popups. Luckily I don’t see any of that crap in Safari).

The Big Picture
Alan Taylor’s new project on Boston.com, in which he tells news stories using big, beautiful pictures. It’s really quite compelling.

Rating the top 25 newspaper websites
An interesting overview of the top 25 newspaper websites. Hooray for NYTimes.com getting an A, but I think Boston.com deserves better than a C.

Hello everyone!

I’ve been writing about my life and thoughts since March 2001 over on my personal blog at JPButler.com. Since my daughters came along, my blog has become overrun with posts about bottles, diapers and other topics that endlessly fascinate my sisters, but are less interesting to my colleagues and the rest of the technical world.

So, I’ve decided to create a separate blog for my thoughts about product management, technology, business, media, and other assorted geekery. That way, my mother doesn’t have to suffer through search diatribes and my colleagues aren’t subjected to the tiniest advances of the little baby. Everybody wins!

I’d like to be able to share some of what I’ve learned, and force myself to clarify some of my thinking by putting it down in writing. We’ll see how it goes. I hope to be a good web citizen, and I hope to be able to contribute something to the conversation. You can always email me at Jason at Serendeputy.

What’s Serendeputy? It’s my new startup. I hope to be able to open up the private beta within the next few months.

Cheers!
Jason