I went Indie!

So I did it – I took the leap – I quit my day job at webMethods doing design, architecture and Java programming. It was a wild ride as I started as employee 80 (+/- 1), worked my butt off, experienced the thrill of IPO at the height of the tech boom and the period now known as the tech bust. I left behind a bunch of friends but I think it was time that I ventured out on my own.

So what am I doing with my time now? A few things actually. I’m doing Ruby on Rails consulting. I’m working on my Cocoa apps, specifically WebnoteHappy and hopefully other apps in the future. Plus some audio editing. And I’m still open to doing Java as well. All this while working for my own company, Happy Apps LLC.

Basically I’ve taken all my passions that I did in my spare time and figured out a way to do all my passions during the day instead of cramming them into the few hours I have a night before I crash.

Plus I’ve thought more about what’s important to me: along with doing what I like to do, I also want to spend time with my kids – especially since they’re all under 7. I think that’s a special time and so I’m striving to only do remote consulting and product development.

Urs Hoelzle – A look behind the scenes at Google – liveblog

8:10AM – Google’s mission is much bigger than search. It is to make the world’s information available in a usable and accessible way. That’s a huge mission.

Urs is going to talk about Google scales its infrastructure.

Google uses commodity PCs, because you get more for your money. However, you need to figure out how to make them work together.

3 pictures on how the data centers have changed from 1998 (just random desktops), 1999 (bare motherboards in a rack, pretty randomly wired), 2000 (blades in racks), and now they’re 2U high.

They want to make each computer like an iMac, with only 2 cables: power and ethernet.

8:30AM – Google has so many computers that they figured out that each computer lasts on average 3 years and that if you have 1000 computers, that 1 will fail every day.

They solve the expected 1/1000 computer failures in an automated way. I’m a big believer in automation. Its better to make computers for you, rather than the other way around. Google solves this via software and with replication and redundancy. This solution also helps with serving capacity and performance.

Fault tolerant software makes cheap hardware practical.

He goes over how Google builds the Search index via the Page Rank. The index is split up into chunks called shards and then distributed among servers. There’s replication of shards between some of those servers. I think most of this is documented elsewhere, but I could be wrong.

Google started off doing it from scratch and then realized it could reuse some core components among its different types of webapps.

One of those is the Google File System (GFS). There’s a Master that manages metadata, which points to (typically) 64MB chunks, which are triplicated (is that a word?) across three Chunk Servers for reliability. Clients talk to the Chunk Servers directly (after consulting the Master) to transfer data. Hmm… I bet the masters are triplicated too. Note that the triplication also helps with throughput, so that if many people are accessing a blog, you can load balance them among the different Chunk Servers. There’s more info from a paper from SOSP 2003.

At this point, I’m wondering… I wonder if Virginia Tech does things the same way. They’re not using the cheapest hardware, but instead are using Apple XServe G5s. So I would expect those to be more reliable. That must affect their software strategy, but how much?

8:45AM – MapReduce is a parallel functional programming framework that they use at Google, which basically takes in key/value pairs, maps them, and then reduces them. It works for a certain class of problems. He showed a C++ example. That’s not too bad since we’ve got the CDT and many Java programmers have a C++ background, but it seems like that might get booed at JavaOne.

Scheduling happens with one Master, many Workers. Master assigns either a Map task or a Reduction task to each worker.

Urs comparing MapReduce to Eclipse: If you create a generally useful tool, you will find that there are many people that will adopt it.

You can read more about MapReduce at Google Labs.

9:00AM – Urs pulls out some amusing slides like the searches for “eclipse”, with spikes every time there is a solar (or maybe lunar) eclipse. Also another slide with the many ways that people try to type in “Britney Spears” while searching for it. (BTW, Google is a nice adhoc spell checker/dictionary.)

Google has a way to train computers to correlate words by putting many many workers over many CPU years at a massive amount of data to produce clusters. He showed a demo which showed us that eclipse is about 80% correlated with the car produced by Mitsubishi. (I wrote about this problem early on in this blog… “The Problem with Common Naming”.) It makes more sense now why Google Mail can be accessed via POP without having ads inserted, because the value to Google is not just the advertising, but the additional data that is already in context.

Q: How do you handle changes to a running system?
A: You don’t. Take a replica that is not being used out of service and use it for testing. Then if it is good, then repeat with more and more replicas.

Q: How long does it take to index the web?
A: Its complicated because visiting each site every day would actually be bad because of bandwidth usage (even though they could). This is nice to hear that they are being considerate of bandwidth. Brent Simmons is the same way and has designed NetNewsWire appropriately.

Xbox Power Cords recalled

Maybe it’s time to buy the kids that Nintendo Gamecube they’ve always wanted. Then again, they’ve been enjoying the Sega Dreamcast I unpacked recently. ;)

Microsoft says that you need to check your manufacture date to see if it is before October 23, 2003 (except in Continental Europe, which the faulty power cord was used until January 13, 2004). If so, then order a replacement.

[Via News.com]:

The company announced the recall on Thursday, saying it is aimed at protecting the consoles from electrical-component failures that can pose a fire hazard. So far, such failures have been reported in 30 consoles, causing minor injury or property damage, Microsoft said. Seven customers reported burns to their hands. In the rest of the cases, the defect caused smoke damage or minor damage to a carpet or entertainment center.

In Europe, consoles made before Jan. 13, 2004, will need replacement cords, the company said. In all other areas, the recall covers consoles manufactured before Oct. 23, 2003.

Orders for replacement cords can be placed at the Xbox Web site. New cords will take up to four weeks to arrive. Until the new cord comes, customers should turn off consoles when not in use, the company said.

Technorati tags:

43 Things

43 Things is a cool site that lets you keep track of your goals, share them with others, see what other people’s goals are and how they’re doing on them. I guess you could say it is “Social Goal Setting (and hopefully Reaching)”.

It’s built on RubyOnRails and it has really come a long way from the initial prototypes that I saw. It’s got an interesting tagging system that also links to Flickr and Delicious.

I personally like the way that it gives you suggestions of goals that are related to whatever goal you’re on right now. Even better is that the more popular the goal is, the bigger its font is, so you can see at a glance what other people are doing.

You can check out my 43 Things (well, haven’t quite gotten to 43 yet) at: http://www.43things.com/people/view/luis. You can also subscribe to the RSS Feed.

Some cool things about the guys behind 43 Things, which is The Robot Co-op: they’ve all got Macs. Check out the group picture on their blog. Looks like a good set up for agile development: everyone w/a PowerBook all sitting around a big conference table. (Well, except for one guy who seems to be reading some books – I’m guessing he uses the iMac in the corner?)

Oh yeah – and these guys get paid to code in Ruby. (Yes, contrary to my prediction, there are some Ruby jobs out there.)

Oh and they’ve got some funding from Amazon. So they’ll probably be around while we’re waiting to finish some of our initial goals. ;)

Get the weather via RSS

Light Snow 34°F

It’s snowing here in Northern Virginia (or it was just a little while ago). How do I know? Well, I guess I could look out the window, but sometimes you’re deep into your code. So here’s a way to get the weather without looking up from your computer.

It’s in beta (but hey, lots of cool things are) and it’s called RSS Weather, available from http://www.rssweather.com.

It’s very convenient, at least if you’ve got a news reader like NetNewsWire. If so, you’ve got weather coming to you instead of having to sit through forecasts from around the country on The Weather Channel (which I think is the most watched cable channel). The forecasts seems reasonably accurate, but it could be updated more often and it could be nicer looking. Well, its in beta, so I’ll give it the benefit of the doubt.

A really nice touch is that you can get localized weather down to the zip code. Once you get to the web site, enter your zip code. Then you will see a page that shows your localized weather. There you’ll see the familiar RSS “XML” graphic that links to the feed.

This reminds me of PointCast, circa 1995. That is, there are some things that are useful to be “pushed” to you. Only this time, network admins aren’t screaming at you that you’re clogging up the network. Ahh, the wonders of RSS (and Atom and RDF.)

BTW, the National Weather Service also has RSS feeds, but it seems to be limited to those extreme weather alerts, which also are for huge regions/states.

Expanding to TextDrive

This blog is going to stay here at TypePad, which is a great blog host. I haven’t had to worry about comment spam or upgrading to the latest Movable Type or other admin tasks. Instead, I’ve been able to concentrate on the content.

On the other hand, I’ve wanted to expand my horizons on the web beyond just blogging.

So I’ve chosen a host for my non-blogging activities: TextDrive. Its quite a nice one, especially from the specs. An interesting part of it is that you can choose to support an open source project with 50% of the profits! I’m choosing to support RubyOnRails.