Category: Technology

Spolsky, Lookout and when Better is actually Worse

mike 7 Comments Lookout, Technology

Today, Joel on Software wrote about Outlook 2007’s recent search performance fixes, and mentioned Lookout.  I had a couple of thoughts about the article.

First – I’m so pleased to hear such nice things about Lookout when it hasn’t been updated for nearly 3 years 🙂  Thanks! 

Second, Joel wrote that “Microsoft finally put Lookout up for download…”.  He makes it sound like it was down for a long time; I think the total down period was between 4-7 days.  I know people like to think Microsoft buys products to squash them, and it’s easy to cite this downtime as an example of it.  But I know for a fact that was not true in this case.   I hope people won’t conclude that.

Third, while lots of people really liked Lookout, it did lack some key features which were put into Windows Desktop Search.  Most notable is that WDS actually indexes new & changed email in real-time.  Google Desktop Search does this too, but Lookout never did. 

But it brings up an interesting point which requires some background. Building incremental indexing in Outlook is quite tricky.  The problem is that changes can happen in your exchange mailbox while Outlook is not running.  When you next start Outlook, you need to make sure these changes are reflected in the index.  Because Outlook/Exchange don’t have APIs to do this efficiently, the only real way to do it is to start combing through folders trying to find what changes exist.  There are a whole bunch of nuances (cached mode, exchange mode, internet mode, online/offline, etc) and many APIs available (OOM, MAPI, CDO, ECS, etc).  When you try to make it work across the 4 different Outlooks (OL2000, OL-XP, OL2003, OL2007), each one presents a new set of problems.

So, Lookout punted on real-time indexing, and elected a “wake up every hour when the user is idle and index then.”  While this approach isn’t perfect, it is much much simpler.  And, it turns out that users usually don’t need to search for things which were sent to you less than an hour ago (although some did complain).  But the biggest benefit is that because it’s not trying to be real-time, it is much less prone to bogging down your system doing indexing.   And “system hogging” is a problem which both WDS and GDS took a long time to get right.  Reading reviews of these products indicates that even today, they still are pretty resource intensive.

In my opinion, this is a classic example of “Better is the Enemy of Good”.  The indexer doesn’t really need to be real time, even though it is “better” to be real-time.  If you weren’t confined by the limits of Outlook & MAPI, my thoughts on this would probably be different.  The fact that people are still talking about Lookout after both Microsoft and Google have continued to work on real-time indexers for 3 years is proof to me that in this case, better is actually worse.

Windows XP and the 1GB limit

mike 2 Comments Editorial, Technology

Every morning, when I come into work and login to my machine with 3GB of RAM running windows XP, I start waiting.  Each application takes 30-40 seconds (literally!) to “come back to life” as I try to use it.  Each one slowly swaps in.

What is going on?  I’m only using 1GB of ram in my applications, with 2GB of free memory.  I left the computer on, and idle.  There are no background file scanners, background indexers, or other software running on the machine.

The answer is the Windows XP 1GB limit.  It’s not in the documentation from Microsoft, but XP has a fundamental limit that it can’t use more than 1GB of RAM without swapping.  It was designed in 1998 when the average computer had about 128MB of RAM.  So, when memory use goes above 256MB of RAM, it automatically starts paging out, assuming you will soon run out of memory.  Unfortunately, it does this without checking how much free RAM you have, which is the cause of this slowness.

A second issue is the IBM large-memory DRAM patent.  The patent has a claim on software algorithms that concurrently access more than 1GB of data within any 24 hour period.  To avoid paying royalties on this patent, Windows intentionally tries to not use the additional RAM in your system.

A 3rd reason my machine is slow in the morning is because of Microsoft’s energy-saving memory technology.   It turns out that using fewer RAM slots in your system will significantly reduce the power consumption inside your PC.  So the software is optimized to try to contain all the in-use memory to the first RAM slot.  Unfortunately this leads to a fragmentation issue much like a disk that needs defragmenting; because it takes longer for the machine to lookup the larger RAM addresses than the lower ones.

Lastly, Microsoft tries to keep a significant amount of RAM free in case you try to run Microsoft Flight Simulator.  Flight Simulator, being an early app to consume large amounts of memory, acquired an early license from IBM on the RAM patent when it was cheap, and also is a showcase of Microsoft Technology.  So, it turns out that if you run Flight Simulator, it will startup extremely fast, and use the remaining RAM in my system.   Unfortunately, I don’t play flight simulator, so I can’t access that memory.

There are probably other causes of the Windows XP 1GB limit.   Anyone want to share?

The DMCA is Good. Mark Cuban is Wrong.

mike 2 Comments Editorial, Technology

Marc continues his anti-Google/Youtube arguments (see also, and also, and also).  He is so upset that his earlier predictions were wrong, that he can’t admit his error.  Overall, Mark claims that what Youtube is doing is illegal.  But, he’s talking through both sides of his mouth.

The fact is that Youtube is not breaking the law.  There is a law, called the DMCA, which specifically says that online service providers are not liable if their users upload copyrighted materials as long as the service provider promptly removes content when requested by the copyright holder.  This is exactly what Youtube does, and it is legal.  It’s the law.

You may not like the DMCA, but that is the law.  It’s legal, and there is no debate about it. 

Marc says that Youtube ought to be able to recognize which material is copyrighted as it is posted, and not allow the post to occur in the first place.  That might be doable, or that might not, (I don’t think it is realistic, and Marc has yet to propose an answer to that) but that is not relevant anyway.  The law has very clearly specified how service providers must deal with copyrighted content, and YouTube obeys the law.  

Marc knows this, stating that Youtube is “invoking Safe Harbor”.  There is no “invoking” here.  The law is clear about what service providers should do.  So how can he claim that this is illegal?  He obviously knows it is not.  He is just jealous or irrational or something.

Marc – if you don’t like the law, go write to your congressman.  Or better yet, get him a hooker and season tickets to the Mavs, and get him to change the law.  Keep in mind, however, that if you eliminate DMCA, you’ll not only take down YouTube, but you’ll take down MySpace, Facebook, Yahoo!, Hotmail, and hundreds of other sites.  The DMCA exists for a reason – and the DMCA allows most of us that don’t download or upload copyrighted  content still be able to benefit from a host of online services that otherwise couldn’t exist because they’d get sued to smithereens.

Pick Your Favorite RSS Reader

mike 0 Comment Technology

For quite some time, I used Rojo as my RSS Reader.  I still Rojo quite a lot.  But I did switch to Google’s Reader quite a while back, because it’s frankly just a far superior interface.

New stats from TechCrunch (via Feedburner) show that users agree.  While Google Reader is the newest entry into the market, it far surpasses Bloglines, Newsgator, Rojo, and Live.com for readership.  They published lots of other interesting stats too. 

I’m sure the competitors are going to copy Google Reader’s infinite page layout.  I wonder what the stats will look like in a year or two? 

Who Reads Your Code? Man or Machine?

mike 0 Comment Technology

I spent entirely too much time debugging the following today:

 

int foo(int num, int l) {
  int baz = 1;
  int bar = l;
  return bar + baz;
}

What gets returned for foo(2,2)?

If you guessed 2, you are dumb like me.  If you guessed 3, you noticed that the author of this broken code used a variable named “l”, and despite the fact that it looks like a “1”, it is not!

It’s easy to write code that machines understand.  The hard part about being a software engineer is writing code that both machine AND man can understand. 

Outlook Performance Tip – Word Editor

mike 1 Comment Technology

A little known thing about Outlook is that by default, when you edit a message, you are actually typing your message into Word rather than Outlook.

Starting Outlook using the default settings causes both Outlook and Word to be loaded at startup.  Editing messages through Outlook invokes Word as the editor.  At startup, Outlook uses 24MB of RAM and Word uses 18MB.

If you’ve got a system without a lot of RAM, I strongly recommend turning off Word as the editor.  Most people never use the Word features anyway.  Even without using Word as the editor, you can still format your messages with rich text- bold, italics, mixed fonts, etc. 

To turn off Word, go to Outlook’s Tools menu, select Options -> Mail Format, and uncheck “Use Microsoft Office Word 2003 to edit e-mail messages”. 

When you startup, you’ll find that Word is not invoked, you still have all the rich-text editing features, that Outlook still only consumes 24MB of RAM.  This can be a significant performance boost.

If any readers know of good reasons to use Word as Editor, it would be great to know! I’m sure there is a reason Microsoft uses this option by default, and I would like to know why too.

Here are images of Outlook 2003’s Compose Window in each mode.  As you can see, they look practically identical.  Quiz – which is Word?

Microsoft Visual Studio 2005 SP1 Troubles

mike 0 Comment Technology

I just went to download the new Service Pack for Visual Studio which was released about a month ago.  It’s pretty beefy – 430MB to download! 

After installing, I got this error:

 “The installation source for this product is not available.  Verify that the source exists and that you can access it.”

I tried it twice (it sits for about 10 minutes before displaying that error) and cursed a lot more, but finally found the solution which worked for me posted at ScottGu’s blog:

About the ”The installation source for this product is not available” problem – if you are you using Windows Server 2003:

If you extract the .exe file and see the properties on the .msp file under “Digital Signatures” you will see that the signature/certificate is not valid. (ScottGu: I presume this is a bug?).

To ”fix” (ignore) this:

1) Open local security setting: Start -> Programs -> Administrative Tools -> Local Security Policy.

2) Click on “Software Restriction Policies” and if it shows “No Software Restriction Policies Defined” then insert a new (right click the left pane node).

3) 2 x click on the “Enforcement” entry and under “Apply software restriction policies to the following users:” section change to “All users except local administrators”.

4) Maybe reboot

This should solve the problem (it did for me)..

/Anders

.NET 3.0 install time

mike 0 Comment Technology

Earlier I wrote about the speed of .NET2.0 installation.  On that same machine I just finished installing .NET2.0 from scratch, and then .NET3.0.

Turns out .NET3.0 is a little bigger than .NET2.0, it’s a 52MB download instead of .NET2.0’s 23MBs.

Install time is also a little slower:

    .NET2.0 download + install time:  8 minutes
    .NET3.0 download + install time: 26 minutes

This might make it hard to deploy commercial applications on Windows XP with .NET3.0.

Google and You

mike 0 Comment Editorial, Technology

If you live in the  bay area, and you are a software engineer, there are only two good choices about where to work.

Either you:
1) Start your own company and work for yourself
2) Work at Google

Google is a phenominal place to work.  The founders are engineers, and they care about engineers.  More importantly, they care about our users.  And despite their successes, they are not changing.  These guys want to make great software that delights users.  It’s so simple, and makes so much sense, and yet it is for some reason very unheard of.  Usually, after companies are successful, they move into “market domination”, sales and marketing mode.  Google hasn’t done that.  It just wants to keep doing simple stuff that delights users.  Is Google fast enough?  Nope.  Is it relevant enough?  Nope.

I’ve worked for lots of other companies.  Aside from HP and Microsoft, my past 10 years have been spent working at startups.  And, I’ve never been at a startup that failed (4-for-4, yeah!)  But, despite success at startups, there is no doubt in my mind that Google is better than all of them – combined.

Here is why you should work at Google.

1) Google trusts You.  Google hires smart people.  Smart people define their own jobs because smart people are smart enough to know what matters and what doesn’t.  

2) Google puts users first.  Users are first priority.  It’s not “winning”, it’s not making money, and it’s not ego that is top priority.  Just make users happy, and everything else will follow.  It’s absolutely true.

3) Innovation counts.  There has never been a company optimized to sustain innovation over the long haul.  This is what Google is doing, and it’s never been done before.  Do you like to come up with new ways of doing things?  Is it always possible to do something smarter?  Yes, while it is true that we stand on the shoulders of Giants – those who have paved the way for us – we can still do better.  That is the goal.  Better.

4) The people really are top notch.  No matter how good you are, prepare to be humbled.  The people at Google (not including me) are smarter than you.  I didn’t believe it either – but when you join, you’ll see it too.  On top of that, they probably work harder, and have more passion than you do too.  Think you can compete?  Maybe so.  Prepare for a challenge. 

5) There is a lot of room for growth in this company.  Google has had tremendous success, but there is a lot more coming.  I can’t say more than that, but I have faith.  Maybe I’ve had too much kool-aid, I don’t know, but I am a believer.

6) Google will continue to change the world.  The projects Google is tackling are unbelievable.  You think the environment doesn’t matter?  Our competitors may think so, but we don’t.  You think society can’t be better?  Our competitors are too focused on making money, but we aren’t.  You think technology is good enough?  We don’t, we think it can be a lot better.  If you believe that computers have a lot of room for improvement, you’re thinking like Google does.  We’re not satisfied with the way things are – we’re hope to make it just a little bit better.  We can’t do it alone.  We need help.  We need people that “think different” (to steal from IBM), and we need people that don’t like the status quo.  Microsoft isn’t worried about Google because of what it has done.  Microsoft is worried about Google because of what it is about to do.  But, we’re not interested in displacing Microsoft.  Microsoft is great.  We just want to build the stuff Microsoft can’t and that users love.

7) The benefits are to die for.  Nobody has benefits better than Google.  Whether it’s your salary, your bonus, your free food, they gym, the shuttle, or the people that want you to work on what you enjoy most, Google has the best benefits.  If you don’t believe me, ask Fortune magazine.

I’ve gone on long enough now that if you aren’t submitting your resume to Google right now, you must either not think much of your own skills or you must not think much of me.  That’s fine.  Please don’t interpret this as being arrogant.  Google is not about arrogance at all – it’s about humility, respect, and making great products.  So, if you are an engineer,  come work at Google.  My endorsement won’t get stronger than this.

Patent Office is Under-Performing

mike 1 Comment Editorial, Lawyers, Technology

Protecting Their Intellectual Assets: Appropriability Conditions and Why US Manufacturing Firms Patent (or not)

I read this paper by W Cohen, R Nelson, and J Walsh.  It was published in 2000.  It confirms a lot of things that we all know.  They present evidence that patents are used primarily for reasons other than what the Patent Office was created for, and that the Patent Office is probably not achieving it’s goals.

Patents are currently filed for two primary reasons:
   1) To block a competitor from getting the patent and suing you.
   2) So that if you get sued over a different, unrelated patent, you can counter-sue with an arsenal of equally unrelated patents.  

According to the USPTO, their purpose is to

“promote the progress of science and the useful arts by securing for limited times to inventors the exclusive right to their respective discoveries (Article 1, Section 8 of the United States Constitution)”

However, Cohen, Nelson &Walsh present compelling evidence that these days, companies rely heavily on secrecy and time-to-market as their primary competitive differentiation to advance science, and that patents offer little, if any additional “progress of science”.  Although the paper doesn’t show it, I postulate that patents actually hinder progress of science due to all the wasted resources that go to lawyers instead of R&D.  (e.g. Blackberry builds a phenomenal, billion dollar business, and lawyers, who produce & invent nothing, steal $600M).

The fact is that the patent office is no longer necessary.  Businesses don’t even value it.  While there may be some isolated cases where it helps individual inventors, by-and-large it is excessive and wasteful.  As a society, we would all benefit from more scientific progress if we just left inventions laissez-faire.