I am using Excel 2003 and the built-in IDE for programming VBA, but a lot of things bug me. Obviously, the built-in nature of the VBA IDE is a big plus. Is there a (free?) alternative out there?
Do you need to code in VBA? You can code in C#. Here's a post from Joel's Software Discuss board.
However the C# I write is not embedded in Excel. It write it as a class library in Visual Studio .NET.
If you want embedded C# I suggested you buy Visual Studio Tools for Office. This makes life a bit easier for you as you don't have to learn how to write C# as a COM Interop component.
Maybe you should check out Office Development with Visual Studio.