I use both CS4 and CS3 (different locations) . Honestly both programs are amazing but here are some comments. CS4 has many major improvements, but also some things worth noting.
The biggest thing to be aware of is what processor you are using. CS4 is really designed for Core 2 Duo, or Quad core processors. CS4 on a Pentium 4 or Pentium D is going to be rough. I found CS4 unusable on a Pentium D with 2GB of RAM. I now use Intel Quad cores and both run smooth, although I have to say CS3 runs much snappier.
I have not run either on AMD Athlon, Phenom, or Phenom II.
CS4 issues I have are:
1) When 3D acceleration is enabled, text typing and manipulation is slow.
2) General movement of text is slower, layer alignment is slower.
3) My OnOne plugins are now spread out. i.e. Focal Point is under Automate, Mask Pro is under Filters. I liked them collected as in CS3.
4) Web galleries have been removed.
5) Images are opened in the same tabbed window, this creates extra steps when working with multiple images. I like the old way better. Tabbed is great for browsing web pages, not for working with images.
CS4 goodness:
1) CS4 Bridge is awesome compared to CS3
2) Having ACR 5 is almost worth the cost. The Adjustment brush(es) in ACR 5 run flawlessly.
3) The whole look of CS4 is very refined and clean.
4) Technical improvements like using 64bit, and RAM usage. (this is big)
5) Huge 3D improvements.
Differences aside and depending on how and what you work with, and if you have the computing power, I say go for CS4. It is a great upgrade that utilizes new and improved computing architecture. I have some issues, but indeed as I remember, when CS3 came out, I had a hard time leaving CS2.