Comment More Practical Suggestion (Score 2) 131
I already use my Android phone to do some light remote work. I use ConnectBot http://code.google.com/p/connectbot/ to SSH into a remote workstation. For graphical apps, I set up port forwarding for VNC (there's a menu option for it in connectbot) and use AndroidVNC http://code.google.com/p/android-vnc-viewer/
I have my VNC server set to only accept connections from localhost (and it's firewalled, too), so that only connections which are forwarded and encrypted via SSH wind up being accepted. This way I get secure remote access, the VNC protocol tends to be less bandwidth-intensive than raw X, and it preserves my session in case I get disconnected.
Don't get me wrong, an X server on Android is a cool technical achievement, but existing SSH and VNC clients for Android are a more practical and secure way of accessing your workstations/servers on the go.