I found myself pretty darned frustrated when my arrow keys didn't work in iTerm in vi/vim or other ncurses-based applications. However, give this a shot in an iTerm if you find yourself in the same predicament:
export TERM=linux
Then open something in vi/vim or run an ncurses application. It should let your arrow keys work normally now. To make the setting stick, just do this:
echo "TERM=linux" >> ~/.profile

You can also do set term=linux in your .vimrc I found export TERM=linux changed the default syntax highlighting in vi to something I didn't like