To meet the increasing demand to expand wavelength division multiplexing (WDM) transmission capacity, ultrahigh spectral density coherent optical transmission employing multi-level modulation formats has attracted a lot of attention. In particular, ultrahigh multi-level quadrature amplitude modulation (QAM) has an enormous advantage as regards expanding the spectral efficiency to 10bit/s/Hz and even approaching the Shannon limit. We describe fundamental technologies for ultrahigh spectral density coherent QAM transmission and present experimental results on polarization-multiplexed 256 QAM coherent optical transmission using heterodyne and homodyne detection with a frequency-stabilized laser and an optical phase-locked loop technique. In this experiment, Raman amplifiers are newly adopted to decrease the signal power, which can reduce the fiber nonlinearity. As a result, the power penalty was reduced from 5.3 to 2.0dB. A 64Gbit/s data signal is successfully transmitted over 160km with an optical bandwidth of 5.4GHz.