Abstract

We demonstrated reduction of quantum fluctuations by creating macroscopic entanglement in an ensemble of 7 × 10^5 rubidium 87 atoms via optical quantum non-demolition measurements. Intrinsic quantum fluctuations of a coherent spin state were reduced by 2.9 dB. The metrological precision of a broad-band magnetometer was improved beyond the standard-quantum limit by 2.0 dB. The major steps toward this first successful realization of spin squeezing in a magnetically sensitive system with MHz-bandwidth are discussed. First, a magnetic field imaging technique was developed to monitor the magnetic field environment of the atomic ensemble. The technique has sensitivities down to sub-microGauss at 100um spatial resolution. It reveals intrinsically both the magnitude and the direction of the magnetic field. Second, we establish a technique to overcome fundamental limitations in the dispersive spin measurement in multi-level atoms, caused by tensorial light shifts. Third, we use a maximally mixed state as noise reference to calibrate the quantum non-demolition measurement reliably.