Engineering A Noise-Free Quantum Memory For Temporal Mode Manipulation