Cosmological Inference Beyond the Two-Point Correlation Functions