Which one is more important: more parameters or more computation? (2021)

ParlAI research shows that model parameters and computation can be decoupled in deep learning. Hash layers use simple hashing-based mixture-of-experts routing to increase model capacity without additional computation, while staircase attention stacks Transformers with shifted alignment to increase computation without adding parameters. The approaches show orthogonal improvements when combined.