I think any next token predictor will benefit. Iiuc mamba is a next token predictor.
I just skimmed the gradient article, but if their only change is swapping out the transformer block for the mamba block, I don't think it's already using this optimization
I think any next token predictor will benefit. Iiuc mamba is a next token predictor.
I just skimmed the gradient article, but if their only change is swapping out the transformer block for the mamba block, I don't think it's already using this optimization