F+
If you look at the drawings for this you will see an inner bushing (Spacer), inside og the bearings. The nut butts up to this which then again butts up to the casing. This way tou can tighten it up, but the swingarm still rotate free. So your nut might be ok tight and spacers tight up against the engine. If your needle rollers are knackered you can tighten this till it breaks but it will still be sluggish. Take a look at the drawings and you will se, or even better, take it apart.
You might have taken this too far allready, and as Taffy says them spacers will have ground into the engine...
Hope not !! Good luck,
Anders, DK