Changing the velocity after noteon doesn't really seem to work on fluidsynth (may be dependent on soundfont file) - seems like the panning changes when velocity is changed. So I disabled that, after which everything works fine.