Should I adjust the feed rate before the finish pass?

Yes – you should slightly decrease the feed rate before the finish pass.