Problem turned out to be with session variables. I have a session variable to set what country the user wants to see. Unfortunately the crawler does level 1 links, then level 2 links, etc, which doesn't retain the country session variable. (whereas a user would click on a level 1 link - sets country variable - level 2 link - same country variable - level 3 link gets reached with the appropriate country variable.)

Can I expect googles crawler to act similar to the sitemap crawler, or does it actually follow links in succession?

I hope this makes sense...?