Was the spider eating your bandwidth or something? Otherwise I can't see a reason personally.

Just wondering what the reasoning was behind it :)

Wish I could still see some of my old HUB entries that were deleted without warning several years ago.

I think it had something to do with the Wayback mahine indexing and archiving every hub page BYOND has. Which is a lot of pages. And I believe it also does it with forum posts. And that's even more pages it has to download.

I think it was a bandwidth issue associated with the aforementioned. It was brought up a while back, anyway.
BYOND blocked ALL spider bots, this includes Google and whatnot too I believe. Mostly for bandwidth reasons, partially because it was indexing things it shouldn't have.
In response to Nadrew
Can't robots.txt be set up to only block specific directories?