Seems like there are dozens (or hundreds) of travel meta-search engines like fly.com, Kayak, TravelPost, etc.
I can't imagine any of the "big" sites like Expedia, Travelocity, etc allowing their content or searches to be indexed and stored like that. Yet, it happens.
Do most of these travel sites have deals with these companies? What are the rules about that? I know about following the
robots.txt file but I would guess many of the big sites have theirs so that you can't search their site.
Thanks for any clues.