Perhaps the idea of two seperate queues might be worth exploring. One for the intro page with the most popular 5-10 servers (shmack, bigfoot, flanders, whatever the most popular CA/RA servers are, etc) which runs more frequently and one for the much larger server browser which could run seperately but less frequently (perhaps every 10-15mins) or maybe only on request/refresh?
If you are using Qstat it should support multiple instances and it certainly supports on request polling from a browser as I have coded a page which does exactly this previously.
Just my humble suggestion