Several days ago, the primary server hosting all of the data comprising the php.net site for synchrony with all of the mirrors around the world became completely inaccessible. Due to security policies with the provider hosting the server, it was some time before we were able to have the machine returned to normal operational status. As a result, network content became stale, and automated tests on the mirrors saw them as outdated and deactivated them. It pointed out a flaw that, though this time was just an inconvenience, has the potential to grow into something more serious – including a sort of self-denial-of-service, if you will, if it went unnoticed for several days and all mirrors were seen as outdated.
Mark Scholten from Stream Service, an ISP based in the Netherlands and provider of an official mirror for their countrymen at nl.php.net, offered to put up a second rsync server, which gave me an idea: take the load off the primary server by distributing it across three regions.
(Click the image to view the full size version.)
Mark set up the European (EU) box in their Amsterdam datacenter, we (Parasane) had already set up an emergency rsync mirror in case the primary dropped out again which would be repurposed for the Americas (AA), and I contacted Chris Chan at CommuniLink in Hong Kong for what would become the Asia-Pacific (AP) region. Chris had submitted an application to the official waiting list to become an official PHP mirror back in February of 2010.
Compiling data over the course of the last 12 months from mirrors in our network which had it readily available, accurate, and up to date, I drew out a plan for the regions so as to limit the load and stress on each new mirror. Thus, the tri-colored map above. I also learned in the process that we will have served roughly 223 gigabytes of data over HTTP, network-wide, by the end of January, 2011, which averages out to about 1.9GB per mirror, per day, with the 115 active mirrors we have worldwide as of right now.
Setting myself an arbitrary date of 30 April, 2011, the goal is to have all existing official mirrors flipped over to using the rsync server designated for their country. Visitors to php.net should see no difference and should experience no negative impact during the transition, but the benefits will be great: far less of a likelihood of a mirror being automatically dropped from rotation due to stale content; the ability of the maintainer to decrease the amount of time to synchronize their mirror to hourly, providing the freshest content as it becomes available; less latency and greater speeds for many of those who are far from the current central rsync server; far, far less stress on our own network.
The immediate goal is ensuring that there are no snags, and that we can successfully synchronize all of the data to the website mirrors without omission. Beginning right away, I’ll be coordinating privately and directly with a few mirrors around the world to beta test the new layered design according to the rsync distribution plan. By 12 February of this year – a bit more than two weeks from now – I hope (and expect) to have all of the kinks straightened out. After that, we’ll begin migrating the rest of the network in its entirety to the new design.
All new mirrors from that point forward will be instructed to use their local rsync mirror as well, as defined by the map above.
It’s no large task, of course, but I’m hoping that the addition of just three new servers will help to ensure the health and stability of the network as a whole for years to come. While I don’t expect anyone to notice any difference – good or bad – in the user experience, behind the scenes I think we’ll not only see some differences in operations, but also begin to come up with even more ways to improve performance in the future.