Chat Now!

Happy anniversary, PonyChat

This month marks the third anniversary of PonyChat. Three years ago two small groups of fans decided that it was time for a dedicated network for My Little Pony to be built that catered to fans of all kinds. In the beginning, both groups independently and without knowledge of the other started working on building the best network they could, and when they finally learned of each other it was decided that it would be in everyone’s best interest to merge, so that fans of MLP would not be split between networks.

Those networks called BronyChat and PonyIRC thus fused to become PonyChat. Meshing together the best of talents and dedication to the fan base a strong foundation was formed, a foundation which has overcome many obstacles and is still here today. When we began this journey most of us would have been happy for the network to survive one year, now we are confidently looking forward to our fourth year and hopefully many more.

It has been quite a journey and we can't wait to see what the future holds. From all the staff at PonyChat we thank you for sharing this journey with us, for sharing your good times and your bad, for showing us the pinnacle of talent and the depth of your friendship. We would not be here today if not for all of you.

Time Moves On and Things Change Forever

With a degree of sadness we must announce the discontinuation of three of our servers. Rainbowdash, Rarity, and Twilightsparkle. They were our original servers and the ones PonyIRC were founded on and which evolved into the PonyChat that you know today. Changes within staff have resulted in a strong reduction in budget for services. The three aforementioned servers are our most expensive and therefore must go.

As a result tonight around midnight there will be a net-split as these servers are removed from service. Those connected directly to one of these servers will need to choose a new server to connect to from now on.

Applejack, Fluttershy, and Pinkiepie will remain online and will be valid choices for direct connection should users wish. Users connecting via a more general address such as irc.ponychat.net will automatically be reconnected to a valid server after a brief disconnect.

PonyChat remains committed to bringing you the best pony and mlp themed IRC available anywhere, with the most professional and friendly staff. We see these events as needed to ensure this level of quality can continue. Do not worry as the remaining servers are more than enough to handle our users for the foreseeable future.

As a side issue these changes have resulted in the loss of our IRIS web-chat feature. We hope to have this resolved soon if possible.

From all of the staff we thank you for making PonyChat possible, for being awesome users, and for continuing to trust in us!

Season 3 Finale

The Princess Coronation. It's the episode we've all been waiting for — whether we were looking forward to it (I was!), or dreading it.

This season's finale is probably the most talked-about episode of the show ever. For weeks, it has seemed to be the only topic of discussion in many channels on PonyChat. Some foresaw the end of the fandom, while others saw this as the only logical next step. Whether or not it is well-received is yet to be seen.

One thing is for sure, though: it sure generated a lot of chat.

Statistics

Note: All times here are UTC.

During Friday and Saturday, we saw 10,229 unique IP addresses on the IRC network. During this period, our servers sent approximately 39 gigabytes of data to users. During the actual episode broadcast, our servers sent an estimated 14 gigabytes to users.

For reference, in January 2012, Wikipedia's English text totaled 9.7 gigabytes.

As with the previous finale, we saw greatly varied demographics.

Top 10 Countries

Full graph: PNG SVG TXT

A Side Note Regarding Network Capacity

Some of our larger communities generate many thousands of hits per second on our web site by embedded our web chat. In the past year, traffic on this portion of our web site increased to the point that it completely crushed our old web server. We spent weeks optimizing a memcache-based, reverse-proxied system to handle the load. That worked well, until we had 6,000 users online at once. Then, when someone posted an emoticon, our web server got 6,000 perfectly simultaneous requests. In one instance, memcached actually crashed, and the kernel logged some surprising TCP-related errors. To mitigate that, we have moved our entire web chat to Amazon's CloudFront content distribution network, which can handle many thousands of requests per second.

Another recent issue was an IRC server misconfiguration. Specifically, our IRC client servers had a file descriptor limit of 1,024. What that means is we could only accept just over 1,000 connections per server, even though each server has the computing and network capacity for a far greater number. This led to some really nasty-looking errors during an episode broadcast. We bumped that limit to 10240 users per server, and do not expect to see any further issues. In the event that we reach 61,440 users on the network, we will just add more servers. (Invite all your friends.)

The Map

Once again, I have generated a map containing approximate user locations.

IMPORTANT NOTE: The points on this map are very approximate and are nowhere near exact locations of users. Map markers are not guaranteed to be accurate, but they will generally be in the correct city or state. Even then, because of the portability of IP addresses, that may not even be correct. Therefore, you cannot use this map to locate any user's homes, neighborhoods, and in many cases, cities or states/provinces.

Also, this map is generated by your computer on load. There are thousands of unique map markers on here, so very old computers may run slowly or freeze while generating or viewing the map. Because of that, I have not embedded the map, but you may view it by clicking the link below.

Map Preview

Click here to view the map (SSL).

Web Site Changes — New Features, Faster Service

We have just completed the switch to our new web site!

There were some changes under the hood, and some changes to the front-end. Here is a run-down of what was done:

  • Front-End
    • HTML and CSS now valid.
    • Graphics were tuned for size and load speed.
    • Donations button is finally live.
    • Channel List Page
    • Channel Detail/Statistics Page
    • User Detail Page
    • Network Ban and Blacklist Checker Page
    • Web Chat Embed Creator Page
  • Back-End
    • No more WordPress! We're now powered by WolfCMS.
    • No more memcached. Caching is now to static gzipped files.
    • Various performance improvements in non-cached pages.

More Information

While there are no substantial changes to our appearance, we did clean up all of the old HTML and CSS. This will improve interoperability and make it simpler for us to fix the site when things are acting up.

The change from WordPress was definitely the biggest change. Our previous web designer chose that platform not for its strengths, but for familiarity so that he could more quickly and effectively launch and maintain the web site. Since his departure, none of the remaining staff have wanted to attempt to work with such an awful piece of software, so it was left largely untouched.

Some quick facts:

  • When we were still hosting on DreamHost, WordPress took between 2 and 9 seconds to load a web page. This was partially due to the slow MySQL server provided by that host.
  • After moving to our own infrastructure, page load times were down to 0.5 to 2.5 seconds.
  • Full-page caching with memcached further improved that to 60 milliseconds load time under normal load.

We have dropped MySQL entirely, now, and are using sqlite3. Even without caching, simple pages load (request to complete render) in well under 100 milliseconds.

Caching

The memcached solution was not a permanent one for us. A correct deployment of any daemon-based caching requires a more advanced configuration than we have. When an episode is airing, our web site and web chat embed gets many hundreds — sometimes thousands — of hits per second. Our web server can handle this, but the number of memcached instances we can currently run was only able to handle about 1,500 hits per second before some page loads would fail. We mitigated this by putting our high-traffic sites on custom, non-memcached embeds.

The new web site has been configured such that pages are immediately committed to disk and compressed. When a client attempts to load a page, the HTTP server sends the compressed file (if the browser supports compression). Compressing files saves disk space, decreases the time it takes to read it from disk, and decreases page load time for users. When a page is accessed frequently, it is cached in memory and can be immediately transmitted many thousands of times per second.

If you want more information about this change, stop by #geek and ask me (Kabaka).

Insider Information: Network History and Staff

Lately, I've noticed a trend: People are curious about who "owns" PonyChat, who pays the bills, and what roles everyone plays. There are some common misconceptions that I'd like to clear up, as well as some general information on this topic I'd like to share.

There is a lot to tell on this topic, but I'll try to keep it all fairly succinct.

This will also be the first post in a series of behind-the-scenes write-ups I'll be doing. Next time, I'm planning on doing an illustrated (with ponies!) post on the technical aspects of the IRC network and web site.

History

Originally, two separate groups of people set out to create an IRC network for My Little Pony fans. One group created BronyChat, and the other created PonyIRC.

Both networks existed for several weeks, basically unaware of each other, each marketing their chat service to various web sites and building a strong user base. Hosting the chat for streaming communities was a good, early boost to population. My perspective on this was a bit limited. (I'm Kabaka, if anyone missed the author on this post.) I wasn't there for the founding of either network.

I've been running or helping run IRC networks for years, and I've been an IRC user for over a decade. When I first got into My Little Pony, I immediately considered starting an IRC network dedicated to it. Instead, I opted to run a #mylittlepony channel on one network.

After a few weeks, it became clear that the network didn't have sufficient population to sustain a pony channel. I was aware of pony channels on other networks, but even those felt insufficient for the needs of the community — especially with the allegedly anti-pony staff on those networks.

So, after a quick Google search, I arrived on BronyChat. After chatting with the owners, I joined the team and started to contribute in what ways I could. A few days later, we began merge talks with PonyIRC. It only took a day or two for us to agree on the basics and begin building a brand new IRC network. The network that was created at this point is the one we have today.

Staff

One of the things discussed during the merge talks was staff positions and ranks: no member of staff is different from any other. There are no leaders, no subordinates, and no individual owner.

From the start, I have often seen users asking "who owns PonyChat?" Normally, people answer "Kabaka" or "Cassy." While Cassy and I are often the driving force behind changes, we don't really own PonyChat. Sometimes, the answer is "DJMidgetBrony," which is slightly more correct since he invests the most money in the network every month. Everyone holds different pieces of the whole thing, but none of us feel the need to declare ownership.

I believe the origin of this misconception is that Cassy and I have the most technical knowledge regarding the IRC network and its servers. We quickly jump on support questions, and questions that require a more technical answer are often deferred to us, giving us the appearance of being in charge.

Since I have been so obsessively IRCing for so long, a lot of it usually winds up on my plate, specifically, which is likely why everyone says it is my network. Even some of the staff say they often think of me as being in charge, probably because I am constantly working to get things done. (I don't mean to imply that others are idle, just that I have no life.)

In reality, anyone on staff can propose something or take immediate action on critical events, and everyone on staff is very involved in making decisions. Aside from things that need fast responses, we discuss every tiny thing at least a little.

Of course, over time, we have all developed habits or found things that work for us. There still isn't a true hierarchy. There have been questions about this, and some erroneous assumptions, so I'll try to clear that up now by explaining what each of us does.

Before reading this list, it is important to realize that all of us have the same basic set of responsibilities which we take seriously and handle any time we're on-line: end-user technical support, network security (flood protection, account security, and attack monitoring), and service stability monitoring.

IRC Nickname Original Team About
aji PonyChat The most recent addition to the team, aji is learning the technical aspects of operating the network and should soon be one of those responsible for server maintenance.
Awesomeshy BronyChat Awesomeshy contributes applejack.ponychat.net. At the time of writing, he is currently on an extended hiatus for off-line matters.
bicyclerepairman PonyChat Recently brought on to fill a gap in our staff coverage, he is one of our most active chatters, and a very fast learner.
Cassy PonyIRC Cassy was PonyIRC's technical expert, and continues to use his skills to help PonyChat. He manages the servers themselves.
MakerDusk PonyIRC Our resident drama wrangler and political expert. Without him, the network would likely come apart at the seams and then burst into flames.
DJMidgetBrony PonyIRC If asked what he does, he would say that he throws money at servers. MidgetBrony contributes celestia.ponychat.net (back-end server), rainbowdash.ponychat.net, rarity.ponychat.net, and twilightsparkle.ponychat.net.
Kabaka BronyChat Along with Cassy, Kabaka manages the network's servers. He also maintains and updates the web site. Kabaka contributes fluttershy.ponychat.net, luna.ponychat.net (back-end server), and pinkiepie.ponychat.net.
klaxa PonyIRC klaxa helps out with a little of everything wherever he can. He's tech-savvy and a fast learner.
Lyude PonyIRC Lyude focuses on on-IRC support and management, but has Linux knowledge enabling him to assist with server management.
Xiodine FlutterNET Xiodine arrived after a merge with his small IRC network, FlutterNET. Though he is most often busy with PonySquare, he still actively helps with PonyChat throughout the week.

If you want to reach staff, you can do so by emailing staff@ponychat.net. You can find individual email addresses by clicking on names on the staff page. You can always find us in #help or #ponychat on the IRC network.

Former Staff

Along the way, people come and go. It is worth mentioning everyone that has been a part of the team. Without them, we wouldn't be what we are today.

IRC Nickname Original Team About
haymaker PonyIRC haymaker was one of the creative users on the original team and provided help with growth and design.
Pontang BronyChat Pontang pushed us to grow and expand, helping to get us the growth we needed to sustain a useful network.
Purist PonyIRC One of our former technical experts. Purist helped create and manage PonyIRC.
SunshineSmiles PonyIRC SunshineSmiles did some of the early work getting our name out there so that we'd actually have some users.
zahqo PonyIRC zahqo (originally known as zaco) is the brilliant graphic designer that originally created this web site.

RSS Feed Valid HTML 4.01 Strict Valid CSS! Valid RSS Feed