The Princess Coronation. It's the episode we've all been waiting for — whether we were looking forward to it (I was!), or dreading it.

This season's finale is probably the most talked-about episode of the show ever. For weeks, it has seemed to be the only topic of discussion in many channels on PonyChat. Some foresaw the end of the fandom, while others saw this as the only logical next step. Whether or not it is well-received is yet to be seen.

One thing is for sure, though: it sure generated a lot of chat.

Statistics

Note: All times here are UTC.

During Friday and Saturday, we saw 10,229 unique IP addresses on the IRC network. During this period, our servers sent approximately 39 gigabytes of data to users. During the actual episode broadcast, our servers sent an estimated 14 gigabytes to users.

For reference, in January 2012, Wikipedia's English text totaled 9.7 gigabytes.

As with the previous finale, we saw greatly varied demographics.

Full graph: PNG SVG TXT

A Side Note Regarding Network Capacity

Some of our larger communities generate many thousands of hits per second on our web site by embedded our web chat. In the past year, traffic on this portion of our web site increased to the point that it completely crushed our old web server. We spent weeks optimizing a memcache-based, reverse-proxied system to handle the load. That worked well, until we had 6,000 users online at once. Then, when someone posted an emoticon, our web server got 6,000 perfectly simultaneous requests. In one instance, memcached actually crashed, and the kernel logged some surprising TCP-related errors. To mitigate that, we have moved our entire web chat to Amazon's CloudFront content distribution network, which can handle many thousands of requests per second.

Another recent issue was an IRC server misconfiguration. Specifically, our IRC client servers had a file descriptor limit of 1,024. What that means is we could only accept just over 1,000 connections per server, even though each server has the computing and network capacity for a far greater number. This led to some really nasty-looking errors during an episode broadcast. We bumped that limit to 10240 users per server, and do not expect to see any further issues. In the event that we reach 61,440 users on the network, we will just add more servers. (Invite all your friends.)

The Map

Once again, I have generated a map containing approximate user locations.

IMPORTANT NOTE: The points on this map are very approximate and are nowhere near exact locations of users. Map markers are not guaranteed to be accurate, but they will generally be in the correct city or state. Even then, because of the portability of IP addresses, that may not even be correct. Therefore, you cannot use this map to locate any user's homes, neighborhoods, and in many cases, cities or states/provinces.

Also, this map is generated by your computer on load. There are thousands of unique map markers on here, so very old computers may run slowly or freeze while generating or viewing the map. Because of that, I have not embedded the map, but you may view it by clicking the link below.

Click here to view the map (SSL).

We have just completed the switch to our new web site!

There were some changes under the hood, and some changes to the front-end. Here is a run-down of what was done:

• Front-End
  • HTML and CSS now valid.
  • Graphics were tuned for size and load speed.
  • Donations button is finally live.
  • Channel List Page
  • Channel Detail/Statistics Page
  • User Detail Page
  • Network Ban and Blacklist Checker Page
  • Web Chat Embed Creator Page
• Back-End
  • No more WordPress! We're now powered by WolfCMS.
  • No more memcached. Caching is now to static gzipped files.
  • Various performance improvements in non-cached pages.

More Information

While there are no substantial changes to our appearance, we did clean up all of the old HTML and CSS. This will improve interoperability and make it simpler for us to fix the site when things are acting up.

The change from WordPress was definitely the biggest change. Our previous web designer chose that platform not for its strengths, but for familiarity so that he could more quickly and effectively launch and maintain the web site. Since his departure, none of the remaining staff have wanted to attempt to work with such an awful piece of software, so it was left largely untouched.

Some quick facts:

• When we were still hosting on DreamHost, WordPress took between 2 and 9 seconds to load a web page. This was partially due to the slow MySQL server provided by that host.
• After moving to our own infrastructure, page load times were down to 0.5 to 2.5 seconds.
• Full-page caching with memcached further improved that to 60 milliseconds load time under normal load.

We have dropped MySQL entirely, now, and are using sqlite3. Even without caching, simple pages load (request to complete render) in well under 100 milliseconds.

Caching

The memcached solution was not a permanent one for us. A correct deployment of any daemon-based caching requires a more advanced configuration than we have. When an episode is airing, our web site and web chat embed gets many hundreds — sometimes thousands — of hits per second. Our web server can handle this, but the number of memcached instances we can currently run was only able to handle about 1,500 hits per second before some page loads would fail. We mitigated this by putting our high-traffic sites on custom, non-memcached embeds.

The new web site has been configured such that pages are immediately committed to disk and compressed. When a client attempts to load a page, the HTTP server sends the compressed file (if the browser supports compression). Compressing files saves disk space, decreases the time it takes to read it from disk, and decreases page load time for users. When a page is accessed frequently, it is cached in memory and can be immediately transmitted many thousands of times per second.

If you want more information about this change, stop by #geek and ask me (Kabaka).

Lately, I've noticed a trend: People are curious about who "owns" PonyChat, who pays the bills, and what roles everyone plays. There are some common misconceptions that I'd like to clear up, as well as some general information on this topic I'd like to share.

There is a lot to tell on this topic, but I'll try to keep it all fairly succinct.

This will also be the first post in a series of behind-the-scenes write-ups I'll be doing. Next time, I'm planning on doing an illustrated (with ponies!) post on the technical aspects of the IRC network and web site.

History

Originally, two separate groups of people set out to create an IRC network for My Little Pony fans. One group created BronyChat, and the other created PonyIRC.

Both networks existed for several weeks, basically unaware of each other, each marketing their chat service to various web sites and building a strong user base. Hosting the chat for streaming communities was a good, early boost to population. My perspective on this was a bit limited. (I'm Kabaka, if anyone missed the author on this post.) I wasn't there for the founding of either network.

I've been running or helping run IRC networks for years, and I've been an IRC user for over a decade. When I first got into My Little Pony, I immediately considered starting an IRC network dedicated to it. Instead, I opted to run a #mylittlepony channel on one network.

After a few weeks, it became clear that the network didn't have sufficient population to sustain a pony channel. I was aware of pony channels on other networks, but even those felt insufficient for the needs of the community — especially with the allegedly anti-pony staff on those networks.

So, after a quick Google search, I arrived on BronyChat. After chatting with the owners, I joined the team and started to contribute in what ways I could. A few days later, we began merge talks with PonyIRC. It only took a day or two for us to agree on the basics and begin building a brand new IRC network. The network that was created at this point is the one we have today.

Staff

One of the things discussed during the merge talks was staff positions and ranks: no member of staff is different from any other. There are no leaders, no subordinates, and no individual owner.

From the start, I have often seen users asking "who owns PonyChat?" Normally, people answer "Kabaka" or "Cassy." While Cassy and I are often the driving force behind changes, we don't really own PonyChat. Sometimes, the answer is "DJMidgetBrony," which is slightly more correct since he invests the most money in the network every month. Everyone holds different pieces of the whole thing, but none of us feel the need to declare ownership.

I believe the origin of this misconception is that Cassy and I have the most technical knowledge regarding the IRC network and its servers. We quickly jump on support questions, and questions that require a more technical answer are often deferred to us, giving us the appearance of being in charge.

Since I have been so obsessively IRCing for so long, a lot of it usually winds up on my plate, specifically, which is likely why everyone says it is my network. Even some of the staff say they often think of me as being in charge, probably because I am constantly working to get things done. (I don't mean to imply that others are idle, just that I have no life.)

In reality, anyone on staff can propose something or take immediate action on critical events, and everyone on staff is very involved in making decisions. Aside from things that need fast responses, we discuss every tiny thing at least a little.

Of course, over time, we have all developed habits or found things that work for us. There still isn't a true hierarchy. There have been questions about this, and some erroneous assumptions, so I'll try to clear that up now by explaining what each of us does.

Before reading this list, it is important to realize that all of us have the same basic set of responsibilities which we take seriously and handle any time we're on-line: end-user technical support, network security (flood protection, account security, and attack monitoring), and service stability monitoring.

IRC Nickname	Original Team	About
aji	PonyChat	The most recent addition to the team, aji is learning the technical aspects of operating the network and should soon be one of those responsible for server maintenance.
Awesomeshy	BronyChat	Awesomeshy contributes applejack.ponychat.net. At the time of writing, he is currently on an extended hiatus for off-line matters.
bicyclerepairman	PonyChat	Recently brought on to fill a gap in our staff coverage, he is one of our most active chatters, and a very fast learner.
Cassy	PonyIRC	Cassy was PonyIRC's technical expert, and continues to use his skills to help PonyChat. He manages the servers themselves.
MakerDusk	PonyIRC	Our resident drama wrangler and political expert. Without him, the network would likely come apart at the seams and then burst into flames.
DJMidgetBrony	PonyIRC	If asked what he does, he would say that he throws money at servers. MidgetBrony contributes celestia.ponychat.net (back-end server), rainbowdash.ponychat.net, rarity.ponychat.net, and twilightsparkle.ponychat.net.
Kabaka	BronyChat	Along with Cassy, Kabaka manages the network's servers. He also maintains and updates the web site. Kabaka contributes fluttershy.ponychat.net, luna.ponychat.net (back-end server), and pinkiepie.ponychat.net.
klaxa	PonyIRC	klaxa helps out with a little of everything wherever he can. He's tech-savvy and a fast learner.
Lyude	PonyIRC	Lyude focuses on on-IRC support and management, but has Linux knowledge enabling him to assist with server management.
Xiodine	FlutterNET	Xiodine arrived after a merge with his small IRC network, FlutterNET. Though he is most often busy with PonySquare, he still actively helps with PonyChat throughout the week.

If you want to reach staff, you can do so by emailing staff@ponychat.net. You can find individual email addresses by clicking on names on the staff page. You can always find us in #help or #ponychat on the IRC network.

Former Staff

Along the way, people come and go. It is worth mentioning everyone that has been a part of the team. Without them, we wouldn't be what we are today.

IRC Nickname	Original Team	About
haymaker	PonyIRC	haymaker was one of the creative users on the original team and provided help with growth and design.
Pontang	BronyChat	Pontang pushed us to grow and expand, helping to get us the growth we needed to sustain a useful network.
Purist	PonyIRC	One of our former technical experts. Purist helped create and manage PonyIRC.
SunshineSmiles	PonyIRC	SunshineSmiles did some of the early work getting our name out there so that we'd actually have some users.
zahqo	PonyIRC	zahqo (originally known as zaco) is the brilliant graphic designer that originally created this web site.

Season two is over! If you haven't watched the finale yet, you are missing out. It was amazing.

And it was PonyChat's biggest day yet, too. We reached a new record number of clients on the network: 4,607. We hit that right before the episode began.

Because of the way we currently collect statistics, this graph is somewhat approximate, which is why it does not reflect the new record.

The drop-off right when the episode began does not seem to have been a technical problem on our end, if you were wondering: our IRC servers were operating well within their load capacity. It may have been due to BronyState's web site intermittently losing service when it was placed under the DDOS-like high load created by so many users accessing the site at once — BronyState users know it is normal to Ctrl-F5 when the show is about to start, and that sudden influx of simultaneous hits is tough on servers. Our tiny web chat embed sure was hit hard!

If you enjoy graphs, here is another one: This is a graph of the network throughput on fluttershy.ponychat.net for the last 24 hours. Times are UTC-5.

If you're unsure how to read that graph, I'll put it simply: that's a lot of text about a kid's show. And this only comprises one sixth of the total network traffic since our client load is spread across six servers.

So who were all these people? If you haven't noticed, I'm a big fan of graphs and statistics. In the past, I've generated graphs of BronyState's channel activity during episodes, lists of the most commonly referenced ponies in my IRC logs, and even videos of connection maps.

As soon as I recovered from the post-episode excitement, I went and checked how many unique IP addresses had visited the IRC network and web site, today. Over 13,000! And that's not including the hits resulting from the pre-episode streams the night before. On a normal week, we normally serve about 10,000 unique IPs from Sunday to Saturday, and we blew past that in a single day. This definitely called for some more investigation.

42% of the users we had were in the USA. The UK, Russia, Canada, and Germany filled the next slots, from 5 to 7% of our total hits.

Here are the top ten countries (by unique IP):

To see the full country graph, click here (note: it is a fairly large PNG). For SVG format, click here. For the raw data, click here.

As I said before, have created maps of user connections in the last. Equestria Daily once featured a map I created as a proof-of-concept for the YouTube video I eventually created. That map was so popular that I thought this would be a great opportunity to make another.

IMPORTANT NOTE: The points on this map are very approximate and are nowhere near exact locations of users. Map markers are not guaranteed to be accurate, but they will generally be in the correct city or state. Even then, because of the portability of IP addresses, that may not even be correct. Therefore, you cannot use this map to locate any user's homes or even neighborhoods.

Also, this map is generated by your computer on load. There are thousands of unique map markers on here, so very old computers may run slowly or freeze while generating or viewing the map. Because of that, I have not embedded the map, but you may view it by clicking the link below.

Click here to see the full map.

It's April Fool's Day!

April 1st is the day when the Internet becomes a little less usable. Web sites change their layouts, things get renamed, jokes are added here and there — everything acts a little different, and it is often pretty hilarious.

This was our first April Fool's Day on PonyChat. Playing April 1st pranks on on-line communities is not a new thing for me, but I wanted to do something unique this time. IRC pranks are tough: some networks will broadcast announcements that they are merging with a rival network, sending users into a political panic; others will go so far as to grant channel operator power to all users in key channels, resulting in a hopeless battle for dominance.

We decided to do something a little different. Ever since my first April Fool's Day as an IRC operator, I've wanted to cleverly modify user messages. And what better way to do that than replace random words with "pony" on a pony-related IRC network?

<aji> ;define forest
<Terminus-Bot> aji: forest (noun): large pony of land covered with trees

<ChanServ> [#BronyState] Welcome to the #bronystate main room, the 
            number of messages you can send may be restricted to
            limit spam. Feel pony to pony unrestricted in one of the
            pony chats.

<brando753> It hasnt affected me I pony think,,,

<aji> "Kabaka: pony do you really pony to ask me?" <-- is pony what
        it says for anybody else??

The code required to pull this off was written two weeks in advance. It was roughly based on nenolod's message interception module for Charybdis which is able to snag PRIVMSGs as they are sent. I modified it, adding code to randomly replace words.

Since this is a fairly invasive prank, it was not left on for the entire duration of April 1st (in fact, it was still not yet April in some locations when it was removed).

The word replacement is simple: The server looks at each word in every message. If the word is 4 letters long, there is a 1 in 25 chance it will be replaced with "pony" before moving on. Each IRC server decides whether or not to replace words, so users connected to different servers may get different messages.

Unfortunately, a few users did need for private messages to be unmodified, something we probably should have foreseen. After receiving a few requests for this, I patched the modification to only tamper with channel messages.

After mentioning my abuse of nenolod's code for this purpose, he committed a better alternative that would have made this way easier to write. If you're looking to accomplish something similar, I recommend looking at that.

I hope everyone enjoyed the joke — I know I did! If not, don't worry: this was a one-time thing. We will likely do another prank next year, but it will be completely different.