The Archive of our Own will be down for planned maintenance for approximately three hours from 15.00 UTC on Friday 20 July (see what time this is in your timezone). During this period we'll be installing some new RAM and performing some other maintenance (more details below for the curious!).

We'll keep users updated on our Twitter AO3_Status as the work progresses. Thanks for your patience while we complete this work!

New RAM

Our database server looking grumpy about having too little RAM!

We're doubling the RAM in our database server and in our two application servers. Increasing RAM will help our system cope with more users: for example, it will allow us to run more unicorn workers, which serve up the content you're trying to access. This should help site performance as the site expands.

You can imagine the unicorns lining up in the hall of RAM to fetch you things from the treasure trove of fanworks: if there aren't many unicorns, you have to wait till one can serve you, which sometimes means you get a 502 error. We can increase the number of unicorns to make things go faster for you, but if the hall is too small (there isn't enough RAM) then things get crowded and inefficient and everything slows down again. More RAM allows us to increase the number of unicorns without slowing things down. (For the interested, this more technical explanation of Unicorn isn't exactly the way things are set up on the AO3, but will give you an idea.)

New drives

We're also installing some new drives in our two oldest machines. Both these machines have room for six drives; currently they each have four installed. Information is mirrored on the drives so that if one goes down, the system continues to work. At the moment, one machine has a broken drive. We'll be replacing the broken drive, and at the same time adding two spares to both machines so that we have more backups if anything else breaks.

Our two original machines preparing to nom their new drives

Mirrored from an original post on the Archive of Our Own.

Key information: As an emergency measure to deal with recent performance issues, we will shortly be disabling the browsing filters on the Archive of Our Own (the grey box of choices which appears on work index pages). This is a temporary measure to ensure that as many people as possible can access the site. You can still use our tags and advanced search feature to find the works you want. Read on for more information!

Key information: As an emergency measure to deal with recent performance issues, we have disabled browsing filters on the site (the grey box of choices which appears on work index pages). This is a temporary measure to ensure that as many people as possible can access the site. You can still use our tags and advanced search feature to find the works you want. As an additional bonus, removing the filters has allowed us to remove the 1000 works cap on lists of works, so you can browse through all the works in your fandom! Read on for more information!

What's happening

As detailed in our recent post on performance, our coders and sys-admins are continuing to work on the performance issues we've been experiencing. We've made some server adjustments which have alleviated some of the worst problems, but we still need to make some substantial changes to fix the issues. We're aware that lots of users are still unable to access the site; as an emergency measure, we've decided to disable tag filters, which put a very heavy load on our servers. This means that the grey box with tags you can check to filter a list of works will no longer appear on the work index pages. We know this will be an inconvenience for many users, but the filters are really the 800-pound gorilla sitting on top of our database. Removing them for now will mean that people can access the site, even if they can't browse quite as easily as usual.

We've been working on significantly redesigning the part of our code that handles filtering for a while - because it's a major performance burden on some of the most popular pages of the site, refactoring this code to make it more efficient has been a priority for some time now. We're almost done with the rewritten version, but it needs more work and extended testing before we roll it out. (We want to be sure it doesn't introduce new bugs.) So, the filters will go away for a few weeks, and will then be replaced by the new, rewritten version.

One major disadvantage of the way the filters were designed was that they needed to retrieve the tags from the list of works found in order to build the filter options. This meant that we had to limit the number of works returned at one time to 1000, because otherwise building the filters would take too long. A side bonus of removing the filters is that we've been able to remove the 1000 works cap! The browsing redesign in progressaims to work around this issue, so we hope to avoid re-introducing this limitation when filtering returns.

How can I find the works I want?

Although the removal of the filters will make it harder to browse the works listings for specific things, there are still lots of ways to find the works you need.

Fandoms page

If you're looking for a specific fandom, you can browse the Fandoms page. Fandoms are organised by media type; the easiest way to find a particular fandom is to use Ctrl + F (or Command + F on a Mac) to search the page in your browser. The fandom pages will give you a list of all the works in your fandom; unfortunately there will be no way to filter that list down further.

Tags

Clicking on any tag will still bring up works with that tag, or with any tag marked as a synonym. So, if you click on Riza Hawkeye you'll get all the works tagged with 'Riza Hawkeye', 'Riza', 'Riza is awesome', etc. Again, while the filters are disabled there'll be no way to filter this list further.

Advanced Search

If you want more refined control over which works you find, you will need to use our Work Search. This feature could use a little bit of prettifying, but the underlying search is quite powerful. Use the following tips to help you find exactly the works you want:

• A space equals AND. So, entering Fluff Friendship would find you works tagged with both 'fluff' and 'friendship'
• | equals OR. So, entering Homestuck | My Little Pony will find you works tagged with 'Homestuck' AND/OR 'My Little Pony'
• - equals NOT. So, entering Supernatural - Castiel/Dean Winchester will find works tagged Supernatural, but will exclude those tagged Castiel/Dean Winchester.
• Fandom, Character, Relationship, Rating, Category, and Warning are all classed as tags (as well as the 'Additional tags'). So, you can search for works which are Explicit, or exclude works tagged 'Major Character Death'.
• Using quotes around a phrase will search for that exact phrase. So, "Harry Potter" will get works tagged with 'Harry Potter', whereas Harry Potter will get works tagged with 'Harry' and works tagged with 'Potter'.
• Entering a term in the tag field will only find works with exactly that tag - so searching for Charles/Erik will bring up only the few works tagged with exactly that tag, not the ones tagged 'Erik Lehnsherr/Charles Xavier' (whereas if you click on the 'Charles/Erik' tag you'll get works with all variations of that pairing).
• The search has trouble with tags which have dashes in them. If you search for X-Men, for instance, you noticed you'll get lots with X and no X-Men. To get around this, put the tag in quotes: "X-Men".

As well as searching tags, titles, and authors, you can also search for specific word counts, hits, kudos, and dates - including ranges, which is a useful tool for finding fics in a fandom. For example, you can search for all Stargate Atlantis fics published 5-6 years ago.

Some search examples!

• Find an explicit Fullmetal Alchemist work with the pairing Riza Hawkeye/Roy Mustang, with no Archive Warnings: Enter "Fullmetal Alchemist" "Riza Hawkeye/Roy Mustang" "No Archive Warnings Apply" Explicit.
• Find works with Rodney McKay but without John Sheppard: Enter "Rodney McKay" -"John Sheppard".
• Find works tagged with "Alternate Universe" in either the Homestuck or White Collar fandoms: Enter "Alternate Universe" Homestuck | "White Collar".
• Find all explicit works tagged as angst, but excluding M/M pairings: Enter Angst Explicit -M/M

Search bookmarklets

If you find yourself re-using the same search parameters (only T-rated works, only works under 5,000 words, only works with over 10 kudos) for new fandoms or characters you fall in love with, you could give these custom search bookmarklets a try. They are not official AO3 tools, but made by one of our own and utilizing the Advanced Search functionality. Think of them as a saved search that lets you enter a keyword (such as a fandom name or specific kink) and spits out only the kind of work you want to see. For help in putting together your own saved search, don't hesitate to comment on the post or here.

What next?

This is definitely a short term measure, but we think it will have a big effect on site performance. In a few weeks we hope to deploy our all new search and browse features, which will restore more browsing functionality without placing the same load on the servers. We thank you for your patience and understanding while we continue to work on the problem areas.

Mirrored from an original post on the Archive of Our Own.

Since last month, we've been experiencing frequent and worsening performance problems on the Archive of Our Own as the site has expanded suddenly and dramatically. The number of new users joining the site doubled between April and May, and we currently have over 17,000 users waiting for an invitation. We've been working hard to deal with the 502 errors and site slowdowns, and we've implemented a number of emergency fixes which have slightly alleviated the issues, but these haven't been as effective as we'd hoped. We're confident that we will be able to fix the problems, but unfortunately we expect the next round of fixes to take at least two weeks to implement.

We know that it's really frustrating for users when the site is inaccessible, and we're sorry that we're not able to fix the problems more quickly. We wanted to give you an update on what's going on and what we're doing to fix it: see below for some more details on the problems. While we work on these issues, you should get better performance (and alleviate the load on the servers) by browsing logged-out where possible (more details below).

Why so many problems?

As we mentioned in our previous post on performance issues, the biggest reason for the site slowdowns is that site usage has increased dramatically! We've almost doubled our traffic since January, and since the beginning of May the pace of expansion has accelerated rapidly. In the last month, more than 8,000 new user accounts were created, and more than 31,000 new works were posted. This is a massive increase: April saw just 4,000 new users and 19,000 new works. In addition to the growing number of registered users, we know we've had a LOT more people visiting the site: between 10 May and 9 June we had over 3,498.622 GB of traffic. In the past week, there were over 12.2 million page views - this number only includes the ones where the page loaded successfully, so it represents a lot of site usage!

This sudden and dramatic expansion has come about largely as a result of changes on Fanfiction.net, who have recently introduced more stringent enforcement of their policies relating to explicit fanworks which have resulted in some fans no longer being able to host their works there. One of the primary reasons the AO3 was created was in order to provide a home for fanworks which were at risk of deletion elsewhere, so we're very keen to welcome these new users, but in the short term this does present us with some challenges!

We'd already been preparing for site expansion and identifying areas of the site which needed work in order to ensure that we could grow. This means some important performance work has been ongoing; however, we weren't expecting quite such a rapid increase, so we've had to implement some changes on an emergency basis. This has sometimes meant a few additional unexpected problems: we're sorry if you ran into bugs while our maintenance was in progress.

What we've done so far

Our sys-admins and coders have implemented a number of things designed to reduce the load on the site over the last week:

• Implemented Squid caching for a number of the most performance intensive places on the site, including work index pages. For the biggest impact, we focused on caching the pages which are delivered to logged-out users. This is because all logged-out users usually see the same things, whereas logged in users might have set preferences (e.g. to hide warnings) which can't be respected by the cache. We initially implemented Squid caching for individual works, but this caused quite a few bugs, so we've suspended that for now while we figure out ways of making it work right. (You can read more about what Squid is and what it does in Release Notes 0.8.17.
• Redistributed and recalibrated our unicorns (which deliver requests to the server and retrieve the data) to make sure they're focused on the areas where we need them most. This included setting priorities on posting actions (so that you're less likely to lose data when posting or commenting), increasing the numbers of unicorns, and adjusting the time they wait for an answer.
• Simplified bookmark listings, which were using lots of processing power. We'll be looking into revamping these in the future, but right now we've stripped them back to the basics to try to reduce the load on the site.
• Cached the listing of guest kudos so the number doesn't have to be fetched from the database every time there are new kudos (which caused a big strain on the servers)

Implementing these changes has involved sustained work on the part of our sys-admins, coders and testers; in particular, the Squid caching involved a great deal of hard work in order to set up and test. Several members of the team worked through the night in the days leading up to the weekend (when we knew we would have lots of visitors) in order to implement the performance fixes. So, we're disappointed that the changes so far haven't done as much as we'd hoped to get rid of the performance problems - we were hoping to be able to restore site functionality quickly for our users, but that hasn't been possible.

What we're going to do next

Although the emergency fixes we've implemented haven't had as much impact as we'd hoped, we're confident that there are lots of things we can do to address the performance problems. We're now working on the following:

• New search and browse code. As we announced in our previous post on performance issues, we've been working for some time on refactoring our search and browse code, which is used on some of the most popular pages and needs to be more efficient. This is almost ready to go -- in fact, we delayed putting it onto our test archive in order to test and implement some of the emergency fixes -- so as soon as we have been able to test it and verify that it's working as it should, then we will deploy this code.
• More Squid caching. We weren't able to cache as many things as we'd initially hoped because the Squid caching threw up some really tricky bugs. We're continuing to work on that and we'll implement more caching across the site once we've tested it more thoroughly.
• More servers. We're currently looking at purchasing a more robust database server and moving our old database server (aka 'the Beast') into an application slot, giving us three app servers. We'll also be upgrading the database software we use so that we can make the most of this server power.

When we'll be able to implement the fixes

We're working as fast as we can to address the problems -- we poured all our resources into the emergency fixes this week to try to get things up and running again quickly. Now that we've implemented those emergency fixes, we think that we need to focus on making some really substantive changes. This means we will have to slow down a little bit in order to make the bigger changes and test them thoroughly (to minimise the chances of introducing new bugs while we fix the existing problems). Buying servers will also take us some time because we need to identify the right machines, order them and install them. For this reason, we expect it to take at least two weeks for us to implement the next round of major fixes.

We're sorry that we're not able to promise that we'll fix these problems right away. We're working as hard as we can, but we think it's better to take the time to fix the problems properly rather than experimenting with lots of emergency fixes that may not help. Since the AO3 is run entirely by volunteers, we also need to make sure we don't burn out our staff, who have been working many hours while also managing their day jobs. So, for the long term health of the site as a whole, we need to ensure we're spending time and resources on really effective fixes.

Invitations and the queue

As a result of the increasing demand for the site, we're experiencing a massive increase in requests for invitations: our invitations queue now stands at over 17,000. We know that people are very disappointed at having to wait a long time for an invitation, and we'd love to be able to issue them faster. However, the main reason we have an invitations system for creating accounts is to help manage the growth of the site -- if the 16,000 people currently waiting for an invitation all signed up and started posting works on the same day the site would definitely collapse. So, we're not able to speed up issuing invitations at this time: right now we're continuing to issue 100 invitations to the queue each day, but we'll be monitoring this closely and we may consider temporarily suspending issuing invitations if we need to.

Until recently, we were releasing some invitations to existing users who requested them. However, we've taken the decision to suspend issuing invitations this way for the present, to enable us to better monitor site usage. We know that this will be a disappointment to many users who want to be able to invite friends to the site, but we feel that the fairest and most manageable way to manage account creation at present is via the queue alone.

What can users do?

We've been really moved by the amount of support our users have given us while we've been working on these issues. We know that it's incredibly annoying when you arrive at the Archive full of excitement about the latest work in your fandom, only to be greeted by the 502 error. We appreciate the way our users have reached out to ask if they can help. We've had lots of questions about whether we need donations to pay for our servers. We always appreciate donations to our parent Organization for Transformative Works, but thanks to the enormous generosity fandom showed in the last OTW membership drive, we aren't in immediate need of donations for new servers. In fact, thanks to your kindness in donating during the last drive, we're in good financial shape and we're able to buy the new server we need just as soon as we've done all the necessary work.

As we've mentioned a few times over the weekend, we can always use additional volunteers who are willing to code and test. If this is you or anyone you know, stop by Github or our IRC chat room #otw-dev!

There are a few things users can do when browsing which will make the most of the performance fixes we've implemented so far. Doing the following should ease the pressure on the site and also get you to the works you want to see faster:

• Browse while logged out, and only log in when you need to (e.g. to leave comments, subscribe to a work, etc). Most of our caching is currently working for logged-out users, as those pages are easier to cache, so this will mean you get the saved copies which come up faster.
• Go direct to works when you can - for example, follow the feeds for your favourite fandoms to keep up with new works without browsing the AO3 directly, so you can click straight into the works you like the sound of.

Support form

Our server problems have caused some problems accessing our support form. If you have an urgent query, you can reach our Support team via the backup Support form. It's a little more difficult to manage queries coming through this route, so we'd appreciate it if you'd avoid submitting feature requests through this form, to enable us to keep on top of bug reports. Thanks!

Thank you

We'd like to say a big, big thank you to all our staff who have been working really hard to address these problems. A particular shoutout to James, Elz, Naomi and Arrow, who have been doing most of the high level work and have barely slept in the last few days! We're also incredibly grateful to all our coders and testers who have been working on fixing issues and testing them, to our Support team, who have done an amazing job of keeping up with the many support tickets, and to our Communications folk who've done their best to keep our users updated on what's going on.

We'd also like to say a massive thank you to all our users for your incredible patience and support. It means so much to us to hear people sending us kind words while we work on these issues, and we hope we can repay you by restoring the site to full health soon.

A note on comments: We've crossposted this notice to multiple OTW news sites in order to ensure that as many people see it as possible. We'll do our best to keep up with comments and questions; however, it may be difficult for us to answer quickly (and on the AO3, the performance issues may also inhibit our responses). We're also getting lots of traffic on our AO3_Status Twitter! Thanks for your patience if we don't respond immediately.

At the recent April Showers Import Party, the topic of tagging came up. Unsurprisingly — there were fanworks finding homes on the Archive of Our Own, after all. And one of the most amazing things about the AO3 is definitely its tagging system.

What makes this particular tagging system so amazing? It's specifically designed so that users can use any tag, in exactly the form they want it on their works, while keeping those works as organized as they would be in a strictly classification-based archive. Perhaps even better organized, since AO3 tagging not only allows users to search for works using tags based on traditional classifications like fandom, character and relationship, but also tens of thousands of canonical "additional tags" that go far beyond the limits of genre. And the more you tag, the better it works overall (more on why that's so later).

So, you may be asking, what’s a canonical additional tag? And how does a tag come to be one?

First, a bit of tagging history is in order. The indexing of information using keywords isn’t a new practice, of course. When del.icio.us launched in 2003, the new part of their model wasn’t the link collections or the keywords themselves — those had been around on the Internet pretty much since the start. Their innovation was to give users the power to attach keywords to those link collections. Nearly ten years on, the AO3 has made a great start at putting that sort of descriptive tagging power in the hands of fan creators when it comes to archiving their fanworks.

A great deal of that power comes from additional tags (originally called freeform tags). They're tags that don’t fall into the standard fandom/character/relationship groupings, and include kinks, tropes, genres, story elements, word counts, recording lengths, video formats, fan art media, POVs, episode tags, additional warnings, and whatever else users can think of! And all those additional tags gain their useful descriptive power when they're made canonical, appearing in the search filters and the auto-complete box as the most useful, general forms of particular tags, with many other synonymous tags linked to them. Tag wranglers — fans who have volunteered to curate the tags belonging to particular fandoms — do the linking, so for a tag to have been made canonical means that a wrangler has either recognized it as complying with tag wrangling guidelines or created it in compliance with those guidelines specifically so they can attach another tag to it.

What that means is that when an additional tag appears in the archive, a tag wrangler assesses whether people searching for works would like to be able to search for works tagged with it. Often the answer is yes, but sometimes it's no — and that's fine! That doesn't mean that you shouldn't tag your works using whatever tags you like; those tags will still bring up works, after all, even if they aren't canonical! A tag wrangler making your tag canonical just means that they judged it a useful addition to the filters — and one that other people might want to have the option of tagging works with, as well.

So if you can tag with whatever you want, how do you effectively use additional tags to shine a spotlight on your work?

This is where we return to the question of how more tagging makes the system work better. You see, additional tags make it possible to bring up works tagged everything from Accidental Marriage to Zombies — and everything else in between. So, using tags which highlight tropes or kinks (among other things), means that people interested in these topics can more easily find your works containing them. And this isn't just limited to fanfic — there are additional tags which describe aspects of other types of fanworks that people might be looking for, as well — whether it’s a crochet pattern, a short podfic, or an example of digital fan art.

What it comes down to is this: your were-creature accidental marriage fic/podfic/painting/quilt with zombies and a female BAMF character of color lead might be exactly the work Fan X was looking for. And they might never know it exists, if you don't tell the world exactly what awesome stuff it contains. There're a lot of awesome fanworks featuring a lot of awesome stuff on the AO3 just waiting for people to spotlight those awesome qualities, so go forth and tell people that your fic is epistolary or your comic features a mythical being or creature or that your podfic is a cast recording.

While many of us love the diversity that additional tags bring, if you don't like seeing additional tags on works, you can always choose to ignore them, or even hide them completely using a custom skin and the Blurblings Hide freeform tags skin.

*****

If you'd like to know more about tagging on the AO3, or about what tag wranglers do, the Tag Wrangling Committee is having an open house on Sunday, April 22, from 19:00 to 21:00 UTC (see when this is in your timezone) in OTW's public chatroom on Campfire. The chatroom can be accessed at: https://fanarchive.campfirenow.com/c6440 Feel free to drop by at any time during the session to ask questions or just to hang out.

The AO3 and its tagging system are funded by fans, for fans. To help keep it growing, please donate today.