Archive Update

As many of you will have noticed, we had some site slowdowns and 502 errors on the Archive of Our Own over the first couple of days of the year. Apologies for the inconvenience! If you've run into this problem and been wondering what was going on, you might be interested in this:

Yes, it looks like lots of fans decided to celebrate the New Year with some delicious fanworks. On Monday 2nd January we had 182,958 visits, and over 1,066,216 pageviews! Furthermore, an octopus swam off with our servers - volta_arovet's Texts From Cephalopods has had 46,301 hits at the time of writing! So, our servers had plenty of work to do!

Over 2000 new users have joined the Archive in the last couple of weeks, and we have hosted several great challenges, including Yuletide (2598 works!), Due South Secret Santa (a more modest 34 works), and Homestuck Ladyfest Exchange (124 works). So, while we're sorry to have had some slowdowns, overall we are super pleased with how well our shiny new servers have held up - those of you who were with us during the holiday season in previous years will remember that the high traffic of holiday challenges made our old servers very sad.

Looking forward, we're not too worried about performance in the immediate future - there are some code improvements we know we need to make which will improve matters a lot, so those will be high priority. If the AO3 continues to expand at the same rate as this year, we will be looking at more servers sooner rather than later. But in light of the graph above, we're pleased that while we certainly slowed down, we didn't grind to a halt! Thanks to all the coders and sysadmins who did the work to make this possible, and thank you to all the OTW members whose donations helped us buy those hardworking servers (we are always grateful for volunteers or donations)! And, of course, thanks to everyone who reads and posts on the AO3 - we're excited to welcome so many of you!

Once again, apologies to those of you who have been affected by the slowdowns - but hurray for so much beautiful fannish activity!

Mirrored from an original post on the Archive of Our Own.

2011 was an amazing year for the Archive of Our Own, and we wanted to take a moment to look back and to thank everyone involved, including all of our users and volunteers! AO3 started its open beta about two years ago, towards the end of 2009. That year, we were really still putting the pieces together, building out the core functionality. In 2010, we started to pick up more momentum with people posting their works and archiving their older fic and art. We added gift exchange challenge hosting, kudos, downloads and skins. This year, we've done a lot of work on site performance and infrastructure, usability improvements, and new features like subscriptions and prompt meme challenges. We're looking forward to expanding on that next year and continuing to build a great, stable home for all kinds of fanworks!

The Accessibility, Design and Technology committee oversees technology-related projects within the OTW. Currently we are responsible for designing and building the Archive of Our Own. Our regular meeting updates keep you informed about developments on the AO3!

This was our final meeting of the year: the OTW takes an end-of-year break during which committees dissolve and are reformed, and committee members take a well-earned rest! We've had an action-packed year, so we're all ready for a break (from meetings at least - a lot of our work goes on as usual). We'll resume in January - we don't take on any new volunteers during our hiatus, so if you volunteer between now and then (and we hope you will - as you can see below, there are several areas we're really keen to build up), you'll have to wait a little while to get started.

Meeting highlights!

Fandom landing pad!

AD&T co-chair and senior coder Elz has been working for a while on improvements to browsing on the Archive. One thing she's been working on is a new 'fandom landing pad' so that browsing to a fandom will give you the option to browse to some important areas relating to that fandom. In this meeting we previewed her new design - going to a fandom landing page will give you a list of pairings and relationships in the fandom, and a list of authors and artists who have created work in that fandom, along with some basic information about the canon source. It's not quite ready for primetime yet, but it's looking very nifty!

Issues for Love!

Issues for love are the requests submitted by users via Support. We try to work through a few of these each meeting: we're working on ways of making it easier for people to see what has been suggested and what has been decided about the suggestions, but for now we'll include a round-up of our discussions in our meeting updates. Note that a decision to implement something does not necessarily mean it will be implemented soon - we have many issues to work on and a limited number of coders! If you want to see the full (and lengthy!) list of things logged for coders to work on you can check out our Google Code Issues. If you'd like to adopt an issue, we welcome new coders!

• Request to add a setting to prompt-meme challenges to disallow anonymous prompts. This seemed like a handy extra feature without too much coding complexity, so we have logged it as an issue for a coder to work on.
• Request to add an option to hide 'Share' buttons on works to reclaim screen real estate. It's already possible to disallow use of the share button on your own works, but you still see the button. We sympathise with the desire to reclaim the screen real estate, but we decided that added a user preference to hide the buttons would add too much complexity (the more user preferences there are, the more complicated it becomes for people to figure out what they can set in preferences, so we try to limit the options to things where there is a lot of demand for a setting). Instead, we added some extra code to our buttons so that they can be selected with CSS, so that people can build skins which hide the 'Share' button (or indeed any other button).
• Suggestion for a 'challenge calendar' listing opening and closing dates for challenge sign-ups, and dates for assignments due, works revealed, authors revealed, etc, which can be opted in when a mod creates the challenge. We loved this idea, but it is fairly complex to implement. Our lovely co-chair Amelia has volunteered to put together a design, so this is something we'll introduce in the future - but probably not for a while.
• Request for a way to mark WIPs as abandoned, and a way to offer abandoned WIPs up for 'adoption' so that someone can finish them. We all agreed it would be really nice to have a quick way to flag that a WIP would never be finished, so we've logged that as an issue for a coder to implement. The idea of offering works up for adoption seems like it might have more limited appeal, so we agreed that for now, it would be better to leave this as something which people can simply indicate in the tags they use, if so desired (you can add 'Adopt this story' or indeed any other tag you wish as an additional tag to your work).

Reflecting on Release 0.8.9

As most of you reading this will know, we had a big release of new code at the beginning of November. This release included a lot of exciting new stuff; unfortunately, it didn't go as smoothly as we had hoped. In this meeting we reflected on problematic areas and ways that we can improve in future:

• We combined two big new features: the redesign of our front-end code and the new tag sets code for challenges and collections. We had decided to combine the two because the tag sets needed some front-end work anyway, and at the time we made the decision it made sense to roll the two things into one. However, the tag sets code was time sensitive: because it offers a new system of challenge nominations which significantly reduces the pressure on tag wranglers, we wanted to implement it in time for the big holiday challenges such as Yuletide. This meant that when we combined the two features, we had a lot more stuff to get ready within a set amount of time, which made everything more difficult. When we decided to merge the two, it didn't seem as if this was going to be a problem - but one thing we've learnt is that any deploy can bring unexpected hitches, so in the future if there's a time-sensitive feature we'll be trying to keep that as separate from other code as possible.
• This was a big visual change, which meant that it had an impact on a large number of users: visual bugs tend to be encountered by lots of users, and even if there are no bugs, people still have lots of feedback about visual changes. We were aware of this; however, given the scale of the response to this deploy we realise we weren't prepared enough. We'll be doing more testing of interface changes in future, and exploring ways of beta-testing them with more users.
• Since one thing about visual changes is that lots of people just prefer the design they are used to, one thing we could have improved on was providing a way of going back to the old default design. We tried to provide for this with the One Point Faux option, but it had quite a few problems. So, in future this is something we'll be paying more attention to: if we introduce a big change, we'll try to provide ways of opting out. The good news is that going forward, this will actually be easier, because the new skins system is much more lightweight and it should be easier to provide some backwards compatibility (one reason this was problematic this time is because the underlying code for the old system was less than ideal, so everything had been completely rewritten).
• We didn't have as much support documentation and information as we really needed for this deploy. In particular, we needed much fuller documentation on the new skins features so that people could try them out more easily and our Support team could point to useful information when helping people. There were several factors which led to a lack of documentation: crucially, several of the team who would normally take care of this were dealing with RL issues which limited the amount of time they could spend on it. In order to help avoid problems like this in future, we're building a deploy checklist which includes documentation, to make sure that we've considered whether we need additional documentation regardless of who is available to work on any given deploy. We're also aiming to build up a proper documentation team so that this work is less likely to fall through the cracks: if you're interested in being involved in this team, get in touch with our Volunteers and Recruitment Committee and let them know. We'd love to welcome new people to the team!
• We also needed more documentation on the new features for testers, so that it was clearer what people needed to test and what they should expect it to do. This is an ongoing aim - we're working to improve our documentation across the board. Improving documentation for testers will also help us to address another issue, which was that feedback from testers got a bit scattered - having clear docs to start with would have helped us make it clearer what feedback needed to go where. Again, we're working on building up our testing procedures generally - if you're interested in getting involved with testing, let us know!

While the problems we had with this deploy did highlight a number of areas where we need to work to improve, it's not all doom and gloom! There were also a number of things that went right with this deploy - we were able to fix critical bugs within 48 hours of the deploy, the Support team did a wonderful job keeping up with the many Support requests, and there was a huge amount of awesome code in the deploy itself. One reason the site is still in beta is that we're still learning the best processes for development (as well as because our code is new and rapidly changing): in the last four years we've gone from being a tiny group working on coding a 'closed' site (i.e. for the first two years we were just writing the code and testing, we didn't have any real users) to being a much larger group catering for a site of over 28800 users! So, we're still figuring things out - objects may still shift in transit! We're pleased that we've been able to keep the site up and running, and everything largely functional, even though we've had the odd bump along the way. Thanks to everyone who has worked hard to make this true!

Next deploy

We're hoping to get one last deploy in before the end of 2011! It will include some updates to our HTML parser, some improvements to our static pages for collections and challenges, and Atom feeds for fandom tags! (YEAY!)

News from our sub-committees

• Coders have been working on polishing off the issues to go in the next deploy. We're particularly excited about the forthcoming addition of Atom feeds for fandom tags - having tested this out for a good while now on the F/F tag, we think we can implement feeds without too much additional strain on the servers, and since this is a very popular request we're excited about launching it!
• Testers have been testing the issues for next deploy, and discussing how they'd like to see the subcommittee develop next year. There have been some great discussions on what worked and what didn't this year, how we can build a stronger testing community, and how we can support our testers.

News from our sister committees

• Support have continued to work amazingly hard keeping up with a high number of tickets. Looking forward, they're also thinking about our documentation needs and places we need more information for users.
• Tag wranglers have been discussing needs for next year with AD&T - the two committees will be meeting in the new term to talk over technical needs for tag wrangling. They've also been surveying all tag wrangling volunteers about their experiences this year, with a view to figuring out what works well and what can be improved on.

If there are things you'd like to do or say, please share them in comments, via the AO3 support and feedback form, by volunteering (we won't be taking on new volunteers until the new term, but you can get in touch now to let us know you're interested), or in whatever medium you feel comfortable with. Everyone is welcome to this party!

This meeting round-up by Lucy

Mirrored from an original post on the Archive of Our Own.

The Accessibility, Design and Technology committee oversees technology-related projects within the OTW. Currently we are responsible for designing and building the Archive of Our Own. Our regular meeting updates keep you informed about developments on the AO3!

AD&T and our associated committees and subcommittees have been very busy recently working towards our latest deploy and then working on issues arising from that. This one didn't go as smoothly as we had hoped (understatement!); we knew that there would be bugs revealed by practical use that didn't appear in testing, though there were more than we had anticipated and we have been working hard to fix the immediate issues. We're happy to say that we were able to fix the most pressing problems within 48 hours; a week on from the deploy we've been able to address quite a few more, so those fixes will be deployed soon. We're really grateful to everyone who worked hard on this deploy and on addressing the issues subsequent to it. We are planning to do a thorough review of the deploy and think about the lessons learned and ways we can improve. This week, however, we focused on working through some outstanding business and outlining the tasks we need to complete before the end of the year.

The day is finally here - after much hard work from our coders and testers, we are ready to roll out our exciting new code on the Archive of Our Own! We will be deploying shortly, and the AO3 will go down for a while while we do so. When we come back, there will be some big changes which you need to know about:

New site scheme

Front-end coder lim, working with back-end coder Naomi, has reworked our site HTML and CSS to make it more accessible, more functional, and all-round nicer. So, you'll see some visual changes across the site (and if you're using assistive technology or special browser settings you should find it behaves better - if not, please tell us). Don't be alarmed if things don't look how you're used to!

New skins

The new CSS has allowed us to massively overhaul our skins functionality to make it way more flexible - check out our post about the new skins for a bit more background. This will allow for a lot more cool site mods in the future! The unfortunate side-effect is that old skins will break. To make the transition smoother, we'll be doing the following:

• Disabling all old skins and setting all accounts to use the Archive default, so that noone gets stuck on a broken skin.
• Loading up a whole bunch of new skins, including new versions based on the most popular public skins in use at the moment. So, many of you will be able to switch immediately to a new skin which replicates the one you were using before. We'll post a list of the new skins right after the deploy.
• Holding an open house for skins support in the next week or so (time tba).

We'll be posting more docs and help over the coming hours and days! There may be the odd bit of unexpected behaviour along the way (with skins especially, the diversity of user-created skins means we can't prepare for every possible scenario) but we think the new functionality will give you much much more awesome! Cinderella is getting her ballgown on and the good news is that for the shiny new skins, midnight will never come!

We'll tweet to AO3_status when we start the deploy, so you'll have a heads-up before the site goes down. Download your long works now to tide you over while our fairy deploymother is working her magic!

Mirrored from an original post on the Archive of Our Own.

The good news!

Shed your skin! We're about to have a massive deploy of new code on the Archive of Our Own that will bring us - among other things - radically new skins! The underlying code for the Archive has been streamlined and modified to make it more accessible and maintainable, and much easier to override. This is going to allow for fantastically beautiful and awesome customizable looks for the Archive; fanartists, prepare to go wild! As a bonus, we've also added some skins options for logged-out users (you'll find these in the footer)!

The bad news :(

Since we had to radically retool the underlying code to make this all work well, existing skins are not going to work right out of the box. (We did mention that this was an archive in beta, right?) We're really sorry for the inconvenience, but we had to sacrifice consistency in the short term to make the skins feature more awesome and ensure our code remains maintainable in the long term.

To make sure that no one comes to the archive and finds a broken skin, we'll be temporarily turning off the existing skins when we deploy the new code - don't be alarmed if you come to the Archive to find it looks different! You'll be able to preview your old skins to see how they'll look with the new code, so you can fix the private ones before turning them back on if necessary. We've also already upgraded a bunch of the existing public skins and added some new ones-- for instance a new light-on-dark skin, and a new plain text skin. We'll be posting a list of these skins as soon as the code deploys.

What's so great about the new skins?

LOTS OF STUFF! \0/ One major improvement is that you'll no longer have to use the dreaded !important for most of your overrides to the default skin. In fact, if you want to you can strip away our default skin altogether and write your styles without using any of our CSS at all. But that's only the beginning! There are a whole bunch of new features:

• Modular system so you can make your skin a 'child' of another skin. This means that when you find that skin which has a perfect layout but a color you hate, you can apply that skin for your layout, then add another one on top which just changes the color. (And that's just the beginning of what you can do with the modular skins - we'll have more in a future post explaining the cool possibilities.)
• Make different skins for different devices (this is courtesy of the modular skins) - you'll be able to make a custom skin for when you're browsing on your computer, and another one for when you're browsing on your phone, and have them kick in automatically when you're logged in to your account (providing your phone respects mobile stylesheets). Customization wherever you are!
• An array of images you can use to customize your skins! You can already use images hosted offsite in your own skins (but not in public ones because we need to be sure the images won't change or break). Now you'll be able to pull from a bunch of stock images on the Archive to make your skins even prettier.

Essentially, the new system is aimed at giving you control of ALL THE THINGS! We'll be posting a list of pre-loaded skins as well as loads of shiny new skins over the coming days. We'll also post tutorials and examples so you can get to grips with the new shiny!

O NOES MAH SKINS ARE BROKEN

All may not be lost! If your skins are broken or you think they're likely to be, you can do some things to prepare. We'll be holding a Skins Open House in a week or two where we'll go over how to make new skins and fix old ones - watch this space for news on that.

Accessible skins

One last note - one reason it was important to us to make this change is that it makes it MUCH easier to customize skins to meet specific accessibility needs. If you are using one of the public skins geared towards particular accessibility needs (e.g. low contrast, plain text, etc), then we have either fixed this to work right out of the box under the new system or (where that wasn't feasible) added a new skin to do the same job. We're adding new accessible skins for logged-out users too - you'll be able to access a Low Vision skin and a Light on dark Small Text skin in the footer even when you are logged out. If you need a customized skin for accessibility reasons and our existing skins don't meet that need, please get in touch with support and let us know what you need - we'll do our best to help! (Also, if you run up against an accessibility issue anywhere on the Archive, please let us know - we're committed to keeping the site as accessible as possible, but inevitably there's stuff we miss!)

Stay tuned for more info about skins!

Mirrored from an original post on the Archive of Our Own.

Are you interested in volunteering as a coder or tester for the Archive of Our Own? Do you have questions about our development process, or would you like to learn more about how archive features go from idea to reality? The Accessibility, Design and Technology Committee will be holding an open house to talk about our work and answer any questions you may have!

All are welcome! The chat will be held on Sunday, October 30th at 20:00 UTC (what time is it in my timezone?) in OTW's public chatroom on Campfire. The chatroom can be accessed at https://fanarchive.campfirenow.com/e79cc.

Accessibility, Design, & Technology is the guiding body that coordinates software design, development and testing on behalf of the Organization for Transformative Works. Currently we are responsible for designing and building the Archive of Our Own.

Mirrored from an original post on the Archive of Our Own.

The Accessibility, Design and Technology committee oversees technology-related projects within the OTW. Currently we are responsible for designing and building the Archive of Our Own. Our regular meeting updates keep you informed about developments on the AO3!

This is a busy time for AD&T, because we're coming closer to our big deploy! We've been working on some major changes and they are almost ready to go onto the main Archive - this means lots of testing and bugfixing and general activity!

The Accessibility, Design and Technology committee oversees technology-related projects within the OTW. Currently we are responsible for designing and building the Archive of Our Own. Our regular meeting updates keep you informed about developments on the AO3!

This was a nice busy meeting - 11 of our 14 staffers were able to attend! Managing different timezones and meeting times, as well as other commitments, means that it can be challenging for everyone to attend every meeting, so it's nice when lots of us make it (although some of us were off celebrating birthdays *looks about shiftily*).

Meeting highlights!

Improving our code commits and deploy process

This is an important subject to us, although one that isn't immediately apparent to our users! When a coder writes some new code, they 'push' it to our code repository and then submit a 'pull request' to ask for it to be included in the main project. Another coder then looks at the code and (if they don't spot any problems) merges it into the main project; when we have a few changes to that, we put all the code on our Test archive for testers to work on. Ideally, this whole process should be pretty quick - it's nice for coders to have their code accepted quickly (or sent back for changes) and it's nice for users if we get new code tested quickly and thus out onto the live site!

Right now, this process isn't as fast as we'd like it to be. Unfortunately, our automated tests currently take a significant amount of time to run, and that causes a bit of a chain reaction: coders run the tests as they work, run all the tests before they push their code out, run all the tests before they approve somebody else's code, and have a job set up on our development server to run all the tests every day and email the whole team with any failures. While we work on making the tests faster, we're hoping to do some reshuffling, where we reduce some of the duplication and the time required to merge in code, while doing more manual review. We'll be aiming to get code merged in and out to our testers on a quicker and more regular basis, so that ultimately users will have faster access to fixes and new features.

As most people are sadly only too aware, the Archive of our Own has been experiencing some performance issues recently. The sad 502 page is showing up increasingly frequently, to the frustration of everyone concerned! We have also had a couple of instances of downtime. We are working on the problem, but we know that our users are wondering what is going on, so we want to give a bit more information on what is going on behind the scenes.

Why is this happening?!

The main reason for 502 errors is the sheer load on the site. We've seen a massive increase in user numbers over the past few months - on Sunday 25th September we had 105,000 visits, of which 46,000 were unique visitors. These visitors racked up an impressive 575,000 page views! This is now a pretty average day for us - we're thrilled that so many people are enjoying and using the site, but we've expanded a bit more rapidly than we expected, so it's a little bit challenging for our servers.

Our recent server outages were caused by a problem with the servers themselves; our Systems team are tracking this to its source and making some changes to fix it.

Why do I sometimes get a 502 as soon as I click on a page?

Since 502 errors are associated with site slowness, it can be a bit unexpected to get a 502 as soon as you click. The reason this happens is that our servers are set up to keep an eye on how many requests are in the queue for them to handle. If there are more than a certain number, then the chances that the page will time out before it can be delivered are high. So, the server gives the 502 page right away instead so that you don't wait a long time only to be disappointed.

Didn't you just buy new servers recently?

Yes, pretty recently! Thanks to the generosity of fans who donated to our parent Organization for Transformative Works, we were able to purchase 5 new servers at the beginning of the year. These are actually doing a great job (if we were still on our original two servers, the site would have keeled over completely by now). However, the speed of our expansion means that we will need to purchase more servers sooner rather than later. When we do this, we'll also have to make some big changes to the underlying infrastructure - one option is shard the database, which means we'd split it up into separate chunks so each server only has to deal with a bit of it (if you're interested in the problem of scaleability and how sites deal with lots of users, this post on how LiveJournal handled it is a good read). We're actively researching now to figure out the best way of doing this and the kind of hardware we'll need.

What are you doing to fix the problem?

One reason the high loads on the site are having such a drastic effect is that some of our code is really optimised for fewer users (since we weren't expecting to have so many this soon!). So we're currently making a number of changes in the code to make it more efficient and reduce the number of database reads/writes (these are the things that tend to cause slowdowns) and updating the application software to one which should be more efficient. Our Systems team are also investigating updating the server operating system which might be necessary to take care of the issue which caused the server outages (it's hard to reproduce the conditions on the real site in our test site). The current set of new code has required intensive testing, which means we haven't been able to roll out the changes as quickly as we would like, but we hope to make these updates in early October. Longer term, we'll be doing some sustained work on scalability so that we can restructure things and buy more servers.

What can I do to help?

We really appreciate how understanding users have been about the problems - it helps our team a lot! If you want to do more and you have some financial resources, then a donation to the OTW will help ensure that we are able to continue expanding our servers (money donated to the OTW supports all OTW projects, but the vast majority is spent on servers and other running costs for the AO3). Finally, if you happen to be an experienced sys-admin or database expert - especially one with experience in performance tuning and scalability - and you would be willing to donate some time, we would welcome the additional expertise and support. (If you're interested, get in touch with our Volunteers and Recruitment Committee.)

And finally...

We'd like to say thanks to everyone who has supported the site in various different ways - while the 502 errors are HUGELY annoying, they do show that people enjoy using the AO3. Thanks also to all our users for your patience while we get to grips with our new success and deal with the performance problems. Finally, an HUGE thank you to our Systems team, who bear the brunt of the work on these issues - the wonderful Sidra has been woken in the night more than once to deal with server issues, and we appreciate her hard work and dedication more than we can say. ♥

We are working hard to resolve the performance issues, and we'll keep updating users as we have more news. The latest site status updates can be found on our Twitter AO3_Status.

Mirrored from an original post on the Archive of Our Own.