Viddler

Viddler’s core architecture is in the middle of an enormous overhaul that will provide reliability to 50+ internal and external services as well as moving to a world-class datacenter, Internap. This multi-month project will dramatically increase our uptime ensuring your videos get delivered when your viewers need them. I sat down with our Systems Architect, Todd Troxell, to talk about the upgrade.

What is the main goal Viddler is looking to accomplish by making this upgrade?

Our customers have spoken and they want greater uptime and service reliability. This is the theme of 2011 and the goal of this project.

Why did Viddler choose Internap?

We chose Internap for their superior network architecture and network engineering staff. We were burned in the past with mis-managed route failovers and slow unreliable links. This is a move to a very well respected name in Internet transit in a world-class faciity. This means faster uploads, greater scalability, and outrageously greater service uptime.

What new hardware has Viddler installed?

Here is a diagram.

In most cases we are doubling our hardware specs or greater in terms of raw CPU, memory, and IO. It’s a factor the market since newer hardware for the same price == faster, however this may have a noticable affect on site performance. We don’t have an accurate test platform to know for sure until it is operational.

In terms of *new* hardware, we have 6x the database capacity, redundant secondary databases, more complete staging setup, redundant networking gear, redundant links to the internet, redundant power in all servers, redundant logging servers, redundant monitoring and adminstration servers, redundant sending of email, thumbnail generation, video recording, transcoding- in total adding redundancy to 50+ components. All of this is new in Viddler NYC.

Viddler has been making great strides over the years to build out their back end hardware and increase site reliability. How will these upgrades affect reliability?

Our upgrades will ensure higher uptime of our entire platform from the core application to our video delivery mechanism and 50+ ancillary services. This is a many-month effort to build enterprise reliability into our once fledgling platform. They will also allow us to iterate on our platform and build out future offerings with far lower friction.

How will it affect speed?

All of the hardware is faster and some of it is of higher quality (Cisco switches for instance). We’re not investing in speed really here, but it may be a by-product.

What does this installation overall mean in terms of how it effects Viddler’s ability to grow in the future?

This architecture is fully modular and repeatable and can be replicated in micro-clusters around the world. There is even a possiblity that it could be pushed onto appliances for users who would prefer to run Viddler in a closed environment. In this new architecture, we will be able to add capacity very easily meaning we can scale our video delivery to a very high number of users. Being repeatable, it will also allow us to replicate and test load and stage new versions with far greater confidence and reliability. This change is not only about reliability but a core fix to our engineering culture, adding repeatability and modularity and paving the way for architectural iteration and experimentation.

Bryan and I getting ready to do some heavy lifting. Todd plotting his attack!

Here at Viddler we have a fairly sophisticated software development process.  We design, review, build, review, test, launch to development/staging servers, test, and then there’s Quality Assurance (QA).  We coordinate our pushes to update the live site (Viddler.com) with our developers, sys admins, testers, and QA. We do a live update every 3 weeks or so.

I’m pretty adamant about sticking to the process – no patches should be made to the live site in between formal updates.  Any exception, the patch needs thorough testing on the development servers and then testing and QA after the patch is pushed to our live site.  I tell the story of the AT&T programmer who only changed 2 lines of code – and brought down the entire northeast phone network for an afternoon. Folks here snicker – but I saw it happen.

Today I broke my own rule.  We had a bug on our reseller’s dashboard.  Our developer found the bug, fixed it and tested.  And then asked me, “Should we patch the live site now or wait till our next push?”  Without hesitating (and without testing) I said push it live.  It was pushed at 6:30PM EDT – and I didn’t test after the push.  And it broke logging in.  No one, not anyone, could log into Viddler.com.  It was literally one line of code.  I was living my own worst nightmare.

Thankfully, we have an amazing team – which was rallied (some out of bed in the middle of the night), and by 11PM a 2^nd fix was made (turns out it wasn’t the one line of code, but the restarting of some new processes during the push) and everyone can once again log into Viddler.com.

I am dreading going to the office tomorrow.  Maybe I can call in sick?  Think they’ll be suspicious?

The moral is: never, ever, push a code change live without thorough testing – even if its one line of code. And there is a reason for a process.

Thank you and good night.

Greetings, internets! Andrew here. I’m Viddler’s “Lead Designer”, but sometimes—when nobody’s looking—I write code. I hope you won’t tell. You won’t, will you? Of course not.

Anyway, sometimes my cold web designer heart shows a glimmer of humanity and I release some of the code I write. For free. And not the kind of “free” where it’s really not free because you have to pay for it. The real kind of free. Free-of-charge free. That kind of free.

Okay, moving on. So today is one of those times. The times I release software, I mean. This bit of software I have dubbed “Pack”.

(Warning: jargon ahead. If your family has a history of jargon-allergies, you may want to consult a physician and skip the next few paragraphs).

Basically, Pack is a command-line tool that lets you combine and compress CSS and JavaScript files in automated but very customizable ways. It’s really handy for when you want your website’s CSS and JavaScript files to be as optimized as possible.

Pack is written in Ruby and uses Google Closure to compress Javascript. I’ve released it on GitHub here: http://github.com/ashrewdmint/pack. Since it’s written in Ruby, it’s a snap to hook up with the popular Ruby deployment framework, Capistrano.

If you made it this far and haven’t lost consciousness, congratulations! Here’s a picture of a cat using a computer: