Building a Website to Scale to 100 Million Page Views Per Day and Beyond (open to the public)

Eric Pickup (29.Feb.2012 at 17:00)
Keynote at ConFoo 2012 (English - US)

Rating: 4 of 5

Over a six month period, I lead the project to rewrite a top 100 website using a new software stack. Doing so, we used HAProxy, Varnish, Nginx, PHP-FPM, Symfony2, Syslog-ng, Redis and MySQL to create a platform that handles 100 million page views per day and has room to grow.

In this presentation, I’ll cover what we did, the challenges we faced and the solutions we found.

Twitter: EricPickupYP

Who are you?

Claim talk

By clicking this button you are declaring that you are the speaker responsible for it and a claim request will be sent to the administrator of the event.

If the claim is approved you will be able to edit the information for this talk.

Are you sure?

 
Comments closed.

Comments

Rating: 3 of 5

29.Feb.2012 at 18:46 by Anonymous

This would have been a perfect session talk. The topic was interesting as was the content. The speaker could use quite a bit more practice at public speaking. I think some more visual slides (no, not implying the porn), showing how the caching and traffic is actually routed and managed would have made more sense. I'm still unclear as to whether or not this was a feat because you did with a dynamic language like PHP, or just because you were able to take so many requests...it sounds like the videos aren't really handled by this application at all.

Also would like to see this developer (and maybe the company) more active in the PHP community & contributing back.

Rating: 4 of 5

29.Feb.2012 at 19:03 by Kevin Bond (20 comments)


Rating: 4 of 5

29.Feb.2012 at 19:04 by Anonymous

This presentation was pretty good and very interesting. Is there any possibility to download the slides or to view them online? Thank you!

Rating: 4 of 5

29.Feb.2012 at 20:13 by Enzo Rizzo (27 comments)

The topic was EXCELLENT and very instructive, specially for the people with growing websites

Rating: 4 of 5

29.Feb.2012 at 20:35 by John Kary (62 comments)


Rating: 4 of 5

29.Feb.2012 at 21:03 by Nicolas Deslandes (6 comments)


Rating: 4 of 5

29.Feb.2012 at 21:47 by Jean-Marc Fontaine (18 comments)

Nice talk. I would have loved to get more technical details but time was lacking.

Rating: 4 of 5

29.Feb.2012 at 21:50 by Gourav Tiwari (3 comments)


Rating: 4 of 5

29.Feb.2012 at 22:47 by Hugo Hamon (124 comments)

This closing keynote was very enjoyable and impressive! The website has to deal with so many huge amounts of hits (+100M HTTP requests) and data (8 to 15Gb of logs per hour!) a day. It was very interesting to discover the architecture behind the scene with HA Proxy, Varnish, PHP & Symfony, Redis, MySQL... softwares. It also showed that choosing a web framework like Symfony2 doesn't mean killing performances as long as you use a solid architecture in front of it.

The talk was really good but I expected a bit more technical and business details. By chance, we had time to know more at the speakers dinner.

Rating: 4 of 5

01.Mar.2012 at 16:34 by Michael Peacock (17 comments)


Rating: 4 of 5

02.Mar.2012 at 16:58 by Anonymous


Speaker comment:

02.Mar.2012 at 19:31 by Eric Pickup (10 comments)

Thanks for all the feedback. I've never spoken to such a large group of people in the past and I've never once actually used Power Point before (I usually use Prezzi but I had no idea how I would be able to upload those slides). The positive feedback has been a great relief, I was mentally prepared for a fail LOL.

I'll be posting the slides as soon as they confirm my ownership of the event. The whole thing was also filmed by the Confoo guys and that should be online pretty soon (the Confoo guys are probably up to their ears with work right now).

Thanks for attending everyone. If you have questions, I have created a Twitter account (never had one before) and can be reached at EricPickupYP or you can email at the address in the slides. The Twitter account is only for technical stuff so don't worry about the content.

Rating: 5 of 5

02.Mar.2012 at 21:12 by Mathieu Godon (11 comments)

Great presentation.

Rating: 5 of 5

02.Mar.2012 at 23:35 by Olivier Leger (9 comments)

Useful talk and like others I would had have a little bit more technical details on the architecture.
I've sent you an email for the questions I had few hours after your talk.

Speaker comment:

03.Mar.2012 at 12:08 by Eric Pickup (10 comments)

The slides are online.

I had to walk a fine line between the details and time. And this was about an entire stack of software. Future presentations more focused on specific elements would be more detailed.

Rating: 4 of 5

04.Mar.2012 at 14:00 by Guilherme Blanco (25 comments)

Great talk, very depth in infrastructure architecture, but I only think it lacked a bit in application architecture.

You covered a lot the infrastructure, but the only parts that you touched you mentioned things like "we had to make many optimization at Symfony code"... other one: "We rewrote a few Symfony components for our needs".

I saw that you have put your team to deliver some enhancements back to Open Source community. It's a great move, but maybe you could provide more if it's not business specific code? =)

Rating: 5 of 5

05.Mar.2012 at 09:23 by Anonymous

Great presentation!

Rating: 4 of 5

05.Mar.2012 at 09:24 by Anonymous


Rating: 5 of 5

05.Mar.2012 at 09:57 by Anonymous

Congratulations Eric!!! You're the BEST... as usually =)!

Rating: 4 of 5

05.Mar.2012 at 15:38 by John Ashton (1 comments)

Great Job!

Would have been nice to include more info graphics to make the slides more exciting.

Rating: 5 of 5

06.Mar.2012 at 07:48 by Jean-François Côté (14 comments)

I was looking forward to see how you structured the architecture and how fast it is. I was surprised!

Rating: 5 of 5

09.Mar.2012 at 17:46 by Wil Moore III (28 comments)

Eric,

Excellent job; you solidified a few things for me with this talk that I had been contemplating. Thanks a lot for giving back by sharing your experience. The community will be better for it.

BTW, congrats on re-proving that high-traffic sites/applications are just fine with correctly architected PHP.

Speaker comment:

15.Mar.2012 at 20:23 by Eric Pickup (10 comments)

Guilherme, given the scope of the talk, the fact that organizers suggested not showing code because of the size of the room, plus Hugo being there providing very detailed Symfony information I decided not to spend too much time on Symfony2 or our implementation. Anything I covered would require an understanding of how Symfony works so I would lose a lot of the audience.

We have buy in on from high for giving back our improvements. Often they are too unique to be of interest to the general public. For example our Redis service is very tiny and only supports the features we needed, it isn't robust enough to be of interest to anyone else yet.

Developers from Manwin have posted patches to JQuery and other open-source projects though. Not all devs use their Manwin email addresses when doing so but it happens and is supported.

Speaker comment:

15.Mar.2012 at 20:34 by Eric Pickup (10 comments)

Wil, thanks. My goal more than anything was to show people that there are software packages beyond the standard LAMP base that offer great benefits if you spend a bit of time on them. While it wasn't the best fit for our requirements, MongoDB is also excellent.

If our requirements change, there is no reason why MongoDB might not become part of our stack. I know other projects at Manwin are experimenting with it. I'll always use the best tool for the job and not try force a tool I know to do things it wasn't designed to do.

We are always experimenting with interesting new technologies that might help us. A lot don't scale or are too buggy. But we try them all. Hopefully we can continue to give back useful information about what we learn.

Given that Facebook is written in PHP, I doubt that PHP really needed any more proof IT'S scalability but hopefully I helped with some ideas about other techs that we have learned do scale.

Rating: 5 of 5

24.Mar.2012 at 16:56 by Anonymous

Thank you very much. Very interesting information about your project!

Rating: 4 of 5

24.Mar.2012 at 17:09 by Anonymous

For your reverse caching, you should rather have looked at trafficserver or G-Wan.

Rating: 4 of 5

25.Mar.2012 at 10:26 by Matt Ryanczak (1 comments)

Is there a video of this talk?

Rating: 1 of 5

30.Apr.2012 at 10:31 by Anonymous

You could save alot more overhead by getting rid of varnish and handling the caching in redis. You can also get rid of the bloat of Symphony to gain another 5% performance

Speaker comment:

17.May.2012 at 21:01 by Eric Pickup (10 comments)

Video is online at http://www.youtube.com/watch?v=RlkCdM_f3p4

Speaker comment:

17.May.2012 at 21:07 by Eric Pickup (10 comments)

Symfony is bloated and we could have cut a lot more than 5% in webserver performance by cutting it. Our tests indicated a 30% performance penalty compared to our in-house framework which was very lean. With Varnish, we pay much less of a price.

However, Symfony allows us to develop faster and adding an extra couple of servers to handle a heavy framework makes business sense since servers are cheap compared to the man-hours of not using it.

© Joind.in 2014