I spent this summer programming for the New York Times. The experience was valuable in quite a number of ways, and I really enjoyed myself. Here’s how it went down.

Sometime in March, I saw a post on the Times’ Open Blog, a blog dedicated to technical topics, and written by developers - not suits. The post I noticed explained how the development group in the Times was looking for several technical interns for the summer. I decided to apply, and I emailed the provided address with my resume and some more information. The blog post explained that, in order to help the recruiting team decide on a candidate, any applicants should provide a short explanation of a new idea the Times might implement. My submission is reproduced below.

One thing that I think has been lost in the transition from print newspapers to online news is the collection of newsboys on city streets shouting the headlines of breaking news. However, blasting breaking news to everybody is crude, annoying, and expensive - with the Internet, we can do better. I’d love to be able to subscribe to some sort of SMS service, Twitter bot, or somesuch to deliver breaking news directly to me, as it happens, unobtrusively.

Problems include:

• obtaining feedback for training the relevancy filter
• determining which is breaking news (“beta” users get news first?)
• infrastructure (how can we decentralize?), and
• revenue model (if any).

Current trends include:

• the ‘breakingnews’ Twitter bot
• Google News’s analysis, and
• RSS readers of all kinds.

Personalization options:

• the aforementioned relevancy filter
• personal stock alerts
• keyword alerts, and
• presence management.

I think this is a relatively unexplored area, an interesting problem, and a useful tool for news consumers and producers/creators/publishers.

Unfortunately, I didn’t get to work on anything related to this at all.

I had applied to several other companies, but after a pretty technical phone interview from my future boss Elizabeth Baron, the Times made me an offer.

I had several reasons for accepting the offer from the Times. First, the Times has a very large and performant website, and I was intrigued about how they solve their scaling problems (a CDN, internal load balancers, plenty of hardware, a MySQL access layer, and some other things). Second, the technologies used by the Times (Linux, Apache, MySQL, PHP, SVN, and other Free/Open Source software) were a perfect match to my expertise, and I had a very easy time getting started. Finally, I was impressed by the freedom given to developers at the Times. In retrospect, the Times seems to have an efficient development group for its size, and at the start of the summer I noticed several signs that clued me in to this (Agile/XP practices, an SCM system, generous use of external libraries and software, and transparency of development with, e.g. the Open blog).

So, what did I work for 3 months? Nominally, one REST API and an application that uses it, but, in reality, several things. My (and my fellow interns’) main project was to create an online submission form for Wedding Announcements to be published in the Style Section of the Sunday Edition. In addition to the public-facing submission form, there was a back-end database and file storage infrastructure, a REST service that re-exposed the information internally, and an internal application that allowed editors, reporters, writers, fact-checkers, and a myriad of other people to collaborate and collectively create a story about the wedding.

The project was written in PHP, with the data stored in MySQL, cached in memcached, and photos were stored on the filesystem. However, over the course of the project, I ended up using Amazon’s S3 an EC2 Web Services as well. I had never worked with AWS before, and I was amazed at how easy it was to have my managers agree to offload a pretty significant part of our functionality to Amazon’s servers. I ended up learned a lot about S3 and EC2!

Another valuable learning experience came from creating the REST API. Our API implemented all the details of REST (as opposed to being a simple read-only API) and combined textual data as well as binary photos. Building this API forced me to learn about all the implementation issues with REST, and I actually had the opportunity to make some improvements to the internal REST framework used by all the NYTimes.com APIs. I wasn’t given direct commit access, but my patches were reviewed and incorporated quickly after they were ready.

I also got a chance to hack on some other cool things. For example, I installed a Jabber server on an internal box, and I wrote a small Ruby bot that would let all the interns and other people working on the Weddings project know whenever there was a commit made to our module. That was a fun little script to hack on!

The development team is pretty well structured at the Times. Everyone I interacted with knew their domain quite well, and there was plenty of communication between teams. My meetings were informative and short, and I had no problems getting my questions about the internal setup answered. My boss and the other managers were helpful in connecting me to the people I needed to talk to, and guiding the scope of the project. Our team met once a week and reviewed all the tasks that we had worked on, and an internal install of XPlanner helped keep us all on task. Most everyone in the department simply used GMail, Google Calendar, and AIM or Jabber/Google Talk to communicate, so I fit right in.

The working environment was top-notch - mostly because I simply provided most of it myself. I brought in my laptop every day, and I used my own keyboard (the excellent Das Keyboard, much better than the flimsy mushy Dell keyboards that are provided by default). I could have of course used the provided PC, but working on my laptop (with their external monitor) was much better for me, and I didn’t have to install Linux on their desktop machine.

By the end of the summer, I was reluctant to leave (the Times and my East Village apartment paid for by the Times), but I was grateful for the incredible learning experience. I learned all about S3 and EC2, all the intricacies of REST, and more about the internal quirks of PHP than I ever wished to know. I helped build an application and API that utilized all the internal infrastructure of the Times, including the Linux servers, the database access layer and caching mechanism, the web server, and the PHP libraries, so I became familiar with the total infrastructure used to serve the extremely popular NYTimes.com. All the technologies and techniques I learned were helpful and interesting.

I look forward to applying everything I learned in future internships and jobs, and I consider this summer spent in New York City one of the funnest experiences I’ve had.

This post has also been cross-posted at the Open blog: “Summer Internship Wrap-up”.