Let’s Ride: Streaming Media services Architecture
An Executive on a motorcycle
The filming of a company video for IT onboarding was nearly wrapped, and I stood with a co-worker outside a company facility gathering some final footage of the CTO as he donned a helmet and rode away on his motorcycle. After a couple of good takes, we checked the camera and wrapped. The CTO asked how we thought it went, so I told him everything looked good and we would have it all edited in a couple of days. Then I shrugged and told him that it was too bad we had no way of sharing the video link when it was done. Because of necessary data security policies, we were unable to use public services like YouTube for sensitive internal content and could only distribute the large file through direct downloads, tying up the network and delivering poor quality of service.
Surprised that we lacked the technology in-house, he asked what it would take to make this content shareable and I suggested we could set up a streaming server on the network and self-host the content. He suggested contacting our IT director and gave us $10k for a proof of concept. Having built streaming services previously for a startup, this was enough to stand up a virtual Linux server and NAS drive with enough left over to purchase a perpetual license for a Wowza Streaming Engine server.
we’re mass communicatinG!
With a single streaming server and connected network storage to host video files, we were able to send out the video and soon hosted many more learning videos. This was an amazing capability available with the new system, but we also gained the ability to stream live video and soon were presenting live streams of “town hall” style events, virtual classes, and guest speakers to any company computer connected to our secure network domain.
Growing capacity and new capabilities
With the addition of some embedded code, chat services could be incorporated into the streaming player pages to connect a live conversation to the video stream. With a custom tuned low-latency RTMP transport and fast encoding, viewers in Wisconsin or Florida offices were able to ask questions to speakers streaming from the company’s headquarters in Kentucky. After a brief delay of a few seconds, their questions could be answered and issues addressed. At the time there were few services available that could provide this function, and the price for each vendor-streamed live event could pay for a server. By this time, it was obvious that we were already saving money, not to mention avoiding security risks like hijacked livestreams and leaked connections to sensitive presentations. The small bet was paying off!
The service was growing in popularity with leaders in the company, and we were asked to host larger events with more viewers. The single server was configured to handle about two thousand simultaneous streams, but we soon began to exceed this capacity. Clearly, it was time to grow. With leader support and a $20k budget, the system was expanded to an origin-edge configuration with four additional servers to redistribute streams like spokes from a hub. I wrote code to distribute the load across the edge servers and expanded the system to handle ten thousand viewers in total. Additional software was written to index recordings and generate shareable links, which helped simplify the hosting process as the userbase continued to grow.
Over the next few years, I continued to train other content producers across the company to host their own live and on-demand streams with this system. This network of streaming communicators gradually expanded to support over 200 departments and enterprise organizations, with live events hosted weekly. A live company radio station was built on this platform, and offered streaming audio 24×7 with custom programming for employees. Humana Radio was eventually streamed to thousands of company-authenticated devices including laptops, phones, and intercom systems.
An additional infusion of $40k purchased more NAS for on-demand hosting and seven more server licenses. At this point, live content could be streamed to two origin servers, supported by eight additional edge servers spread across two datacenters. This allowed redundancy and failover capabilities required for true enterprise-scale architecture. For example, if connection to one server was broken, or an entire datacenter went offline, it wouldn’t stop a stream in progress. This basic 10-server CDN allowed us to grow the capacity to support 6 million annual streams alongside over 5,400 hours of on-demand content. These numbers are small by YouTube standards with a global userbase, but the program’s impact for 55,000 employees was significant.
Calculated savings and ongoing cultural change
The system was decommissioned in 2021, as other services were contracted to provide live event streaming and on-demand video hosting. Based on the monthly contract for these new vendor services, I was able to calculate cost-avoidance for our self-hosted streaming service based on billing for file storage, bandwidth, and concurrent connection capacity. Conservatively, the same services provided by my system architecture would have cost approximately $3.3M over 12 years. This does not include the cost of travel required previously for training events, town halls, and all-staff meetings which had transitioned to streaming broadcasts. In addition, we would have avoided the cost of event production by external vendors by training internal staff. But the greatest achievement of this system was a cultural change that encouraged greater communication from company leaders, with a platform that allowed direct audience feedback. This change had laid the groundwork for future behavior when services like Zoom and Microsoft Stream were made available, with participatory broadcasts made regularly across the enterprise.