Client Story: PayPal M&O Stamp of Approval
"Uptime Institute’s M&O Stamp of Approval helps us to validate our enterprise data center operations practices and shows us, through third-party validation, that we are indeed going in the right direction, and doing things the right way. I think of it as trust but verify."
Richard Reyher
Sr. Director of Data Centers
PayPal
Overview
Fueled by a fundamental belief that having access to financial services creates opportunity, PayPal (NASDAQ: PYPL) is committed to democratizing financial services and empowering people and businesses to join and thrive in the global economy. Our open digital payments platform gives PayPal’s 277 million active account holders the confidence to connect and transact in new and powerful ways, whether they are online, on a mobile device, in an app, or in person. Through a combination of technological innovation and strategic partnerships, PayPal creates better ways to manage and move money, and offers choice and flexibility when sending payments, paying or getting paid. Available in more than 300 markets around the world, the PayPal platform, including Braintree, Venmo and Xoom, enables consumers and merchants to receive money in more than 100 currencies, withdraw funds in 56 currencies and hold balances in their accounts with PayPal in 25 currencies.
Opportunity
The Uptime Institute M&O Stamp of Approval process is designed to incorporate the learnings of over 20 years of site visits at critical facilities around the world into best practices to help data center owners and operators introduce consistency, transparency, and industry best practices into data center management and operations processes.
Uptime Institute has analyzed countless data and has found that the majority of reported data center outages are directly related to human error, whether in the form of operator or management error in decisions regarding staffing, maintenance processes, training, or overall rigor of operation. These types of outages can occur despite the infrastructure topology of the data center. On the opposite side, a well-managed data center can achieve availability to the business well beyond expectation and typically based solely on the topology of the installed infrastructure.
Since human error is responsible for the majority of all data center incidents, taking a holistic approach to the assessment of staffing and organization practices, maintenance and operations activities and management and planning made sense to Shawn Tugwell, Director of Data Center Architecture & Engineering at PayPal. He saw this as a chance to make sure his team went over every detail, and to use the M&O Stamp of Approval assessment as an important tool to cover every aspect of operations with the 3rd party perspective and expertise of Uptime Institute.
Tugwell wanted to keep pushing his team to make sure no stone was unturned, asking, “How can I make this harder to be sure that nothing was missed?” He also wanted to look at how he could in turn take this same methodology and apply the M&O Stamp of Approval and independent assessment to the colos that work with PayPal to be able to determine how each of the individual colos working with PayPal were doing, and what could be improved upon.
Solution
The M&O Assessment is a way of directly measuring the operational effectiveness of the teams within the data center by focusing on behaviors within five “components” and dissecting each identified behavior to determine if that behavior meets the three principles of Operational Sustainability – Proactive, Practiced, and Informed. The five components these behaviors align with are 1) Staffing and Organization; 2) Maintenance; 3) Training; 4) Planning, Coordination, and Management; and 5) Operating Conditions. The PayPal Data Center Operations team in PHX01 is a seasoned team with years of experience under their belts, but they had never been assessed by Uptime Institute. And with Paypal’s recent commitment to five-nines (99.999%) of availability across all data centers, regardless of Tier level, the assessment was a way of validating that the team could achieve their goal.
The process began in early 2018, when PayPal Data Center Services received the Uptime Institute M&O Stamp of Approval for the PHX01 data center upon completion of the assessment. The company received 96.2%, one of the highest first-time scores ever received, a great achievement that clearly illustrated PayPal’s leadership in implementing management and operational best practices. The outcome is a direct result of the tenacity of the DC Ops team in Phoenix, Arizona and reflects the team’s dedication to process-driven operations.
Uptime Institute’s team was also impressed with PayPal’s implementation of the ServiceNow platform. The DC Ops team leverages ServiceNow for procedural approval workflows, maintenance management for data center critical infrastructure, incident management, as well as space and power planning. When PayPal explained that ServiceNow had only been implemented for a matter of months within DCS, the Uptime Institute representatives were quite pleased with the system’s level of maturity.
PayPal then put PHX02 and PHX04 through the same M&O Stamp of Approval process, and each site received another industry leading first-time scores of 100%. With these assessments, the competitive PayPal team continues to raise the bar, with a clear understanding of how to get to the next level.
PayPal’s commitment to Five-Nines of availability require what is known in the Data Center industry as “24 by forever” staffing. Within the component of “Staffing and Organization” it was identified that the Facility Operations group met this staffing criteria by having at least three facilities technicians on site at all times. These technicians undergo rigorous initial and continuing training to ensure they possess a comprehensive and in-depth knowledge in various disciplines and a variety of systems and equipment.
The Facility Operations group is also responsible for the successful execution of preventive and corrective maintenance at the Data Center. The team creates and maintains on-site maintenance standards and procedures to ensure maintenance is successfully completed and documented. The high standards of the Facilities team, in partnership with PayPal’s Data Center Management team, resulted in zero deferred maintenance items at all Data Centers. A thorough and effective Preventive Maintenance Program allows PayPal to keep equipment in like-new condition, which is essential to deliver Five-Nines of availability.
Training is at the forefront of PayPal’s Data Center Services Team. New hires within the Facility Operations Team are required to complete an initial training program prior to being permitted to work on shift. In addition to the initial training program, a vendor training program also exists for vendors who will be performing on-site maintenance. The vendor training includes required training topics and various work rules for the Data Centers.
Effective Planning, Coordination, and Management is another key aspect of delivering PayPal’s uptime objectives. The team has various Procedures and Standards on-hand, available in the ServiceNow Knowledge Base, to ensure consistency across the Data Centers and to reduce the risk of human error events at the site. In addition, PayPal’s financial team creates, reviews, and tracks budgets to ensure each Data Center has an adequate budget to support business objectives. These budgets are maintained independently from site to site to ensure the proper focus and attention can be placed on specific data centers.
Lastly, PayPal utilizes consistent and documented load limits on components to drive successful Operating Conditions. Setpoints associated with components and equipment are established as risk mitigation tools as well as focus on the overall cost of operations at the Data Centers. Alarm setpoints at the facility are set to allow reserved capacity for switching and failover, while setpoints are established in partnership with an engineering consulting firm.
As the organization continues to grow, consistency becomes even more important as the team adds colocation companies to their data center mix. The Data Center Services team then took their success and rolled the M&O Stamp of Approval program over to the organization’s colocation data center vendors. Now, new contracts require data center vendors to have achieved the Uptime Institute M&O Stamp of Approval and maintain a valid award status while PayPal is a customer.
The Data Center Services Team’s overarching goal is to continue to reduce cost to the business, and any data center downtime is a loss to business. The team also wants to make sure that all colos are in a position to do their best, and the M&O Stamp of Approval assists in these goals. The M&O Stamp of Approval also helps teams to make sure that no surprises occur as new teams and operational practices come into play. PayPal was also pleased with the excellent support they received from their BGIS team for critical staffing and program maintenance which was also key to the program’s success.
PayPal’s excellent scores also provided the team with validation of a job well done and assist the data center management team in clearly illustrating to their executive team that they are doing a tremendous job and can ensure their customers their payments are safe with the PayPal digital payments platform.
“Our culture meshes well with the M&O Stamp of Approval, helping us to makes sure that processes are fully in place to ensure everything is running smoothly, from both our enterprise and colo teams.”
“We are proud of our teams and are excited about achieving industry-leading 100% scores in our PHX/02 and PHX/04 data centers, our teams care about continuing to achieve our best work, and Uptime Institute helps us in doing just that.”
Shawn Tugwell Director of Data Center Architecture & Engineering
PayPal