Business Continuity in Any Situation

Migdal, Israel’s leading insurance company, selected MedOne’s data center to host its disaster recovery site.When Migdal designed its strategy of business continuity in any situation, with no compromises or shortcuts, it also decided to locate its main backup site in the MedOne’s data center in northern Israel.  “Migdal’s strategy is to generate a competitive advantage, based on reliability and availability at the highest possible levels – usually at even higher levels than required by the regulator – out of a profound concern for and commitment to the policyholders’ rights and in order to maintain a high level of service even in a disaster,” says Shai Basson, Migdal’s Deputy CEO and Head of Technology and Organizational Resources Division. Risk management is the axis around which insurance companies’ business revolves.  Their expertise in minimizing their exposure to operating, financial, and environmental risks is what enables these companies to “buy” the risks from companies and individuals – at a price worthwhile to the customer, who can manage risks in the same manner – and still make an overall profit.  A significant element in insurance companies’ risk management strategy is strong protection for their information systems in order to ensure business continuity under any situation, including large-scale natural disasters, war and terrorism, criminal actions, and even human error. The threshold level set by the regulator for protection of the insurance companies’ information systems ensures that service to the policyholders will continue in such cases. However, insurance companies guide by a strategy of operational excellence will not suffice with the regulator’s requirements. “The companies invest extensive resources in creating reliability and availability which will enable them to continue providing services to their clients in any situation,” says MedOne CEO Ronnie Sade.“ Our hosting facilities in central and northern Israel are the only ones that provide a solution to the growing need to ensure business continuity and protection of information systems at the highest global standards.”


Risk Management According to the Black Swan Theory

Where clients are concerned, explains Basson, “the rationale for insuring their business is the critical need to prepare a safety net to protect against the possible results of destructive events.  It is clear from this definition why the most critical feature of the safety net is its ability to survive in full working order and to continue to function also in an event causing damage to the entire infrastructure.  For example, an earthquake insurance policy is of no value if the information systems of the insurance company itself cannot withstand earthquakes.  This is the reason why Migdal’s management decided a few years ago to comprehensively revise the company’s business continuity plan and to adapt it to ‘extreme’ scenarios, even if the probability of occurrence is low.  To illustrate this point, experience shows that companies which were unable to get their systems up and running again within 72 hours of November 9, 2001 (since their DR systems were housed in the same office buildings which collapsed) did not resume business as usual even two or three years later.”  “In the 21st century, every business continuity plan (BCP) must be based on a disaster recovery program (DRP) for the information systems,” says Sade, CEO of MedOne. “At MedOne, the infrastructure of a protected disaster-proof information system is an alternative data center site specifically designed to survive any challenge, be it acts of nature or mankind.  The activity can be transferred to this site from the main production site safely and within a very short time, with no tangible disruption of the ongoing work. Building such sites, operating them professionally, and hosting the main information systems, their production, and the DRP infrastructure of commercial companies are MedOne’s speciality.” It is well known that information storage and processing is the “production floor” of companies in the financial sector in general, and of insurance companies in particular. For us, the survival of the historic information in the storage systems is even more critical than for the banks,” says Basson, “because we are not only handling recent transactions.  We have policies that are 50 years old or more that are still a significant part of the clients’ hedges, and we can’t let the clients down being our business partners, for all intents and purposes.  For us, history is alive and kicking in our daily operations, and the preservation of the information is therefore priceless and of supreme importance.  This insight led Migdal’s management to define an especially ambitious target for disaster recovery and to make storage protection the key element.  The model selected is duplication and full redundancy of infrastructure and systems at two operationally independent physical sites over 70 kms apart. Each site is capable of providing full service at a high performance level independent of its mirror image.”

Zero Error Culture

The plan was launched in early 2011. Requirements were studied and specified in the first half of the year, and tenders were published and the second site replacing the main site on extremely short notice was selected in the second half of the year.   “Our requirements from the DR site were stringent and meticulous, because we realized that there could be no compromising on the quality of the facility to which we would shift in a disaster,” said Basson.  “In addition to withstanding the full gamut of natural disasters from earthquakes to floods, we required protection and security against acts of hostility and war damage; an independent power supply that will last at least 72 hours; 24/7 monitoring, management and control, and no less important –  a zero error operating culture.  The site we chose as most suitable was the MedOne hosting facility in the north.”   MedOne’s facilities were constructed as special data center facilities; the company focused its business on hosting information systems for clients like Migdal, which could appreciate the benefits of complete reliability for business continuity.  “Our hosting facilities enable an organization to apply a solution that ensures business continuity of the highest quality, and provides a solution for the organization’s changing needs in the future.”  “The facilities were built in compliance with international standards, and are today available for hosting main computer production sites in addition to hosting DR sites.  Many companies concluded that it is economically more viable to rent space and purchase services on our data center sites than to build their own facilities.” 


Tie-breaker on a Small Scale

In addition to its site in northern Israel, MedOne operates two secure sites in the Greater Tel Aviv metropolitan area. MedOne offers its clients a total over 13,000 sq.m. in floor space in a secure and protected underground structure with power backup.  The facilities can work independently of the Israel Electric Corporation and other infrastructure providers for up to 72 hours.  The storage space is divided into “cages” and “hosting theaters” which are locked and prepared for hosting servers and storage systems of any configuration.  These sites provide 24/7 outsourced hosting services, an economic concept which draws its power from expertise, economies of scale, business focus, and operational flexibility. “The transition to a model of renting managed data center services has gained great momentum in recent years,” Sade says.  “Setting up, maintaining, upgrading, and managing a data center is very distant from the business focus of all our clients, who usually have no interest in owning real estate properties, and certainly not properties dedicated for the specific purpose of storing information systems.  The economies of scale and specialization are a ‘tie-breaker’ on a relatively small scale, because massive investment in a high-quality facility is a heavy financial burden for a business seeking flexibility and agility.”  “The absolute advantages of having the main site hosted in a managed and secure data center are reflected in the overall cost of the business continuity plan, which is unwilling to compromise or cut corners,” Sade says.  “If you want to guarantee continuous services in any situation, including natural disasters on a nationwide scale or an all-out war, the idea of doing everything within the organization requires very large investments to implement a BCP solution, which most organizations in Israel cannot afford.  A cost-benefit BCP account will show you that it is better to also place your production site in an external data center facility, and not just the DR site.”

Replacing Airplane Engines in Mid-flight

Migdal’s information system obviously serves all the company’s 3,400 employees, but it also serves fifty times as many direct and indirect users.  These include over 10,000 insurance agents, who rely on Migdal’s services in all their daily activities, including policy proposals, clarifying redemption values, collection, payment, and any disruptions of work continuity, and approximately 1.7 million private and business clients. In terms of the overall re-hosting of the systems, we achieved a return to operational routine within 10 hours, less than half the time to which we are committed according to the policy defined by company management.”  In setting up the data centers in the DRP project managed at Migdal, all the infrastructure storage and communications systems were also significantly upgraded.  A key element in the solution is a switch to 98% virtualization of the computer environment in updating and replacing the backup storage systems.  At the same time, an independent robust communications apparatus was established between the facilities, based on a dark fiber link in two independent routes. “The project in general involves an overhaul of all of Migdal’s operational elements,” Basson explains.  We left nothing untouched. We replaced the classic servers with blade servers, by replacing the storage farm (total capacity of 1.3 petabytes) with modern high-end solutions; we switched to an Exadata platform for Oracle databases, and we converted all the applications servers from a Unix environment to a virtual Linux environment. What Migdal’s management still finds astonishing is that all this was done without disrupting the organization’s regular business.  To illustrate the point, imagine replacing two engines of a passenger jet during a flight where neither the passengers nor the pilots are aware that anything unusual is going on outside.”  In November, re-hosting of the secondary site of MedOne’s data center facility in northern Israel was completed, following a series of ultra-realistic integrity? tests.  “The operational integrity? of a DR site has two aspects:  one, immediate backup – a mirror image of each system using a reflection mechanism, and two, the ability to relocate the company’s activity from the main site to the secondary site within a short time (less than one workday), with almost no loss of information, meaning that transactions that took place during the re-hosting were not lost,” says Basson. “In our case, this involves re-hosting of a number of separate work environments, each of which is essential for operations and all are interdependent. These include VMware on Wintel servers, Solaris on Sun servers, Exadata on Oracle servers, and AS/400 on IBM computers. It was proven in trials that we can switch between the sites in an extremely short time without the users being aware of any change in performance.”

MedOne Data Center Services

Migdal’s investment in information infrastructure, which will guarantee business continuity under any scenario, even the most extreme, has not escaped the notice of the rating company that keeps a table of survivability for the financial sector.  Migdal is so far the only company in Israel with an AAA rating – the highest rating grade.  By analogy, this high rating reflects the strength of three key elements in the investment we made,” Basson says.  We formulated a holistic BCP concept which does not confine itself to the regulator’s requirements, but is guided by organizational DNA – the aspiration to excellence in customer service – and is reflected in every detail of the solution that we developed in cooperation with MedOne. We applied for the first time innovative technologies meeting the definition of business continuity as a complete operative solution for continuity in organizational functioning under any challenging scenario.  We upgraded and updated the basic hardware and software in order to minimize the complexity of management at a general system-wide level by a synchronized transition to standard virtual platforms that ensure business agility and scalability without disruptions.  “I’d like to note that MedOne’s engineers were partners in the successful conclusion of the project.  These success stories reflect a blend of high professional capabilities and  advanced technology.  In our case as well, the physical features of the data center and the provider’s business commitment to be an important player in the data center segment led us to choose MedOne to host the Migdal site. However, in order to implement major infrastructure modification plans, with the IT manager taking responsibility for finishing the task within a tight and rigid timetable without disrupting ongoing work continuity, we need partners with a highly developed service culture, not just suitable physical assets.  I met these partners at MedOne, and I can only give them equal credit for the success.” “The service culture which Shai Basson is talking about is a reflection of the business base on which MedOne was founded,” Sade says.  “We are at the crossroads of four forces accelerating the demand for hosting services for organizations’ information systems:

  • Global regulatory trends
  • Israel’s unique geopolitical position
  • Added economic value (replacing capex with opex)
  • Increased awareness on the part of management of the critical dependence on information systems as a prerequisite for ensuring business continuity in both routine situations and emergencies.

“In order to achieve the highest levels of reliability and availability and make it possible for clients to continue their activity in any situation and under any condition, we have implemented the most stringent international standards in setting up the data center.  These standards include the building itself, the auxiliary and energy facilities, and meticulous work and security procedures.  It is clear that the human factor plays a key role in the company’s ability to fulfill SLA availability commitments to its clients.  We also invest in people, not only equipment, and develop human resources in taking a long-term view. Good employees dedicated to service and committed to professional excellence are what make the difference in quality – and clients like Migdal will testify to the effectiveness of our service attitude.”