The biggest change for Shopify was the requirement to store personal information in Canada. This required Canadian-specific infrastructure that we were able to develop through our recent move to the Cloud and Google Cloud Platform’s new region in Montreal. Using this platform as our foundation, we created a new instance of Shopify, in an entirely new region, to meet the needs of this industry. In our migration, we built several new Google Cloud Platform projects (all based in the Montreal region) which included key projects housing Shopify’s core infrastructure such as PCI compliant payment processing infrastructure and a regional data warehouse.
The core infrastructure, which runs on a mixture of Google Kubernetes Engine and Google Compute Engine, already existed in our other regions which meant adding another region was relatively straightforward. We used Terraform to declare and configure all parts of the underlying infrastructure, like networks and Kubernetes Engine clusters. We also took advantage of improved resiliency features in Google Cloud Platform, such as regional clusters. We structured our compute node clusters to segregate workloads, minimizing the noisy neighbour problem to ensure maximum stability and reliability. After a few months of building out this infrastructure, configuring and testing it, we had the first working version of this new regional infrastructure running test shops with a functional storefront and admin. That’s when we faced our next major challenge: scaling.
A major factor in our scaling ability were social factors—particularly, determining the behavior of cannabis consumers, an area with little to no available research. Most research focused on cannabis producers, whereas Shopify needed to figure out the behavior of cannabis consumers. We modeled a number of different traffic scenarios and provisioned enough infrastructure to ensure we could handle the peak traffic from each one. Some of the possibilities we considered included:
- A strong initial, worldwide surge of interest on storefront pages as curiosity about a government-run online cannabis store peaks
- Waves of traffic based on multiple days of media coverage across the world, with local timezone spikes
- Very strong initial sales in the first minutes and hours of store openings as Canadians rush to be one of the first to legally purchase recreational cannabis
- Possible bursts of denial of service attacks from malicious actors
We went through multiple cycles of load testing using a mix of different storefront traffic patterns, varying the relative percentages of search, product browsing, collection browsing and checkout actions to stress the system in different ways. Each cycle included different fixes and configuration changes to improve the performance and throughput of the system until we were satisfied that we would be able to handle all possible traffic scenarios. In addition, we modeled and tested different types of bot attacks to ensure our platform defenses were effective. Finally, we conducted multiple pre-mortem discussions and built out mitigation plans to address any scenario which would cause downtime for our merchants.
At the same time, we were solving how to keep personal information contained in Canada. This was extremely challenging as Shopify was built from day one with a number of storage and communications systems located outside of Canada, such as our data warehouse and network infrastructure. We examined each system for personal information to ensure that this information remains stored in Canada.
We ensured there were protections for regional storage in multiple places: inside the application, within the hosts, and at the network/infrastructure level. For our main Ruby on Rails application, we:
- Built a library which captured network requests and verified the requested host belonged to a list of known safe endpoints.
- Utilized strict network firewall rules and minimized interconnections to ensure that data wouldn’t accidentally traverse into other jurisdictions.
- Deployed the containers which house the main application with the absolute minimum number of secrets necessary for the service to function in order to ensure that any service outside the jurisdiction reached in error would simply reject the request due to insufficient credentials.
- Ensured the infrastructure used unique SSL certificates so data would not cross-pollinate between internal pieces of the system.
- Deployed all these protections, in combination with monitoring and alerting, ensuring the teams involved were notified of potential issues.
As launch day neared, we reduced the amount of change we applied to the environment to minimize risk. While the merchants were in their final testing cycles, we continued to perform load testing to ensure that the environment was optimally configured and ready. Having a successful launch day was critical for our merchants and we decided to scale the environment to handle five times the traffic and sales volume projections for launch day. Internally, we ran a series of game days (a form of fault injection where we test our assumptions about the system by degrading its dependencies under controlled conditions) for core infrastructure teams to validate that system performance and alerting was sufficient.
On launch day, merchants chose to take full advantage of the excitement and opened their stores one minute after midnight in their local time zones. That meant we’d see both retail and online launches starting at 10:31 PM EDT on October 16th (Newfoundland and Labrador) and continue through every hour until 3:01 AM EDT on October 17th (British Columbia). And at 12:01 NST, the first legal sale of cannabis in Canada was made on Shopify’s point of sale in Newfoundland followed by successful launches in Prince Edward Island, Ontario and British Columbia — all with zero downtime, excellent performance and secure storage and transmission of personal information within Canada.
Being part of launching a new retail industry and acting as a trusted partner with multiple licensed sellers while building infrastructure with regional data storage requirements, all on a strict deadline, was quite a challenge which required coordination across all Shopify departments. We learned a lot about what it takes to support regulated industries and restricted markets, knowledge which will help us support similar markets in the future, both in Canada and throughout the world. A number of the technologies and processes we developed during this project will continue to be improved and reused to support future deployments with similar requirements. Overall, it was incredibly rewarding to be part of a historic launch by contributing to and supporting the success of licensed recreational cannabis retailers throughout the country.
Intrigued? Shopify is hiring and we’d love to hear from you. Please take a look at the Production Engineer and Senior Technical Security Analyst roles available.