As published in NZ Reseller news, 1 December 2021
As a customer loyalty specialist dealing in huge volumes of data, Loyalty NZ knows a thing or two about system performance, lessons sometimes learned the hard way.
The company's multi-year shift from on-premises systems onto Amazon Web Services' (AWS) cloud, therefore, is instructional. But no more so than its advice on how on how to build a mutually beneficial vendor relationship in the era of the cloud.
In the lead-up to AWS's annual conference today, Loyalty NZ doubled down on its engagement with AWS, announcing it is developing a customer engagement platform to deliver highly automated, personalised and relevant offers to Flybuys members.
The new platform comes as Loyalty NZ neared the end of a multi-year project to migrate all its systems to the AWS cloud.
Brian Ferris, Loyalty NZ's chief technology and data officer, said when he looked back, Loyalty NZ had hit the limit of what it could do with its heritage on-premises systems a few years ago. It took 27 hours to do the daily data load in the old data warehouse, for instance.
"You can do the maths on that and figure out the obvious problem there," Ferris said at an AWS press conference. "Some of the analysts' queries were taking six hours to run."
When business partners and the organisation needed queries answered, often only parts could be executed, and even then slowly.
Not sustainable
Fifty-five TB of data was growing by around 2GB a day and Loyalty NZ found it was having to economise on other foundational jobs, such as model training, because these were taking so long to run.
"That wasn't sustainable," Ferris said. "That wasn't going to get us in place for another 25 years. We knew we needed to make a move."
A shift to the cloud was targeted to fix some of those challenges and it was "really successful".
"We've got the results that we wanted and they are business outcomes, not just technical ones," Ferris said.
The overnight load now takes four hours, real-time data feeds have been developed and query speeds have been reduced by up to 99 per cent. That in turn changed the way the team could respond to the organisation with the scalability of the cloud allowing Loyalty NZ to pump up capacity on demand whenever needed.
"For Black Friday just gone, we ramped up the capacity of our e-commerce site by a factor of six and then we put it back down again when demand was over as the demand tailed off," Ferris said. "It's just not something that can be done in an old-school datacentre on-prem."
It was not just about the speed either but supporting a total change to the way Loyalty could operate and providing the ability to respond to opportunities.
One challenge, however, is the cloud space is fast moving and complex. A constant flow of new services and capabilities were hard to keep up with and it was hard to understand what Loyalty should be doing and evolving.
RFIs and RFPs don't cut it
To deal with that, users needed a partner, not just a vendor, and needed to invest in that relationship.
"What that means is we have a strategic partnership rather than a transactional purchase of services," Ferris said. "I think organisations that want to nickel-and-dime every decision with these big bloated RFIs and RFPs being floated out in the marketplace can't expect great partnership.
"When AWS got involved and helped look at what we were doing and what we were planning to do, that took six months off our project plan and more than US$50,000 in savings, just from them seeing what we are doing and from their knowledge and experience identifying better ways to do it."
AWS reviews of usage and architecture had also cut Loyalty NZ's spend by US$10,000 a month.
Partnership was the key and not just externally facing.
"We don't do any tech for tech's sake," Ferris said. "In my opinion that is just a waste of money building shelfware.
"Everything we do or are going to do is directly tied to enabling our mission to build NZ's leading data driven customer engagement business, one that is valued by our partners because we positively impact their customers."
Bridget Lamont, chief customer officer, said Loyalty NZ believed technology and business strategy was all one thing.
"It may sound a little bit like semantics, but it's very real at Loyalty NZ," she said. "We don't think this is particularly common. You don't often see disparate parts of an organisation, particularly the marketers, the tech people and the data people truly working in partnership to get the most out of that really important tech and data investment."
At the end of the day, it was about benefiting Flybuys members, partners and consumers.
"Our learnings from this would be: don't transact," Ferris said. "Invest in building trusted partnerships both internally and externally and then you will get the most form these amazing technical advances we are seeing from our partners like AWS."
How cloud enables partnerships
Ferris said while AWS was not a charity and was out to make sales, the way the company engaged was to actively try and help customers grow. From that, AWS gets a share of a bigger pie as opposed to others that constantly tried and talk customers into making purchases in order to shift licenses or products.
"As an example, our solution architect has a KPI on him at AWS to reduce our spending with AWS," Ferris said. "He is being incentivised to actively find ways to save money where we are spending."
That was smart because it meant Loyalty NZ grew faster and invested more, bringing in more services and doing more things.
"We don't have that feeling we are going to be ambushed in every conversation or tricked into buying new licenses we may or may not need," Ferris said. "It tends to be more organic. To be honest the investment AWS has put into trying to find solutions to our business problems I found very refreshing."
Both sides decided to take a bit of a risk and it had paid off, he said.
Loyalty NZ stores customer data on Amazon simple storage service (S3) with Amazon managed workflows for Apache airflow, a managed orchestration service that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale.
The company will be completely out of its on-premises data centre by the end of February next year, slightly behind schedule due to the impact of COVID-19, when it shifts its Flybuys website.