The Challenge
The retailer’s legacy integration platform, Microsoft BizTalk, had become a significant bottleneck in their operations, presenting several non-functional challenges:
- Lack of maintainability: Troubleshooting and maintaining BizTalk required highly specialized skills, which created dependency on a few experts and slowed down operations.
- Poor resilience and transparency: Errors within BizTalk were difficult to diagnose, leading to frequent disruptions.
- High costs: BizTalk’s licensing model made it an unsustainable solution for long-term use.
With the need for improved transparency, scalability, and modernized infrastructure, the company turned to Create IT for a seamless migration to a custom-built solution, which could replace BizTalk while maintaining existing integration functionalities.
"This project was filled with significant hurdles, from the lack of documentation and the need for constant investigation to navigate through old workflows, to the uncertainty of the issues we encountered along the way. With tight deadlines and many unknowns, we had to be agile, adapt quickly, and problem-solve on the go." says David Botelho, Solutions Architect at Create IT.
The Solution
Create IT partnered with the retailer to design and implement a modern, tailored-made integration platform. The solution aimed to replace BizTalk's functionality while addressing its key weaknesses.
- Cloud-native architecture: The new solution leverages RabbitMQ for message queuing, adopting a publish-subscribe model for integration flows. This architecture ensures efficient communication between systems and significantly enhances the platform's resilience.
- Modular workers in .NET: The new system utilizes .NET workers as consumers and publishers, providing flexibility and better control over integration processes. These workers are deployed in Docker containers on a Linux environment, enhancing portability and scalability.
- Improved monitoring and visibility: BizTalk’s difficult monitoring was replaced by a combination of New Relic for real-time visibility into system performance and issues, and Hangfire for scheduling and managing background jobs. This provided a major leap forward in troubleshooting capabilities and overall system transparency.
- Generic coding approach: Create IT developed a foundation layer - Base Worker - that consolidates integration logic into reusable components. This generic approach improves the maintainability of the system, allowing for quicker onboarding of new developers, reducing the system’s overall complexity and implementing new and faster business workflows.
- Seamless integration with cloud services: The new solution integrates with several Azure services, including Blob storage and File Shares, while also utilizing SQL for database operations. This ensures that data is securely stored and easily accessible.
"This project allowed us to introduce several key innovations, including being the first at the client to implement RabbitMQ as the backbone of the integration architecture. Additionally, the generic Base Worker, as a reusable, versatile and modular solution, allows the client to streamline message processing not only within our solution, but across different workflows. We’ve definitely set a new standard for any integration needs." explains José Coelho, Software Developer at Create IT
