Azure Stream Analytics: The Secret Weapon for Real-time Insights That's Often Overlooked

A while ago, I worked with a client on their product, which was a complex mix of  different parts like windows applications, web applications, microservices, and console applications. The problem was that there was no proper way to track the data flowing into the system. So, if anyone wanted to find out what happened to a specific piece of data, it was nearly impossible due to the involvement of multiple components. This led to the need for a centralized system that could store information about what happened at a specific date and time – basically, a centralized system for managing events. Since there was an existing Azure Postgres DB server in place, it was determined to utilize this resource for storing the events. The remaining aspect that required attention was determining the most effective method for managing the event streaming process.

I was given the responsibility to analyse and explore the technology and tools needed to implement this event streaming process that met the following high-level requirements:

1. Platform-as-a-Service (PaaS) Solution: The system had to be built using a PaaS offering in the Azure cloud.
2. Limited Development Effort: Considering the limited availability of resources, it was essential to minimize significant development efforts.
3. Rapid Time-to-Market: The goal was to ensure a quick time-to-market for this system, reducing the overall implementation timeline.

My task involved identifying a suitable PaaS solution in Azure that aligned with these requirements while addressing the challenges posed by resource constraints and the need for timely delivery. In my quest to identify suitable tools and technology, I considered the following approach:

1. Event Hub for ingesting the events: To handle the event streaming, the plan was to leverage Azure Event Hub. Each component in the system would be responsible for pushing its respective events to Azure Event Hub. This would ensure a centralized entry point for all events, facilitating efficient and reliable data ingestion.
2. Azure Function for receiving the events: To process and store the events, I initially considered implementing an Azure Function. The idea was to create an Azure Function using C# that would act as the receiver, responsible for capturing the events from Azure Event Hub and storing them in the existing Azure Postgres DB.

If you want to try out how to create an azure event hub, here is the link for you - 
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create

If you want to try out the integration of Event hub with an Azure function, here is the link for you - 
https://thegreenerman.medium.com/trigger-azure-functions-on-event-hub-1e1df0402ba4

While the concept of utilizing Event Hub seemed promising, I realized that developing and deploying this Azure Function would require substantial development effort, which was not aligned with the resource bandwidth constraints and the need for a quick time-to-market. This led me to explore further alternatives that would meet the requirements efficiently. After thorough research, I came across an ideal solution: Azure Stream Analytics!

Azure Stream Analytics turned out to be a pretty straightforward, no nonsense solution for my problem. The way it works is -

1. Set up Azure Event Hub:

• Create an Azure Event Hub resource in the Azure portal.
• Configure the Event Hub with the desired settings, such as partitions, throughput units, and access policies.

2. Create an Azure Stream Analytics job:

• Create an Azure Stream Analytics job in the Azure portal.
• Specify the input source as the Azure Event Hub you created earlier.
• Define the desired query to process and transform the incoming events.

3. Configure the output sink for Azure Stream Analytics:

• Create or use an existing PostgreSQL database to store the processed events.
• Obtain the necessary connection information for the PostgreSQL database, such as server name, credentials, and database name.

4. Configure the output sink in Azure Stream Analytics:

• In the Azure Stream Analytics job settings, specify the output as the PostgreSQL database.
• Provide the connection details, including the server name, credentials, and database name.
• Define the table structure or mapping for storing the events in the PostgreSQL database.

5. Start the Azure Stream Analytics job:

• Start the Azure Stream Analytics job to begin processing events from the Event Hub.
• The events will be transformed according to the defined query and stored in the PostgreSQL database.

6. Monitor and troubleshoot:

• Monitor the Azure Stream Analytics job and verify that events are being processed correctly.
• Monitor the PostgreSQL database to ensure the events are being stored as expected.

If you want more detailed steps, visit
https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics

Azure Stream Analytics offers a turnkey solution that requires minimal development or deployment effort. Think of it as a ready-made Azure Function specifically designed for streaming analytics. When I discovered Azure Stream Analytics, it proved to be the exact solution I had been seeking. It fulfilled the requirements of requiring no significant development effort and allowed us to launch quickly, especially for the initial phase.

I presented both approaches, namely utilizing Event Hub with an Azure Function and utilizing Event Hub with Azure Stream Analytics job, to the stakeholders. Without a doubt, the second approach—the integration of Event Hub with Azure Stream Analytics—received their approval. This decision validated the efficacy and suitability of Azure Stream Analytics for our project.

If you have any queries or any specific topics you'd like me to write about, please feel free to email me at sha.unchained@gmail.com.