Apache Beam and Dataflow Pipeline

Our recommendation system leverages Apache Beam and Dataflow for efficient data processing.

A simulated streaming source sends data to Pub/Sub every 10 minutes and the pipeline streams it to BigQuery.

For the detailed implementation, refer to the following notebooks:

• App Usage
• Calls
• Locatioins

Data Serving

• BigQuery: Tables are partitioned to minimize query cost and materialized views are created for Dashboard Connection.

• Dashboard: Hosted on Looker Studio with cross filtering.

• Recommendation System: Simple Collaborative Filtering with Matrix Factorization, new model is rolled out daily with Blue-Green Deployment.

Embedded Dashboard

Recommendation System (Daily Updated)

Select a user (with previous activities) from the dropdown:

Click the button to get recommendations: