Name Email
Bharath Gajula gajula.b@northeastern.edu
Divya Sri Bandaru bandaru.di@northeastern.edu
Henil Gajjar gajjar.he@northeastern.edu
Nishith Reddy Pochareddy pochareddy.n@northeastern.edu
Soumyae Tyagi tyagi.so@northeastern.edu
Fatima Jawadwala jawadwala.f@northeastern.edu

Table of Contents

  1. Introduction
  2. Scoping
  3. Data
  4. Modeling
  5. CI/CD
  6. Logging and Monitoring
  7. Cost Analysis

Demo Video

Introduction

1. Problem Statement

Customers often struggle to find accurate and personalized answers to their product-related questions. Product descriptions can be overwhelming, and user reviews are scattered and hard to analyze. Businesses also face challenges in ensuring fair and unbiased responses while maintaining the reliability and scalability of their systems.

This project aims to create an AI-powered chatbot that combines product details and user reviews using a multi-agent workflow and a vector store for efficient retrieval. The solution leverages GPT-based LLMs for natural language understanding and CI/CD pipelines for seamless deployment. It focuses on delivering fast, reliable, and unbiased information while being easy to integrate into various platforms through a FASTAPI serverless API.

2. Methodology

1. Data Collection and Preprocessing: For this project, we utilized the Amazon Product metadata and review dataset for the Toys & Games and Video Games categories. Following this, the data was cleaned and converted into a schema-based format, which was then utilized for our project.

2. Model Implementation: We used four different LLMs in our application: GPT-4o Mini as a supervisor agent, Hugging Face for generating embeddings, Llama 3.1 8B as an agent for summarizing and follow-up question generation, and Llama 3.1 70B as the responding agent.

3. Model Evaluation: Used RAGAS to evaluate our model’s performance for specific prompts based on the following metrics: Context, Precision, Answer Relevance.

4. Bias Detection: For the given prompts, we conducted sentiment analysis and calculated the percentage distribution across three sentiment categories: positive, negative, and neutral. By rerunning this module, any changes in this distribution can be detected. If significant changes are observed, it may indicate the presence of bias.

5. CI/CD/CM: GitHub Actions is used to implement CI/CD processes, enabling us to deploy and run our project on GCP using Cloud Run. For monitoring, we utilized GCP Cloud Monitoring and Langfuse for tracking model responses.

3. Goals

  • Enhance the eCommerce shopping experience by addressing challenges in navigating product reviews and information.  
  • Develop an intelligent chatbot using Retrieval-Augmented Generation (RAG) to streamline the process of finding relevant insights from vast review datasets and product metadata, providing real-time, context-specific answers to customer queries.  
  • Improve customer satisfaction by providing personalized recommendations and accurate responses to nuanced questions.  
  • Integrate seamlessly with existing eCommerce platforms to create a scalable, impactful solution.

The source code for our project can be found here: GitHub.

The source code for the E-Com web application we developed can be found here: GitHub-UI-Code.

4. Tools Used for MLOps

5. Project Architecture

Project Achitecture