ConvoInsight is a state-of-the-art Customer Conversational Intelligence Platform powered by Large Language Models (LLMs) and advanced Natural Language Processing (NLP) techniques. This platform analyzes customer interactions across diverse channels to extract actionable insights, enabling businesses to optimize customer service processes and enhance the overall customer experience.
- Project Overview
- Key Features
- Technology Stack
- Project Structure
- Key Components
- Installation
- Usage
- Current State and TODO
- API Documentation
- Contributing
- Datasets
- License
- Acknowledgements
- Disclaimer
ConvoInsight leverages cutting-edge NLP techniques and Large Language Models to analyze customer conversations from various sources such as chatbots, call centers, emails, and social media. The platform provides real-time insights, including sentiment analysis, intent recognition, topic modeling, and agent performance evaluation.
- Develop a robust conversational intelligence platform using Django and Django Channels
- Implement advanced NLP techniques using LangChain and custom LLM models
- Provide real-time analysis and recommendations for customer service interactions
- Create a scalable and efficient system using Celery for background task processing
- Utilize PostgreSQL with pgvector for efficient storage and retrieval of conversation data and embeddings
- Integrate with SageMaker for ML workflow automation and model deployment
- Multi-channel conversation analysis (chat, voice, email, social media)
- Real-time sentiment analysis with granular emotion detection
- Intent recognition for tailored responses
- Topic modeling and trend identification
- Agent performance evaluation
- LLM-driven real-time recommendations for customer service agents
- Interactive dashboards for insights visualization
- Scalable architecture for handling large volumes of conversations
- Fine-tuning capabilities for LLM models
- Integration with SageMaker for model training, deployment, and monitoring
- General AI assistant functionality with text and voice input
- Order management system with integrated support chat
- Backend Framework: Django
- Asynchronous Support: Django Channels
- Database: PostgreSQL with pgvector extension
- Task Queue: Celery with Redis as message broker
- LLM Integration And LLM-Agent Creation: LangChain and LangGraph
- Web Servers: ASGI (Daphne) for WebSocket, WSGI (Gunicorn) for HTTP
- Frontend: Django templates with JavaScript and jQuery (future plans for React.js)
- Containerization: Docker (for deployment)
- Machine Learning:
- Custom fine-tuned LLM models
- SageMaker for ML workflow automation
- Hugging Face models for various NLP tasks
- NLP Libraries: NLTK, Gensim, transformers, BERTopic
- Cloud Integration: AWS S3 for data storage
convo-insight-platform/
├── config/ # Main Django project directory
├── apps/ # Django apps
│ ├── accounts/ # User account management
│ ├── analysis/ # Analysis and metrics calculation
│ ├── api/ # API endpoints
│ ├── convochat/ # Core conversation handling
│ ├── dashboard/ # User dashboard and visualization
│ ├── general_assistant/ # General AI assistant functionality
│ ├── llms/ # LLM development, integration and management
│ ├── orders/ # Order management
│ └── products/ # Product management
├── data_processing/ # Scripts for data ingestion and preprocessing
├── static/ # Static files (CSS, JS, images)
├── templates/ # HTML templates
├── tests/ # Test cases
├── manage.py
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
├── CONTRIBUTING.md
└── README.md
- LLM Configuration and Integration: Utilizes LangChain for LLM integration with support for multiple Hugging Face models (Mistral, Mixtral)
- Real-time Communication: Uses Django Channels for WebSocket support in chat functionality
- Background Tasks: Celery tasks for processing conversations, generating metrics, and managing ML workflows
- Performance Evaluation: AgentPerformanceEvaluator for assessing conversation quality
- NLP Tasks: Sentiment analysis, intent recognition, and topic modeling using various ML and deep learning models
- Data Storage and Retrieval: PostgreSQL with pgvector for efficient vector storage and similarity search
-
Clone the repository:
git clone https://github.com/rampal-punia/convo-insight-platform.git cd convo-insight-platform -
Set up a virtual environment:
python -m venv venv source venv/bin/activate # On Windows, use `venv\Scripts\activate` -
Install dependencies:
pip install -r requirements.txt -
Set up PostgreSQL and create a database for the project.
-
Install the pgvector extension in your PostgreSQL database.
-
Set up environment variables (create a
.envfile in the project root). -
Run migrations:
python manage.py migrate -
Start the development server:
python manage.py runserver
-
Access the admin interface at
http://localhost:8000/admin/to manage data and configurations. -
Use the API endpoints to interact with the platform programmatically.
-
Access the frontend interface at
http://localhost:8000/to view dashboards and insights. -
To start a new conversation, navigate to
http://localhost:8000/convochat/new/. -
To create dummy data for testing:
python manage.py create_random_users 50 python manage.py generate_dummy_data -
To fine-tune the LLM model:
python manage.py fine_tune_llm -
To train and deploy a model using SageMaker:
python manage.py train_deploy_model [model_type] [script_path] [train_data_path] [output_path] [endpoint_name] -
To monitor model performance:
python manage.py monitor_model [endpoint_name]
- Basic dashboard with general assistance and customer support interactions
- User authentication and profile management
- General AI assistant with text and voice input
- Initial implementation of NLP tasks (sentiment analysis, intent recognition, topic modeling)
- SageMaker integration setup (pending authentication)
- Order and product management systems
- Implement the main Customer Support chat pipeline with real-time NLP analysis
- Develop and integrate more advanced NLP models using transformers
- Enhance the dashboard with detailed analysis and visualizations
- Develop comprehensive API endpoints
- Implement more robust error handling and logging
- Enhance test coverage
- Optimize database queries and indexing
- Implement more sophisticated recommendation systems
- Enhance voice processing capabilities
- Develop a comprehensive admin interface for system management
For a complete list of TODO items, please refer to the project documentation.
Detailed API documentation can be found at /api/docs/ when running the server. Key endpoints include:
/api/conversations/: Upload and retrieve conversation data/api/analysis/sentiment/: Get sentiment analysis for conversations/api/analysis/intent/: Get intent recognition results/api/analysis/topics/: Get topic modeling results/api/recommendations/: Get real-time recommendations for agents
We welcome contributions to ConvoInsight! Please refer to our CONTRIBUTING.md file for detailed guidelines on how to contribute to the project.
The project utilizes the following datasets for training and testing:
- Relational Strategies in Customer Interactions (RSiCS)
- 3K Conversations Dataset for ChatBot from Kaggle
- Customer Support on Twitter Dataset from Kaggle
For more details on these datasets, please refer to the project documentation.
This project is licensed under the MIT License - see the LICENSE file for details.
- This project is part of the Advanced Certification Course on Computational Data Science at IISC with TalentSpirit.
- We acknowledge the use of open-source libraries and frameworks that made this project possible, including Django, LangChain, Hugging Face Transformers, and BERTopic.
- Special thanks to the open-source community for their invaluable contributions to the tools and technologies used in this project.
This project is intended as a learning exercise and demonstration of integrating various technologies. While it showcases the integration of Django, Django Channels, LangChain, Hugging Face models, and AWS SageMaker, it is not designed or tested for production use. It serves as an educational resource and a showcase of technology integration rather than a production-ready web application.
Contributors and users are welcome to explore, learn from, and build upon this project for educational purposes. However, please exercise caution and perform thorough testing and security audits before considering any aspects of this project for production environments.