Analyzing the Evolution of the Phishing Threat Landscape using LLM Technique
| This develops an AI cybersecurity system that merges Retrieval-Augmented Generation (RAG) technology with a fine-tuned BERT model to solve present-day cybersecurity problems that include user query management and phishing attack Analysis. A combination of discrimination and generation AI methods enables the system to provide detailed threat awareness together with precise threat identification.
The BERT-based phishing Analysis module functions as an advanced security barrier to identify phishing threats present in emails and URLs and SMS messages with high accuracy. Users receive warning alerts through this system to make immediate prevention possible. While operating alongside the “RAG-based chatbot” functions to provide users with context-specific information through real-time assistance along with dynamic statistical analyses about cybersecurity development. This two-part architecture design provides a powerful combination of defense capabilities because BERT delivers accurate threat Analysis from data while RAG creates valuable real-time feedback for users through its informative system. The system operates as a strong instrument for contemporary Security Operations Centers (SOCs) and threat monitoring setups through its combined functionality. The project methodology followed three structured phases of operation. The initial phase concentrated on data acquisition and preparation where data was obtained from repositories PhishTank, OpenPhish and Hugging Face. The analysis incorporated 18,000 emails with 5,971 SMS messages and 800,000 URLs and 80,000 website entries. The system used PyPDF2 and BeautifulSoup to parse PDFs and HTML followed by LangChain text chunking and OpenAI and Hugging Face models for dense vector embedding conversion of content before storage in FAISS. The vectors were placed into a FAISS database to enable quick semantic searches. The RAG pipeline operated during system architecture by obtaining document fragments from FAISS and producing responses using GPT-3.5 Turbo with BERT-based threat analysis integration. The BERT model (`bert-base-uncased`) received 15,756 balanced samples to develop its capacity to differentiate between phishing and benign inputs. The model performance received enhancements through applied feature engineering methods which included URL-based attributes in addition to NLP preprocessing. Evaluation results demonstrated strong performance. The RAG module showed strong context precision at 0.83 yet its answer relevancy scored 0.66 alongside low factual accuracy that resulted in a faithfulness score of 0.14. The exceptional performance of the BERT model showed 94.2% accuracy together with 93.7% precision and 92.5% recall and 93.1% F1-score, indicating very few mistaken classifications.
|