Notes and projects on AI, LLM systems, data engineering and machine learning by HyunSuk Lee. https://nockda.github.io/ How to use the Spark - Using Language and Tools Language: Python Tool: Spark (PySpark) Apache Spark has emerged as one of the most prominent tools in the realm of big data processing. In this guide, I’ll provide an introduction on how to utilize Apache Spark, particularly catering to beginners. Apache Spark API Reference: Spark Python... Wed, 03 Jan 2024 00:00:00 +0000 https://nockda.github.io/2024-01-03-Spark-Introduction/ https://nockda.github.io/2024-01-03-Spark-Introduction/ AutoEncoder & IF & LOF - Using language and Tools Python, Scikit-learn, matplotlib Goal of this project In today’s society, the frequency of credit card transactions has increased significantly. Credit card companies bear the responsibility to protect the customers’ assets and must ensure the swift identification of every transaction. Each customer presents distinct features, leading credit... Sun, 10 Dec 2023 00:00:00 +0000 https://nockda.github.io/2023-12-10-Credit-Card-Fraud-Detection-Model/ https://nockda.github.io/2023-12-10-Credit-Card-Fraud-Detection-Model/ Analysis the customer action with RFM method. Using language and Tools Python, Pandas-profiling, Pandas, SKlearn, Matplotlib Goal of this project The objective of this project is to analyze e-commerce customer action data and classify customers based on the characteristics identified in the data. Dataset Description This data is archived from... Wed, 29 Nov 2023 00:00:00 +0000 https://nockda.github.io/2023-11-29-customer/ https://nockda.github.io/2023-11-29-customer/ AutoEncoder & IF & LOF - Using language and Tools Python, TensorFlow, Scikit-learn, matplotlib Goal of this project Fraud detection in financial transactions is of utmost importance worldwide. The main objective of this project is to develop a model that can identify outliers in financial transactions. By implementing this model, financial companies can effectively prevent fraud,... Tue, 20 Jun 2023 00:00:00 +0000 https://nockda.github.io/2023-06-20-Model-for-Fraud-Detection-System/ https://nockda.github.io/2023-06-20-Model-for-Fraud-Detection-System/ From multi-output to single-output - Using language and Tools Python, XGBoost, lightBGM, matplotlib, joblib, pandas, sklearn Goal of this project There are 8 main steps involved in the manufacturing of memory semiconductors. Among these processes, the Photo and Etching process stands out as the most crucial. When it comes to quality control, it is essential... Mon, 12 Jun 2023 00:00:00 +0000 https://nockda.github.io/2023-06-12-Semiconductor-Manufacturing-Analysis/ https://nockda.github.io/2023-06-12-Semiconductor-Manufacturing-Analysis/ Focuses on designing, implementing, and testing a complex Spark application for Big Data analysis. Evaluations will be based on functionality, code quality, efficiency. Using language and Tools Java, Apache spark, Python Goal of this project This project is to design and implement a batch-based text search and filtering pipeline with... Thu, 08 Jun 2023 00:00:00 +0000 https://nockda.github.io/2023-06-08-Big-Data-Engineering/ https://nockda.github.io/2023-06-08-Big-Data-Engineering/ Understanding Model Performance and Feature Interpretation using Captum Using language and Tools Python, Numpy, Pandas, PyTorch, SKlearn, Ray Tune, Captum, Matplotlib Goal of this project Train two deep neural network which can take images with a cell nuclei and classify it into one of the following types which are shown... Mon, 05 Jun 2023 00:00:00 +0000 https://nockda.github.io/2023-06-05-Classification/ https://nockda.github.io/2023-06-05-Classification/ This is Full-stack coding. Building Airbnb service with Django and React. It covers the backend and frontend components, including app functionalities and their interaction. Airbnb Clone Coding Using Language and tools : Django(Python), Rest API, React, Poetry, Git Goal of this project : The project aims to gain a thorough... Thu, 01 Jun 2023 00:00:00 +0000 https://nockda.github.io/2023-06-01-Airbnb-Clone-Coding/ https://nockda.github.io/2023-06-01-Airbnb-Clone-Coding/