Beginning with over 35 Gigabytes of data, we predict which books will be predicted over the next month. This is a binary classification problem, where we utilized PySpark, Amazon EMR, and H2O. Through feature engineering and testing several different machine learning algorithms (including logistic regression, random forests, gradient boosted trees, and H2O's AutoML), we were able to achieve over 0.91 ROC AUC and 0.80 PR AUC.
- Preprocessing: Converts CSV data to HDFS .parquet files and hosts them on Amazon S3.
- Feature Engineering: Creates new features, splits a training and testing set, and writes HDFS .parquet files.
- Final Algorithms: Tests several machine learning classifiers with AUC ROC and AUC PR as validation metrics.
- Visualizations: Plots EDA and some of the results from our models.