A/B Testing & User Behavior Analysis

Project Overview

This project analyzes the results of a marketing A/B test to evaluate whether displaying an advertisement increases user conversion compared with a PSA (Public Service Announcement) control group.

The analysis also examines user behavior patterns related to ad exposure, day, and hour in order to identify patterns that can support future advertising decisions.

Business Question

Does the advertisement increase user conversion compared with the PSA control, and what user behavior patterns can help guide future advertising decisions?

Dataset

The dataset contains 588,101 users and 7 variables describing:

• Experimental group assignment
• Conversion status
• Number of ad exposures
• Day of highest ad exposure
• Hour of highest ad exposure

The experiment consists of two groups:

• Ad group: 564,577 users (96%)
• PSA control group: 23,524 users (4%)

Each user appears only once in the dataset, with no duplicated user IDs.

Analysis Approach

The analysis includes:

1. Data validation and quality checks
2. Experimental group distribution analysis
3. Conversion rate comparison
4. User behavior analysis by ad exposure
5. Conversion analysis by day and hour of highest ad exposure
6. A/B testing using a Two-Proportion Z-Test
7. Effect size and confidence interval analysis
8. Business impact evaluation
9. Business recommendations

Key Findings

Conversion Performance

The ad group achieved a higher conversion rate than the PSA control group:

• Ad group: 2.55%
• PSA control group: 1.79%

This corresponds to a 0.77 percentage-point increase in conversion rate for the ad group.

The statistical analysis was used to determine whether this difference was unlikely to be explained by random variation.

Ad Exposure

Conversion rates generally increase as the number of ad exposures increases, with the strongest rates appearing among users exposed to the ads more frequently.

However, this relationship should not be interpreted as proof that additional exposures directly cause higher conversion, since users who receive more exposures may differ systematically from users who receive fewer exposures.

Timing

Conversion rates vary across both the day and hour of highest ad exposure. The analysis identifies periods with relatively higher conversion rates that may be useful for future campaign targeting and scheduling.

Business Recommendations

Based on the analysis, the advertising treatment appears to have a positive effect on conversion and can be considered for continued use.

Future campaigns should:

• Continue testing the advertisement against a control group rather than relying only on observational differences.
• Monitor ad exposure frequency to identify an effective exposure range.
• Consider timing campaigns around periods with stronger observed conversion rates.
• Use further experiments to test whether increasing exposure directly improves conversion and to identify potential diminishing returns.

Tools

• R
• RStudio / Google Colab
• dplyr
• ggplot2
• readr
• Statistical hypothesis testing

Project Structure

AB-Testing-User-Behavior/

├── data/
│   └── marketing_AB.zip
│
├── image/
│   └── conversion_rate_by_group.png
│
├── AB_Testing.ipynb
│
└── README.md