Conversation-level toxicity data gets big fast — and the tooling has to keep up. This project explored 28,823 Twitter conversations with toxicity scores to understand how toxic threads differ structurally from healthy ones, using Spark so the same code scales past a single machine.
A SparkSession pipeline loads and cleans the conversations, derives boolean toxicity flags from the continuous scores, and compares conversation structure across the two groups with groupBy aggregations. Each analysis is written twice on purpose — once in the DataFrame API and once in SparkSQL — verifying both paths return identical results.
The structural signal is clear in the descriptive statistics: toxic conversations run longer and pull the original author back in more often (more author replies) than non-toxic ones. As an EDA study it stops at description rather than prediction — the honest scope for the data available — and demonstrates the Spark workflow end-to-end: ingestion, cleaning, feature derivation, and dual-API analysis.
Describe your data and the decision it should support — you'll get a feasibility answer and a fixed quote.
Start a project →