Six models, one question: which predicts schedule overrun best?
What my second paper found when we compared six machine learning algorithms on 4,517 software projects.
By Abdelkarim Sweilem ·
After my first paper showed that a gradient boosting model could predict schedule overrun, the obvious next question from PMO leaders was: why that model? Would a neural network do better? Would something simpler be good enough? My co-authors and I set up a fair comparison to find out.
A fair race
We took 4,517 software-project records and tested six algorithms: linear regression, a regression tree, random forest, support vector regression, gradient boosting and a deep neural network. Every model got the same predictors and its default settings, and we evaluated all of them with 10-fold cross-validation, so no model had an advantage from tuning.
The results
- Gradient boosting won on error. It had the lowest RMSE (4.578) and the lowest MAE (3.503).
- The deep neural network came very close and had the highest squared correlation (0.875).
- Random forest ranked third.
- Support vector regression was weakest with default settings.
Why I still recommend gradient boosting
Accuracy is only half of the decision. A PMO has to explain to a steering committee why a project was flagged. Gradient boosting gives near-best accuracy and still shows which factors drove the prediction. That makes it easier to trust and easier to act on than a neural network.
The same three signals, again
Across the interpretable models, the same predictors kept rising to the top: risk factor, duration and client satisfaction. Two studies pointing at the same signals is a strong hint about where project controls should focus.
What to do with this
- Use gradient boosting as the starting point for an early-warning model.
- Invest in the quality of your risk data, because it carries most of the signal.
- Use the predictions for prioritization and recovery planning, not as automatic verdicts.
Next steps for research are independent organizational datasets, tuned models and consistent explainability across algorithms. If your organization wants to pilot this on its own data, I'd be glad to talk.