This year Louis has the Boys category but had a tough time picking his final six acts.
The crowd made their opinions known at every turn, encouraged by Simon Cowell who told the audience to make it “the worst day of [Louis'] life”.
And it all ended with a dramatic three-way sing off.
The Daily Mirror reports that among Louis' decisions was sending home one of two brothers.
The pair had auditioned as a duo in the groups category - this year mentored by Simon Cowell - before being split up into soloists in the boys category.
Louis is said to have given the younger brother a chair before replacing him with his own sibling in what could be the cruellest six chair challenge yet.
"It’s the most savage six chair challenge ever," a source told the newspaper. “Louis knew what he was doing and thought it might cause a reaction but at the same time he wanted to have the strongest six he could for his boys section and there wasn’t enough room in his line up for both brothers.”
Meanwhile, when it came to Nicole Scherzinger's turn in the hot seat, things also got rather tense, The Sun reports.
"Nicole left the set upset as the situation was so intense and it all got too much for her," a source claimed. “It was a long and stressful day and it wasn’t helped by the fact the crowd were not on Nicole’s side for some of her choices."
The insider explained: “Before filming had commenced, the audience were encouraged by the show’s production staff to be vocal about what they thought of her choices.
“They took this on board and were very loud, but unfortunately for Nicole at times it did mean a deafening shower of boos and jeers.”
Those who make it through the six chair challenge will make it to judges' houses.
The X Factor will air on ITV in the summer.
No exact start date has been confirmed just yet but typically the show launches at the end of August.