The music mogul told the Irish Herald that bosses were currently in negotiations with five stars to join the panel this summer.
It’s widely expected that Kelly will not return to the show this year, with talks having stalled with producers.
Louis Walsh let slip that Nicole Scherzinger, Katy Perry, Mel B, Anastasia or former judge Sharon Osbourne were all possible replacements for the Destiny’s Child star.
"These are the five people that they're looking in terms of a replacement for Kelly," he told the newspaper.
He explained: "I've worked with Nicole and Katy before and I would love to work with them again, they were fantastic. Mel B has done the X Factor Australia and could easily do it in Britain as she knows the format so well.
"Anastasia has sold millions of records and is a big star and I would give anything to work with Sharon again."
In other reports today however it's been claimed that former judge Dannii Minogue may also be set to return.
So far only Gary Barlow has been officially announced as part of [cat]The X Factor 2012[/cat] panel, with further news expected this week.
Auditions start filming on May 15 in Cardiff.