Even so, they can take pride in the history they made by playing at the level, as the first team relegated from the top-flight all the way down to non-league, to be promoted back to the top-flight.
With that journey taking the Kenilworth Road club over 30 years, there were plenty of players who played a part for them in that time.
One individual who had a particularly significant influence during his time with the club, was Andre Gray.
Luton Town got plenty of goals from Andre Gray
Back in March 2012, the Hatters completed the signing of Gray on loan until the end of the season, after he had impressed while playing against them for Hinckley United in the FA Trophy.
That move saw the striker make the step up from the Conference North to the Conference Premier, and he instantly settled in at Kenilworth Road.
Gray scored seven goals in 12 appearances for Luton by the end of the 2011/12 season, as they agonisingly missed out on promotion after defeat in the play-off final.
Even so, the forward’s scoring record meant it was no surprise to see them sign him permanently in the summer of 2012.
It was reported that they paid a fee in the region of £30,000 to sign the forward, in what turned out to be a major bargain.
His first full season with the club saw Gray net 17 times in 44 league appearances for the club, although there would be no promotion back to the Football League that time round.
A seventh place finish, 13 points adrift of the play-offs, left the club some way adrift of those in the mix for the main honours.
However, Luton and Gray would not be denied the next time round, winning promotion as champions in the 2013/14 campaign in emphatic style, surpassing the 100-point mark.
Gray himself played a big part in that, with the striker netting a remarkable 30 goals over the course of the campaign.
However, he would not then get the chance to play a part for Luton in League Two, as his career continued to move on a rapid trajectory.
Andre Gray quickly moved up the divisions
With his exploits having caught the eye in helping the Hatters to win promotion back to the Football League, the striker earned a big move of his own.
In the summer of 2014, Gray joined then Championship side Brentford, and although he was effectively stepping up three divisions, he did not struggle to settle at all.
Over the next year, the striker scored 20 goals in 52 appearances in all competitions for the Bees, which meant he was again quickly on the move.
In August 2015, Gray joined Burnley, who had been relegated from the Premier League the season before.
The Clarets were reported to have paid £9million for the striker, with Luton apparently receiving £1.1million through sell-on clauses.
Combined with the £500,000 Brentford were said to have paid to the Hatters previously, that ensured that plenty of profit had been made on Gray by the club, considering what they paid for him.
As for the striker himself, he continued to excel in the Championship, scoring 23 goals to help Burnley to an immediate promotion back to the Premier League.
Gray also won both the Championship Player of the Year and Golden Boot awards that season, further proving his quality as a player at that level.
The following campaign saw the striker find the net nine times in 32 top-flight appearances for Burnley, helping them to safety with a 16th place finish in the 2016/17 season.
It was at that point in the summer of 2017, that Gray would return to some familiar territory, signing for Luton’s local rivals Watford, only this time a move to that part of the world for much more money.
