The first major withdrawals to take place amongst the class came with the opening of the Tyne & Wear Metro system. Many North Eastern allocated vehicles were taken out of service and some reallocated to the London Midland Region. The LMR vehicles never saw any form of 'mass withdrawals', being taken out of service at a steady rate from the early 1980s. Those vehicles which did not go through works for overhaul and asbestos removal were the first to go. Doncaster, Derby, Swindon and Glasgow Works were responsible for these overhauls, which ran through until 1987 for the LMR vehicles.

By the start of the summer 1989 timetable there were around 50 vehicles still in service. Those vehicles still to survive at Newton Heath were reallocated to Chester, where there were already some 104 trailers working between 108 power cars which came from Buxton. With the continued introduction of second generation DMUs, the surviving 104s at Chester were withdrawn at an increasing pace towards the end of 1989. The allocation from Scotland of several 101s to Chester drove more nails into the coffin for the 104s, although the surviving vehicles still managed to work as far afield as Birmingham New Street, Derby, Holyhead, Leicester, Rugby, Sheffield, Skegness and Stafford, to name but a few. The last two sets allocated to Chester were CH610 M53447 + M53531 and CH611 M53451 + M53529 which saw out there final days on Chester to Helsby services during June 1990, the month they were taken out of service. This left just the ten vehicles on Network SouthEast still in service. Of these, the two trailers were the first to be withdrawn, 59206 on October 1, 1990 and 59163 (due to fire damage) on October 22, 1990.

The withdrawal of the DMBS vehicles started with 53437 and 53479 on January 18, 1992. Two days later, Class 115 DMUs replaced the 104s on the Barking to Gospel Oak services and the remaining six vehicles were transferred to Thames Line duties. During the rest of 1992 five of the six vehicles fell victim to the continued introduction of new Class 165 Thames Turbo units to Thames Line services. 53540 was the last vehicle in service into 1993 and the only vehicle to be properly fitted with a high density halogen headlamp (the "Mexican Bean's" was fitted on the roof domes). It was withdrawn in October 1993.