Built to 4 mm/ft scale with EM (18.2mm) gauge track, the layout is closely modelled on period photographs of the original railway, and almost every detail is drawn from reality. The time period modelled focuses on the five years from 1928-33, which spans the death of Colonel Stephens in October 1931 and his succession by W H Austen.

Because of space constraints Stephen needed to model a terminus, which the K&ESR did not have. He opted for the East Sussex Light Railway from Northiam to Rye, which was one of a number of proposed branches and extensions to the K&ESR. The Light Railway Order authorising the line to Rye was approved in 1902, but the branch was never built. Stephen's model assumed that the line had been built along the Rother Valley route - as shown on the K&ESR maps of the period - but terminated some distance short of its intended destination. By good fortune the final layout also fits well with the Colonel's intended terminus location between the River Tillingham and Ferry Road.

The main elements of the model were drawn from the stations at Northiam and Bodiam, but the final track plan closely replicates the original Rother Valley Railway terminus at Tenterden (subsequently renamed Rolvenden after the line was extended in 1904), but with fewer servicing facilities. Instead of the large engine shed at Rolvenden, Rye Town has a small servicing point and a single carriage siding which (for the revised location) approximates to the position of the intended interchange connection to the South Eastern & Chatham Railway station at Rye.

The rolling stock used on the layout is also modelled closely on the original K&ESR fleet, and visiting 'main line' vehicles that are known to have used the railway. Much of the stock was hand-built or heavily modified from kits by Stephen.

Rye Town was exhibited at many shows across the southeast of England between 1990 and 2005. It has collected a number of trophies over the years, culminating with 'best in show' at the EM Gauge Society's Golden Jubilee ExpoEM event in May 2005. However, in 2003 Stephen decided to move on to a new project, and his assistant operator Chris Jackson, a fellow member of the Colonel Stephens Society, offered to take over Rye Town.