The two-storey building would replace greenhouses used by the horticultural department on Exhibition Road, and the full planning application will be made public in the coming days.
It has been designed for students with learning difficulties. If the building gets the go-ahead from Bradford Council the college hopes to start construction in June, and open in April, 2015.
Saltaire Village Society held an “extraordinary” meeting with college principal Nav Chohan to discuss the plans on Tuesday evening.
Although described as “heated” by some who attended, the society has said will not likely be putting forward an official stance on the plans - its chairman said members had differing views on the proposals.
Chairman Vanessa Pilny said: “I don’t think there is going to be a unanimous position from the society. Some people are quite concerned about this, others are more supportive of it.”
She said some of the objectors had organised a meeting for next week, by which point the planning application will likely have been made public.
Mr Chohan said: “Shipley College is privileged to be based in the heart of Saltaire World Heritage Site and deeply committed to preserving the buildings as a community asset for the people of Bradford and beyond. We aim to design a modern day build which will blend sympathetically with the surrounding area and all its rich history. The development is critical to maintaining the College’s ability to meet future demands, both educational and economic.
“The proposed new build will cost £1.9 million and the target given is for work to be completed in March 2015. This is a great opportunity for high need learners.
“The additional students we can teach and develop will secure the College funding so that we can maintain these beautiful buildings.”
Andrew Brown, from Maddocks Street, said the site was an “antique garden” and believes the modern building, on the doorstep of Salts Mill, would spoil the feel of Saltaire.