The first winning "S-Class" idea will receive 50 million euros (£42m) and will be readied for launch in 2017.
It is a departure for the organisation that is used to spending hundreds of millions of euros on flagship planetary probes and space telescopes.
Esa hopes particularly to encourage young scientists and research teams from smaller member states.
"We see a dearth of people being trained in space science in satellites, and this can pick those people up and help bring them on so that by the time they get to work on some of our big missions they already have experience," said Prof Mark McCaughrean, head of Esa’s Research and Scientific Support Department.
"Also, it gives our smaller member states the opportunity to lead missions, which is something that is very difficult for them to do. The ‘big four’ (France, Germany, Italy, UK) have no problem, but it is harder for the likes of Spain, Sweden and Switzerland.
"We want to hear from innovative people who’ve got a clever idea that doesn’t need a billion or half a billion euros, but can be done with a much smaller amount."
Interested parties will have to move fast, however. Expressions of interest must be submitted by 23 March, followed by a defined proposal in June.
A clutch of the best ideas will probably be selected for study in November, with a winner chosen for implementation 12 months after that.
There is no directive on what an S-Class mission should do, other than it fit within Esa’s "Cosmic Vision" – the strategy document that governs all the agency’s space science endeavours and which covers fundamental, but fairly broad, themes about the Universe and our place within it.