University College London researchers found a 3.6% decline in mental reasoning in women and men aged 45-49.
They assessed the memory, vocabulary and comprehension skills of 7,000 men and women aged 45 to 70 over 10 years.
The Alzheimer's Society said research was needed into how changes in the brain could help dementia diagnoses.
Previous research had suggested that cognitive decline does not begin much before the age of 60.
But the results of this study show that it could in fact begin in middle age.
This is important, the researchers say, because dementia treatments are more likely to work at the time when individuals start to experience mental impairment.
The UCL researchers tested the cognitive functions of 5,198 men and 2,192 women aged 45 to 70, who were all UK civil servants, from 1997 to 2007.
Individuals were tested for memory, vocabulary and aural and visual comprehension skills.
Differences in education level were taken into account.