A great post in moz by @ipullrank which i admit it will take me days to digest. Basically it lists all the main reasons why SEO is not only about Content and link building.
I had a similar discussion with a colleague recently who was talking to me about his furstration on why people believe that SEO is just about link building. To be honest i feel that this is something we are responsible also as SEO’s as we don’t explain our work perfectly.
Additionally, sometimes we focus mostly on that factor (links) as it is the most important rather than the overall performance.