Born in London to bohemian parents - her mother was a novelist and her father a journalist who worked with Ian Fleming and Graham Greene in Naval intelligence during the War – Avery received a First Class degree in art history at SOAS. She then held a fellowship at Brown, and studied Japanese at ICU University in Tokyo, before becoming an art historian and writer.