I've started reading more in my mid twenties. Prior to that I've mostly read fantasy, and never knew what is out there.
Having read business, self-help, big history and biographical books over a couple of years, and then switched to parenting books when I started getting kids - I've never took notes and realized I forget 99% of what I've read. I can't even remember all the titles.
The best day to start taking notes would be 10 years ago for me, and the next best day is today.