To the best of my understanding a ".ics" file only contains event data such as time, date, title, description, location, reminders, etc. Your mapping software is responsible for getting you a route not the .ics file. If you wanted to use OSM for sure, convert the address to GPS coordinates with something like https://gps-coordinates.net and use those as the location in the calendar event.
That conversion isn't accurate at all, at least in the cities I have been. I used to trust OSM a lot, but it led me to wrong places a few times. I use Organic Maps, which is based on OSM, for general navigation, but I have to rely on Google Maps for finding precise locations. I use web interface only, for google maps, but OSM can be a pain in the butt, especially if others are dependent on me navigating them.
Odd, to me wherever I have been Google Maps is vast areas of uniformly coloured nothing only whereas OSM has details of every house, every footpath,...
Odd, i have always found that method to work correctly. If OSM has a POI for the place i use that first, but if nothing else i do it the way i mentioned and have had success with it. Then i add the location to OSM when i go their.
Since Google makes money off of you using Maps, and makes third-party apps pay a fee for using their API, I don't think they have an incentive to let you download navigation data to use it into another company's software