If it's possible to do a reservation for a train right now in general, it should be also possible when holding a rail pass. And in most cases there should be no difference in availabilty. But it depends on train and route. And a general "90 days in advance" isn't correct - it depends. So if you did find a train that is already available there is a big chance you are able to do a reservation with your railpass, too.
Regarding your Venice to Vienna reservation: Unfortunately, you didn't find a cheap way to get your reservation, you just found a wrong way. I hope you didn"t buy one yet. For the direct trains, you'd need a supplement as a railpass holder that includes a reservation. That's 10 € when bought with ÖBB (that's possible but not that way you obviously used) - interrail / eurail charges an additional 2 € reservation fee - that's why it is 12 € there. But if you bought a reservation for 3 €, you'd still need a supplement for 10 € which would include the reservation, so you'd pay 13 € all together.
Regarding availabllity: There are countries/ trains where there is just a quota of places for railpass holders, for instance with Eurostar to and from London. So you may get places when you'd buy a normal ticket but not as a railpass holder. So the app may also be right that the seats for railpass holders are sold out. That has not to mean that the train is sold out. But there are also countries/trains where there is no such difference.
Going from Nice to Venice without a reservation is possible but it means more changes (Ventimiglia - Genova - Milano - Verona) and is slighlty longer (to get you an idea: when leaving Nice Ville at 8.50, you'd arrive Venice Santa Lucia at 19.50).