Depends on train and company and country.

But nowadays most often these special trains have gates-which you cannot pass without the needed RES, That certtainly applies to €* and AVE and most TGV on starting point.

In other cases as soon as staff finds you they may tell you to get off next stop, or fine you as ´blackrider=this may mean a fine of up to 200€ in some cases. Or they can confiscate the pass ´ to have it for report´. Or nothing at all happens. 

While a slightly different angle to what you ask, some trains (much to Eurailpass holders’ chagrin) have ‘quotas’ so if the number of Eurail/Interrail passholders who have booked a seat has reached its limit, you won’t be able to travel.

This policy is ludicrous.  Why not release unoccupied seats to passholders the day before, unless SNCF etc. “know” from past history there’s a huge likely last minute rush of passengers with revenue tickets who book onb the day before, or day of, travel?

Eurail passholders in particular often come from Australian, Canada, the United States, New Zealand and Chile/Brazil/Argentina, and it’s not always easy to book in advance.  Given Global passes supposedly offer flexibility, ‘quota controls’ are not fair, especially when on top of the significant daily amount for consecutive or even greater for ‘4 days in a month’ passes, one has to pay a substantial supplement to travel on some high speed or other trains.