1. It's a shared 4-person compartment. But if you book for 4, then you'll be booked into one compartment, if possible, so you'll still have a private one. Pass reservations cost 4 x €39 = €156 (plus booking fee if any). There may be saver fares that are only a bit more expensive, especially when booking a long time in advance, in which case you could save a travel day.
2. AFAIK compartments can be locked.
3. This train is perfectly safe.

Für 172€ kann man in Palermo auch 2-3 Nächte ein Appartment beziehen, es wäre finanziell schlauer für 20€ Sitzplatzreservierung zu kaufen, aber vielleicht geht das mit Kindern nur schwer. Viel Spaß!

Sitzplatzreservierungen kosten nur €3 p.P., z.B. über tickets.oebb.at.

I assume you are referring to the day train, not trying to travel through the night in a standard seat - even if possible, which I doubt.

Although I agree that accommodation in Palermo (or any other town/City) is far better than onboard a train (except maybe the Orient Express) I doubt there is a decent apartment in Palermo for a family where 2 - 3 nights would be less than 172 euros. I would be budgeting 100 euros per night.

This is still well below the cost of what I would expect to be an uncomfortable night for a family in a space about 2 metres square, bunk beds, a small toilet (if lucky) and multiple stops through the night, but as I have never travelled on a night train this is all from reading reviews and youtube.

Yup, 172 euros for a couchette is a bit of expensive … 

Train is an experience.. Wanna try it as long as it is safer.. :)