The legislators took the view that 40+ year old vehicles are mostly owned and driven/ridden by enthusiasts who are able to, and do, keep their vehicles to a high standard of maintenance.
Modern vehicle testing is more and more moving into the realms of high tech testing equipment, and more and more of the test centres are no longer equipped to check old vehicles.
Thus the decision to change the rules.
As for insurance companies withdrawing their policies after an accident, well that truly is over thinking and a real herring of a red hue. The situation with the 40 year rule is no different to the current rule for vehicles manufactured before the 1st January 1960. No MOT means simply that, no more and no less. Insurance companies cannot insist that owners carry out further testing if it is not mandatory.
The roadworthy "condition" exists even when you do not, and do test vehicles both at a mandatory and insurance level, so it is and always has been the rider/drivers responsibility.
I haven't had many accidents, I am happy to say, but I have never experienced an insurance company wanting to avoid their responsibilities due to non-roadworthiness. Indeed my insurers have always taken on board and paid for the legal guys, including barristers at one time, to make claims on my behalf. IMHO many of the suggestions about insurance companies wanting to avoid paying for their insured vehicles are simply unfounded and are based upon urban myth rather than reality.
Finally on a political note, if we were to follow the EU decisions on this matter it would have been a 30 year rule rather than a 40 year rule. It's all in the document published to record the results of the consultation, and the governments rationale.
Finally, I replied at some length as part the consultation. How many others did the same?