As you will know the specially shaped washer is also a basic wear indicator,, the felt washer is to help keep out dust and water. Personally I think you should fit them both.
It's a fiddly job made difficult as the brake lever splines will be tight from the clamping of the bolt, worse if overtightened in the past. I would ensure the splines are clean on the shaft, rust free.
As you will have found out as you slide it in place the shaft moves into the hub. I had one of those tools that is like a screwdriver except it's a pointed tapered end. I was able to push the tapered point into clamp bolt hole and this pulls the shaft out of the hub as the taper meets the bolt recess part of the shaft.
It might help to ease open the lever arm if it's a very tight fit in the shaft.
If they are available a new lever arm might be needed if it's been continually overtightened in the past,.