Why is my wisteria leaf brown?

My wisteria is 6 years old, growing in a container.
The leaves (about 10% of all, seems to be the younger ones) are turning brown on the main part of the leaf (not the edges).
Some leaves have spots on them- not sure if that's a first step before they turn brown.
Any thoughts on what this might be and what to do about it would be appreciated!
Thank you!

 

Update- the leaves are now dried up and curled. :(

I sprayed with Safer's but I didn't see aphids.

Any advice would be appreciated!

 

Is this occurring over the entire plant? This may be reaching the limits of its growth in the container.

 

Dear Daniel,
Thank you so much for your response!
It's not really all over, just on the smaller leaves. Would adding fertilizer help?

I did see one ant today but I still haven't spotted any aphids.

Thanks again,

Sarah

 

Do you have a hand lens or other magnifier? Have a closer look -- might be mites.