It sounds like you have a good handle on e-collar use but just to clarify, it's not a punishment and shouldn't cause pain at all just stimulation. I didn't get the sense that you are, I just want to make sure since it's fairly critical here. The corrections, particularly in this case, should be very mild.
What I would do is just take his training back down a level and make sure you're solid on the basics of walking on lead. The most important part being attention. It normally gets overlooked but in this case it's particularly important that when you have him on lead, his attention is solely on you.
The easiest way to test and reinforce this is by making random direction changes on lead. If he's not 100% focused on you he won't catch them. When that happens, give correction. I prefer to go with the continuous setting starting out and turning away from him, give a little over a second to respond then give the correction and release it when he turns as well. It's important to use the lowest setting that he notices. It's a bit tricky though because you really have to be observant to get the timing right on the release. When he's at the point that he's responding very quickly, the momentary is more practical.
The first couple times turn away from him to pick up his attention level. Usually they'll start heeling closer to you in anticipation of getting turned. If that happens after the first time or at whatever point he starts picking it up really well, start to turn into him. You have to be careful not to step on his feet so it's best to kind of push him with your knee as you're turning. You don't want to knock him over you just want to get him out of the way.
Then you can add stops with him sitting every time you stop and then moving forward again. I use two bumps on the e-collar for enforcing the sit when you stop but obviously you have to introduce him to that if he's not used to it already.
You can also just change walking speed or side step. With every skill you should decrease the amount of time you give him to respond as he understands it. Obviously there's a practical limit to the response speed that you can expect but it's pretty brief, though.
Once he's starting to get a good handle on it, mix it all together randomly with pretty frequent changes. Then start increasing the time between changes to challenge the amount of time he stays focused. Give him praise when he responds to your changes instantly and say nothing, just give correction, when he doesn't.
Any time he's on lead his shoulder should be 6 inches off your knee and if your standing still, he should be sitting. A good test and reinforcer is to have him sitting and you side step several feet. He should shift over and sit down. It just highlights the importance to him that no matter what happens, he should be in the proper place.
Once he's got that down, introduce some mild distractions like other people. If you get the impression that he's not focusing on you, do a direction change or stop. Then gradually increase the level of distraction.
Obviously the maximum level of distraction is going to be another dog. At this point he should be 100% focused on you in all other situations. Start with a dog he knows with a helper holding it on lead, standing still. When the dog comes into view give an attention check. Don't use the same attention check every time or he'll anticipate it. After he's passed that continue to approach and give frequent attention checks. Work closer and closer to the other dog. Once he's done it with a dog he knows than you can switch to a strange dog. Again move a bit closer every time he does it perfectly. Obviously it's always best to keep them out of reach of each other just in case.
As you've already identified, catching it early is key. If he's worked up, mild corrections are ignored and stronger corrections make things worse. If he gets to the point where he's worked up, just stop where you are, hold him by the lead but don't react in any way until he stops. Any change or action you make is either going to reward the behavior, like walking the other dog away, or make it worse, like any kind of correction since you're adding stimulus to a situation that's already stressing him. Just wait it out. When he settles enough that you're convinced he'll respond, make him sit and continue on a line so you're not 90 deg from looking at the dog so you can see where his attention is very clearly.
Eventually you just slowly work up to the point that his focus is on you any time he's on lead. From what you're saying he should probably get a good handle on the basic stuff pretty quickly but working around the dog might be slow going. Every once in a while mix in distraction free sessions to reinforce and don't work him too much in one session as they can get bored and productivity drops significantly at that point. Work sessions should be followed by fun sessions.
Socialization will also help speed things up. Any dog interaction has to be off lead though so as not to interfere with your training but the more dogs and people he can meet, the smoother it'll go. Also, working in odd places will help his general confidence. Some play grounds work great when they are empty like the bridges they have that kind of flex when you walk on them. Man hole covers, metal grates with small holes, or metal stairs also work well.
Best of luck with it.