Sunday, October 21, 2007

Downloading Pages via SSL and proxy using LWP Perl

If you use LWP modules in perl then you can download pages via HTTP easily using a proxy. However if you wanna download pages with HTTPS through a proxy then it becomes a problem.
This is because both proxy and HTTPS are not implemented simultaneously in LWP modules although individually they work quite well.
But there is a way around. To download paged via HTTPS you have to give LWP the impression that you are not using any proxy. Thus you set the environment variable HTTPS_PROXY as a valid proxy and dont set anything as the proxy in the request object for https. This would make it work like a charm. :)
Link to above Funda.