Access and download files with dynamic cookie using cURL-CodePudding

I have a perl script in bash which uses cURL to download the files over HTTPS using username/password and cookie. However, I found that the cookie is dynamic thus hardcoding it would not work. Thus, I would like to inquire if you have a better solution for this problem.

Below is my current code.

    # This just pulls the webpage. Will need to parse out the HTML Table chunk into a CSV
        print "Downloading $target_file.html\n";
                if  ($filespec eq "file1")  {
my $curl_code = `curl "https://sample.com/wells/CompositionReportResult.asp?ReportId=123"   -H 'Cookie: SMDEVICE=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.0lMlSBb0mzYbuV6M0P9Qst0JijUl2YvuVxaBtAsVEfcUkKLRsO2p1w.PC
Y7zSD0HHuUCZZSsEbnFg.Z5QUEHlGNDQDoL8UsIKnakAxalwcGU7CX5rUerKKbrem25tc8GyIcmxPasp4KzFF9caR20lH7_lc6M1QlYssBEYEcI4DM50P-t8Mtr-PhDA.-Vdgrpt4t96Lk_N1hPCMYg; SMUSRMSG=; Navajo= eGoE8a0fQmcgzcWwCp5hHDE/d2qhD 3PWhAGpDtbjdcarcbVJ04ReK6NKYsEQp
6Rnew1GOvnH0-; SMSESSION=LtzURcRBZWhHbcJTfzHBMmf TKgP6SLlM5r1l6HXDUB4xsOPw G0lJleBwYDcLPotLXzu5 arzlceezQP2oc/cCmsi9K86aIzn9DkNlJhZUUPDX6SzvSxRxZDJDvsyAsDFv Tn 0hixJ4f7fHk0ATYFMUjyy7nM3DnW9kxHUuwfrAxHGnj5zvBRKzMTV0jM6Eotgc8jUJS6ehEwZX
GEIVC7QIuk3dtJqaSEXzkI6m4Bxi7VUq522ajQ618rsC3ICS3gy/y/ RnpfAHEFkJHwWpCP4kz1byYdSjKmDaq8cxnIJWXENCxUj2OuKEkJw8izZfum47 5FfI2gAPnv1aq5W 902r/AwSOSJfT8YBCCWHARbf1hab2Jwk9XfFxmWUDNsVa6oxTLc3/LQ1fEO7rd9lVqlyPUD9q U53I5IZ6UybyrNzRNrGY11tftc
Glo0FDlVpmpboebeJXnIk JPDB9azB8/QC1cKoGU 3VvmikUsvMx2/I6ZNRbagJKMUo4zSEsOcaLMgV8WrLVDifg7brUvASDQSXsKgWisIoQfVhiVGWsKJl0ovQ9Eeih1iTQpt05FpDXiigsb0cFz/vcRCt6chszLT99kKoO29Ck0abro4Tir4ZTOXSAAyu108C0 f1sWCjvM/FX4Myer/pgSDQfXzy5w/7mdVLSlQd
SNBIAzjDZdrAk2vxsLeYR6UjJLJocPeC/YOBwSC438IVw0LSay1oBsLv0OlQukMvxKuPnvROPHvIecXNN3PrYenR4ctYtw rLZCAVh0RGuKzkgjC1DMxk9g1mPmFGPmmUI9nRQ NI2BD1CQt2Np1zI29Ow7bOqc6OhobsV2C5wpmKZwEqRjeNqaNJGuwYYJwId6vmKgq5wq0J60CcOXOCjN/EX5YwVbLERPAN/3F1h
n3h/c52odG2CTkGwq1B4ns/6Uc06XW3SSB olODwgLmRqUQgrlajyPdYiub6JJMpA hf4iGSiz3LV7s4efSPPIUCM/5DKy9kcoQWmjsbPVoBCpS4Aywnc0xAhRpbK6KeO6dzndVvoZC8D9aQj4MBMoTTJBZGrAIr2WkE71msMIvg2BmFRegsTqODLHNJqxZfBeNxVHcO bk8RUbje3 QmCmY6Tc6Nl9jbfH3I5k0zB
g9SQ/x6HWNOKv7NuTZHQ9uiC6DslxLXX87eZV0L 4SF2QMtkwl82pHXGtcMkk2F9desk0dHB406LCRCki3 DYQTuQMfDH/HYIuQzvpuCrzIRpHZvIpoA7sR tgEkUFVhibK/gVM'  --compressed >  "$httpsconf->{localdir}/$target_file.html"`;
                } elsif ($filespec eq "file2") {
my $curl_code = `curl "https://sample.com/wells/MainReportResult.asp?ReportId=456"   -H 'Cookie: SMDEVICE=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.0lMlSBb0mzYbuV6M0P9Qst0JijUl2YvuVxaBtAsVEfcUkKLRsO2p1w.PCY7zSD0H
HuUCZZSsEbnFg.Z5QUEHlGNDQDoL8UsIKnakAxalwcGU7CX5rUerKKbrem25tc8GyIcmxPasp4KzFF9caR20lH7_lc6M1QlYssBEYEcI4DM50P-t8Mtr-PhDA.-Vdgrpt4t96Lk_N1hPCMYg; SMUSRMSG=; Navajo= eGoE8a0fQmcgzcWwCp5hHDE/d2qhD 3PWhAGpDtbjdcarcbVJ04ReK6NKYsEQp6Rnew1G
OvnH0-; SMSESSION=2a3mfLKCU39E56N 0e9O8WDQTAQ3RZ9FCxsd0khmmKUMmfg1NOzRVAE/KJLvqSTmoowZkHZlh4BMCchhlv Ej2ePzTdo1fzWLAIrGZwiPOobjqJnOZ8Xb RluEXcZxsHoGVSLpcT51x3uw5Xd9E39XdupiYRoqkEQHRpC4KgsJts6lsU1WIXGfnskAxLIwt bXSMLQVoccQkjnR0bQRI1RME
apxW0DV RZDLjbiZTqC7PmXhbLmFLfONZxD0B2TPeH9w8YtL8pBNardx35CQyZ/j3ICutB/Qt/kTirFAotJhYAKQNG2O9vdmoiny3quPX3d87AhM2bmsIOAUetw5imTkwaU3KEgY5upSlXKTqvpqjcgQubVGlXBwPOzJJcLvEQJ4i4IwU6neUv5SvjuzOGhs3buyMbr63P2s1/3pyUOnn955Qmte/joWWoxVq 85AkJ
hkgM3wrlj9DMwsLzbqwU1JCbU1lDwjkNZzQci5s3QMaJjq27PmudijeZQ5cT3ZlgBjCef40AlJuMOzo6BfnS98nGWxvPHO2KIwXYloYx6u9uDQGbm6PUVDSwQSk5Nk5aPs53fvUO47AhjudjCs/l nxgOW1uxNL6iZSDyKwXYnzLkLCnK/qPPvUlnULbjIGNNKxgtcH yIdwFM9pUfEfIhaTthR1wK7MLKzFsqRijy
EiQF4wBrTuSaFjt84O3RqnNai/3iMUrZ8ajS1AasWaiZI tAHaJIAQMceyzdHEOh6jdI2z7Mah6Gbu yKMMX3AlotYidfaNcGtbieazaDgGe7oCpncpw4 y3OnLQy3eEfu7swg/Ty7IXZV2c6gzcoXzxar7mcP1stdJfovfciz2 pCLuJnL3pcAP7atGgmoJdK7MtgbA3GNgw/sj7vcqUgNoyKyqoTbfRqqZwaAgsL
kSdwXH6UAfxqiF2xo34nxyrFdaMkzEYNZCiU VTQRrpjx Y lACTIhRTeThdPfc1Gy0CUs3bEzQiznlffhoYKwq/RNv43ySxY4QUb3iYE07JgfpXj5FdzL5PuMr1Y48ZWXf84OE0UMJ5gi31Fp85c2ewDLjm3I7iQhVQO6BsC9vXpjERWFwmDmx1mZUkgLBLjl4u7sSYIJER2GphP8G3 bt4D WC5lCzUShK/PNOSx
SCZcSy8oMS 6BNbDUO1qU6w5QERvi5hkklKiQXhLTR4JJzi22tcyyHIcxNTTwKX6MPHKi9cjvHpQjzyILhg4SPzaTrtCY ZUuFtqfblpt3jYGPfehDFO3grLK0J8AzFm'  --compressed >  "$httpsconf->{localdir}/$target_file.html"`;
                } elsif ($filespec eq "file3") {
                my $curl_code = `curl "sample.com/wells/DisplayMDXResult.asp?ReportId=789"  -H 'Cookie: SMDEVICE=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.i0N-N4zs25tS1YyxFsPgghtTvb-j1yoJlNcMwFJt0YlczWTtzJbLyQ.VaOp-3bOsKZsFZK3J-5Ttw._d967mUZH2Oly5YVAZvxwbeud4Y86gy3K5r8dDpupC3aLnhYd7ZTB1r-4wsuvVzDtUGATftexCVNPMItz0AxwjQ8zuHpfEbNXVlgCEYwWDw.thTbm9lUA_B7C62X4mIa6g; SMUSRMSG=; Navajo=HghuGrY7Sps24/1DNa6srjbey aj UXdfPIARKvhZrg3dVBKiKYosBr/onJVE2iz0DdUAuty48M-; SMSESSION=RnIiliYFFe4EwQNRMIsgzMY7Vjf21dA8tuvUkLRO1S6Uql21o15cEaoj0ChiZ1V1dSwhCUmLog4EbcIxRlMx8qoQLd05tCN5/dAaspIdVez4D6hV42Ih6drEyX188wn2ZjPIDHet8Rxl XHHKv4EHK6zB5ECCS S4HB M/fihEjaKA8o9ny8KvhGa5msJn1KXkhSvTByUpT78MdCLBghhlZeHGxapQhAU42zansZui4vIX3iUFmI9WQw0i3vDm0APoanQRW580YtXLxR//pxvUKpOjrsTW3DxcQ0W7Iib8/7xX7Sa8CnQhMG heaHJ8LooaKBLGgEI9e1/52Un67YrJR6pgpk5VqB4Kkf1EQdzGWDvQkiBY8z2EDtitSh0EdKTVBiE0h9Voom7AkV9jxtRawpUYr0ktYwQEWQGmItcg10TrNUshTio/p5tkS/5Pmuy0e rmTZbtiEEMpVeu/Yg/AHpYzfEuUxJtXyXQX4WNEwdiEAKH cCObmunZ9n5jw  GLZFrQghszYeoDR16vmUPtW8XRx5teXSQkstUTIq chuyYdPpPZmeo6tYHj8bO9lAsxKD3boeax7NxMfO2ev4EAl8J8sHPku2tM7ZRHi05ciEqLASDQjXIK1QTDpnPBstt2gxNWDXp84ml2fAlakzQ4RePtvYtJedjxaf6I CW7vXG2IoxxLTCJtWUcMwq1iIBFFjYQj2e j1YVYQIQXuRbwDuDVpsmUVdwh0GeyJiY/BUQRG27y1xom4jzAXOk8fRIUUaj2uIkjiJOS2wzr2rKnckm5H5xl88xialsrkGnXGtN4FBVzUQpx/AuQO01OFdUsAD9EXYlSaxjPJSYh788VooaY/v7SurbKcXhqJSNt4VTnAyidpQGn5dHRuklp5hdFxqfS2Oxt0sAJdFgAVJ38a qf9NbM7Gh3L DC/ovgBGrSn8HRQkw3U/0yZZZUtVJLhQF MO44qkiFF5n/fHJTUNzA u4sB6rmQt0TMB9a3AV48OwafqkVubGvPasxmmdQp4kewVzbGoNHH4jjy65BBHsUXAlp79LD1vMS4Ig/VhMkhiqK QJktyd8R5PX0 nOmTTAuVDEsNrWyYtJiIC338DvuMLYHjQsPet09XH54ufoF/21GIz IA 6X90AXdXAurE3n7mZSFfmvJfs9taTg0jwRuRuQcojBY4c9g2oTp/cz4Q8GXnCrWCSwA7onrxOPelXOYUiLakBM8pxSJcUZhAb/mc0kZKD1WbNThYiriKIivIrv7ivzFrKre'  --compressed > "$httpsconf->{localdir}/$target_file.html"`;
                }

However, I found that the cookie is changing everyday thus hardcoding it would not work. With this, is there any suggestion on how to

CodePudding user response：

Should be able read and write the same cookie, like

curl --cookie cookie_file --cookie-jar cookie_file https://...

or just

curl -b cookie_file -c cookie_file  https://...

In principle you'd use -c option to write a cookie file when logging in with your credentials and then use -b to read from it for the next request(s) but it should work in one as well.

See cookies in curl docs, and the Cookies chapter in Everything curl book.

CodePudding user response：

Alternatively, consider using the Perl modules LWP::UserAgent or WWW::Mechanize.