Getting a lot of errors when crawling issues

We are crawling issues daily of our repository to make them better searchable and analyzable: https://github.com/timroes/issue-crawler

Every night a couple of requests fail (we currently retry each–modified–issue page 3 times). Still occasionally there are pages even failing 3 times in a row and thus breaking our “build”. You can see all past logs in the travis ci job: https://travis-ci.org/timroes/issue-crawler

Usually the failure is either a 502 or 500, though it seems to have shifted more towards 502 in the past.

Here is a list of failed requests, but feel free to look into travis ci for more up to date issues:

D66A:77A0:16CC2A:2C5B23:5D1A499F

D69A:60AA:17137A:2CF49A:5D1A49D0

D6A0:3E94:168C80:2BD151:5D1A49DF

D6A4:3FD3:15B909:2A2BD4:5D1A49E9

D6A8:6042:62E03:104750:5D1A49F3

D6AE:1F7E:114E94:24ECF3:5D1A4A08

D6D2:4478:BB605:1AFC9D:5D1A4A4B

D6F4:779D:AED7D:1A1D07:5D1A4A81

D700:0808:188766:2E48EA:5D1A4A93

AC7C:4478:BD258:1B4176:5D1A4AD4

ECCA:5EE1:1F92661:3D947E9:5D17A65F
ECE0:623A:727520:12E6417:5D17A66F
ECFC:15FD:177399F:329903A:5D17A693
D40E:3D79:1FA536A:3D55A59:5D17A6AB
D41A:434D:19F4EE0:37058AD:5D17A6B9
D42E:5606:F42284:25B1D33:5D17A6DF
D456:2BD7:DA172A:21D4475:5D17A722
D472:34CE:DDB191:2242C5C:5D17A742
D4B6:72E9:20EA0EE:3FCDC13:5D17A789
D4F0:5989:1F3258B:3D9D710:5D17A7C8

8BF0:6240:865DD9:1092026:5D13B205
D810:10B7:871084:F1B52E:5D13B220
D814:77BA:64C156:DA0EE2:5D13B22B
8C32:46E1:860D06:10A906B:5D13B25D
8C36:563F:86108F:1065FF8:5D13B269
8C62:1EAF:69928E:E34C15:5D13B2BA
8C72:5CD5:836267:101D7C5:5D13B2D4
8C7C:1E61:64F2D3:D938CD:5D13B2E2
D886:46DE:6A16F4:E14731:5D13B306
D8D4:1878:40036F:97984D:5D13B363

These are from the past couple of days. I would appreciate any help on getting those failures down a lot.

A couple of more failures from yesterdays run:

AC3E:4A33:B8C0E4:187BB8C:5D1E3BEF
AC4C:47B2:6CA605:108CCF2:5D1E3BF9
91E8:2315:10EB9E7:1F98865:5D1E3C04
9206:50FE:F4C383:1D20F6B:5D1E3C40
920C:68EE:CAA7BA:1A49E2B:5D1E3C4F
B4F6:7E42:1E417C:2E2A1B:5D1E3C86
B4FA:64BF:22D651:32AD2A:5D1E3C92
B4FC:04B6:194AB2:243D7A:5D1E3C9C
B500:2B72:16E431:21AB8D:5D1E3CA7
9234:417F:C74202:1A77B17:5D1E3CB6
92C0:6FC8:64000B:F69E7E:5D1E3D4D

Thanks for taking the time to report this, the team will take a look.