We started seeing this failure sporadically with our spark builds ubuntu 18.04, but it shows up all the time now. Fixed the
spark 3 tests by explicitly setting
SPARK_LOCAL_IP=127.0.0.1 in spark_env. But, the
spark 2 builds are still broken even with the env explicitly set and calling spark-submit with
--conf "spark.driver.bindAddress=127.0.0.1. Running
lsof -iTCP -sTCP:LISTENe before spark-submit show all the 4K ports to be free. See this broken build for instance.
Jenkins builds seem to be fine.
2021-04-19T17:42:28.3311657Z 21/04/19 17:42:27 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. ... 2021-04-19T17:42:28.3342144Z 21/04/19 17:42:27 ERROR SparkUI: Failed to bind SparkUI 2021-04-19T17:42:28.3344354Z java.net.BindException: Cannot assign requested address: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
Any pointers to working around this issue for