news-crawl 2.x Broken when using multiple workers (across multiple hosts)

[jnioche]

Discussed in #62

^{Originally posted by alextechnology December 8, 2023}
I spent today merging our local news-crawl codebase to the 2.x branch. We run only 1 nimbus and 2 supervisors. While the topology seems to work fine with a single worker, it does not work at all with two workers despite all the configuration being the same and working fine on news-crawl 1.2.4.

The first issue seems to be related to local worker not being able to connect to the remote supervisor/worker. Once the topology is submitted to storm, the worker log will scroll for over a minute with 140+ attempts like the following:

2023-12-08 21:14:17.456 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 147 to Netty-Client-xxxxxxxxxx failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: xxxxxxxxxxxxxxxx

It will eventually start just trying to do the crawl but there will be lots of the followings kinds of logs indicating failures:

2023-12-08 21:14:20.720 o.a.s.m.n.Client client-worker-1 [ERROR] failed to send 1 messages to Netty-Client-xxxxxxxxx java.nio.channels.ClosedChannelException
AND

2023-12-08 21:14:20.735 o.a.s.m.n.Client Worker-Transfer [INFO] Dropping 1 messages
2023-12-08 21:14:20.735 o.a.s.m.n.Client Worker-Transfer [INFO] Dropping 2 messages
2023-12-08 21:14:20.735 o.a.s.m.n.Client Worker-Transfer [INFO] Dropping 2 messages
2023-12-08 21:14:20.735 o.a.s.m.n.Client Worker-Transfer [INFO] Dropping 2 messages

The second issue (which I assume is related to the first) is this will inevitably lead to an exception with the java.util.ConcurrentModificationException error and crash the worker:

2023-12-08 21:14:23.316 o.a.s.m.n.Client Worker-Transfer [INFO] Dropping 3 messages
2023-12-08 21:14:23.316 o.a.s.u.Utils Thread-14-feed-executor[70, 71] [ERROR] Async loop died!
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
md (com.digitalpebble.stormcrawler.Metadata)
	at org.apache.storm.executor.Executor.accept(Executor.java:301) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) [storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
md (com.digitalpebble.stormcrawler.Metadata)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557) ~[kryo-4.0.2.jar:?]
	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:118) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:555) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.boltEmit(BoltOutputCollectorImpl.java:112) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.emit(BoltOutputCollectorImpl.java:65) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:93) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:42) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:77) ~[storm-client-2.5.0.jar:2.5.0]
	at com.digitalpebble.stormcrawler.bolt.FeedParserBolt.execute(FeedParserBolt.java:104) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	... 6 more
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1511) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1544) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1542) ~[?:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:99) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:39) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557) ~[kryo-4.0.2.jar:?]
	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:118) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:555) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.boltEmit(BoltOutputCollectorImpl.java:112) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.emit(BoltOutputCollectorImpl.java:65) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:93) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:42) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:77) ~[storm-client-2.5.0.jar:2.5.0]
	at com.digitalpebble.stormcrawler.bolt.FeedParserBolt.execute(FeedParserBolt.java:104) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	... 6 more
2023-12-08 21:14:23.320 o.a.s.e.e.ReportError Thread-14-feed-executor[70, 71] [ERROR] Error
java.lang.RuntimeException: java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
md (com.digitalpebble.stormcrawler.Metadata)
	at org.apache.storm.utils.Utils$1.run(Utils.java:413) ~[storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
md (com.digitalpebble.stormcrawler.Metadata)
	at org.apache.storm.executor.Executor.accept(Executor.java:301) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
Caused by: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
md (com.digitalpebble.stormcrawler.Metadata)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557) ~[kryo-4.0.2.jar:?]
	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:118) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:555) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.boltEmit(BoltOutputCollectorImpl.java:112) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.emit(BoltOutputCollectorImpl.java:65) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:93) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:42) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:77) ~[storm-client-2.5.0.jar:2.5.0]
	at com.digitalpebble.stormcrawler.bolt.FeedParserBolt.execute(FeedParserBolt.java:104) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1511) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1544) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1542) ~[?:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:99) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:39) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557) ~[kryo-4.0.2.jar:?]
	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:118) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:555) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.boltEmit(BoltOutputCollectorImpl.java:112) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltOutputCollectorImpl.emit(BoltOutputCollectorImpl.java:65) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:93) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:42) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.task.OutputCollector.emit(OutputCollector.java:77) ~[storm-client-2.5.0.jar:2.5.0]
	at com.digitalpebble.stormcrawler.bolt.FeedParserBolt.execute(FeedParserBolt.java:104) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more

Thankfully, a single topology worker appears to run much faster in 2.x than 2 workers did in 1.2.4, so we're ok with that for the moment but would be nice to find out why this cannot run with 2 or more topology workers.

EDIT:

I forgot to mention that the remote worker/supervisor experiences similar errors, though it does receive the topology and tries to crawl the pages, but fails with dropped messages and during Worker-Transfer

[jnioche]

Hi @alextechnology

The 2 issues might not necessarily be related, so we have

1. Connection problem between workers / nodes
2. ConcurrentModificationException on the Metadata when emitting tuples from the FeedParserBolt

Focusing on the first one for now, how do you set the cluster up? Do you install Apache Storm manually on the nodes? Do you run Docker?

[alextechnology]

Hi @jnioche,

Very much appreciate your work on this and assistance.

I'm currently running two separate Ubuntu Pro 22.04 VMs on GCP - node Alpha and node Beta. Both have openjdk-8 and openjdk-11 installed (via aptitude) with the latter currently set as JAVA_HOME system (previously, openjdk-8 was set to run Storm 1.2.4 for news-crawl). Alpha also has elastic and kibana installed locally and configured to bind to the virtual private IP. I'm also running an independent 6 node Zookeeper cluster (in the same cloud and project). No docker involved at all.

I'm running Alpha as a storm nimbus seed (also bound to the private IP) and a supervisor node via supervisord with openjdk-11 set as the JAVA_HOME in the command string. I have Beta running only as a supervisor node and both instances using the same storm.yaml config. Both are running out of a storm user account created for this which has ownership of the storm binaries and the news-crawl code base. I haven't modified any of the news-crawl code (build completes successfully) but I have modified the es-conf and crawler-config to point to the correct resources for elastic. I have parallelism set to 32 everywhere and topology.workers set to 2. A few other minor config tunes as well but only the values intended to increase the performance a bit. The same config works fine for the current news-crawl master branch or for 2.x when I change topology.workers to 1. There is no firewall at all between Alpha and Beta. I've also added a /etc/hosts entry on Alpha for Beta's private IP to map Beta's system configured domain name - though not sure if this helps since both nodes' workers seem to fail to communicatie with each other.

Happy to share more details, so let me know anything that may help

To upgrade I manually merged 2.x into master locally and only had some minor conflicts to resolve in the news-crawl yaml configs.

[jnioche]

Just to check - Storm 2.5.0 is installed on these nodes, right?

[alextechnology]

Just to check - Storm 2.5.0 is installed on these nodes, right?

Correct, pre-built binary archive downloaded from archives.apache.com. Nimbus and supervisor both appear to startup and connect to zookeeper correctly

[jnioche]

The 2 issues might not necessarily be related, so we have

Actually the 2 issues might be related. If a worker crashes, obviously the other one can't connect to it...

The ConcurrentModificationException will happen only if there is serialization and Kryo serialization will happen only if a worker needs to send tuples to another worker

[jnioche]

The serialization issue seems to happen at line 104, which is surprising because once we get to that point nothing else should be accessing the Tuple.
@alextechnology could you please run the topo with log level DEBUG and check whether this happens when a tuple is not a feed?

[jnioche]

one way of working out what modifies the objects as they are being serialised would be to use the lock method on the metadata object as it is being sent. One way to do this would be to copy the class FeedParserBolt, modify around line 104 to use the lock, modify the topology class or Flux file so that it uses your copy of the FeedParserBolt then restart the topology and check the logs.
@alextechnology do you think you could give it a try?

[alextechnology]

Hi @jnioche - let me try both of these now and get back shortly. However, I did just try a few other configurations:

1. Run only nimbus on Alpha and run only supervisor on Beta. This fails as the supervisor tries to load the seedlist from the local disk which doesn't exist as news-crawl isnt deployed on this node. I did place the seedlist in the same path and tried again, and it does seem to start up but ultimately fails because the WARC output path doesn't exist. I do see it begin fetching pages without an issue and pretty sure this will work if I just recreate the file path for the WARC

2. Run nimbus and supervisor on Alpha but change topo workers to 2. This seems to run without an issue suggesting that the workers can communicate fine on the same host so there's some issue with workers communicating with the remote workers only.

Finally, before these tests, I did a storm kill CrawlTopo -w 30 to stop the running crawl and noticed that the shutdown doesn't seem so graceful - am attaching that log in case that's unexpected
shutdown.txt

[jnioche]

Finally, before these tests, I did a storm kill CrawlTopo -w 30 to stop the running crawl and noticed that the shutdown doesn't seem so graceful - am attaching that log in case that's unexpected

Killing a topology is a messy business -;)

The Thread Dump seems to be a normal behaviour, see https://github.com/apache/storm/blob/9a238c448d7849b908b77d929b682e74926c01ac/storm-client/src/jvm/org/apache/storm/utils/Utils.java#L347

[alextechnology]

@jnioche So I'm a bit confused here - do all the resources (ie. seedlist and output folder) need to exist on all supervisor nodes? Because I just started the same problematic config (ie. Nimbus + Supervisor on Alpha, Supervisor on Beta) and:

1. Same issue on Alpha worker at start up - 147 lines of not being able to connect to Beta worker before it seems to give up and start running the topo. Later I do see the same lines in the log suggesting it cannot connect to the Beta:

TIME(sampled): null TASK: 57 DELTA: -1
2023-12-11 16:31:59.754 o.a.s.e.b.BoltOutputCollectorImpl Thread-56-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[5, 5] [INFO] BOLT ack TASK: 5 TIME: -1 TUPLE: source: status:115, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: status[115]}, [[waitAck = 0]
]] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:31:59.754 o.a.s.e.b.BoltExecutor Thread-27-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[43, 43] [INFO] Execute done TUPLE source: status:115, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: status[115]}, [[waitAck = 0]]] PROC_START_TIME(sampled): null EXEC_START_
TIME(sampled): null TASK: 43 DELTA: -1
2023-12-11 16:31:59.754 o.a.s.e.b.BoltExecutor Thread-56-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[5, 5] [INFO] Execute done TUPLE source: status:115, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: status[115]}, [[waitAck = 0]]] PROC_START_TIME(sample
d): null EXEC_START_TIME(sampled): null TASK: 5 DELTA: -1
2023-12-11 16:32:04.592 o.a.s.e.Executor Thread-46-fetch-executor[73, 73] [INFO] Processing received TUPLE: source: __system:-1, stream: __tick, id: {}, [5] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: -2 
2023-12-11 16:32:04.592 o.a.s.e.b.BoltExecutor Thread-46-fetch-executor[73, 73] [INFO] Execute done TUPLE source: __system:-1, stream: __tick, id: {}, [5] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 73 DELTA: -1
2023-12-11 16:32:04.864 c.d.s.e.p.AggregationSpout Thread-36-spout-executor[91, 91] [INFO] [spout #9]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.865 c.d.s.e.p.AggregationSpout Thread-35-spout-executor[95, 95] [INFO] [spout #13]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.865 c.d.s.e.p.AggregationSpout Thread-48-spout-executor[85, 85] [INFO] [spout #3]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.866 o.e.c.RestClient I/O dispatcher 41 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A9%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel case
 format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.866 o.e.c.RestClient I/O dispatcher 47 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A13%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel cas
e format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.867 o.e.c.RestClient I/O dispatcher 34 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A3%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel case
 format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.867 c.d.s.e.p.AggregationSpout I/O dispatcher 47 [INFO] [spout #13]  ES query returned 9 hits from 2 buckets in 2 msec with 9 already being processed. Took 0.22222222 msec per doc on average.
2023-12-11 16:32:04.867 c.d.s.e.p.AggregationSpout I/O dispatcher 41 [INFO] [spout #9]  ES query returned 15 hits from 6 buckets in 2 msec with 15 already being processed. Took 0.13333334 msec per doc on average.
2023-12-11 16:32:04.867 c.d.s.e.p.AggregationSpout I/O dispatcher 34 [INFO] [spout #3]  ES query returned 12 hits from 4 buckets in 2 msec with 12 already being processed. Took 0.16666667 msec per doc on average.
2023-12-11 16:32:04.876 c.d.s.e.p.AggregationSpout Thread-50-spout-executor[93, 93] [INFO] [spout #11]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.876 c.d.s.e.p.AggregationSpout Thread-52-spout-executor[97, 97] [INFO] [spout #15]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.876 c.d.s.e.p.AggregationSpout Thread-32-spout-executor[87, 87] [INFO] [spout #5]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.877 o.e.c.RestClient I/O dispatcher 34 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A11%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel cas
e format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.877 o.e.c.RestClient I/O dispatcher 41 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A5%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel case
 format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.877 o.e.c.RestClient I/O dispatcher 47 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A15%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel cas
e format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.877 c.d.s.e.p.AggregationSpout I/O dispatcher 34 [INFO] [spout #11]  ES query returned 8 hits from 1 buckets in 1 msec with 8 already being processed. Took 0.125 msec per doc on average.
2023-12-11 16:32:04.878 c.d.s.e.p.AggregationSpout I/O dispatcher 41 [INFO] [spout #5]  ES query returned 10 hits from 4 buckets in 2 msec with 10 already being processed. Took 0.2 msec per doc on average.
2023-12-11 16:32:04.878 c.d.s.e.p.AggregationSpout I/O dispatcher 47 [INFO] [spout #15]  ES query returned 33 hits from 5 buckets in 2 msec with 33 already being processed. Took 0.060606062 msec per doc on average.
2023-12-11 16:32:04.880 c.d.s.e.p.AggregationSpout Thread-31-spout-executor[83, 83] [INFO] [spout #1]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.881 o.e.c.RestClient I/O dispatcher 47 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A1%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel case
 format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.881 c.d.s.e.p.AggregationSpout I/O dispatcher 47 [INFO] [spout #1]  ES query returned 3 hits from 2 buckets in 1 msec with 3 already being processed. Took 0.33333334 msec per doc on average.
2023-12-11 16:32:04.891 c.d.s.e.p.AggregationSpout Thread-51-spout-executor[89, 89] [INFO] [spout #7]  Populating buffer with nextFetchDate <= 2023-12-11T16:30:34Z
2023-12-11 16:32:04.892 o.e.c.RestClient I/O dispatcher 47 [WARN] request [POST http://10.128.0.34:9200/status/_search?typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512&preference=_shards%3A7%7C_local] returned 1 warnings: [299 Elasticsearch-7.17.15-0b8ecfb4378335f4689c4223d1f1115f16bef3ba "Camel case
 format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."]
2023-12-11 16:32:04.893 c.d.s.e.p.AggregationSpout I/O dispatcher 47 [INFO] [spout #7]  ES query returned 24 hits from 7 buckets in 2 msec with 24 already being processed. Took 0.083333336 msec per doc on average.
2023-12-11 16:32:05.062 o.a.s.m.n.Client Netty-ChannelAlive-Timer [ERROR] connection to Netty-Client-beta.com/10.142.0.21:6800 is unavailable
2023-12-11 16:32:05.095 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 215 to Netty-Client-beta.com/10.142.0.21:6800 failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: beta.com/10.142.0.21:6800
2023-12-11 16:32:05.695 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 216 to Netty-Client-beta.com/10.142.0.21:6800 failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: beta.com/10.142.0.21:6800
2023-12-11 16:32:06.295 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 217 to Netty-Client-beta.com/10.142.0.21:6800 failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: beta.com/10.142.0.21:6800
2023-12-11 16:32:06.895 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 218 to Netty-Client-beta.com/10.142.0.21:6800 failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: beta.com/10.142.0.21:6800
2023-12-11 16:32:07.495 o.a.s.m.n.Client client-worker-1 [ERROR] connection attempt 219 to Netty-Client-beta.com/10.142.0.21:6800 failed: org.apache.storm.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: beta.com/10.142.0.21:6800
2023-12-11 16:32:08.050 o.a.s.d.w.WorkerState Netty-server-localhost-6800-worker-1 [INFO] Sending BackPressure status to new client. BPStatus: {worker=6bdbc725-3533-454e-a317-15dc0f8c6a5f, bpStatusId=20, bpTasks=[], nonBpTasks=[1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65
, 67, 70, 71, 73, 75, 77, 79, 81, 83, 85, 87, 89, 91, 93, 95, 97, 99, 108, 109, 110, 111, 112, 113, 114, 115]}
2023-12-11 16:32:08.095 o.a.s.m.n.Client client-worker-1 [WARN] Re-connection to beta.com/10.142.0.21:6800 was successful but 993 messages has been lost so far
2023-12-11 16:32:09.242 o.a.s.e.Executor Thread-47-partitioner-executor[77, 77] [INFO] Processing received TUPLE: source: __system:-1, stream: __metrics_tick, id: {}, [10] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: -2 
2023-12-11 16:32:09.242 o.a.s.d.Task Thread-47-partitioner-executor[77, 77] [INFO] Emitting Tuple: taskId=77 componentId=partitioner stream=__metrics values=[TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: partitioner[77]}, [[URLPartitioner = {}]]]
2023-12-11 16:32:09.242 o.a.s.e.ExecutorTransfer Thread-47-partitioner-executor[77, 77] [INFO] TRANSFERRING tuple [dest: 7 tuple: source: partitioner:77, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: partitioner[77]}, [[URLPartitioner = {}]]] PROC_START_TIME(sampled): null EXEC_START_TIME(sample
d): null]
2023-12-11 16:32:09.242 o.a.s.e.ExecutorTransfer Thread-47-partitioner-executor[77, 77] [INFO] TRANSFERRING tuple [dest: 57 tuple: source: partitioner:77, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: partitioner[77]}, [[URLPartitioner = {}]]] PROC_START_TIME(sampled): null EXEC_START_TIME(sampl
ed): null]
2023-12-11 16:32:09.243 o.a.s.e.Executor Thread-44-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[57, 57] [INFO] Processing received TUPLE: source: partitioner:77, stream: __metrics, id: {}, [TASK_INFO: { host: dcrawler01.c.resonant-matter-144117.internal:6800 comp: partitioner[77]}, [[URLPartitioner = {}]]] PROC_START_TIME(sampled)
: null EXEC_START_TIME(sampled): null for TASK: 57 

2. Beta seems even more odd - it takes forever to start up the worker, though clearly it does communicate fine with the Alpha nimbus. But I suppose it never connects with Alpha workers, so it immediately throws an error about the missing seedlist:

2023-12-11 16:28:20.107 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[94, 94]
2023-12-11 16:28:20.107 o.a.s.e.b.BoltExecutor Thread-43-ssb-executor[98, 98] [INFO] Preparing bolt ssb:[98]
2023-12-11 16:28:20.107 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:83, stream: __ack_init, id: {}, [-4123990916148515094, -8114022221506181452, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.107 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:83, stream: __ack_init, id: {}, [-3378199865264390574, 8307993500274236253, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.108 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:83, stream: __ack_init, id: {}, [-3378199865264390574, 8307993500274236253, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.085 o.a.s.u.Utils Thread-15-filespout-executor[74, 74] [ERROR] Async loop died!
java.lang.RuntimeException: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
	at com.digitalpebble.stormcrawler.spout.FileSpout.open(FileSpout.java:160) ~[stormjar.jar:?]
	at org.apache.storm.executor.spout.SpoutExecutor.init(SpoutExecutor.java:142) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:152) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:50) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:393) [storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:?]
	at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
	at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
	at com.digitalpebble.stormcrawler.spout.FileSpout.populateBuffer(FileSpout.java:133) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.spout.FileSpout.open(FileSpout.java:158) ~[stormjar.jar:?]
	... 5 more
2023-12-11 16:28:20.108 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:83, stream: __ack_init, id: {}, [-3378199865264390574, 8307993500274236253, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.109 o.a.s.e.Executor main [INFO] Finished loading executor spout:[94, 94]
2023-12-11 16:28:20.109 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[90, 90]
2023-12-11 16:28:20.109 o.a.s.e.s.SpoutExecutor Thread-44-spout-executor[94, 94] [INFO] Opening spout spout:[94]
2023-12-11 16:28:20.108 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-4123990916148515094, -6603443033961019697] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.110 o.a.s.e.e.ReportError Thread-15-filespout-executor[74, 74] [ERROR] Error
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
	at org.apache.storm.utils.Utils$1.run(Utils.java:413) ~[storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
	at com.digitalpebble.stormcrawler.spout.FileSpout.open(FileSpout.java:160) ~[stormjar.jar:?]
	at org.apache.storm.executor.spout.SpoutExecutor.init(SpoutExecutor.java:142) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:152) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:50) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:393) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
Caused by: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:?]
	at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
	at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
	at com.digitalpebble.stormcrawler.spout.FileSpout.populateBuffer(FileSpout.java:133) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.spout.FileSpout.open(FileSpout.java:158) ~[stormjar.jar:?]
	at org.apache.storm.executor.spout.SpoutExecutor.init(SpoutExecutor.java:142) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:152) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.spout.SpoutExecutor.call(SpoutExecutor.java:50) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:393) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
2023-12-11 16:28:20.110 o.a.s.e.Executor main [INFO] Finished loading executor spout:[90, 90]
2023-12-11 16:28:20.112 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[36, 36]
2023-12-11 16:28:20.112 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: 0 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-4123990916148515094, -6603443033961019697] PROC_START_TIME(sampled): 1702312100112 EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.112 o.a.s.e.s.SpoutExecutor Thread-45-spout-executor[90, 90] [INFO] Opening spout spout:[90]
2023-12-11 16:28:20.114 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[36, 36]
2023-12-11 16:28:20.114 o.a.s.e.Executor main [INFO] Loading executor tasks __system:[-1, -1]
2023-12-11 16:28:20.115 o.a.s.e.b.BoltExecutor Thread-46-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[36, 36] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[36]
2023-12-11 16:28:20.115 o.a.s.e.Executor main [INFO] Finished loading executor __system:[-1, -1]
2023-12-11 16:28:20.115 o.a.s.e.Executor main [INFO] Loading executor tasks partitioner:[76, 76]
2023-12-11 16:28:20.115 o.a.s.e.Executor main [INFO] Finished loading executor partitioner:[76, 76]
2023-12-11 16:28:20.116 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[60, 60]
2023-12-11 16:28:20.116 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[60, 60]

However, then Beta's seems to start processing tuples (is this because I've enabled sampling?):

2023-12-11 16:28:20.116 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[64, 64]
2023-12-11 16:28:20.117 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-4123990916148515094, -6603443033961019697] PROC_START_TIME(sampled): 1702312100112 EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.118 c.d.s.i.DummyIndexer Thread-43-ssb-executor[98, 98] [INFO] Mapping key parse.title to field title
2023-12-11 16:28:20.118 o.a.s.e.b.BoltExecutor Thread-50-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[60, 60] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[60]
2023-12-11 16:28:20.118 o.a.s.e.b.BoltExecutor Thread-48-partitioner-executor[76, 76] [INFO] Preparing bolt partitioner:[76]
2023-12-11 16:28:20.118 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3378199865264390574, 6837611288718397660] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.118 c.d.s.i.DummyIndexer Thread-43-ssb-executor[98, 98] [INFO] Mapping key parse.keywords to field keywords
2023-12-11 16:28:20.118 c.d.s.b.URLPartitionerBolt Thread-48-partitioner-executor[76, 76] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.118 o.a.s.e.b.BoltExecutor Thread-46-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[36, 36] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[36]
2023-12-11 16:28:20.119 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[64, 64]
2023-12-11 16:28:20.118 c.d.s.i.DummyIndexer Thread-43-ssb-executor[98, 98] [INFO] Mapping key parse.description to field description
2023-12-11 16:28:20.120 o.a.s.e.Executor main [INFO] Loading executor tasks fetch:[72, 72]
2023-12-11 16:28:20.120 o.a.s.e.b.BoltExecutor Thread-43-ssb-executor[98, 98] [INFO] Prepared bolt ssb:[98]
2023-12-11 16:28:20.119 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3378199865264390574, 6837611288718397660] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.121 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-3378199865264390574, 6837611288718397660] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.121 o.a.s.e.b.BoltExecutor Thread-48-partitioner-executor[76, 76] [INFO] Prepared bolt partitioner:[76]
2023-12-11 16:28:20.121 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [-2068343561169177225, -531835215444606166, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.119 o.a.s.e.b.BoltExecutor Thread-50-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[60, 60] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[60]
2023-12-11 16:28:20.121 o.a.s.e.b.BoltExecutor Thread-53-fetch-executor[72, 72] [INFO] Preparing bolt fetch:[72]
2023-12-11 16:28:20.121 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [-2068343561169177225, -531835215444606166, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.121 o.a.s.e.Executor main [INFO] Finished loading executor fetch:[72, 72]
2023-12-11 16:28:20.122 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[56, 56]
2023-12-11 16:28:20.122 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [-2068343561169177225, -531835215444606166, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.122 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-4123990916148515094, -725436098569196388] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.121 o.a.s.e.b.BoltExecutor Thread-51-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[64, 64] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[64]
2023-12-11 16:28:20.123 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-4123990916148515094, -725436098569196388] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.123 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[56, 56]
2023-12-11 16:28:20.123 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-4123990916148515094, -725436098569196388] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.123 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[52, 52]
2023-12-11 16:28:20.123 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3378199865264390574, -2606298094286109596] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.123 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3378199865264390574, -2606298094286109596] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.123 o.a.s.e.b.BoltExecutor Thread-51-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[64, 64] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[64]
2023-12-11 16:28:20.124 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-3378199865264390574, -2606298094286109596] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.124 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:83, stream: __ack_init, id: {}, [-3478115639169539415, 7278560857325105651, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.124 o.a.s.e.b.BoltExecutor Thread-54-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[56, 56] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[56]
2023-12-11 16:28:20.124 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:83, stream: __ack_init, id: {}, [-3478115639169539415, 7278560857325105651, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.125 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[52, 52]
2023-12-11 16:28:20.125 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:83, stream: __ack_init, id: {}, [-3478115639169539415, 7278560857325105651, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.125 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[44, 44]
2023-12-11 16:28:20.125 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-2068343561169177225, 1145033480577199324] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.126 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-2068343561169177225, 1145033480577199324] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.126 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-2068343561169177225, 1145033480577199324] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.126 o.a.s.e.b.BoltExecutor Thread-54-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[56, 56] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[56]
2023-12-11 16:28:20.126 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[44, 44]
2023-12-11 16:28:20.126 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [-1801313057165177831, -1465155875809605520, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.126 o.a.s.e.b.BoltExecutor Thread-56-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[52, 52] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[52]
2023-12-11 16:28:20.126 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[40, 40]
2023-12-11 16:28:20.126 o.a.s.e.b.BoltExecutor Thread-58-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[44, 44] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[44]
2023-12-11 16:28:20.126 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [-1801313057165177831, -1465155875809605520, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.127 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[40, 40]
2023-12-11 16:28:20.127 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [-1801313057165177831, -1465155875809605520, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.127 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[48, 48]
2023-12-11 16:28:20.129 o.a.s.e.b.BoltExecutor Thread-56-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[52, 52] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[52]
2023-12-11 16:28:20.130 o.a.s.e.b.BoltExecutor Thread-59-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[40, 40] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[40]
2023-12-11 16:28:20.130 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:83, stream: __ack_init, id: {}, [-148650565251030105, 776269881405876811, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.131 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:83, stream: __ack_init, id: {}, [-148650565251030105, 776269881405876811, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.131 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[48, 48]
2023-12-11 16:28:20.131 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[84, 84]
2023-12-11 16:28:20.132 o.a.s.e.b.BoltExecutor Thread-58-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[44, 44] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[44]
2023-12-11 16:28:20.132 o.a.s.e.b.BoltExecutor Thread-59-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[40, 40] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[40]
2023-12-11 16:28:20.132 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:83, stream: __ack_init, id: {}, [-148650565251030105, 776269881405876811, 83] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.132 o.a.s.e.b.BoltExecutor Thread-62-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[48, 48] [INFO] Preparing bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[48]
2023-12-11 16:28:20.133 o.a.s.e.Executor main [INFO] Finished loading executor spout:[84, 84]
2023-12-11 16:28:20.133 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[88, 88]
2023-12-11 16:28:20.133 o.a.s.e.Executor main [INFO] Finished loading executor spout:[88, 88]
2023-12-11 16:28:20.134 o.a.s.e.Executor main [INFO] Loading executor tasks sitemap:[80, 80]
2023-12-11 16:28:20.134 o.a.s.e.s.SpoutExecutor Thread-65-spout-executor[88, 88] [INFO] Opening spout spout:[88]
2023-12-11 16:28:20.134 o.a.s.e.b.BoltExecutor Thread-62-__metrics_org.apache.storm.metric.LoggingMetricsConsumer-executor[48, 48] [INFO] Prepared bolt __metrics_org.apache.storm.metric.LoggingMetricsConsumer:[48]
2023-12-11 16:28:20.132 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [4970240361946410058, -1146044696927907618, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.134 o.a.s.e.Executor main [INFO] Finished loading executor sitemap:[80, 80]
2023-12-11 16:28:20.135 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[28, 28]
2023-12-11 16:28:20.135 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [4970240361946410058, -1146044696927907618, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.135 o.a.s.e.b.BoltExecutor Thread-67-sitemap-executor[80, 80] [INFO] Preparing bolt sitemap:[80]
2023-12-11 16:28:20.135 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [4970240361946410058, -1146044696927907618, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.135 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[28, 28]
2023-12-11 16:28:20.135 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [-7813587939482510894, -283538427390482334, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.135 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[32, 32]
2023-12-11 16:28:20.135 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [-7813587939482510894, -283538427390482334, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.135 o.a.s.e.b.BoltExecutor Thread-68-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[28, 28] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[28]
2023-12-11 16:28:20.135 o.a.s.e.s.SpoutExecutor Thread-64-spout-executor[84, 84] [INFO] Opening spout spout:[84]
2023-12-11 16:28:20.136 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[32, 32]
2023-12-11 16:28:20.136 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [-7813587939482510894, -283538427390482334, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.137 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[12, 12]
2023-12-11 16:28:20.137 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [2286059023404455111, 2600024955830255420, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.137 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [2286059023404455111, 2600024955830255420, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.137 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[12, 12]
2023-12-11 16:28:20.137 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [2286059023404455111, 2600024955830255420, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.138 o.a.s.e.b.BoltExecutor Thread-69-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[32, 32] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[32]
2023-12-11 16:28:20.138 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:95, stream: __ack_init, id: {}, [2946569250910592797, -1400580135842812437, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.138 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[4, 4]
2023-12-11 16:28:20.138 o.a.s.e.b.BoltExecutor Thread-70-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[12, 12] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[12]
2023-12-11 16:28:20.139 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:95, stream: __ack_init, id: {}, [2946569250910592797, -1400580135842812437, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.139 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[4, 4]
2023-12-11 16:28:20.140 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[8, 8]
2023-12-11 16:28:20.140 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:95, stream: __ack_init, id: {}, [2946569250910592797, -1400580135842812437, 95] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.140 o.a.s.e.b.BoltExecutor Thread-71-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[4, 4] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[4]
2023-12-11 16:28:20.140 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-2068343561169177225, 6573333448291401933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.140 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-2068343561169177225, 6573333448291401933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): 1702312100140
2023-12-11 16:28:20.141 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-2068343561169177225, 6573333448291401933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): 1702312100140 TASK: 2 DELTA: 1
2023-12-11 16:28:20.141 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[8, 8]
2023-12-11 16:28:20.141 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[20, 20]
2023-12-11 16:28:20.142 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[20, 20]
2023-12-11 16:28:20.142 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[16, 16]
2023-12-11 16:28:20.142 o.a.s.e.b.BoltExecutor Thread-72-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[8, 8] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[8]
2023-12-11 16:28:20.143 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[16, 16]
2023-12-11 16:28:20.144 o.a.s.e.Executor main [INFO] Loading executor tasks __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[24, 24]
2023-12-11 16:28:20.145 o.a.s.e.b.BoltExecutor Thread-73-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[20, 20] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[20]
2023-12-11 16:28:20.145 o.a.s.e.b.BoltExecutor Thread-74-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[16, 16] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[16]
2023-12-11 16:28:20.145 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3478115639169539415, -5709649264682537008] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.145 o.a.s.e.Executor main [INFO] Finished loading executor __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[24, 24]
2023-12-11 16:28:20.146 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3478115639169539415, -5709649264682537008] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.146 o.a.s.e.Executor main [INFO] Loading executor tasks warc:[116, 116]
2023-12-11 16:28:20.147 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-3478115639169539415, -5709649264682537008] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.147 o.a.s.e.b.BoltExecutor Thread-75-__metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[24, 24] [INFO] Preparing bolt __metrics_com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[24]
2023-12-11 16:28:20.148 o.a.s.e.Executor main [INFO] Finished loading executor warc:[116, 116]
2023-12-11 16:28:20.147 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-1801313057165177831, -4745064018105203615] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.148 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[96, 96]
2023-12-11 16:28:20.148 c.d.s.u.URLPartitioner Thread-28-spout-executor[82, 82] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.148 c.d.s.u.URLPartitioner Thread-30-spout-executor[86, 86] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.148 o.a.s.e.Executor main [INFO] Finished loading executor spout:[96, 96]
2023-12-11 16:28:20.148 o.a.s.e.Executor main [INFO] Loading executor tasks spout:[92, 92]
2023-12-11 16:28:20.148 c.d.s.u.URLPartitioner Thread-65-spout-executor[88, 88] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.148 c.d.s.u.URLPartitioner Thread-45-spout-executor[90, 90] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.149 c.d.s.u.URLPartitioner Thread-64-spout-executor[84, 84] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.149 o.a.s.e.Executor main [INFO] Finished loading executor spout:[92, 92]
2023-12-11 16:28:20.149 c.d.s.u.URLPartitioner Thread-44-spout-executor[94, 94] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.148 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-1801313057165177831, -4745064018105203615] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.149 o.a.s.e.Executor main [INFO] Loading executor tasks feed:[68, 69]
2023-12-11 16:28:20.150 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-1801313057165177831, -4745064018105203615] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.150 o.a.s.e.Executor main [INFO] Finished loading executor feed:[68, 69]
2023-12-11 16:28:20.151 o.a.s.e.Executor main [INFO] Loading executor tasks status:[100, 107]
2023-12-11 16:28:20.151 o.a.s.e.b.BoltExecutor Thread-76-warc-executor[116, 116] [INFO] Preparing bolt warc:[116]
2023-12-11 16:28:20.151 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-148650565251030105, -7054458867463269924] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.151 o.a.s.e.s.SpoutExecutor Thread-77-spout-executor[96, 96] [INFO] Opening spout spout:[96]
2023-12-11 16:28:20.151 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-148650565251030105, -7054458867463269924] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.152 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-148650565251030105, -7054458867463269924] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.152 c.d.s.u.URLPartitioner Thread-77-spout-executor[96, 96] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.152 o.a.s.e.b.BoltExecutor Thread-79-feed-executor[68, 69] [INFO] Preparing bolt feed:[68, 69]
2023-12-11 16:28:20.152 o.a.s.e.Executor main [INFO] Finished loading executor status:[100, 107]
2023-12-11 16:28:20.157 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [4970240361946410058, -1393566806601548956] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.157 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [4970240361946410058, -1393566806601548956] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.155 o.a.s.e.b.BoltExecutor Thread-80-status-executor[100, 107] [INFO] Preparing bolt status:[100, 101, 102, 103, 104, 105, 106, 107]
2023-12-11 16:28:20.157 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [4970240361946410058, -1393566806601548956] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.158 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-7813587939482510894, 7480049982378529504] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.158 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-7813587939482510894, 7480049982378529504] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.158 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-7813587939482510894, 7480049982378529504] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.158 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2286059023404455111, 2066116417734427933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.158 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2286059023404455111, 2066116417734427933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.158 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [2286059023404455111, 2066116417734427933] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.158 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [3800453218928820137, -118032834169845226, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.159 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [3800453218928820137, -118032834169845226, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.159 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [3800453218928820137, -118032834169845226, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.159 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [-2634672587229201707, -1726815147112707869, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.159 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [-2634672587229201707, -1726815147112707869, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.159 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupCpuStat is disabled. cpu is not a mounted subsystem
2023-12-11 16:28:20.159 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [-2634672587229201707, -1726815147112707869, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.159 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [8946397892981490109, 1319004779640931907, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.160 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [8946397892981490109, 1319004779640931907, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.160 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [8946397892981490109, 1319004779640931907, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.160 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2946569250910592797, -6897279057914939938] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.160 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2946569250910592797, -6897279057914939938] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.160 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [2946569250910592797, -6897279057914939938] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.160 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:87, stream: __ack_init, id: {}, [4256344492579818105, -5624555335493571757, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.160 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:87, stream: __ack_init, id: {}, [4256344492579818105, -5624555335493571757, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.160 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:87, stream: __ack_init, id: {}, [4256344492579818105, -5624555335493571757, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.160 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupMemoryLimit is disabled. memory is not a mounted subsystem
2023-12-11 16:28:20.161 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:97, stream: __ack_init, id: {}, [-7267420976913106448, -1663728895472928813, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.161 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:97, stream: __ack_init, id: {}, [-7267420976913106448, -1663728895472928813, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.161 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:97, stream: __ack_init, id: {}, [-7267420976913106448, -1663728895472928813, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.161 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [4572311561040840987, -6788911851055926022, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.161 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [4572311561040840987, -6788911851055926022, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.161 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [4572311561040840987, -6788911851055926022, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.161 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:97, stream: __ack_init, id: {}, [4981792382413578668, -3074453321169276904, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.161 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:97, stream: __ack_init, id: {}, [4981792382413578668, -3074453321169276904, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.161 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupCpuGuaranteeByCfsQuota is disabled. cpu is not a mounted subsystem
2023-12-11 16:28:20.162 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:97, stream: __ack_init, id: {}, [4981792382413578668, -3074453321169276904, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.162 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:87, stream: __ack_init, id: {}, [-3490743804298062617, -7275838135438203156, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.162 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:87, stream: __ack_init, id: {}, [-3490743804298062617, -7275838135438203156, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.162 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:87, stream: __ack_init, id: {}, [-3490743804298062617, -7275838135438203156, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.162 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:93, stream: __ack_init, id: {}, [-4074006197233086992, -4335374168709552037, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.162 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:93, stream: __ack_init, id: {}, [-4074006197233086992, -4335374168709552037, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.162 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:93, stream: __ack_init, id: {}, [-4074006197233086992, -4335374168709552037, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.163 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:93, stream: __ack_init, id: {}, [-9061266249914929912, -5546759073310690614, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.163 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupMemoryUsage is disabled. memory is not a mounted subsystem
2023-12-11 16:28:20.163 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:93, stream: __ack_init, id: {}, [-9061266249914929912, -5546759073310690614, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.163 o.a.s.e.s.SpoutExecutor Thread-78-spout-executor[92, 92] [INFO] Opening spout spout:[92]
2023-12-11 16:28:20.163 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:93, stream: __ack_init, id: {}, [-9061266249914929912, -5546759073310690614, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.164 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [5301100988092975999, 6835809773324968329, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.164 c.d.s.u.URLPartitioner Thread-78-spout-executor[92, 92] [INFO] Using partition mode : byDomain
2023-12-11 16:28:20.164 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [5301100988092975999, 6835809773324968329, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.165 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupCpu is disabled. cpuacct is not a mounted subsystem
2023-12-11 16:28:20.165 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [5301100988092975999, 6835809773324968329, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.165 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [6760544673998728235, 4712299035249289647, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.165 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [6760544673998728235, 4712299035249289647, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.166 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [6760544673998728235, 4712299035249289647, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.166 o.a.s.m.c.CGroupMetricsBase Thread-47-__system-executor[-1, -1] [WARN] CGroupCpuGuarantee is disabled. cpu is not a mounted subsystem
2023-12-11 16:28:20.166 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [2209434805601426965, -2651182582903056162, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.166 o.a.s.e.b.BoltExecutor Thread-47-__system-executor[-1, -1] [INFO] Prepared bolt __system:[-1]
2023-12-11 16:28:20.166 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: 0 TUPLE: source: spout:89, stream: __ack_init, id: {}, [2209434805601426965, -2651182582903056162, 89] PROC_START_TIME(sampled): 1702312100166 EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.167 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [2209434805601426965, -2651182582903056162, 89] PROC_START_TIME(sampled): 1702312100166 EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.167 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [-3521069700462821725, -4121685141838493521, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.168 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [-3521069700462821725, -4121685141838493521, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.168 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [-3521069700462821725, -4121685141838493521, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.168 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:87, stream: __ack_init, id: {}, [-5099824388405106009, -7814831663344770237, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.169 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:87, stream: __ack_init, id: {}, [-5099824388405106009, -7814831663344770237, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.169 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:87, stream: __ack_init, id: {}, [-5099824388405106009, -7814831663344770237, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.169 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:91, stream: __ack_init, id: {}, [-6736403382555792190, -4970367296170810415, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.170 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:91, stream: __ack_init, id: {}, [-6736403382555792190, -4970367296170810415, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.170 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:91, stream: __ack_init, id: {}, [-6736403382555792190, -4970367296170810415, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.171 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:93, stream: __ack_init, id: {}, [3866775734294629399, -6768188872314748886, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.171 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:93, stream: __ack_init, id: {}, [3866775734294629399, -6768188872314748886, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): 1702312100171
2023-12-11 16:28:20.172 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:93, stream: __ack_init, id: {}, [3866775734294629399, -6768188872314748886, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): 1702312100171 TASK: 2 DELTA: 1
2023-12-11 16:28:20.172 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:91, stream: __ack_init, id: {}, [-4180928043309386618, -5043785088620001322, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.173 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:91, stream: __ack_init, id: {}, [-4180928043309386618, -5043785088620001322, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.173 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:91, stream: __ack_init, id: {}, [-4180928043309386618, -5043785088620001322, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.174 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:97, stream: __ack_init, id: {}, [-7886972260148926948, 4188480566748697533, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.174 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:97, stream: __ack_init, id: {}, [-7886972260148926948, 4188480566748697533, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.174 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:97, stream: __ack_init, id: {}, [-7886972260148926948, 4188480566748697533, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.177 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:93, stream: __ack_init, id: {}, [-4098573674829406881, 826968679567669069, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.177 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: 0 TUPLE: source: spout:93, stream: __ack_init, id: {}, [-4098573674829406881, 826968679567669069, 93] PROC_START_TIME(sampled): 1702312100177 EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.177 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:93, stream: __ack_init, id: {}, [-4098573674829406881, 826968679567669069, 93] PROC_START_TIME(sampled): 1702312100177 EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.178 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3478115639169539415, -8730761081484476922] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.178 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3478115639169539415, -8730761081484476922] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.178 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-3478115639169539415, -8730761081484476922] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.178 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:93, stream: __ack_init, id: {}, [6237080999574947757, -238454198009739507, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.178 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:93, stream: __ack_init, id: {}, [6237080999574947757, -238454198009739507, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.179 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:93, stream: __ack_init, id: {}, [6237080999574947757, -238454198009739507, 93] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.179 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:97, stream: __ack_init, id: {}, [6084453207936611306, -3914889148982770675, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.179 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:97, stream: __ack_init, id: {}, [6084453207936611306, -3914889148982770675, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.179 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:97, stream: __ack_init, id: {}, [6084453207936611306, -3914889148982770675, 97] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.179 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:87, stream: __ack_init, id: {}, [-8043919662105889343, 4431343109975724737, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.179 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:87, stream: __ack_init, id: {}, [-8043919662105889343, 4431343109975724737, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.180 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:87, stream: __ack_init, id: {}, [-8043919662105889343, 4431343109975724737, 87] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.180 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-7267420976913106448, 3481212360140942143] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.180 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-7267420976913106448, 3481212360140942143] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.180 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-7267420976913106448, 3481212360140942143] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.180 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:89, stream: __ack_init, id: {}, [3858824648568270060, -3559849241998194517, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.180 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:89, stream: __ack_init, id: {}, [3858824648568270060, -3559849241998194517, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.181 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: spout:89, stream: __ack_init, id: {}, [3858824648568270060, -3559849241998194517, 89] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.181 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: spout:91, stream: __ack_init, id: {}, [6935833859334561755, -8472375794900500271, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.181 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: spout:91, stream: __ack_init, id: {}, [6935833859334561755, -8472375794900500271, 91] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.181 c.d.s.u.URLPartitioner Thread-80-status-executor[100, 107] [INFO] Using partition mode : byDomain

But then after a bit of time it loads the topo config - isn't this out of order:

2023-12-11 16:28:20.244 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [6084453207936611306, 1594212693638942219] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.244 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [6084453207936611306, 1594212693638942219] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.244 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5236018837279089367, 4910516185876199059] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.244 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5236018837279089367, 4910516185876199059] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.245 o.a.s.d.w.Worker main [INFO] BackPressure status change checking will be performed every 50 millis
2023-12-11 16:28:20.245 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-5236018837279089367, 4910516185876199059] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.245 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [6935833859334561755, 3759631466051588138] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.245 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [6935833859334561755, 3759631466051588138] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.245 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [6935833859334561755, 3759631466051588138] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.245 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-8927812074023104766, 2878556287460316803] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.245 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-8927812074023104766, 2878556287460316803] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.245 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-8927812074023104766, 2878556287460316803] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.245 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2556569793293798762, 7498757573609867582] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.245 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [2556569793293798762, 7498757573609867582] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.246 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [2556569793293798762, 7498757573609867582] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.246 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-8935012321312715528, -56508434523359153] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.246 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-8935012321312715528, -56508434523359153] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.246 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-8935012321312715528, -56508434523359153] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.246 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-5627070527711278671, 1549329322995875292] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.246 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-5627070527711278671, 1549329322995875292] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.247 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-5627070527711278671, 1549329322995875292] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.245 o.a.s.d.w.Worker main [INFO] Worker has topology config {topology.backpressure.wait.strategy=org.apache.storm.policy.WaitStrategyProgressive, sitemap.extensions=null, sitemap.schedule.delay=15, storm.zookeeper.auth.user=null, storm.messaging.netty.buffer_size=5242880, topology.name=NewsCrawl, scheduler.adaptive.fetchInterval.ma
x=129600, ui.filter=null, worker.profiler.enabled=false, executor.metrics.frequency.secs=60, fetchInterval.ERROR.isSitemapNews=true=86400, supervisor.queue.size=128, logviewer.cleanup.age.mins=10080, parsefilters.config.file=parsefilters.json, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, es.status.max.urls.per
.bucket=8, indexer.url.fieldname=url, fetchInterval.DISCOVERED.isSitemap=true=300, topology.transfer.buffer.size=1000, storm.oci.nscd.dir=/var/run/nscd, feed.sniffContent=true, nimbus.worker.heartbeats.recovery.strategy.class=org.apache.storm.nimbus.TimeOutWorkerHeartbeatsRecoveryStrategy, logviewer.port=8000, worker.childopts=-Xmx%HEAP-MEM%m -XX:+Hea
pDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.daemon.metrics.reporter.plugins=[org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter], logviewer.childopts=-Xmx128m, topology.ras.acker.executors.per.worker=1, storm.messaging.transport=org.apache.storm.messaging.netty.Context, storm.messaging.netty.authentication=false, fe
tcher.server.min.delay=0.0, storm.cgroup.memory.limit.tolerance.margin.mb=0.0, es.status.concurrentRequests=100, max.fetch.errors=3, storm.metricprocessor.class=org.apache.storm.metricstore.NimbusMetricProcessor, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, fetchInterval.REDIRECTION.isSitemapNews=true=20160, es.status.globa
l.sort.field=nextFetchDate, http.robots.403.allow=true, indexer.text.fieldname=content, indexer.md.mapping=[parse.title=title, parse.keywords=keywords, parse.description=description], supervisor.slots.ports=[6800, 6801, 6802, 6803], storm.local.dir=storm-local, topology.backpressure.wait.park.microsec=100, fetchInterval.ERROR.isSitemapIndex=true=86400
, scheduler.class=com.digitalpebble.stormcrawler.persistence.AdaptiveScheduler, protocol.instances.num=1, fetcher.threads.per.queue=1, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64, worker.gc.childopts=, topology.multilang.serializer=org.apache.storm.
multilang.JsonSerializer, drpc.request.timeout.secs=600, topology.state.synchronization.timeout.secs=60, topology.bolt.wait.progressive.level2.count=1000, topology.worker.shared.thread.pool.size=4, topology.submitter.user=storm, pacemaker.servers=[], supervisor.monitor.frequency.secs=3, storm.auth.simple-white-list.users=[], topology.scheduler.strateg
y=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, storm.zookeeper.port=2181, storm.cluster.state.store=org.apache.storm.cluster.ZKStateStorageFactory, nimbus.thrift.threads=64, es.metrics.index.name=metrics, supervisor.supervisors=[], http.agent.version=118.0, storm.cluster.metrics.consumer.publish.interval.secs
=60, logviewer.filter.params=null, sitemap.discovery=true, protocols=http,https,file, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, topology.backpressure.wait.progressive.level2.count=1000, fetcher.max.queue.size=10, drpc.https.keystore.password=, resource.aware.scheduler.constraint.max.state.search=100000, superv
isor.heartbeat.frequency.secs=5, metadata.track.depth=true, storm.supervisor.medium.memory.grace.period.ms=30000, es.status.recentDate.increase=-1, ui.port=8080, fetchInterval.REDIRECTION.isSitemapIndex=true=20160, fetchInterval.DISCOVERED.isSitemapNews=true=0, warc.rotation.policy.max-minutes=1440, drpc.https.keystore.type=JKS, supervisor.memory.capa
city.mb=4096.0, storm.metricstore.class=org.apache.storm.metricstore.rocksdb.RocksDbStore, pacemaker.client.max.threads=2, okhttp.protocol.connection.pool={connection.keep.alive=300, max.idle.connections=5}, topology.shellbolt.max.pending=100, drpc.max_buffer_size=1048576, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, fi
le.protocol.implementation=com.digitalpebble.stormcrawler.protocol.file.FileProtocol, nimbus.supervisor.timeout.secs=60, http.agent.url=, storm.zookeeper.root=/storm, topology.disable.loadaware.messaging=false, feed.filter.hours.since.published=720, topology.ras.one.executor.per.worker=false, parser.emitOutlinks.max.per.page=-1, nimbus.topology.valida
tor=org.apache.storm.nimbus.DefaultTopologyValidator, fetchInterval.fetch.error=2880, pacemaker.max.threads=50, fetcher.max.crawl.delay.force=false, ui.disable.http.binding=true, supervisor.blobstore.download.max_retries=3, topology.enable.message.timeouts=true, http.robots.headers.skip=false, es.status.addresses=http://10.128.0.34:9200, fetcher.max.t
hrottle.sleep=-1, urlfilters.config.file=urlfilters.json, logviewer.disable.http.binding=true, storm.messaging.netty.transfer.batch.size=262144, http.accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, storm.metricstore.rocksdb.retention_hours=240, supervisor.run.worker.as.user=false, fetcher.threads.number=100, topology.tasks=null
, indexer.text.maxlength=-1, fetchInterval.ERROR.isFeed=true=86400, logviewer.filter=null, pacemaker.kerberos.users=[], es.status.bucket.field=metadata.hostname, supervisor.worker.heartbeats.max.timeout.secs=600, supervisor.worker.timeout.secs=30, topology.worker.receiver.thread.count=1, es.status.routing.fieldname=metadata.hostname, scheduler.adaptiv
e.fetchInterval.min=60, robots.cache.spec=maximumSize=10000,expireAfterWrite=6h, nimbus.file.copy.expiration.secs=600, http.trust.everything=true, nimbus.topology.blobstore.deletion.delay.ms=300000, topology.workers=2, blacklist.scheduler.tolerance.time.secs=300, topology.environment=null, drpc.invocations.port=3773, storm.metricstore.rocksdb.create_i
f_missing=true, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, fetchInterval.FETCHED.isSitemap=true=259200, storm.nimbus.retry.intervalceiling.millis=60000, storm.messaging.netty.server_worker_threads=1, http.content.partial.as.trimmed=true, storm.thrift.socket.timeout.ms=600000, storm.topology.classpath.beginning.enabled=false, es
.status.index.name=status, topology.tick.tuple.freq.secs=null, nimbus.inbox.jar.expiration.secs=3600, storm.zookeeper.retry.interval=1000, storm.messaging.netty.buffer.high.watermark=16777216, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.zookeeper.retry.times=5, nimbus.code.sync.freq.secs=120, topology.component.resources.offheap.mem
ory.mb=0.0, textextractor.skip.after=-1, partition.url.mode=byDomain, spout.ttl.purgatory=600, topology.state.checkpoint.interval.ms=1000, supervisor.localizer.cleanup.interval.ms=30000, storm.health.check.dir=healthchecks, supervisor.cpu.capacity=400.0, topology.backpressure.wait.progressive.level3.sleep.millis=1, fetcher.timeout.queue=-1, http.agent
.description=Firefox 118.0 Linux, status.updater.use.cache=true, storm.worker.min.cpu.pcore.percent=0.0, topology.spout.wait.progressive.level3.sleep.millis=1, track.anchors=true, topology.worker.childopts=-Xmx7g -Djava.net.preferIPv4Stack=true, drpc.https.port=-1, topology.bolt.wait.park.microsec=100, detect.mimetype=true, topology.acker.executors=nu
ll, topology.bolt.wait.progressive.level3.sleep.millis=1, supervisor.worker.start.timeout.secs=120, supervisor.worker.shutdown.sleep.secs=3, logviewer.max.per.worker.logs.size.mb=2048, task.heartbeat.frequency.secs=3, scheduler.adaptive.setLastModified=true, supervisor.thrift.max_buffer_size=1048576, es.status.max.buckets=900, fetchInterval.REDIRECTIO
N.isFeed=true=20160, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, es.status.recentDate.min.gap=-1, topology.backpressure.enable=false, topology.producer.batch.size=1, drpc.worker.threads=64, http.allow.redirects=false, spout.min.delay.queries=15000, blacklist.scheduler.reporter=org.apache.storm.scheduler.blacklist.reporters.L
ogReporter, indexer.canonical.name=canonical, fetchInterval.FETCHED.isSitemapNews=true=720, drpc.queue.size=128, topology.eventlogger.executors=1, topology.spout.recvq.skips=3, storm.resource.isolation.plugin.enable=false, storm.supervisor.memory.limit.tolerance.margin.mb=128.0, storm.disable.symlinks=false, fetchInterval.default=5256000, transactiona
l.zookeeper.servers=null, https.protocol.implementation=com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol, logs.users=null, pacemaker.thrift.message.size.max=10485760, topology.ras.one.component.per.worker=false, es.status.bulkActions=1000, supervisor.thrift.port=6628, dev.zookeeper.path=/tmp/dev-storm-zookeeper, status.updater.cache.spec=ma
ximumSize=1000000,expireAfterAccess=4h, drpc.invocations.threads=64, storm.metricstore.rocksdb.metadata_string_cache_capacity=4000, storm.workers.artifacts.dir=workers-artifacts, storm.cluster.mode=distributed, ui.childopts=-Xmx768m, task.refresh.poll.secs=10, supervisor.childopts=-Djava.net.preferIPv4Stack=true, storm.health.check.timeout.ms=5000, wo
rker.profiler.command=flight.bash, storm.messaging.netty.min_wait_ms=100, http.content.limit=2148576, fetchInterval.FETCH_ERROR.isFeed=true=4320, topology.submitter.principal=, storm.resource.isolation.plugin=org.apache.storm.container.cgroup.CgroupManager, fetcher.server.delay.force=false, metadata.persist=[_redirTo, fetch.statusCode, error.cause, er
ror.source, isSitemap, isSitemapNews, isSitemapIndex, isSitemapVerified, isFeed, numLinks, last-modified, signature, signatureChangeDate, fetchInterval, protocol.etag], fetchInterval.FETCH_ERROR.isSitemap=true=259200, storm.exhibitor.port=8080, topology.bolt.wait.progressive.level1.count=1, pacemaker.auth.method=NONE, storm.oci.cgroup.root=/sys/fs/cgr
oup, storm.id=NewsCrawl-1-1702312011, supervisor.thrift.threads=16, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, supervisor.supervisors.commands=[], storm.cgroup.memory.enforcement.enable=false, drpc.port=3772, supervisor.localizer.update.blob.interval.secs=30, topology.max.spout.pending=120, fetchInterval.ERROR.is
Sitemap=true=259200, worker.metrics={CGroupCpuStat=org.apache.storm.metrics2.cgroup.CGroupCpuStat, CGroupMemoryLimit=org.apache.storm.metrics2.cgroup.CGroupMemoryLimit, CGroupCpuGuaranteeByCfsQuota=org.apache.storm.metrics2.cgroup.CGroupCpuGuaranteeByCfsQuota, CGroupMemory=org.apache.storm.metrics2.cgroup.CGroupMemoryUsage, CGroupCpu=org.apache.storm.
metrics2.cgroup.CGroupCpu, CGroupCpuGuarantee=org.apache.storm.metrics2.cgroup.CGroupCpuGuarantee}, fetchInterval.FETCHED.isFeed=true=360, es.status.bucket.sort.field=[nextFetchDate, url], es.status.sample=true, topology.component.cpu.pcore.percent=10.0, fetchInterval.FETCH_ERROR.isSitemapIndex=true=4320, worker.max.timeout.secs=600, blacklist.schedul
er.resume.time.secs=1800, drpc.childopts=-Xmx768m, nimbus.task.launch.secs=120, storm.supervisor.hard.memory.limit.overage.mb=2024, storm.zookeeper.servers=[10.142.0.16, 10.142.0.17, 10.142.0.18, 10.138.0.52, 10.168.0.11, 10.168.0.12], topology.localityaware.higher.bound=0.8, storm.cgroup.hierarchy.name=storm, blacklist.scheduler.assume.supervisor.bad
.based.on.bad.slot=true, protocol.md.prefix=protocol., warc={fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem}, nimbus.assignments.service.threads=10, es.status.routing=true, worker.heap.memory.mb=4096, http.agent.name=Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0, storm.network.topography.plugin=org.apache.storm.networkto
pography.DefaultRackDNSToSwitchMapping, warc.dir=/data/warc, topology.stats.sample.rate=0.05, topology.ras.constraint.max.state.search=10000, storm.oci.cgroup.parent=/storm, topology.testing.always.try.serialize=false, nimbus.assignments.service.thread.queue.size=100, sitemap.offset.guess=1024, robots.error.cache.spec=maximumSize=10000,expireAfterWrit
e=1h, nimbus.local.assignments.backend.class=org.apache.storm.assignments.InMemoryAssignmentBackend, http.robots.content.limit=-1, storm.group.mapping.service.cache.duration.secs=120, fetcher.max.urls.in.queues=9000, topology.kryo.register={com.digitalpebble.stormcrawler.persistence.Status=null, com.digitalpebble.stormcrawler.Metadata=null}, nimbus.bl
obstore.class=org.apache.storm.blobstore.LocalFsBlobStore, topology.kryo.decorators=[], topology.executor.receive.buffer.size=32768, storm.nimbus.retry.times=5, topology.transfer.batch.size=1, transactional.zookeeper.port=null, storm.zookeeper.retry.intervalceiling.millis=30000, nimbus.thrift.port=6627, blacklist.scheduler.tolerance.count=3, fetcher.m
ax.crawl.delay=40, nimbus.seeds=[10.128.0.34], topology.min.replication.count=1, nimbus.blobstore.expiration.secs=600, storm.nimbus.retry.interval.millis=2000, topology.max.task.parallelism=null, nimbus.credential.renewers.freq.secs=600, es.status.query.timeout=-1, detect.charset.maxlength=10000, storm.thrift.transport=org.apache.storm.security.auth.S
impleTransportPlugin, storm.cgroup.hierarchy.dir=/cgroup/storm_resources, storm.zookeeper.auth.password=null, drpc.authorizer.acl.strict=false, topology.message.timeout.secs=300, topology.error.throttle.interval.secs=10, topology.backpressure.check.millis=50, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, topology.builtin.metrics.bucket.size.secs=60
, topology.spout.wait.park.microsec=100, fetchInterval.error=133920, storm.local.mode.zmq=false, http.protocol.implementation=com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol, parser.emitOutlinks=true, es.status.max.start.offset=500, http.store.headers=true, ui.header.buffer.bytes=4096, jsoup.treat.non.html.as.error=true, http.timeout=30000
, topology.serialized.message.size.metrics=false, topology.metrics.consumer.register=[{class=org.apache.storm.metric.LoggingMetricsConsumer, parallelism.hint=32}, {class=com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer, parallelism.hint=32}], drpc.disable.http.binding=true, storm.codedistributor.class=org.apache.storm.codedistribut
or.LocalFileSystemCodeDistributor, storm.supervisor.cgroup.rootdir=storm, topology.worker.max.heap.size.mb=768.0, indexer.ignore.empty.fields=false, storm.supervisor.hard.memory.limit.multiplier=2.0, worker.heartbeat.frequency.secs=1, storm.messaging.netty.max_wait_ms=1000, topology.backpressure.wait.progressive.level1.count=1, topology.max.error.repo
rt.per.interval=5, nimbus.thrift.max_buffer_size=1048576, fetchInterval.FETCHED.isSitemapVerified=true=720, storm.metricstore.rocksdb.location=storm_rocks, storm.supervisor.low.memory.threshold.mb=1024, ui.pagination=20, http.protocol.versions=null, topology.spout.wait.progressive.level2.count=0, blacklist.scheduler.strategy=org.apache.storm.scheduler
.blacklist.strategies.DefaultBlacklistStrategy, http.retry.on.connection.failure=true, storm.messaging.netty.client_worker_threads=1, supervisor.thrift.socket.timeout.ms=5000, storm.group.mapping.service.params=null, drpc.http.port=3774, scheduler.adaptive.fetchInterval.rate.incr=0.5, transactional.zookeeper.root=/transactional, supervisor.blobstore.d
ownload.thread.count=5, sitemap.sniffContent=true, topology.spout.wait.strategy=org.apache.storm.policy.WaitStrategyProgressive, storm.blobstore.inputstream.buffer.size.bytes=65536, fetcher.queue.mode=byHost, fetchInterval.FETCHED.isSitemapIndex=true=5040, logviewer.max.sum.worker.logs.size.mb=4096, topology.executor.overflow.limit=0, topology.batch.f
lush.interval.millis=1, selenium.tracing=false, pacemaker.port=6699, topology.worker.logwriter.childopts=-Xmx64m, fetcher.metrics.time.bucket.secs=30, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.blobstore.acl.validation.enabled=false, ui.filter.params=null, storm.supervisor.medium.memory.threshold.mb=1536,
 sitemap.filter.hours.since.modified=720, metadata.track.path=true, nimbus.cleanup.inbox.freq.secs=600, topology.fall.back.on.java.serialization=false, storm.nimbus.zookeeper.acls.fixup=true, logviewer.appender.name=A1, ui.users=null, pacemaker.childopts=-Xmx1024m, scheduler.display.resource=false, ui.actions.enabled=true, storm.zookeeper.connection.t
imeout=15000, robots.noFollow.strict=true, topology.debug=true, storm.blobstore.dependency.jar.upload.chunk.size.bytes=1048576, worker.log.level.reset.poll.secs=30, topology.spout.wait.progressive.level1.count=0, fetchInterval.DISCOVERED.isSitemapIndex=true=0, topology.priority=29, es.status.flushInterval=5s, warc.rotation.policy.max-mb=1023, schedule
r.adaptive.fetchInterval.rate.decr=0.2, storm.cgroup.resources=[cpu, memory], topology.classpath=null, storm.nimbus.zookeeper.acls.check=true, num.stat.buckets=20, supervisor.localizer.cache.target.size.mb=10240, spout.reset.fetchdate.after=120, topology.max.replication.wait.time.sec=60, storm.cgroup.cgexec.cmd=/bin/cgexec, topology.users=[null], sele
nium.timeouts={implicit=-1, pageLoad=-1, script=-1}, fetchInterval.DISCOVERED.isFeed=true=0, topology.trident.batch.emit.interval.millis=500, http.accept.language=en-us,en-gb,en;q=0.7,*;q=0.3, supervisor.enable=true, fetchInterval.FETCH_ERROR.isSitemapNews=true=4320, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strate
gies.priority.DefaultSchedulingPriorityStrategy, storm.messaging.netty.socket.backlog=500, storm.cgroup.inherit.cpuset.configs=false, nimbus.queue.size=100000, ui.disable.spout.lag.monitoring=true, pacemaker.base.threads=10, nimbus.childopts=-Xmx1024m, http.robots.5xx.allow=false, storm.zookeeper.superACL=null, nimbus.monitor.freq.secs=10, fetcher.ser
ver.delay=3.0, status.updater.unit.round.date=SECOND, topology.localityaware.lower.bound=0.2, nimbus.task.timeout.secs=30, http.robots.meta.skip=false, urlbuffer.class=com.digitalpebble.stormcrawler.persistence.urlbuffer.SimpleURLBuffer, ui.host=0.0.0.0, topology.bolt.wait.strategy=org.apache.storm.policy.WaitStrategyProgressive, es.metrics.addresses=
http://10.128.0.34:9200, pacemaker.thread.timeout=10, fetchInterval.REDIRECTION.isSitemap=true=259200, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.skip.missing.kryo.registrations=false, storm.zookeeper.session.timeout=20000, http.robots.file.skip=false, topology.component.resources.onheap.
memory.mb=128.0, storm.log4j2.conf.dir=log4j2, task.credentials.poll.secs=30, http.agent.email=, storm.blobstore.replication.factor=3, storm.messaging.netty.buffer.low.watermark=8388608}
2023-12-11 16:28:20.247 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-5236018837279089367, 7955357069768614300] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.247 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-5236018837279089367, 7955357069768614300] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.248 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-5236018837279089367, 7955357069768614300] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.248 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3461902836084744326, -8298087996738896118] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.248 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-3461902836084744326, -8298087996738896118] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.248 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-3461902836084744326, -8298087996738896118] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.248 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-8927812074023104766, -6381006344183577194] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.248 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-8927812074023104766, -6381006344183577194] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.248 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-8927812074023104766, -6381006344183577194] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.248 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-1214025998605477641, 8486828589341367712] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.248 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-1214025998605477641, 8486828589341367712] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.248 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-1214025998605477641, 8486828589341367712] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.249 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-2174812826611686353, 8933910478784277266] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.249 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-2174812826611686353, 8933910478784277266] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.249 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-2174812826611686353, 8933910478784277266] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.249 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [2556569793293798762, 1794687774208397323] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.249 o.a.s.d.w.Worker main [INFO] Worker b656245f-ae7f-4d41-bd0b-98ff985a51ed for storm NewsCrawl-1-1702312011 on 81ea410b-6200-4420-8b2d-452a189f65b7-10.142.0.21:6800  has finished loading
2023-12-11 16:28:20.249 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [2556569793293798762, 1794687774208397323] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.249 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [2556569793293798762, 1794687774208397323] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.249 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5412917930674871460, -2679041900049810362] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.249 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5412917930674871460, -2679041900049810362] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.250 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-5412917930674871460, -2679041900049810362] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.250 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-8935012321312715528, 1410811458580393384] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.250 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-8935012321312715528, 1410811458580393384] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.250 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-8935012321312715528, 1410811458580393384] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.250 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5809973328905888172, -8700995318426037421] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.250 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5809973328905888172, -8700995318426037421] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.250 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-5809973328905888172, -8700995318426037421] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.250 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5160443130954077648, -7999364024869339203] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.250 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [-5160443130954077648, -7999364024869339203] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.250 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [-5160443130954077648, -7999364024869339203] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.251 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3461902836084744326, -3805243869296651998] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.251 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-3461902836084744326, -3805243869296651998] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.251 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-3461902836084744326, -3805243869296651998] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.251 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-1214025998605477641, 7938128007580305540] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.251 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-1214025998605477641, 7938128007580305540] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.251 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-1214025998605477641, 7938128007580305540] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.251 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [7741585547196580345, 5967571995592991844] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.251 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: prefilter:79, stream: __ack_ack, id: {}, [7741585547196580345, 5967571995592991844] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.251 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: prefilter:79, stream: __ack_ack, id: {}, [7741585547196580345, 5967571995592991844] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.251 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-2174812826611686353, -4092819621652857975] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:20.252 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-2174812826611686353, -4092819621652857975] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:20.252 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: partitioner:77, stream: __ack_ack, id: {}, [-2174812826611686353, -4092819621652857975] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:20.252 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: partitioner:77, stream: __ack_ack, id: {}, [-5412917930674871460, 5353909444598418875] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 

Finally, I do see Beta's worker start to emit some errors - logging related? Datagram too big?

protocol.content-type: text/html; charset=UTF-8
] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null]
2023-12-11 16:28:21.470 STDERR Thread-1 [INFO] 	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89)
2023-12-11 16:28:21.470 STDERR Thread-1 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154)
2023-12-11 16:28:21.470 STDERR Thread-1 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140)
2023-12-11 16:28:21.471 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -1602384185032665566] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.471 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -1602384185032665566] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.470 STDERR Thread-1 [INFO] 	at org.apache.storm.utils.Utils$1.run(Utils.java:398)
2023-12-11 16:28:21.471 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -7745260568013248921] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.471 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -7745260568013248921] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.471 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -7745260568013248921] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.471 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [2648580543762087270, 6163409022820091796] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.471 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: 0 TUPLE: source: status:108, stream: __ack_ack, id: {}, [2648580543762087270, 6163409022820091796] PROC_START_TIME(sampled): 1702312101471 EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.471 STDERR Thread-1 [INFO] 	at java.base/java.lang.Thread.run(Thread.java:829)
2023-12-11 16:28:21.471 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [2648580543762087270, 6163409022820091796] PROC_START_TIME(sampled): 1702312101471 EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.471 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 8454087514544735688] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.471 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 8454087514544735688] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.471 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 8454087514544735688] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.471 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 7899198290451746760] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.471 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 7899198290451746760] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.471 STDERR Thread-1 [INFO] Caused by: java.io.IOException: Message too long
2023-12-11 16:28:21.472 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, 7899198290451746760] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.472 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -6200665771136418110] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.472 o.a.s.d.Task FetcherThread #10 [INFO] Emitting Tuple: taskId=72 componentId=fetch stream=status values=[https://www.clickondetroit.com/arcio/sitemap/, url.path: https://www.clickondetroit.com/sitemap.xml
url.path: https://www.clickondetroit.com/arcio/news-sitemap/
depth: 2
isSitemap: true
, DISCOVERED]
2023-12-11 16:28:21.472 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -6200665771136418110] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.472 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -6200665771136418110] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1
2023-12-11 16:28:21.472 o.a.s.e.ExecutorTransfer FetcherThread #10 [INFO] TRANSFERRING tuple [dest: 104 tuple: source: fetch:72, stream: status, id: {5493392416672285611=-3066904289869453196}, [https://www.clickondetroit.com/arcio/sitemap/, url.path: https://www.clickondetroit.com/sitemap.xml
url.path: https://www.clickondetroit.com/arcio/news-sitemap/
depth: 2
isSitemap: true
, DISCOVERED] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null]
2023-12-11 16:28:21.472 STDERR Thread-1 [INFO] 	at java.base/java.net.PlainDatagramSocketImpl.send0(Native Method)
2023-12-11 16:28:21.472 o.a.s.e.Executor Thread-41-__acker-executor[2, 2] [INFO] Processing received TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -670262096599667379] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null for TASK: 2 
2023-12-11 16:28:21.472 o.a.s.e.b.BoltOutputCollectorImpl Thread-41-__acker-executor[2, 2] [INFO] BOLT ack TASK: 2 TIME: -1 TUPLE: source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -670262096599667379] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null
2023-12-11 16:28:21.472 o.a.s.e.b.BoltExecutor Thread-41-__acker-executor[2, 2] [INFO] Execute done TUPLE source: status:108, stream: __ack_ack, id: {}, [6935833859334561755, -670262096599667379] PROC_START_TIME(sampled): null EXEC_START_TIME(sampled): null TASK: 2 DELTA: -1

[jnioche]

Quick answer - about to log off

if the topology is configured to do the injection, then yes, you need the seed files to be accessible for the workers. that's why you are seeing

java.lang.RuntimeException: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory)
if Elastic already contains the seeds then, avoid the injection by omitting the arguments on the command line.
Typically, with StormCrawler the injections tend to be done as a local job, i.e. it doesn't involve the workers and is not distributed.

As for the output folders, you will definitely need them.

Maybe set the log level to DEBUG only for the FeedParserBolt. It gets very verbose otherwise.

[alextechnology]

Quick answer - about to log off

if the topology is configured to do the injection, then yes, you need the seed files to be accessible for the workers. that's why you are seeing

java.lang.RuntimeException: java.io.FileNotFoundException: /home/storm/systems/news-crawl/seeds/20231110.txt (No such file or directory) if Elastic already contains the seeds then, avoid the injection by omitting the arguments on the command line. Typically, with StormCrawler the injections tend to be done as a local job, i.e. it doesn't involve the workers and is not distributed.

As for the output folders, you will definitely need them.

Maybe set the log level to DEBUG only for the FeedParserBolt. It gets very verbose otherwise.

@jnioche Many thanks for answering my questions. Seeds are already in Elastic so I've removed those parameters and no longer see that error. As for the output folders, I assumed the "lead/master" supervisor (ie. the first to start) acted like a sink, writing all the data crawled from all the workers but your comment clearly says otherwise so I've simply created an NFS share out of the data output folder of Alpha which is mounted on Beta at the same absolute path, so that error is gone too - hooray! Though I'm still not sure how they two supervisors coordinate writing to the same WARC file concurrently - or perhaps they write to independent files to avoid this but I don't see that happening and not sure if that's simply due to connectivity issues between the workers.

Unfortunately, neither of the above changes appears to have fixed the main issue which is that the cluster workers can't seem to consistently communicate with each other when they are on different machines. From what I can tell, this seems to lead to a whole number of issues including the java.util.ConcurrentModificationException errors among others. This error appears to arise in the remote worker relative to the nimbus (ie. Beta). I'm fairly certain that part if not all of this is actually in Storm itself and related to Netty because there's no issues with a single worker nor multiple workers running on a single host - both configurations where Netty is not involved. It seems odd that the Alpha worker starts up and spends over a minute failing to connect to the Beta worker which actually hasn't event started yet at that point! After ~140 attempts, the Alpha worker finally starts fetching and only a bit later does the Beta worker start up - yet doesn't display the same behavior of dozens of attempts to connect to the Alpha worker. Eventually, I suppose the two are not communicating correctly and there's a concurrent modification done of metadata which causes the Beta worker to just die and restart, a loop that continues indefinitely.

I'm not sure how else to try and debug this, especially if the issue is in storm itself. Curious why you chose storm 2.5.0 instead of the latest 2.6.0? Any chance the later version is compatible with this news-crawl branch? Perhaps I should try that version instead? Or maybe I should try building storm from source instead of using the pre-built binary?

I really appreciate the work you've done and the assistance and insight you've provided so far. Worse comes to worst, my use case can probably still be fulfilled with a single worker but really limits the scale to a degree that I may need to find another crawling solution entirely.

[jnioche]

perhaps they write to independent files to avoid this but I don't see that happening and not sure if that's simply due to connectivity issues between the workers.

See WARCNameFormat - the file name contains the task index so there can't be a collision

I'm not sure how else to try and debug this, especially if the issue is in storm itself. Curious why you chose storm 2.5.0 instead of the latest 2.6.0?

Simply because 2.6.0 has been releasing since and currently doesn't have a Docker image. I initiated a vote yesterday so that the Apache Storm project releases its own Docker images and we can control when this happens.

The code in this repo uses Docker to simplify adoption but since you install Storm directly, you could try 2.6.0
You would probably need to use StormCrawler 2.11-SNAPSHOT to ensure compatibility. I am planning to release SC 2.11 soon, not sure when though.

I am not sure the ConcurrentException is due to the connection problem, or whether it is the other round. One thing to do would be to try the lock() approach I described yesterday and see where modifies the Metadata object.

You could also try a vanilla WordCount topology on your cluster and check whether it works. Maybe double check the Storm configuration and version of Zookeeper?

BTW it is always interesting to hear about how people use our code or StormCrawler in general. If at all possible, feel free to share your use case with the community.

Thanks!

[alextechnology]

@jnioche I went ahead and made a new FeedParserBolt with a metadata.lock() and metadata.unlock() before and after the collector.emit() + collector.ack() around line 104 and below is the resulting error stack I'm getting (also added a few logging messages to get a better idea). It seems to indicate the StatusUpdaterBolt for elastic is somehow involved? I tried disabling the status.updater.use.cache and reducing the es.status.concurrentRequests but neither change made a difference. I also tried the following setup:

• Alpha node - Nimbus + WARC folder
• Beta node - Supervisor + NFS mount of WARC folder on Alpha
• Topology Workers 1
• No seedlist (already in elastic)

And this works just fine as well - so there's definitely no issue with the remote supervisor being able to communicate with nimbus, run a supervisor + workers, interface with elastic, or write to the WARC data folder configured. Ironically, it's actually a bit faster as it's a more powerful VM.

Any ideas of what could be the issue or anything else I could try?

2023-12-12 20:02:19.299 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Processing https://qz.com/uk-lawmakers-vote-in-favor-of-the-rwanda-migration-bill-1851093410
2023-12-12 20:02:19.299 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Not a feed https://qz.com/uk-lawmakers-vote-in-favor-of-the-rwanda-migration-bill-1851093410
2023-12-12 20:02:19.301 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 1 links, 1 skipped): https://www.cityam.com/sitemap.xml?yyyy=2018&mm=02&dd=17
2023-12-12 20:02:19.301 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] https://www.macworld.com/news-sitemap.xml detected as news sitemap based on content
2023-12-12 20:02:19.301 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Processing https://spaceexplored.com/2023/12/12/blue-origin-plans-to-return-to-new-shepard-flights-next-week/
2023-12-12 20:02:19.302 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Not a feed https://spaceexplored.com/2023/12/12/blue-origin-plans-to-return-to-new-shepard-flights-next-week/
2023-12-12 20:02:19.309 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 32 links, 0 skipped): https://www.macworld.com/news-sitemap.xml
2023-12-12 20:02:19.310 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Processing https://www.foxbusiness.com/economy/november-inflation-breakdown-where-are-prices-still-rising-fastest
2023-12-12 20:02:19.302 o.a.s.u.Utils Thread-77-status-executor[105, 112] [ERROR] Async loop died!
java.lang.RuntimeException: java.util.ConcurrentModificationException: Attempt to modify a metadata after it has been sent to the serializer
	at org.apache.storm.executor.Executor.accept(Executor.java:301) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) [storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.ConcurrentModificationException: Attempt to modify a metadata after it has been sent to the serializer
	at com.digitalpebble.stormcrawler.Metadata.checkLockException(Metadata.java:256) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.Metadata.setValue(Metadata.java:130) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.persistence.AbstractStatusUpdaterBolt.execute(AbstractStatusUpdaterBolt.java:183) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	... 6 more
2023-12-12 20:02:19.310 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Not a feed https://www.foxbusiness.com/economy/november-inflation-breakdown-where-are-prices-still-rising-fastest
2023-12-12 20:02:19.311 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Processing https://www.thedailybeast.com/fbi-arrest-internet-pornography-personality-paul-caloia-for-jan-6-capitol-riot
2023-12-12 20:02:19.310 o.a.s.e.e.ReportError Thread-77-status-executor[105, 112] [ERROR] Error
java.lang.RuntimeException: java.lang.RuntimeException: java.util.ConcurrentModificationException: Attempt to modify a metadata after it has been sent to the serializer
	at org.apache.storm.utils.Utils$1.run(Utils.java:413) ~[storm-client-2.5.0.jar:2.5.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.RuntimeException: java.util.ConcurrentModificationException: Attempt to modify a metadata after it has been sent to the serializer
	at org.apache.storm.executor.Executor.accept(Executor.java:301) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
Caused by: java.util.ConcurrentModificationException: Attempt to modify a metadata after it has been sent to the serializer
	at com.digitalpebble.stormcrawler.Metadata.checkLockException(Metadata.java:256) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.Metadata.setValue(Metadata.java:130) ~[stormjar.jar:?]
	at com.digitalpebble.stormcrawler.persistence.AbstractStatusUpdaterBolt.execute(AbstractStatusUpdaterBolt.java:183) ~[stormjar.jar:?]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:212) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:294) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.5.0.jar:2.5.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:398) ~[storm-client-2.5.0.jar:2.5.0]
	... 1 more
2023-12-12 20:02:19.311 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] https://www.twincities.com/sitemap.xml?yyyy=2011&mm=08&dd=05 detected as news sitemap based on content
2023-12-12 20:02:19.311 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Not a feed https://www.thedailybeast.com/fbi-arrest-internet-pornography-personality-paul-caloia-for-jan-6-capitol-riot
2023-12-12 20:02:19.312 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Processing https://www.pressherald.com/life/arts-entertainment/feed/
2023-12-12 20:02:19.312 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] Found a feed https://www.pressherald.com/life/arts-entertainment/feed/
2023-12-12 20:02:19.312 c.d.s.b.FeedParserBolt Thread-75-feed-executor[69, 70] [INFO] parseFeed for https://www.pressherald.com/life/arts-entertainment/feed/
2023-12-12 20:02:19.328 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 159 links, 159 skipped): https://www.twincities.com/sitemap.xml?yyyy=2011&mm=08&dd=05
2023-12-12 20:02:19.329 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] https://www.ocregister.com/sitemap.xml?yyyy=2011&mm=08&dd=01 detected as news sitemap based on content
2023-12-12 20:02:19.346 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 196 links, 196 skipped): https://www.ocregister.com/sitemap.xml?yyyy=2011&mm=08&dd=01
2023-12-12 20:02:19.347 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] https://www.nbcsportsphiladelphia.com/sitemap.xml?yyyy=2019&mm=11 detected as news sitemap based on content
2023-12-12 20:02:19.367 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 462 links, 447 skipped): https://www.nbcsportsphiladelphia.com/sitemap.xml?yyyy=2019&mm=11
2023-12-12 20:02:19.370 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] https://www.news.com.au/sitemap.xml?yyyy=2018&mm=05&dd=06 detected as news sitemap based on content
2023-12-12 20:02:19.376 o.c.s.n.NewsSiteMapParserBolt Thread-36-sitemap-executor[78, 78] [INFO] Sitemap (found 129 links, 129 skipped): https://www.news.com.au/sitemap.xml?yyyy=2018&mm=05&dd=06