PubSubLite java.lang.NoSuchMethodError

[jhsenjaliya]

I get NoSuchMethodError while trying to read message from pubsublite using spark structured streaming application.

2023-05-30 11:15:05,024 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Reading table [com.google.cloud.pubsublite.spark.PslTable@79708f8] from DataSourceV2 named 'pubsublite' [com.google.cloud.pubsublite.spark.PslTableProvider@64ddf4ea]
2023-05-30 11:15:07,496 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Starting new streaming query.
2023-05-30 11:15:07,498 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Stream started from {}
2023-05-30 11:15:09,308 [INFO  ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Writing atomically to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0 using temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp
2023-05-30 11:15:09,348 [INFO  ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Renamed temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0
2023-05-30 11:15:09,349 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Committed offsets for batch 0. Metadata OffsetSeqMetadata(0,1685470509297,Map(spark.sql.streaming.stateStore.providerClass -> org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider, spark.sql.streaming.join.stateFormatVersion -> 2, spark.sql.streaming.stateStore.compression.codec -> lz4, spark.sql.streaming.stateStore.rocksdb.formatVersion -> 5, spark.sql.streaming.statefulOperator.useStrictDistribution -> true, spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion -> 2, spark.sql.streaming.multipleWatermarkPolicy -> min, spark.sql.streaming.aggregation.stateFormatVersion -> 2, spark.sql.shuffle.partitions -> 200))

Exception in thread "pubsub-lite-futures-1" java.lang.NoSuchMethodError: 'com.google.api.gax.rpc.ApiCallContext com.google.cloud.pubsublite.internal.wire.ServiceClients.getCallContext(com.google.cloud.pubsublite.internal.wire.PubsubContext, com.google.cloud.pubsublite.internal.wire.RoutingMetadata)'
	at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.lambda$null$1(PslReadDataSourceOptions.java:139)
	at com.google.cloud.pubsublite.internal.wire.SingleConnection.<init>(SingleConnection.java:71)
	at com.google.cloud.pubsublite.internal.wire.SingleConnection.<init>(SingleConnection.java:77)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.<init>(ConnectedSubscriberImpl.java:54)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.<init>(ConnectedSubscriberImpl.java:33)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:46)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:39)
	at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.reinitialize(RetryingConnectionImpl.java:106)
	at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.lambda$doStart$0(RetryingConnectionImpl.java:94)

i do see the getCallContext() method with required parameter and classes are present at package com.google.cloud.pubsublite.internal.wire which is available in google-cloud-pubsublite-1.12.5 jar

I also noticed another thing Upon looking at the PartitionSubscriberFactory getSubscriberFactory() method in com.google.cloud.pubsublite.spark.PslReadDataSourceOptions it indeed using different consumer
com.google.cloud.pubsublite.SequencedMessage vs setMessageConsumer expecting com.google.cloud.pubsublite.proto.SequencedMessage

Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...") : PubSub Lite
   General, Core, and Other are also allowed as types
2. OS type and version: Mac OS M1(ventura)
3. Java version: 11
4. version(s):

[dyptan]

I had the same issue. Basically transitive dependencies are missing, so you have to download a fat jar "with-dependencies.jar" and append it to Spark's classpath.

[jhsenjaliya]

I can not find pubsublite-spark-sql-streaming -- with dependency jar on maven central. but i packaged from this source and used it but it does not work. r u using latest version 1.0 ?
I actually already have bigquery with dependency and gcs connector which covers all dependencies ( gax, gax-grpc, etc....)

[jhsenjaliya]

Found the issue, actually the latest change on the google-cloud-pubsublite has changed the SequencedMessage package to use proto, and pubsublite-spark-sql-streaming still uses non-proto version of SequencedMessage.
so you have to use compatible libs: com.google.cloud:google-cloud-pubsublite:1.10.0 along with com.google.cloud:pubsublite-spark-sql-streaming:1.0.0 works. ( as the breaking change in google-cloud-pubsublite is after 1.10.0 )