Watch keynotes and sessions from MongoDB.live, our virtual developer conference.

Decimal values written as NumberDecimal("") - Spark Connector

Recently, I’ve started working on writing some data to MongoDB via Spark. I have developed an application which writes to MongoDB via MongoDB Spark Connector.

I have used MongoSpark.save() to write Dataframe to MongoDB, however the decimal values are being written as NumberDecimal(""), Eg: “Score” : NumberDecimal(“0.3916”)

Instead, I would like to write as “Score” : 0.3916.

On going through articles I got to know that the Decimal is being represented as NumberDecimal, but is there a way where the data can be written without NumberDecimal() represented as like “Score” : 0.3916.

The quick workaround to deal with drive datatype issues is, explicit typecast the column of the dataframe in the notebook before calling save().
ex. dataframe.withColumn(field, dataframe(field).cast( DecimalType )
Please refer: (https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/sql/types/DecimalType.html)))

Thanks