Python is a strongly typed language that often places considerable emphasis on both variables and containers. But sometimes the Python error messages associated with those traits can be somewhat confusing. If you’ve found yourself confronted with an error message of “valueerror: input contains nan, infinity or a value too large for dtype(‘float64’)” then you’re in good company. It’s an error message that brings up a lot of points that aren’t always immediately apparent to Python users who are mainly familiar with the core library. But thankfully, it’s also not too difficult to understand and fix.
What Does the Error Mean?
In simple terms, the Python error is notifying you that an attempt has been made to assign an incompatible value as a 64-bit floating-point number created through the NumPy library. Specifically, the code is attempting to assign NaNs (Not a Number), infinitely long numerical sequences, or a number large enough to push past the float64 memory constraints.
When a Number Isn’t Really a Number
You’ve probably heard people warn against comparing apples to oranges. And that’s essentially what’s going on in this error message. The ValueError is complaining about something that’s similar but ultimately incompatible with an acceptable number being assigned to a float64.
This problem can come about in a number of ways. In this specific case it’s due to the presence of one of three different problematic data types. The first is a Not a Number value. As the name suggests, NaNs are not a number. It can pop up in code for a number of different reasons. One of the most common causes comes down to containers being automatically populated with Not a Number values in order to ensure compliance within otherwise incompatible operations. For example, a column might be automatically populated with Not a Number instances when adding it into a DataFrame.
Infinity and values too large for a 64-bit float are similar in practical respects. The two can be quite different in terms of the way they’re formed or how they’re used in your code. But ultimately, both are incompatible with a numerical format that insists on size restraints.
How To Fix the Error
Unfortunately, there’s isn’t a one size fits all fix for this valueerror. The issue can come about for multiple reasons and requires some specificity to properly fix it. But there are some procedures which you can almost certainly fit into your code. You might simply need to tailor this to your individual situation.
The first step should simply involve a quick code review. If you receive this error when passing a Python generated value then you should ensure those generators are working correctly. You might think that you’re passing a compatible value. But a simple typo is all it takes to turn a standard number into something utterly incompatible with a float64. However, you might also have a situation where you need to format data before passing or assigning it as a 64-bit float. This would be the case for instances where Not a Number values are automatically added into generated containers to compensate for missing values.
But if your python code is generating data as expected you’ll need to manually clean up any Not a Number or infinity value instances. The easiest way to do so involves a NumPy method called nan_to_num. It will try to make a reasonable guess about what you’d like to do with Not a Number or infinity values. But for this example we’ll pass NaNs instances to the conversion method to set all converted values as 1.1. And we’ll first convert any infinite instances to a Not a Number so that they’ll both receive the same underlying conversation logic. This will make it easier to note specific replacements when printing out the results.
If you don’t pass a Not a Number value through the method it’ll simply create new non NaN values as 0 and any infinity instances as a larger random number. You can always get more control over that process by creating your own function with more specific logic that’s customized to your program’s individual needs. However, the following code demonstrates a concise use of nan_to_num for you to build on.
import numpy as np
numNam = 3
inProgress = np.random.randn(32)
nansPoint = np.random.choice(inProgress.size, numNam, replace=False)
infPoint = np.random.choice(inProgress.size, numNam, replace=False)
inProgress.ravel()[nansPoint]=np.nan
inProgress.ravel()[infPoint]=np.inf
print(inProgress)
inProgress[inProgress == np.inf] = np.nan
inProgress = np.nan_to_num(inProgress, nan=1.1)
print(inProgress)
We begin by importing NumPy and stating the number of Not a Number instances we’ll create as numNam. The inProgress variable creates our initial data without any Not a Number or infinity instances. We proceed to find indices to insert our NaNs and infinity values to before printing it to screen for verification. The data then goes to a conversion stage to convert infinite to Not a Number. And all Not a Number instances are then changed to 1.1. Finally, the results are printed to screen for easy comparison to the initial data.
Keep in mind that all of this works using a data sample filled with Not a Number and infinity values. Your own data will, of course, vary from this specific structure. But the effects of the method should remain consistent within any collection of data that contains infinite and NaN value instances.